Non-custodial Wallet Hosting & Sync

(draft)
Motivation
Your wallet file stores important data that apps need to access. There's no convenient noncustodial way to back up your wallet or keep it synced across multiple apps. If you use several LBRY apps, you'll end up with many separate wallets.
Goals

name
description

noncustodial
host has no access to information inside wallet

multiapp
multiple apps can use (odysee, lbry desktop, hound.fm, etc)

onepw
single password for wallet encryption and app login (prolly via lbry.id oauth)

pwreset
password can be changed, and can be reset if forgotten in some cases. some data loss is acceptable here

realtime
wallets synced live between apps (at least it should feel like its live)

decentralized
the whole setup can be run by anyone

userfriendly
losing access to your wallet is as hard as we can make it

Plan
1. move as much stuff out of the wallet as possible

channel keys come from seed (lex already working on this https://github.com/lbryio/lbry-sdk/issues/1433)
subscriptions go on-chain?
app settings stored in encrypted blob in channel?

2. hosted wallet sync and oauth
set up https://lbry.id service to host wallets and provide oauth (should be open-source and documented so anyone can run their own auth service)
support 2fa?
3. client-side signing via lbry.id or js or browser plugin (a la metamask)
Once this sync protocol is adopted, apps will need a way for users to sign transactions to interact with LBRY. This must be done client side to satisfy the noncustodial requirement. This can be done (in increasing order of security) custom in the app via JS, or using a popup like Deso, or a browser extension like Metamask (which could also do hardware wallet support).
Terms

version (uint)
The protocol version used to encrypt the wallet. This document describes version 1 of the protocol.
id (string)
A string that uniquely identifies a user. If this is an email address, we get the benefit of being able to contact the user if necessary.
password (string)
The password as the user enters it.
rootKey (bytes)
A cryptographic key derived from the password using a key derivation function (KDF)
walletKey (bytes)
The first half of the root key. This key is used to encrypt a user's wallet. It is only used locally on a user's device and is never sent to the server.
loginKey (bytes)
The second half of the root key. This key is used to authenticate a user with the server.
sequence (uint)
A counter that is incremented every time a wallet is updated. This prevents race conditions where multiple simultaneous writes overwrite one another.
authToken (bytes)
A token representing a user session.
mfaCode (uint)
The TOTP code from a multi-factor device (Authenticator app, etc)
wallet (bytes)
Plaintext private user data. May include seed phrases, private keys, app info, settings, etc.
encryptedWallet (bytes)
`wallet` data that is encrypted and may be shared securely.

Server Operations
register
Register a new account
params: id, password, version
response: authToken
errors: version not supported, id already exists
oauth
Support the standard Oauth flows. We'll probably integrate with existing IDP project
getWallet
Get the current wallet data
params: authToken
response: version, encryptedWallet, sequence
errors: invalid authToken
putWallet
Store new wallet data
params: authToken, version, encryptedWallet, sequence
response: ok
errors: invalid authToken, version not supported, sequence mismatch
changePassword
This is the same as a putWallet, but also changes the loginKey.
params: authToken, version, loginKey, encryptedWallet, sequence
response: ok
errors: invalid authToken, version not supported, sequence mismatch
info
Get info about a user
params: authToken
response: id, version, sequence
websocket
Connect to websocket to be notified when new wallet data is stored
params: authToken
Common flows
New account
TODO
Connect an app
TODO
Conflicting writes
TODO
I forgot my password, what can I do?

recovery from paper backup
changing password if unlocked wallet is/isnt available
changing password if active auth is/isnt available

TODO
Multiple user apps (desktop + mobile + paper backup)
TODO

how to handle when a password was changed on another app
recovery via another app

UX Considerations
Different wallets have different needs. If you just started an account, its no big deal to lose your wallet. If you have a lot of channels/claims/lbc in there, you want more backups and security.
Security
how to do this so its secure? what's the threat model?
prolly want an audit for this, but shouldn't block deployment
Rabbit holes
password resets
oauth and multiple apps
secure client-side signing via lbry.id
migrating odysee to this system
phishing
do we have to have the same password across multiple apps?
Extras
When unlocking, run sync first to make sure latest version is stored in cloud
Unlock via QR code or copying a string?
optional passcode to wrap root key locally
encrypt data with intermediate key, then encrypt that key with the password. this lets you back up your wallet encrpytion key on paper
does this work with multiple apps? can you have separate passwords across multiple apps? prolly not… so then a malicious app could get your odysee password
can you login with zksnarks? proof you know some encrypted string without revealing that string?
tell ppl when their passwords are poor using haveibeenpwned.com
outline what changes sdk has to make
should we store an updatedAt timestamp for wallet changes?
Alternative ideas
no wallets. everything is on-chain and can be restored from seed

makes every settings change a txn. may lead to scaling issues

we should assume our blockchain will be upgraded to handle this

where do we store the data? its not always tied to channels
plus side: if we can do this, we won't need lbry.id to host wallets. every app can do its own thing.

store wallet in public cloud (s3, google drive, etc). define storage location in channel claim.

doesnt meet onepw requirement
wallets aren't tied to channels, so we'd need a new place on-chain to store them

Related issues
https://github.com/lbryio/lbry-sdk/issues/2641
References

https://docs.standardnotes.org/specification/encryption/
https://github.com/standardnotes/snjs
https://docs.deso.org/devs/identity-api
https://github.com/decentralized-identity/ion
https://identity.foundation/sidetree/spec/

Putting this here so I don't forget: it would be nice if this also worked with social login. Maybe there's a way to get a string from the social login params and use that as a password? Or store a password with the social login service.

https://github.com/lbryio/lbry-desktop/issues/6792
https://github.com/lbryio/lbry-android/issues/1212
I made a suggestion to the LBRY desktop and mobile clients on better wallet management, I am for these features since it will allow your LBRY account/wallet to only have one seed phrase and not a whole bunch of seed phrases over time as you login to new devices.


Fix state machine error handling

The failed test is one that has failed intermittently on Travis for a long time that I've never been able to fully resolve. Will rerun the tests.


Older rosters trump newer rosters

When a roster gets updated in Dropbox, it is not updated in Roster Decoder if cached.

This appears to be resolved by d0cd5030c38d53f7763856950099aa4a83548ff1


Update C++/Obj C++ support for Firebase 9

Fix ObjC++ Firebase.h regression
Copy and adapt Storage and Functions ObjC API tests for Objective C++
Several Firebase 9 release notes updates

cc: @jonsimantov


Android API 16-21

Do apollo-android have support for working between API 16-21? I don't know if there is a minimum API level to be able to work with it. Thanks!

Apollo Android supports API levels 15+. IIRC, you'll need to enable Java 8 language features but that should be the default in the Kotlin plugin these days.

Thanks @martinbonnin


2.0 Release?

What's left to do?

[x] #150
[x] #97
[ ] Check testing coverage and test saneness
[ ] Integrate 2.0 into a few more projects to spot usage issues?

I've deployed the 2.0 branch in my app to production and spotted no crash or anomalies in stats.
@eldk Any report on how it works for you?

Updated the README and the wiki, so that part is ready for 2.0.
Had one more thought. Need to change the Tracker init to use a TrackerConfig so we can extend this without breaking the API in the future.
In anycase there doesn't seem to much activity... Did anyone do anymore testing or code review?
I'll do one more app integration, go on holiday and when I'm back, if there is no opposition, release it?

I'm not testing 2.0 branch anymore.

If there is no opposition, release it?

Sure thing

I merged and released it.

Maybe someone can fluff up the release notes? Link to tickets that were closed?

:tada:  Well done @d4rken  :tada:
FYI using our Github Changelog Generator tool here is the generated changelog of closed issues since July 26th 2016:
piwik/piwik-sdk-android
#16 Offline mode missing [by @d4rken, @n8fr8]
#110 Use Okhttp instead of HTTPUrlConnection [by @d4rken]
#116 Unable to connect local piwik
#117 Problems tracking pages and visitors
#118 "Typo" in readme.md [by @eldk, @d4rken]
#119 API 10 - https PIWIK server - Could not validate certificate signature -
#120 Update dev branch with last commits from 138f8de [by @eldk, @dotsbb]
#121 Update dev branch with master [by @dotsbb, @d4rken]
#122 Why is minSDK to API 10 and not API 9 ?
#123 Gzipped dispatched json [by @eldk, @dotsbb]
#124 Update project to latest tool versions and API targets [by @d4rken]
#125 Add unit test for gzip setting [by @d4rken]
#126 Fixed javadoc issues (Java8 specs) [by @d4rken]
#127 Fixed a few lints [by @d4rken]
#128 UncaughtExceptionHandler fix [by @d4rken]
#129 Updated tools and libs [by @d4rken]
#130 Addresses #110. [by @d4rken]
#131 GZIP setting unit test [by @d4rken]
#132 Update doc : gzipped json dispatch [by @eldk, @dotsbb]
#133 No tracking
#134 Handle search events [by @d4rken]
#135 Added site-search to TrackHelper. [by @d4rken]
#136 Offline mode [by @d4rken]
#137 i get Value not defined in Custom Dimension
#138 Custom variable value in table overwrites when a new event is logged.
#139 FYI: Piwik Tracking API now allows data up to 24 hours in the past [by @eldk, @d4rken, @dotsbb]
#140 compile source for local maven repository
#141 Updated javadocs. [by @d4rken, @dotsbb]
#142 Improved bintray upload script. [by @d4rken]
#143 Up version on dev to v1.1.0 for the next release. [by @d4rken]
#144 Trying to fix travis. [by @d4rken]
#145 Add options for different dispatching modes. [by @d4rken]
#146 Updated javadocs. [by @d4rken]
#148 Architecture overhaul (SDK 2.0) [by @d4rken]
#151 Material design side menu
#155 2.0 Improvements [by @d4rken]
#156 Offline cache bug fixes [by @d4rken]
#157 Introducing TrackerConfig [by @d4rken]
#158 Allow custom identifiers when tracking downloads. [by @d4rken]
#159 2.0 [by @d4rken]
#160 2.0 Release [by @d4rken]


Move from Rails 5.2.X -> Rails 6.0.X

BIG MILESTONE! This commit finally switches the application
from the Rails 5.2.X series to the Rails 6.0.X series.
We've wanted to do this for a while, but other gems (libraries)
had blocked our way. The big problems (that we've been working to
overcome) were:

We had to eliminate and replace the gem fastly-rails.
The gem "fastly-rails" does not work on Rails 6, so we
had to eliminate it to upgrade to Rails 6.
They recommend switching to the "fastly" gem (aka "fastly-ruby"),
but fastly-ruby is not designed to support multi-threading (WHAT?!).
The only workable solution I could find was to modify our code
to directly call the Fastly API instead.
For details, see commit 2c9c0940ce9b42a00e003e6c2e4ec6b1012d9e35.

We had to eliminate the gem minitest-rails-capybara,
which we depended on for system tests, and modify all our system
tests to work with the now-standard Rails system test system.
The gem minitest-rails-capybara support ends with minitest-rails 3.x,
and thus we cannot use Rails 6 as long as we used it for tests.
For details, see commit fdbbc1eddd9ce78a232e6117175b4f80badd6cd0.

There was no rocket science, just a non-trivial amount of tedious work.
The XKCD cartoon "Software Updates" https://xkcd.com/2224/
illustrates the problems we had upgrading Rails, where
"an update finally breaks a feature I'm unwilling to lose"
and thus updating stops. In our case, we were unwlling to lose
our Content Delivery Network (CDN) since it's critical for performance,
and we were unwilling to lose our system tests because they help us
maintain high quality. So we had to put in some effort so that we could
do our update. If your update is difficult, it can create a security
problem because people may find it impractical to update!
Signed-off-by: David A. Wheeler<EMAIL_ADDRESS>

@jdossett - let me know soon if you know of an issue. The change here is relatively small, but that's because we've made many other changes before to get to this point.
Notice I didn't change everything (e.g., I didn't add webpacker). My goal is primarily to update to maintained components, even if we keep doing some things in "older" ways.
I intend to separately update to Rails 6.1, but only after we're confident in this move.

I've decided it's time to pull the trigger. I'm merging it now, hopefully everything "just works".


Review documentation for settings used in call_spark_submit

call_spark_submit turns a couple of kiterc settings into Java properties. For example KITE_HTTPS_KEYSTORE. We have to update the documentation for these now.

https://github.com/lynxkite/lynxkite/blob/5.0.0/tools/call_spark_submit.sh#L73


Deploy resources

Context
New project that needs deployment workflow and associated assets.
Changes proposed in this pull request

Add build and deploy workflow plus terraform & make file
This implementation copies and refines the implementation from access-your-teaching-profile

Sets terraform version in 1 place
Improves makefile formatting
Deploys tagged version of ARM resources from the pipeline

Guidance to review
Include any useful information needed to review this change.
Include any dependencies that are required for this change.
Checklist

[ ] Attach to Trello card
[ ] Rebased main
[ ] Cleaned commit history
[ ] Tested by running locally

 Setup deploy pipeline for Refer serious misconduct


t2 run/push: stub gunzip & extract in remote binary request tests. Fixes gh-711

Signed-off-by: Rick Waldron<EMAIL_ADDRESS>

Ok, now it's dying in provision.js

First run report (to save readers from having to wade through the travis logs):

Node 4, Linux https://travis-ci.org/tessel/t2-cli/jobs/126171222

Pass

Node 5, Linux https://travis-ci.org/tessel/t2-cli/jobs/126171223

Pass

Node 4, OSX https://travis-ci.org/tessel/t2-cli/jobs/126171224

Stopped in provision.js

Node 5, OSX https://travis-ci.org/tessel/t2-cli/jobs/126171225

Stopped in provision.js

Rerun report

Node 4, Linux https://travis-ci.org/tessel/t2-cli/jobs/126171222

Pass

Node 5, Linux https://travis-ci.org/tessel/t2-cli/jobs/126171223

Pass

Node 4, OSX https://travis-ci.org/tessel/t2-cli/jobs/126171224

Stopped in provision.js

Node 5, OSX https://travis-ci.org/tessel/t2-cli/jobs/126171225

Pass

Looks good - what instigated this PR?

My report about tests not completing on Node 5.x. Which @mattsoftware was running into on Slack. Seems there are a few issues open with tests just not completing all the way. Then when that happens Travis still reports it passed, because all the tests did pass, but not all were ran. Hopefully this can put a stop to all of that.
It was very good this was able to be reproduced locally, as it can be iterated on much faster than pushing and waiting for Travis.

It was very good this was able to be reproduced locally, as it can be iterated on much faster than pushing and waiting for Travis.

Sanity saver :)

My tests still seem to die even if I merge this patch into my repo. This time in/after controller.js
Same point if I run node v5.0.0 or v5.11.0
If you need me to provide any debugging etc let me know!

Whats your output if you run grunt -v?

This was only one part of a set of fixes...


Clean up naming conventions

What type of PR is this?
(bug/feature/cleanup/documentation)
What this PR does / Why we need it?
Which Jira/Github issue(s) does this PR fix?
Resolves #
Special notes for your reviewer
Pre-checks (if applicable)

[ ] Ran unit tests locally
[ ] Validated the changes in a cluster
[ ] Included documentation changes with PR

/lgtm
/approve


JedisCluster works in 2.7.x but doesn't work in 2.9.x with JedisConnectionException and Connection refused. (Single node cluster)

Expected behavior
JedisCluster in Jedis 2.9.x works as same as Jedis 2.7.x
Actual behavior
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
	at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:66) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisCluster.get(JedisCluster.java:124) ~[jedis-2.9.x.jar:?]
.........
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused (Connection refused)
	at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.x.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
	at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.x.jar:?]
	... 45 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_144]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_144]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_144]
	at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.x.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
	at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.x.jar:?]
	... 45 more

Steps to reproduce:
Summary: I have the Jedis 2.7.x for several months. But i found I have the same issue as https://github.com/xetorthio/jedis/issues/1625 (JedisPool.getResource block when redis server restart) . So I switched from 2.7.x to 2.9.x. I am using the same way to create JedisCluster. But it failed to connect and get data from Jedis. I created a singleton JedisCluster instance and every time i just use JedisCluster.get() to get data. I don't close connections for cluster.
Please create a reproducible case of your problem. Make sure
that case repeats consistently and it's not random

I have Jedis 2.7.x . Code below worked:

    @Singleton
    JedisCluster jedisCluster(){
            Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
            jedisClusterNodes.add(new HostAndPort("10.xx.xx.xx", 6379));
            GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
            jedisPoolConfig.setMaxTotal(16);
            jedisPoolConfig.setMaxIdle(16);
            return new JedisCluster(jedisClusterNodes, 2000, 1000, 5, jedisPoolConfig);
   }
   //When using cluster
    jedisCluster.set("foo", "bar");
    String value = jedisCluster.get("foo");

I switched to Jedis 2.9.x, below code worked

           Jedis jedis = new Jedis("xx.xx.xxx.xx",6379);
           System.out.println(jedis.get("foo"));

I ssh to the Redis host and run "get foo" worked.

<IP_ADDRESS>:6379> get foo
"bar

Code below is the same as 1 but it doesn't work. I switched back to Jedis 2.7.x and it worked. I tried to switch Jedis version several times and only 2.7.x worked.

    @Singleton
    JedisCluster jedisCluster(){
            Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
            jedisClusterNodes.add(new HostAndPort("10.xx.xx.xx", 6379));
            GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
            jedisPoolConfig.setMaxTotal(16);
            jedisPoolConfig.setMaxIdle(16);
            return new JedisCluster(jedisClusterNodes, 2000, 1000, 5, jedisPoolConfig);
   }
   //When using cluster
    jedisCluster.set("foo", "bar");
    String value = jedisCluster.get("foo");

log from redis side
3088:M 21 Mar 21:42:24.187 - Accepted xxx.xxx.xx.xx:36064
3088:M 21 Mar 21:42:24.197 - Accepted xxx.xxx.xx.xx:36065
3088:M 21 Mar 21:42:24.212 - Reading from client: Connection reset by peer
3088:M 21 Mar 21:42:28.481 - DB 0: 12xxxxxx keys (0 volatile) in 16777216 slots HT.
3088:M 21 Mar 21:42:28.482 - 1 clients connected (0 slaves),<PHONE_NUMBER> bytes in use

Redis / Jedis Configuration
I am testing with one single host and use it as a cluster.
<IP_ADDRESS>:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

Jedis version:
2.7.x works fine
2.9.x having problem
Redis version:
3.2.8
Java version:
I don't know. I am using JDK8

what does your cluster slots command return?

Yeah,
1) 1) (integer) 0
   2) (integer) 16383
   3) 1) ""
      2) (integer) 6379
      3) "c97xxxxxxxxxxxxxxxxxxxxxxxxxx"

I believe that's the reason. Jedis uses cluster slots from 2.8 onwards (more info here: https://github.com/xetorthio/jedis/pull/1211). So your client needs to be able to resolve / connect to the hostname that appears there.

@marcosnils WOW!!!!! This solved my problem!!! Previously I didn't create cluster in my single host and I could directly use it as a cluster in 2.7.x. I justran the below command which solved my problem.
for i in {0..16383}; do bin/redis-cli -h <ipOfHost1> CLUSTER ADDSLOTS $i; done
cluster meet <ip> 6379

I have 3 more questions. Some of them are related to Jedis and some of them are related to Redis. Could you please let me know where is the best place to ask these questions?

I have a program which will use JedisCluster to query data from my redis cluster. I have 3 masters and 3 slaves. But i saw there are some spikes in the latency side no matter how heavy is the load and I looked at my redis log. The timestamps for "Connection rest by peer error" matched the service latency spike.

28071:S 20 Mar 20:36:22.723 - Error writing to client: Connection reset by peer
28071:M 20 Mar 20:39:01.218 . I/O error reading from node link: Connection reset by peer
28071:M 20 Mar 21:57:03.803 - Reading from client: Connection reset by peer

My poolconfig is maxTotal 16 and maxIdle 16. I set the connection timeout = 2000ms and max redirection =5, I tried soTimeout = 2000/1000/300/30. But this problem continues. It happened intermittently and I don't know why. Do you have any clue? or should I ask in Redis github?
GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
jedisPoolConfig.setMaxTotal(16);
jedisPoolConfig.setMaxIdle(16);
JedisCluster jc = new JedisCluster(jedisClusterNodes, 2000, 1000, 5, jedisPoolConfig);

I want some advice on whether I should update my Redis from 3.2.8 to the newest Redis 4.x

I want some advice on how big i should set the soTimeout, maxTotal for poolConfig or how could i find the best value for my use case. I googled and I just want to know whether you have some resources for me to read.


Kit system issues

Enhancement

Default font size for name should increase.
Default font size for number should decrease.

Bug

Horizontal and vertical attributes are reversed on outfield shorts.
Keeper outline color is always black unless you change it to a different outline color.

Fixed.


Initial OCTOSPI support

write_extended and read_extended methods allow more complicated transactions, such as those needed to read/write from flash memories.
Rename the quadspi flag to xspi
Ref #165

@richardeoin I haven't had any luck yet...
I'm not experienced with qspi so maybe I'm just missing some configuration that I need.. should the chip be handling ncs itself if I set the correct alternate function?
I'm not seeing an rcc ahb3enr enable, but maybe that happens behind the scenes somehow?
Anyway, I'm trying to read the product information (in 1 line mode) just for some sanity checking.. i only get zeros?
        let _ncs = pg6.into_alternate_af10().set_speed(Speed::VeryHigh); //QUADSPI_BK1_NCS

        let sck = pf10.into_alternate_af9().set_speed(Speed::VeryHigh);
        let io0 = pf8.into_alternate_af10().set_speed(Speed::VeryHigh);
        let io1 = pf9.into_alternate_af10().set_speed(Speed::VeryHigh);
        let io2 = pf7.into_alternate_af9().set_speed(Speed::VeryHigh);
        let io3 = pf6.into_alternate_af9().set_speed(Speed::VeryHigh);

        let config = Config::new(50.mhz()).mode(QspiMode::OneBit);
        let mut qspi = regs.bank1((sck, io0, io1, io2, io3), config, &clocks, prec);

        let mut info: [u8; 3] = [0xFF; 3];
        while qspi.is_busy().is_err() {}
        qspi.read_extended(
            QspiWord::U8(0x9F),
            QspiWord::None,
            QspiWord::None,
            0,
            &mut info,
        )
        .unwrap();
        while qspi.is_busy().is_err() {}

I'm not sure about the pin speeds.. if I don't set them though I get stuck in the read on while self.rb.sr.read().tcf().bit_is_clear() {} in the HAL..

Ok, just to confirm which QSPI memory part (and/or board) are you using to test this? The chip should be handling the ncs here iirc, but the HAL documentation about this is indeed lacking. In the H7 HAL the AHB enable happens by passing the prec variable, which the HAL then calls enable() on.
Could you try the JEDEC ID read with the old API? Whilst that wouldn't work for most instructions, the old API should work for the JEDEC ID read (this is in the example). I would also suggest trying a lower clock speed first - maybe 1MHz. Once that's working then you can try higher clock speeds.

Ok, just to confirm which QSPI memory part (and/or board) are you using to test this? The chip should be handling the ncs here iirc, but the HAL documentation about this is indeed lacking. In the H7 HAL the AHB enable happens by passing the prec variable, which the HAL then calls enable() on.

I'm using the IS25LP064A 8M flash part on the Daisy Seed board that also has a stm32h750 on it.

Could you try the JEDEC ID read with the old API? Whilst that wouldn't work for most instructions, the old API should work for the JEDEC ID read (this is in the example). I would also suggest trying a lower clock speed first - maybe 1MHz. Once that's working then you can try higher clock speeds.

I haven't tried the old API for the JEDEC ID but I'll give that a go. I did use a lower clock speed for a bit but not as low as 1MHz so I'll try that and report back.
Thanks @richardeoin !

Ok, just to confirm which QSPI memory part (and/or board) are you using to test this? The chip should be handling the ncs here iirc, but the HAL documentation about this is indeed lacking. In the H7 HAL the AHB enable happens by passing the prec variable, which the HAL then calls enable() on.

I'm using the IS25LP064A 8M flash part on the Daisy Seed board that also has a stm32h750 on it.

Could you try the JEDEC ID read with the old API? Whilst that wouldn't work for most instructions, the old API should work for the JEDEC ID read (this is in the example). I would also suggest trying a lower clock speed first - maybe 1MHz. Once that's working then you can try higher clock speeds.

I haven't tried the old API for the JEDEC ID but I'll give that a go. I did use a lower clock speed for a bit but not as low as 1MHz so I'll try that and report back.
Thanks @richardeoin !

I set the clock to 1Mhz and successfully read the JEDEC ID with the old read (still on your branch).

In case anyone else is watching this, I got the quadspi working on the h750 with some alterations to @richardeoin 's branch. The changes are now merged in.
@richardeoin any thoughts on what is next? Would it make sense to replace the read and write with the read_extended etc? Does it seem like this work could get into this HAL?
Considering the instruction on some of the chips (h750 at least) is always 1 byte, would it make sense to have a feature gated Option<u8> for that

Great that you got this working with the Quadspi peripheral as well!
I would say that the read and write functions can remain mostly unchanged, since they provide a simple API for the majority of non-memory ICs. The read_extended and write_extended methods add extra functionality that it would be useful to have in the HAL too imo.
You're right that for the Quadspi peripheral the instruction is always a u8 (for the Octospi peripheral it can be larger). If there's a nice way to simply the API using feature gates then that would be nice to see!

Great that you got this working with the Quadspi peripheral as well!
I would say that the read and write functions can remain mostly unchanged, since they provide a simple API for the majority of non-memory ICs. The read_extended and write_extended methods add extra functionality that it would be useful to have in the HAL too imo.

would it make sense to refactor the read and write to simply call the *_extended methods so that there is less maintenance to consider?

You're right that for the Quadspi peripheral the instruction is always a u8 (for the Octospi peripheral it can be larger). If there's a nice way to simply the API using feature gates then that would be nice to see!

I'll look into that, I was thinking that it could just be a feature gated pub type XSPIAddr = .. or something..

would it make sense to refactor the read and write to simply call the *_extended methods so that there is less maintenance to consider?

The read and write methods already call the begin_read and begin_write methods, so I don't think there's much to be achieved by refactoring them to call something else. If anything, the common code that actually transfers the data could be refactored into a common method.
Something like:
fn transfer_read_bytes(dest: &mut [u8]) -> Result<(), XspiError> {
    // Wait for the transaction to complete
    while self.rb.sr.read().tcf().bit_is_clear() {}

    // Check for underflow on the FIFO.
    if (self.rb.sr.read().flevel().bits() as usize) < dest.len() {
        return Err(XspiError::Underflow);
    }

    // Read data from the FIFO in a byte-wise manner.
    unsafe {
        for location in dest {
            *location =
                ptr::read_volatile(&self.rb.dr as *const _ as *const u8);
        }
    }

    // Wait for the peripheral to indicate it is no longer busy.
    while self.is_busy().is_err() {}
}

Thank you for working on this, this is awesome. Reading the ID worked on my DaisySeed as well. @x37v do you by any chance have a read/write example for the flash on DaisySeed? I'm trying to translate Xilinx driver to rust, but I struggle. In any case, thanks!

Without #235 it's not possible to build or test the OCTOSPI support in this PR. For that reason it's blocked for now, but if it really becomes an issue then we can look at splitting up this PR (or just accepting some code with no CI coverage on master for a bit)

Rebased onto master

bors r+


README.md file should document the usage and configuration of the backend-middleware

The simplest configuration should be provided under quick start section.
The data files and url parameter, resource json object attribute mapping should be briefly explained with a simple example.
Under Advanced Configuration all other configuration options should be explained. How custom handlers could be added to deal with POST, PUT, DELETE, PATCH requests should be exemplified.


Error patterns and handling: design and document

Clear and specific errors can greatly improve the maintainability and usability of a system; the difference between a request failed error or message like Post (ID: 123ASD234) cannot be deleted due to it's state: 'PUBLISHED' is huge. However, the content of an error needs to be keenly aware of it's audience (developer, admin, end user, etc) and security context.
Errors usually contain details about the inner workings of a system (file paths, software version, internal values, etc) that's critically important for some audiences but should be completely censored for others. Additionally, interfaces are often better places to provide clear, relevant error information than the code from which the error originated. These front ends need to be able to reliably recognise different categories of error and display their own, custom messages.
Basically, there's a great deal to consider. We need a clear, documented strategy (or at least some guidelines) describing:

The responsibilities of different areas of the system re: errors (ie. is it ok to throw here? Has this error already been logged or should I log it now?)
Who are the different "error audiences" and how are they bounded by the system (ie. results from GraphQL -vs- stderr/stdout -vs- Keystone error messages)
Conventions around how error content for these audiences is referenced (eg. err.message -vs- err.public?)
A reliable way of identifying specific error states across the system so interfaces can add their own interpretation (Error codes -vs- error keys? eg. [passwordAuth:secretMismatch])
What should and shouldn't be logged? Protection of personal info, etc. (eg. passwords/secrets should never be logged or included in error messages. What about email addresses, IP addressed and other personal info? GDPR makes this a big deal)

Possibly related topics include:

Error logging react front ends? (eg. something like Rollbar?)
Warnings and other debug

What should Keystone warn about on init? (stderr or stdout?)
What are the convention for debug (are we using the debug package throughout? how are messages scoped?)

Other things..?

@MadeByMike Hey I've removed the good first issue label -- this is a ticket about designing and documenting broad-reaching architectural patterns in Keystone. Whoever takes this on is going to need a deep understanding of Keystone and systems development generally.

That was probably a mistake.


Adding Dockerfile for hubot-slack adapter

Added the Dockerfile for use the hubot-slack adapter.

@guttertec can i get some review?


Uncaught Error: Atom can only handle files < 2MB for now.

Uncaught Error: Atom can only handle files < 2MB for now.
Atom Version: 0.162.0
System: linux 3.2.0-23-generic
Thrown From: Atom Core
Steps To Reproduce

...
...

Stack Trace
At /usr/share/atom/resources/app/src/project.js:461

Error: Atom can only handle files < 2MB for now.
  at Project.module.exports.Project.buildBuffer (/usr/share/atom/resources/app/src/project.js:461:15)
  at Project.module.exports.Project.bufferForPath (/usr/share/atom/resources/app/src/project.js:438:63)
  at Project.module.exports.Project.open (/usr/share/atom/resources/app/src/project.js:392:19)
  at Workspace.module.exports.Workspace.openUriInPane (/usr/share/atom/resources/app/src/workspace.js:482:29)
  at Workspace.module.exports.Workspace.open (/usr/share/atom/resources/app/src/workspace.js:409:19)
  at Ipc.<anonymous> (/usr/share/atom/resources/app/src/window-event-handler.js:45:32)
  at Ipc.emit (events.js:110:17)
  at process.<anonymous> (/usr/share/atom/resources/atom/renderer/api/lib/ipc.js:22:29)
  at process.emit (events.js:118:17)

Please update to the latest Atom version (currently 0.187.0) and see #307 for details.


TUN-1.9: GRE inner packet DSCP

This script convers "TUN-1.9: GRE inner packet DSCP" test scenario.
Summary
Verify the DSCP value of original packet header after GRE acl based tunnel encapsulation.
Configuration:

Configure interfaces on DUT.
Configure acl based tunnel configuration with action as encapsulation.
Attach the filter on ingress interface.
Configure the static route for the tunnel end point destination.
Capture packet on ATE on the receiving end(port-2).

Validation Coverage:
No packet drop should be oberserved.
Capture the packet on receiving end and verify the original DSCP value. Original value should not be altered.

Pull Request Functional Test Report for #1972 / 10f389cde4ef4ff04a547cdca61b679d09873c8b
Virtual Devices

Device
Test
Test Documentation
Job
Raw Log

Arista cEOS

TUN-1.9: GRE inner packet DSCP

Cisco 8000E

TUN-1.9: GRE inner packet DSCP

Cisco XRd

TUN-1.9: GRE inner packet DSCP

Juniper cPTX

TUN-1.9: GRE inner packet DSCP

Nokia SR Linux

TUN-1.9: GRE inner packet DSCP

Openconfig Lemming

TUN-1.9: GRE inner packet DSCP

Help

This PR introduced multiple static error checks into the main branch, can someone please resolve this?
@arulkumarsekar @sachendras @xw-g @cfernanz


Release Google.Cloud.Billing.V1 version 3.1.0

Changes in this release:
Bug fixes

More oauth scopes (commit 79b9658)

New features

Added Sku.geo_taxonomy (commit 79b9658)

Created release for Google.Cloud.Billing.V1-3.1.0


regression add delete case for every type

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

#26954


High CPU usage by TypingsInstaller

VSCode Version: Code 1.15.1 (41abd21afdf7424c89319ee7cb0445cc6f376959, 2017-08-16T18:07:25.676Z)
OS Version: Windows_NT x64 10.0.15063
Extensions:

Extension
Author (truncated)
Version

vscode-eslint
dba
1.2.11

pgsql
dou
0.0.8

EditorConfig
Edi
0.9.4

JSDocTagComplete
Hoo
0.0.2

vscode-esformatter
how
1.5.2

output-colorizer
IBM
0.1.2

vscode-puglint
mrm
2.3.0

code-spell-checker
str
1.3.7

(1 theme extensions excluded)

Opening certain projects causes a process to spawn that apparently downloads TS support files. These processes regularly cause extremely high CPU usage over long periods of time. I don't think this is intentional.
The log has a lot of:
Response has been sent.
Closing file watchers for project '/dev/null/inferredProject3*'
Closing file watchers for project '/dev/null/inferredProject3*' - done.
Installing typings ["emitter"]
Entry for package 'emitter' does not exist in local types registry - skipping...
All typings are known to be missing or invalid - no need to go any further

I'm also seeing this, which looks like an error:
Installing typings ["@fairmanager/core-secur","emitter"]
Package '@fairmanager/core-secur' is scoped and currently is not supported
...
Installing typings ["@fairmanager/core-securi","emitter"]
Package '@fairmanager/core-securi' is scoped and currently is not supported
...
Installing typings ["@fairmanager/core-security","emitter"]
Package '@fairmanager/core-security' is scoped and currently is not supported

The package in question is named core-security2.

This issue was moved to Microsoft/TypeScript#18263


Added a serialization context when invoking CustomSerializers

Added functionality for Issue: https://github.com/YAXLib/YAXLib/issues/163
Provides additional information for any custom serializers

This is a breaking change, isn't?

This is a breaking change, isn't?

Yes, it is (ICustomSerializer is changing). ISerializationContext would give more power to implementing custom serializers. For sure it required additional documentation, and for now we don't have tests and samples making use of the ISerializationContext.
@NKnusperer Your opinion?

It is indeed a break change and I did not create additional tests or samples for this logic.
Feel free to disregard this PR if it is inconvenient.
I am not an really an open-source developer so I am not sure how you merge breaking-changes different from non-breaking without forcing an major version bump.

Well breaking changes need a major version bump. It's unfortunately that 3.0 is released only a month ago. So a non breaking change is really preferred (e.g. a new interface next to this one).

It is unlikely that I can change this to a non-breaking change soon. Do you want me to retract this PR?

It is unlikely that I can change this to a non-breaking change soon.

To avoid a breaking change, and in case we would consider the change of ICustomSerializer for .NET5.0+ target frameworks only, could we benefit from the C#8 default interface implementations?

I like that idea!

does .net standard 2.1 also support default interface implementations

Indeed, it does.

Not sure why we would target netstandard2.1 instead of directly going with net5.0.
The concept of  .NET Standard is dead and new development is exclusively happen on .NET Core.
netstandard2.1 is also not supported by the full .NET Framework, so this would a platform breaking change anyway.

.net 5 will be EOL within a year.
.net standard 2.1 is also supported by .net core 3.1 - which isn't EOL

That why I'm talking about .NET Core instead of  .NET Standard.
Tbh I don't see a reason to switch the target framework anyway, we already have the "catch all" framework eg. netstandard2.0.
Changes this just for default interface implementations seems a bit drastic to me.
NET Core 3.1 will  be EOL in 12/2022 and the next LTS release is .NET 6.

switch the target framework

To be more clear about the proposal:
There would not be a need to move from NetStandard2.0 to NetStandard2.1 when using preprocessor directives and C#8 default interface implementations. NetStandard2.1 would be an additional target framework, where the changed interface had effect, while in NetStandard2.0 it had not.

Yes indeed @axunonb. As I'm not keen on releasing YaxLib 3.0 after a month of XayLib 2.0 😅

1. SerializationContext implementation
Proving ICustomSerializers with a SerializationContext will definitely make custom serializers more powerful. Just wondering, whether changing ICustomSerializer is ultimately a must. What if we

left ICustomSerializer as it is
created a class like SerializationContextAbstract<T> implementing ICustomSerializer<T>, and having a constructor with SerializationContext as the parameter, which is stored and usable with a corresponding property.

=> Custom serializers, that require the SerializationContext will derive from SerializationContextAbstract, all others stay untouched. We would have to distinguish these cases when creating an instance of the custom serializer.
2. ISerializationContext

would be good to add a member for YAXSerializationOptions
should go into a separate file

3. Others

Flag new code files with #nullable enable
Replace new object[0] with Array.Empty<object>()

What do you think about that?


Fix crash when parsing URLs with GET parameters without value

I've been receiving some crash reports for an app I'm developing and a subsequent investigation revealed that happened attempting to play YouTube videos whose URLs contain valueless GET parameters results in a crash when parsing the URL.
Here's an example: https://www.youtube.com/watch?v=yhnkD7YxJ1s&feature=youtu.be&a
It all boils down to the fact that the lib assumes every parameter has a specific value, although it may not be the case.

Good safety check, thanks @NSMyself! Thanks for using the library :)

Not a problem, thanks for developing and maintaining it :) Cheers


Fix: update golang version to 1.17.8

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 38.56%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
600

Relevant Lines:
1556

💛  - Coveralls


Compatibility with Django Q?

Hello. I went to kick the tyres of data-browser — looks awesome, thanks for sharing — but ran into an error related to Django Q's admin.py:
Traceback (most recent call last):
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/views.py", line 199, in query_html
    config = _get_config(request)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/views.py", line 113, in _get_config
    orm_models = get_models(request)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/orm_admin.py", line 424, in get_models
    model_admins, admin_fields = _get_all_admin_fields(request)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/orm_admin.py", line 185, in _get_all_admin_fields
    get_common(model_admin, model)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/orm_admin.py", line 175, in get_common
    all_model_fields[model].update(from_fieldsets(admin, include_calculated=False))
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/data_browser/orm_admin.py", line 149, in from_fieldsets
    fields = admin.get_fieldsets(request, obj)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/contrib/admin/options.py", line 337, in get_fieldsets
    return [(None, {'fields': self.get_fields(request, obj)})]
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/contrib/admin/options.py", line 328, in get_fields
    form = self._get_form_for_get_fields(request, obj)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/contrib/admin/options.py", line 667, in _get_form_for_get_fields
    return self.get_form(request, obj, fields=None)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django/contrib/admin/options.py", line 680, in get_form
    readonly_fields = self.get_readonly_fields(request, obj)
  File "/ifs-WfM5IAGo/lib/python3.7/site-packages/django_q/admin.py", line 30, in get_readonly_fields
    return list(self.readonly_fields) + [field.name for field in obj._meta.fields]

Exception Type: AttributeError at /data-browser/
Exception Value: 'NoneType' object has no attribute '_meta'

Got it — thanks for the advice 👍

Maybe I should expose that in settings as a handy way of dealing with issues in third party libs.

I just released that

Awesome! We'll remove our workaround and use that.
Already proving to be a very handy package, by the way. Particularly on a project where we (the developers) are not afforded direct access to the production environments with access to a shell etc.

I'm glad it's working out for you.
It has also become a critical tool at my work.
There's so much more I'd like to do with it, graphs, formulas, etc but unfortunately work takes up most of my headspace.
If you have any requests or issues please do reach out.
Also since you are a developer there are a couple of advanced features not covered in README.rst you might want to know about
1: Take a look at the table in the URL Format section of CONTRIBUTING.rst
2: Drop this into settings.py to enable traversal of "to many" links
def DATA_BROWSER_GET_FEATURE_FLAG(flag, request):
    return flag == "to_many"

This is behind a flag as you need a decent mental model of what it's doing to get correct results. I plan to add some explanatory tooltips and then make it generally available.
3: There is a lot in the readme I would strongly recommend giving it at least a top to bottom skim.


Remove query id in favor of query filename

fixes #735,

Thanks! Fixed a minor typo to help tests pass, and now queued for merge.

Trying to fix the test failures made me realize that this is the wrong approach because we are utilising SemverQuery.id in many places and passing the id around in all those places makes the code worse.
I would close this PR in favor of adding a test as described in #735 if you agree.

Sounds good!
On Thu, Apr 11, 2024, 5:20 PM Pavan Kumar Sunkara @.***>
wrote:

Trying to fix the test failures made me realize that this is the wrong
approach because we are utilising SemverQuery.id in many places and
passing the id around in all those places makes the code worse.
I would close this PR in favor of adding a test as described in #735
https://github.com/obi1kenobi/cargo-semver-checks/issues/735 if you
agree.
—
Reply to this email directly, view it on GitHub
https://github.com/obi1kenobi/cargo-semver-checks/pull/739#issuecomment-2050572208,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR5MSV26UINQUKMPCCVRFTY43477AVCNFSM6AAAAABGBKDZK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQGU3TEMRQHA
.
You are receiving this because you commented.Message ID:
@.***>


SAK-48657 Profile: Restrict photos to site members only

WDYT @juanjmerono ?

@ern Thanks for the review. We're going to share some thoughts about this in the core call, don't merge it yet.

@mpellicer Tested in our instance!. Please merge. Thanks! :)

Perfect @jesusmmp !


Is there any way to dump the whole colormap in a readable file ?

Hi,
I am aware of the command line ggcat dump-colors output.fasta.colors.dat that dump the colors associated to each individual file in a json file; but I was wondering whether there is an option to dump not only the colors associated to each file, but also those associated to the powerset of those colors.
Say, for instance, that some kmer is seen in files {1,2,4}, and that this subset {1,2,4} is associated to say color 8 in the colormap; in the final FASTA file this kmer would have header C:8:1. But without querying this kmer, I cannot know to which subset of colors corresponds 8.
So basically I am interested in accessing the colormap in plain. Is there any way this is possible ?

Hi,
I added an experimental query_colormap api function, to do what you requested.
The only caveat is that reading a single color set from the colormap is very slow (can take up to several ms), and it is much better to batch the queries in groups of subsets that have (almost) consecutive indices, especially for large datasets.


Task - New Taxon Name - part of speech

'Part of speech' should not be on the list of available options.

Thera are 5 options available

Part of speach
*Adjective
*Noun in opposition
*noun in genetive case
*participle

There should be only 4 options. The first one is a parent class. It should not be on the list

part_of_speech.assignable = False
The others are true

Pull, and rebuild.  Works for me.
On Wed, Sep 13, 2017 at 6:39 PM, Dmitry Dmitriev<EMAIL_ADDRESS>wrote:

part_of_speech.assignable = False
The others are true
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/SpeciesFileGroup/taxonworks/issues/209#issuecomment-329327236,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABYSDYdUkE6khZiYWuozwZPwFjpMnFZks5siGeogaJpZM4PWX_a
.


[AutoPR apimanagement/resource-manager] Custom properties were updated to reflect all the possible values

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/6249

This PR has been merged into https://github.com/Azure/azure-sdk-for-js/pull/3595


Include 'provider' column

Unless I'm missing something, I think you forgot the 'provider' column. I'm working on testing something similar out now so please let me know if I am in fact missing something :)

I see now that this is part of OAuthConsumerMixin - apologies!


MAINT: Refactor NI_LineExtend.

Simplify code to extend a buffer beyond the array's boundaries.
It also speeds things up for some of the modes, e.g. see these before/after timings:
a = np.random.rand(100, 10)

%timeit ndimage.uniform_filter1d(a, size=100, mode='nearest')
10000 loops, best of 3: 21.6 µs per loop
10000 loops, best of 3: 20.8 µs per loop

%timeit ndimage.uniform_filter1d(a, size=100, mode='wrap')
10000 loops, best of 3: 22.6 µs per loop
10000 loops, best of 3: 20.5 µs per loop

%timeit ndimage.uniform_filter1d(a, size=100, mode='reflect')
10000 loops, best of 3: 23.4 µs per loop
10000 loops, best of 3: 22.2 µs per loop

%timeit ndimage.uniform_filter1d(a, size=100, mode='mirror')
10000 loops, best of 3: 24.7 µs per loop
10000 loops, best of 3: 22.3 µs per loop

%timeit ndimage.uniform_filter1d(a, size=100, mode='constant')
10000 loops, best of 3: 20.1 µs per loop
10000 loops, best of 3: 20.6 µs per loop

So maybe up to 10% overall speed-up for 'wrap', 'reflect' and 'mirror'.

Have fixed the merge conflict, and I don't think I broke anything in the process, let's see what Travis has to say about it...

Travis is happy, merging.


Try ClimaCore with removed deprecations

This PR tries out https://github.com/CliMA/ClimaCore.jl/pull/1632.

We updated to ClimaCore 0.13, closing this.


"+ doesn't yank to system clipboard (with +clipboard +xterm_clipboard +X11)

Describe the bug
A clear and concise description of what the bug is.
yanking to the "+ register results with text in the "+ register, but not the system clipboard, despite
To Reproduce
Detailed steps to reproduce the behavior:

open terminal, run vim
echo has('clipboard')
echo has('xterm_clipboard')
echo has('X11')
select some text
"+y
:registers -> text is in "+ according to vim
exit vim
9 paste (Ctrl-V in browser, Ctrl-Shift-V in terminal, etc) -> no text

Expected behavior
A clear and concise description of what you expected to happen.
with +clipboard +xterm_clipboard and +X11, according to documentation the "+ register should correspond to system clipboard (gnome)
Screenshots
If applicable, copy/paste the text or add screenshots to help explain your problem.
$ sudo dpkg -l | grep vim
ii  vim-common                                    2:8.0.1453-1ubuntu1.3                            all          Vi IMproved - Common files
ii  vim-gnome                                     2:8.0.1453-1ubuntu1.3                            all          Vi IMproved - enhanced vi editor (dummy package)
ii  vim-gtk3                                      2:8.0.1453-1ubuntu1.3                            amd64        Vi IMproved - enhanced vi editor - with GTK3 GUI
ii  vim-gui-common                                2:8.0.1453-1ubuntu1.3                            all          Vi IMproved - Common GUI files
ii  vim-runtime                                   2:8.0.1453-1ubuntu1.3                            all          Vi IMproved - Runtime files
ii  vim-tiny                                      2:8.0.1453-1ubuntu1.3                            amd64        Vi IMproved - enhanced vi editor - compact version

Environment (please complete the following information):

Vim version
(installed through package vim-gtk and vim-gnome)

$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 18 2020 18:29:15)
Included patches: 1-1453
Modified by<EMAIL_ADDRESS>Compiled by<EMAIL_ADDRESS>Huge version with GTK3 GUI.  Features included (+) or not (-):
+acl               +farsi             +mouse_sgr         -tag_any_white
+arabic            +file_in_path      -mouse_sysmouse    +tcl
+autocmd           +find_in_path      +mouse_urxvt       +termguicolors
-autoservername    +float             +mouse_xterm       +terminal
+balloon_eval      +folding           +multi_byte        +terminfo
+balloon_eval_term -footer            +multi_lang        +termresponse
+browse            +fork()            -mzscheme          +textobjects
++builtin_terms    +gettext           +netbeans_intg     +timers
+byte_offset       -hangul_input      +num64             +title
+channel           +iconv             +packages          +toolbar
+cindent           +insert_expand     +path_extra        +user_commands
+clientserver      +job               +perl              +vertsplit
+clipboard         +jumplist          +persistent_undo   +virtualedit
+cmdline_compl     +keymap            +postscript        +visual
+cmdline_hist      +lambda            +printer           +visualextra
+cmdline_info      +langmap           +profile           +viminfo
+comments          +libcall           -python            +vreplace
+conceal           +linebreak         +python3           +wildignore
+cryptv            +lispindent        +quickfix          +wildmenu
+cscope            +listcmds          +reltime           +windows
+cursorbind        +localmap          +rightleft         +writebackup
+cursorshape       +lua               -ruby              +X11
+dialog_con_gui    +menu              +scrollbind        -xfontset
+diff              +mksession         +signs             +xim
+digraphs          +modify_fname      +smartindent       +xpm
+dnd               +mouse             +startuptime       +xsmp_interact
-ebcdic            +mouseshape        +statusline        +xterm_clipboard
+emacs_tags        +mouse_dec         -sun_workshop      -xterm_save
+eval              +mouse_gpm         +syntax
+ex_extra          -mouse_jsbterm     +tag_binary
+extra_search      +mouse_netterm     +tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK  -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Wdate-time  -g -O2 -fdebug-prefix-map=/build/vim-dfCUyu/vim-8.0.1453=. -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc   -Wl,-E  -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim   -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE  -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl  -L/usr/lib -llua5.2 -Wl,-E  -fstack-protector-strong -L/usr/local/lib  -L/usr/lib/x86_64-linux-gnu/perl/5.26/CORE -lperl -ldl -lm -lpthread -lcrypt  -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -lpython3.6m -lpthread -ldl -lutil -lm -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpthread -lm

OS: [e.g. Ubuntu 18.04, Windows 10 1809, macOS 10.14]

OS: Ubuntu 18.04 bionic
Kernel: x86_64 Linux 5.4.0-42-generic
Uptime: 16d 22h 35m
Packages: 3201
Shell: bash 4.4.20
Resolution: 3840x1200
DE: GNOME
WM: i3

Terminal:
GNOME Terminal 3.28.2 using VTE 0.52.2 +GNUTLS -PCRE2
XTerm(330)

After some digging, found the reason: exiting vim clears "+, and thus the clipboard. I don't know if this is intentional. Closing.

For more info, see
@wsdjeg It is not a feature, it is a consequence of how X11 clipboard is implemented: you do not tell X server “put X into the clipboard”, you tell it “I own the clipboard now”. Application you paste to will then negotiate with application owning the clipboard to get clipboard contents (this is used to paste in the specific format: e.g. gtk can do automatic conversion of the images AFAIR). To fix you need to use clipboard manager which will listen to clipboard owner changes, get clipboard contents and own it itself (at least, when application exits).

Some useful link: https://www.jwz.org/doc/x-cut-and-paste.html (you can get more by searching `X11 clipboard` in Google).

Originally posted by @ZyX-I in https://github.com/neovim/neovim/issues/3895#issuecomment-168447731

yes, that is how inter-process communication works in X11 (see https://tronche.com/gui/x/icccm/sec-2.html#s-2).
If you do not like this, you can use a clipboard manager.


WIP - test line_plot() with metadata

Please confirm that this PR has done the following:

[ ] Tests Added
[ ] Documentation Added
[ ] Description in RELEASE_NOTES.md Added

Adding to RELEASE_NOTES.md (remove section after adding to RELEASE_NOTES.md)
Please add a single line in the release notes similar to the following:
- (#XX)[http://link-to-pr.com] Added feature which does something

Description of PR
Please describe the changes introduced by this PR.

Coverage decreased (-9.8%) to 74.207% when pulling 83eaf22d34f2b9dab44049ca00e8f5655ceea63a on danielhuppmann:line_plot_with_meta into d41c99c398e693ca1ac683d1c26a432badcb9b45 on IAMconsortium:master.


Collect and log CosmosDB query metrics when extra logging is enabled

Collect and logs CosmosDB query metrics when extra logging is enabled
Fixes #4268
Description
if a request is performed with X-OW-EXTRA-LOGGING header set to on like below
curl -k --user "23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP" \
     -H "X-OW-EXTRA-LOGGING: on" \
     "https://<IP_ADDRESS>:10001/api/v1/namespaces/_/activations?limit=30&skip=0"

Then in the logs following entries would logged which provide detailed insight into how query is executed by CosmosDB
[2019-02-13T14:11:48.689Z] [INFO] [#tid_FUn23vWbEbEQBFi5rIcioaZJkB4a2Wfw] [CosmosDBArtifactStore] [QueryMetricsEnabled] Collection [subjects] - Query [SELECT r AS view FROM root r JOIN n in r.namespaces WHERE (NOT(IS_DEFINED(r.blocked)) OR r.blocked = false) AND ((r.uuid = @uuid AND r.key = @key) OR (n.uuid = @uuid AND n.key = @key))].
QueryMetrics
[Retrieved Document Count                 :               1
Retrieved Document Size                  :             428 bytes
Output Document Count                    :               1
Output Document Size                     :             682 bytes
Index Utilization                        :          100.00 %
Total Query Execution Time               :            0.00 milliseconds
Query Preparation Times
    Query Compilation Time               :        1.250000 milliseconds
    Logical Plan Build Time              :        0.440000 milliseconds
    Physical Plan Build Time             :        0.510000 milliseconds
    Query Optimization Time              :        0.070000 milliseconds
  Index Lookup Time                      :        0.640000 milliseconds
  Document Load Time                     :        0.020000 milliseconds
Runtime Execution Times
    Query Engine Times                   :        0.090000 milliseconds
    System Function Execution Time       :        0.000000 milliseconds
    User-defined Function Execution Time :        0.000000 milliseconds
  Document Write Time                    :        0.000000 milliseconds
Client Side Metrics
  Retry Count                            :              10
  Request Charge                         :           15.25 RUs

Partition Execution Timeline
?------------?------------------------------------?----------------?----------------?-------------?-------------------?-----------?
?Partition Id?                         Activity Id?Start Time (UTC)?  End Time (UTC)?Duration (ms)?Number of Documents?Retry Count?
?------------?------------------------------------?----------------?----------------?-------------?-------------------?-----------?
?           0?6d930ac9-39e8-49f9-a0c6-348ddc0a753b?   08:41:47:6920?   08:41:47:9370?        245.0?                  0?          4?
?           0?4a823e13-f723-48e8-a6e9-aaa8c975b3b3?   08:41:47:4540?   08:41:47:6910?        237.0?                  0?          3?
?           0?bc36fe9d-b75c-4a4f-a6d8-27c7aca7c209?   08:41:47:2170?   08:41:47:4540?        237.0?                  1?          2?
?           0?22fd7bb1-1c99-4df1-b934-f8f6ae584f01?   08:41:46:9830?   08:41:47:2160?        233.0?                  0?          1?
?           0?d0ddda24-5954-408e-9e16-038d36805301?   08:41:46:7380?   08:41:46:9720?        234.0?                  0?          0?
?------------?------------------------------------?----------------?----------------?-------------?-------------------?-----------?

Scheduling Metrics
?------------?------------------?-------------?--------------?--------------------?---------------------?
?Partition Id?Response Time (ms)?Run Time (ms)?Wait Time (ms)?Turnaround Time (ms)?Number of Preemptions?
?------------?------------------?-------------?--------------?--------------------?---------------------?
?           0?                 1?          233?             2?                 235?                    1?
?           0?                 1?          232?           248?                 479?                    2?
?           0?                 1?          236?           481?                 717?                    3?
?           0?                 1?          236?           718?                 954?                    4?
?           0?                 1?          244?           956?                1200?                    5?
?------------?------------------?-------------?--------------?--------------------?---------------------?
]
[2019-02-13T14:11:48.690Z] [INFO] [#tid_FUn23vWbEbEQBFi5rIcioaZJkB4a2Wfw] [CosmosDBArtifactStore]  [marker:database_queryView_finish:2342:2241]

Related issue and scope

[x] I opened an issue to propose and discuss this change (#4268)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[ ] I signed an Apache CLA.
[ ] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Codecov Report

Merging #4275 into master will increase coverage by 21.04%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #4275       +/-   ##
==========================================
+ Coverage   59.36%   80.4%   +21.04%
==========================================
  Files         162     162
  Lines        7585    7595       +10
  Branches      500     509        +9
==========================================
+ Hits         4503    6107     +1604
+ Misses       3082    1488     -1594

Impacted Files
Coverage Δ

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0% <0%> (ø)
:arrow_up:

.../org/apache/openwhisk/core/entity/EntityPath.scala
100% <0%> (+1.88%)
:arrow_up:

...a/org/apache/openwhisk/core/entity/Parameter.scala
95.45% <0%> (+2.27%)
:arrow_up:

.../org/apache/openwhisk/http/PoolingRestClient.scala
90% <0%> (+3.33%)
:arrow_up:

...hisk/core/controller/actions/SequenceActions.scala
92.1% <0%> (+3.5%)
:arrow_up:

...pache/openwhisk/core/containerpool/Container.scala
88.6% <0%> (+3.79%)
:arrow_up:

...abase/cosmosdb/CosmosDBArtifactStoreProvider.scala
4% <0%> (+4%)
:arrow_up:

...tainerpool/docker/DockerClientWithFileAccess.scala
96% <0%> (+4%)
:arrow_up:

...rg/apache/openwhisk/core/entity/ExecManifest.scala
97.87% <0%> (+4.25%)
:arrow_up:

...he/openwhisk/core/database/CouchDbRestClient.scala
87.23% <0%> (+4.25%)
:arrow_up:

... and 92 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a8564b...a651847. Read the comment docs.

LGTM Thanks!


[Bug]: SFTP atmoz Test container Permission Denied to delete file

Module
Core
Testcontainers version
1.15.2
Using the latest Testcontainers version?
Yes
Host OS
MAC
Host Arch
ARM
Docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:35 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.1 (90346)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:16 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?
Trying to create an SFTP server in testcontainer.
File is copied to sftp folder using withClasspathResourceMapping. While trying to do an sftp rm, it throws Permission denied issue:
private static final int PORT = 22;
private static final String USER = "cppadm";
private static final String USER_PASS = "cppadm::1000";
private static final GenericContainer sftp =
new GenericContainer("atmoz/sftp:latest")
.withClasspathResourceMapping("II_110227", "/home/" + USER + "/upload/II_110227/", BindMode.READ_WRITE)
.withExposedPorts(PORT)
.withCommand(USER_PASS);
Tried using withCopyFileToContainer and observed same permission denied issue
sftp get is working but sftp rm is not working eventhough Bindmode is READ_WRITE
The file is added to the upload folder with full permission but user and group is cppadm:
-rwxrwxrwx 1 root root 402 Dec 21 06:06 II_110227
Relevant log output
Unexpected exception thrown: com.jcraft.jsch.SftpException: Permission denied

Additional Information
No response

When I try to run in Gitlab, the same permission issue is hit
permission denied while trying to delete the file in upload folder

After reading atmoz/sftp implementation I was able to understand and here it is the code
I have the following src/test/resources/testcontainers/file.txt
@Test
void test() throws Exception {
    try (GenericContainer<?> sftp = new GenericContainer<>("atmoz/sftp:latest")
            .withCopyFileToContainer(MountableFile.forClasspathResource("testcontainers/", 0777), "/home/foo/upload/testcontainers")
            .withExposedPorts(22)
            .withCommand("foo:pass:::upload")) {
        sftp.start();
        JSch jsch = new JSch();
        Session jschSession = jsch.getSession("foo", sftp.getHost(), sftp.getMappedPort(22));
        jschSession.setPassword("pass");
        jschSession.setConfig("StrictHostKeyChecking", "no");
        jschSession.connect();
        ChannelSftp channel = (ChannelSftp) jschSession.openChannel("sftp");
        channel.connect();
        assertThat(channel.ls("/upload/testcontainers")).anyMatch(item -> item.toString().contains("file.txt"));
        assertThat(new String(channel.get("/upload/testcontainers/file.txt").readAllBytes())).contains("testcontainers");
        channel.rm("/upload/testcontainers/file.txt");
        assertThat(channel.ls("/upload/testcontainers/")).noneMatch(item -> item.toString().contains("testcontainers/file.txt"));
    }
}

Please, next time consider to submit a full example in order to reproduce it quickly :)

Thank you so much, it worked :)


Qt deprecation model halting compiltion

i have included json.hpp as a single file in a qt project using the msvc17 compiler in release mode
i get several of these errosr:
attribute 'deprecated'(since 3.0.0 ) cannot be used in this context
I don't know if commenting out the offending lines might cause a mjor disruption in the function of the library.

In order to asses your issue, we need the following information:

What is the issue you have?

Please describe the steps to reproduce the issue. Can you provide a small but working code example?

What is the expected behavior?

And what is the actual behavior instead?

Which compiler and operating system are you using? Is it a supported compiler?

Did you use a released version of the library or the version from the develop branch?

If you experience a compilation error: can you compile and run the unit tests?

compiler error
incldue "json.hpp" in a qt cretor project with msvc2017 32 bit release option enabled
the expected behaviour is to copile without errors
i have uysed the latest ( non develop branch) version
no i can't

What is the exact compiler error message?

the exact compiler message is :
C:\Users\Vincenzo Panella\Desktop\qtprojects\levgen4_8\json.hpp:20486: error: C2416: l'attributo 'deprecated("Since 3.0.0")' non può essere applicato in questo contesto
which translates:
C:\Users\Vincenzo Panella\Desktop\qtprojects\levgen4_8\json.hpp:20486: error: C2416: attribute 'deprecated("Since 3.0.0")' cannot be applied in this context

@v71 It's not a json bug, it's a MSVC bug, known and fixed in the version after:
https://developercommunity.visualstudio.com/content/problem/138957/wrong-compilation-error-for-deprecated-nested-frie.html
Either downgrade or upgrade to whatever version of MSVC has been released in 2019.


TWITTER ERROR: ffmpeg exited with code 1

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.02.07. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[ ] I've verified and I assure that I'm running youtube-dl 2017.02.07

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
C:\Transmogrifier>youtube-dl.py -v "https://twitter.com/ZackFradellaWx/status/82
9007356159737856/video/1"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://twitter.com/ZackFradellaWx/status/8
29007356159737856/video/1']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2017.02.07
[debug] Python version 2.7.6 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-71727-g46778ab, rtmpdump 2.4
[debug] Proxy map: {}
[twitter]<PHONE_NUMBER>59737856: Downloading webpage
[twitter:card]<PHONE_NUMBER>59737856: Downloading webpage
[twitter:card]<PHONE_NUMBER>59737856: Downloading m3u8 information
[debug] Invoking downloader on u'https://video.twimg.com/ext_tw_video/8290072576
22835200/pu/pl/720x1280/d5QIri_JHdHPd8ft.m3u8'
[download] Destination: Zack Fradella - Debris falling on I-55 as possible torna
do passes between Manchac and Ponchatoula. #lawx-829007356159737856.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-
8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0
 (Chrome)
' -i https://video.twimg.com/ext_tw_video/829007257622835200/pu/pl/720x1280/d5QI
ri_JHdHPd8ft.m3u8 -c copy -f mp4 'file:Zack Fradella - Debris falling on I-55 as
 possible tornado passes between Manchac and Ponchatoula. #lawx-8290073561597378
56.mp4.part'
ffmpeg version N-71727-g46778ab Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --
enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-l
ibilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enab
le-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --en
able-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --ena
ble-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc
 --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enabl
e-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --ena
ble-decklink --enable-zlib
  libavutil      54. 23.101 / 54. 23.101
  libavcodec     56. 35.101 / 56. 35.101
  libavformat    56. 31.100 / 56. 31.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 14.100 /  5. 14.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Using network protocols without global network initialization. Please use avform
at_network_init(), this will become mandatory later.
    Last message repeated 5 times
Input #0, hls,applehttp, from 'https://video.twimg.com/ext_tw_video/829007257622
835200/pu/pl/720x1280/d5QIri_JHdHPd8ft.m3u8':
  Duration: 00:00:09.97, start: 0.000000, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 720x1280,
 29.92 fps, 29.88 tbr, 90k tbn, 59.76 tbc
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, mono, fltp,
 126 kb/s
[mp4 @<PHONE_NUMBER>b3e9c0] Codec for stream 0 does not use global headers but cont
ainer format requires global headers
[mp4 @<PHONE_NUMBER>b3e9c0] Codec for stream 1 does not use global headers but cont
ainer format requires global headers
Output #0, mp4, to 'file:Zack Fradella - Debris falling on I-55 as possible torn
ado passes between Manchac and Ponchatoula. #lawx-829007356159737856.mp4.part':
  Metadata:
    encoder         : Lavf56.31.100
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 720x1280, q=2-31
, 29.92 fps, 29.88 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, mono, 126 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @<PHONE_NUMBER>b3e9c0] Malformed AAC bitstream detected: use the audio bitstre
am filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @<PHONE_NUMBER>b3e9c0] Malformed AAC bitstream detected: use the audio bitstre
am filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame=    2 fps=0.0 q=-1.0 Lsize=      42kB time=00:00:00.11 bitrate=2959.6kbits
/s
video:45kB audio:2kB subtitle:0kB other streams:0kB global headers:0kB muxing ov
erhead: unknown
Conversion failed!

ERROR: ffmpeg exited with code 1
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Transmogrifier\youtube-dl.py\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\__init__.py", line 452, in ma
in
    _real_main(argv)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\__init__.py", line 442, in _r
eal_main
    retcode = ydl.download(all_urls)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 1796, in
download
    url, force_generic_extractor=self.params.get('force_generic_extractor', Fals
e))
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 707, in e
xtract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 778, in p
rocess_ie_result
    new_result, download=download, extra_info=extra_info)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 753, in p
rocess_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 1440, in
process_video_result
    self.process_info(new_info)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 1704, in
process_info
    success = dl(filename, info_dict)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 1646, in
dl
    return fd.download(name, info)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\downloader\common.py", line 3
53, in download
    return self.real_download(filename, info_dict)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\downloader\external.py", line
 44, in real_download
    self.get_basename(), retval))
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\downloader\common.py", line 1
62, in report_error
    self.ydl.report_error(*args, **kargs)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 570, in r
eport_error
    self.trouble(error_message, tb)
  File "C:\Transmogrifier\youtube-dl.py\youtube_dl\YoutubeDL.py", line 532, in t
rouble
    tb_data = traceback.format_list(traceback.extract_stack())

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Single video: https://twitter.com/ZackFradellaWx/status/829007356159737856/video/1
Single video: https://twitter.com/TonyRussellTV/status/829015067224113156/video/1

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information
Thanks
Ringo

Guess I had not read through the issues enough...
SOLUTION:
-update FFMPEG & FFPROBE
-add --hls-prefer-native to command
...but updating the FFMPEG/FFPROBE i have reintroduced this NHL problem:
https://github.com/rg3/youtube-dl/issues/9300
very confused


Create full-stack case study

Description: It would be great to have one case study that illustrates our full stack of software, from WS1 to WS4. This would be best in the form of a jupyter notebook, ideally with an interactive part that allows the user to see all the different intermediary spatial data EO data -> classification -> polygons -> graph. Main purpose of this case study is to highlight the hard work done in all workstreams and enable new users to build on it 🚀
Features:
These are ordered, each building on top of the previous.

[ ] Create basic notebook that does the different steps one after the other (with one taking the input from the other). For readability this notebook should be as short as possible.
[ ] Add (time-static) spatial data from different time steps to GeoGraphViewer.
[ ] Figure out how we can make data for this case study accessible to other users (license, loading, hosting, etc.). This will make it easily reproducable and allow the notebook to serve as a great intro for new users to our library.
[ ] Optional: add all spatial data to GeoGraphTimeline, and let the user scroll through time in all spatial data layers.

Roughly what this what the visualisation part of the notebook might look like:

EDIT:
To be more specific. The stich together the most basic version of this case study we would need the following contributions:

WS1:

EO data at a particular time (the visualisation will only use part of it)
A pre-trained model
A small notebook that loads this pre-trained model and applies it to EO data to create landcover data

WS2:

A notebook that runs the basic pylandstats analysis on the created landcover data. (See notebook recently added)

WS3&4:

Create GeoGraph and visualise it using the GeoGraphViewer.

This is no longer within scope, therefore closing this.


form checkbox

form :
<form action="http://localhost:3000/" method="post">

	  <input type="checkbox" checked="checked" name="pets" value="dog">狗
	  <input type="checkbox" checked="checked" name="pets" value="cat">猫
	  <input type="checkbox" name="pets" value="fish">鱼

	  <input type="submit" value="提交"/>

</form>

parsed result by formidable :
{ pets: 'cat' }

actually, the right answer should be as following:
{ pets: [ 'dog', 'cat' ] }

Thanks. Maybe make sense for adding to the docs and the the more extended examples one day.

Natively supported on next major version. We already merged a PR for "fields and files" array-like support.


it's incompatible with angular version 4

Hi
Thank you for good Module but i'm using angular version "~4.2.3" when add your Module to my project get this Erorr:
Unexpected token <
  Evaluating http://localhost:2634/@ng-bootstrap/ng-bootstrap
  Evaluating http://localhost:2634/app/service/during-detail.component.js
  Evaluating http://localhost:2634/app/app.routing.module.js
  Evaluating http://localhost:2634/app/app.module.js
  Evaluating http://localhost:2634/app/main.js
  Loading ../app
    at eval (<anonymous>)
    at evaluate (system.src.js:2821)
    at system.src.js:3620
    at dynamicExecute (system.src.js:1144)
    at doEvaluate (system.src.js:1091)
    at ensureEvaluate (system.src.js:999)
    at system.src.js:1017
    at Object.eval (:2634/app/service/during-detail.component.js:18)
    at eval (:2634/app/service/during-detail.component.js:101)
    at eval (:2634/app/service/during-detail.component.js:102)

Hi
Thank you
Its not related to angular version, I am using it in angular 4.3.1.

Hi
I do any, you say for installation but get that error.
you don't any idea?

according to this question I think there might be a problem with import packages.
please check how you have imported this package
it should be like bellow
    import {DpDatePickerModule} from 'ng2-jalali-date-picker';

and if you use jalali-moment import it as the following code
import * as moment from 'jalali-moment';

Hi
Thanks for attentions,
I exactly do it that way you say but get that error,
can i call you?
On Wed, Aug 2, 2017 at 1:47 PM, mojtaba<EMAIL_ADDRESS>wrote:

according to this question
https://stackoverflow.com/questions/34538615/why-am-i-getting-unexpected-token-errors-in-angular-2
I think there might be a problem with import packages.
please check how you have imported this package
it should be like bellow
import {DpDatePickerModule} from 'ng2-jalali-date-picker';

and if you use jalali-moment https://github.com/fingerpich/jalali-moment
import it as the following code
import * as moment from 'jalali-moment';
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/fingerpich/jalali-angular-datepicker/issues/25#issuecomment-319616962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMg4PAXfN7haMKS_1Z-ZkpYFCeT8pye-ks5sUD6RgaJpZM4OobT9
.

--
With Best Regards
M.Sarmast

Hi
Thanks for attentions,
I exactly do it that way you say but get that error,
can i call you?

09039290480

باز روزی که select شده کلاسش درست Assign نمیشه
[image: Inline image 2]
On Wed, Aug 2, 2017 at 3:14 PM, mojtaba<EMAIL_ADDRESS>wrote:

09039290480
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/fingerpich/jalali-angular-datepicker/issues/25#issuecomment-319637011,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMg4PPH-xjANdpMQcifMQNJ0moHoYfO6ks5sUFMKgaJpZM4OobT9
.

--
With Best Regards
M.Sarmast


Update README.md

Problem
Just noticed that the link to the qdsl-paper (Everything Old Is New Again:
Quoted Domain-Specific Languages) was to a draft. Updated the link to the publised paper.
Solution
Describe the modifications you've done.
Checklist

[x] Update README.md if applicable
[x] Squash commits that aren't meaningful changes

@getquill/maintainers
Replace link from draft to the published version of qdsl.pdf

👍

thanks @berntan! 👍


Add CleanShot to Screnshot Software

Checklist

[x] I made something good today 💐.
[x] I put links in the correct format: [Tool](link) — description.
[x] My description starts with the lower-case letter and ends with the full stop.
[x] I put the tool in the alphabetical order.

Optional

[x] I added the appropriate labels: free, open-source or mac.
[x] I put the tool only to one category.
[x] I followed Contribution Guidelines.

@lukaszmtw thanks for the contribution!


x/time/rate: alternating AllowN before last and after last can create new tokens

What version of Go are you using (go version)?

$ go version
go version go1.18 windows/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
PS D:\code\src\yangr\rate> go env
set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\yangr\AppData\Local\go-build
set GOENV=C:\Users\yangr\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\yangr\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\yangr\go
set GOPRIVATE=
set GOPROXY=https://goproxy.io
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\code\src\yangr\rate\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\yangr\AppData\Local\Temp\go-build38909240=/tmp/go-build -gno-record-gcc-switches

What did you do?

package main
import (
"fmt"
"golang.org/x/time/rate"
"time"

)
func main() {
limiter := rate.NewLimiter(rate.Every(time.Second/1), 10)
t0 := time.Now()
t1 := time.Now().Add(-time.Second * 2)

t := t0
for limiter.AllowN(t, 1) {
	if t == t0 {
		t = t1
	} else {
		t = t0
	}
}

fmt.Println("exit")

}
What did you expect to see?
Exit loop
What did you see instead?
Always loop.
mark:
i also use the fixed(https://github.com/ZekeLu/time/commit/127e7a41be4d57aa3254fa7faa3673dec5ce8a6d) of https://github.com/golang/go/issues/52584, but it did not work.

@bcmill, @ianlancetaylor can you give this a look?

x/time/rate is more @Sajmani...

I think this is work as designed. It's covered by the test  TestLimiterJumpBackwards (https://github.com/golang/time/blob/f3bd1da661afd6357b957af27c50ccdb248b7dd3/rate/rate_test.go#L124-L138). On line 127, it jumps backward from t1 to t0. And then on line 131, it jumps forward from t0 to t1, and it got a new token.
In your demo, every time it jumps from t1 to t0, it gets two new tokens. That's why the available tokens never reduce to 0.
By the way, @tentree please change your comment to put your code into a fenced code block:

```go
package main

import (
	"fmt"
	"time"

	"golang.org/x/time/rate"
)

func main() {
	limiter := rate.NewLimiter(1, 10)

	t0 := time.Now()
	t1 := time.Now().Add(-time.Second * 2)

	t := t0
	for limiter.AllowN(t, 1) {
		if t == t0 {
			t = t1
		} else {
			t = t0
		}
	}

	fmt.Println("exit")
}
```

since t1 to t0 could got new tokens, i think t0 back to t1 also could reduce tokens.  This is what I'm confused about.

I think this is work as designed. It's covered by the test  TestLimiterJumpBackwards (https://github.com/golang/time/blob/f3bd1da661afd6357b957af27c50ccdb248b7dd3/rate/rate_test.go#L124-L138). On line 127, it jumps backward from t1 to t0. And then on line 131, it jumps forward from t0 to t1, and it got a new token.
In your demo, every time it jumps from t1 to t0, it gets two new tokens. That's why the available tokens never reduce to 0.
By the way, @tentree please change your comment to put your code into a fenced code block:
package main

import (
	"fmt"
	"time"

	"golang.org/x/time/rate"
)

func main() {
	limiter := rate.NewLimiter(1, 10)

	t0 := time.Now()
	t1 := time.Now().Add(-time.Second * 2)

	t := t0
	for limiter.AllowN(t, 1) {
		if t == t0 {
			t = t1
		} else {
			t = t0
		}
	}

	fmt.Println("exit")
}

since t1 to t0 could got new tokens, i think t0 back to t1 also could reduce tokens. This is what I'm confused about.

@tentree Please don't quote an entire comment unnecessarily. Jumping backward won't reduce the tokens. It's also covered by that same test (line 127, jump back to t0, two tokens remain).

@ZekeLu thanks , i may understand the designe of limiter.  back in the past just like fixed clock. the func with N（like AllowN） is just for testing easily, Is my understanding biased？

In my opinion, the time.Time parameter in the *N funcs is just for testing. Regarding whether the tokens should be dropped when it jumps backwards, there is some interesting comments here: https://go-review.googlesource.com/c/time/+/16672/2..6/rate/rate.go#b65
Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. Please close this issue if there is not bug. See https://github.com/golang/go/wiki/Questions.


setuptoolsCheckPhase: allow providing arguments via setuptoolsCheckFl…

…agsArray

Motivation for this change
cherry-picked from https://github.com/NixOS/nixpkgs/pull/102306/files#diff-78e272fcdca196c9997e1a2da70702ea519a5ebcea9a396bbb7f876750130844
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Change looks good to me, however, note that setuptools has deprecated support for python setup.py test https://github.com/NixOS/nixpkgs/issues/71897.
For 21.03 we need to change buildPythonPackage default from format = "setuptools"; to format = "pyproject". That also means that, unless this hook is explicitly included, one needs to specify a checkPhase manually.
Therefore, please avoid relying on this test invocation.

Change looks good to me, however, note that setuptools has deprecated support for python setup.py test #71897.
For 21.03 we need to change buildPythonPackage default from format = "setuptools"; to format = "pyproject". That also means that, unless this hook is explicitly included, one needs to specify a checkPhase manually.
Therefore, please avoid relying on this test invocation.

I have no clue if that works with pyobjc or not. When I tried executing the tests with pytest I got a lot of errors.

I have no clue if that works with pyobjc or not. When I tried executing the tests with pytest I got a lot of errors.

You can keep it, but realize that as soon as we change the default, all these packages will fail to build because no tests are run anymore. Of course we can keep that for until then, if its unclear how to solve it now.

I try running it manually to avoid the mass rebuild and future deprecation.

Thanks for the cherry-pick and the quick responses but I decided to avoid the deprecation and mass rebuilding and invoking it manually in checkPhase. This PR can be closed.