Support CORS headers for web browsers

I noticed that https://discovery.etcd.io/new doesn't respond with CORS (Cross Origin Resource Sharing) headers:
$ curl -i -w "\n" https://discovery.etcd.io/new
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: Thu, 02 Oct 2014 09:55:23 GMT
Content-Length: 58
Connection: keep-alive

https://discovery.etcd.io/$token

This means that JS web apps on different domains are unable to get discovery tokens from https://discovery.etcd.io/new. If browsers were able to get tokens, some interesting possibilities would pop up (I'm using DO as an example):

Get unique discovery token from https://discovery.etcd.io/new
Perform OAuth authentication against DO to get an OAuth Access Token
Create new DO droplets with the CoreOS image via their API by POSTing the cloud-config as user data. The cloud-config would include the discovery token generated in (1)

What this basically means: Setting up a CoreOS cluster from a web browser without any setup and just a few clicks. Pretty cool. My motivation for this is a browser based editor I'm working on. It generates tarball archives and sends them to dockerd for building. Being able to set up a (development) CoreOS cluster from a browser with a few clicks would be a first step towards cluster awareness in the dev environment.
DO has good CORS support which is documented here.

+1
Will look at code to see what needs to be done, but this is definitely a blocker for many interesting use cases. Alternative is to setup a proxy, which I really don't want to do, but it looks like I will have to temporarily.
BTW, Docker pulled CORS from Docker Hub last week. I've a support ticket in, but doesn't seem to be going anywhere...

:+1: I'm writing a web app that generates a config file for etcd2. The app runs in any web browser. In the app, I'm making an async request to https://discovery.etcd.io/new to generate a new token, but that request fails because the Access-Control-Allow-Origin header is missing.
Could you add a header for CORS so the /new endpoint can be called by anywhere?

@colhom there is a discussion on this on HN today. Probably should take another look at adding this header on the LB.

@philips: CORS from anywhere for /new on discovery.io has me worried. that's a write operation that we'd be opening up to any javascript code served on the internet. i feel like we're asking for a browser based ddos filling up our etcd cluster.
I feel like, if we're doing CORS from anywhere, we should put in a Referrer header based rate-limit cap on the /new endpoint. If certain domains want to exceed the limit legitimately, we'll have a white-list that exempts domains from the cap.
Thoughts?
\cc @EgoAleSum

DDOS is a legitimate concern, however, the endpoint is already susceptible. Accepting compliant CORS traffic would allow conforming agents to interact with the service. Non-conforming agents and non-browser HTTPS users can already access the endpoint.
A domain-based rate-limit seems reasonable, but it would be nice if you could afford to limit by the sending address. I think there might also be a "reverse-ddos" attack where the referrer is spoofed to exceed the rate of a domain.

@colhom is on the right track here.
The DoS risk isn't affected much by this, but wildcard CORS opens new horizons in DDoS. Malicious actors who can MitM web traffic can inject Javascript, causing browsers to become attack nodes. See for example China's "Great Cannon" system that was used against Github last year.
If we're going to wildcard we should implement rate limiting at the same time.


Make hub env-vars indra-specific

If users have existing env variable that are called the same thing, hub-run scripts will fail.
We should add "indra-" before all indra-specific env variables to avoid collisions.
(This happened with Ujo and the DB-URL variable)

Glad you noticed this, I can fix it today

This has been fixed by Indra commit 59725dbd22ab138b42170c4fc89b9a42b599b8aa

Well, it's definitely fixed when deploying locally using docker. If this problem is still present (eg while deploying locally) then we can reopen & address those issues too


feat: add email validation function to lower bounce rates

The goal is to only return an error when we have a very high confidence the email won't be deliverable.
This is currently going to be added as a draft for the team to review. I haven't actually implemented any paths that call this or configuration around when it is activated.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

31 of 43 (72.09%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 57.05%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

internal/mailer/validate.go
31
43
72.09%

Totals

Change from base Build<PHONE_NUMBER>5:
0.04%

Covered Lines:
9577

Relevant Lines:
16787

💛  - Coveralls


Update package

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

For some reason the bot failed to create an update PR for this package, hence I'm creating it manually. Would you mind taking a look? Thanks.

@conda-forge-admin, please rerender

Thanks @renan-r-santos! Strange that it didn't autoupdate.


Resample / upsample behavior diverges from pandas

I've found a few issues where xarray's new resample / upsample functionality is diverging from Pandas. I think they are mostly surrounding how NaNs are treated. Thoughts from @shoyer, @darothen and others.
Gist with all the juicy details: https://gist.github.com/jhamman/354f0e5ff32a39550ffd25800e7214fc#file-xarray_resample-ipynb
xref: #1608, #1272

The key difference appears to be:

In xarray, .resample(...).interpolate(...) only interpolates over existing gaps in the data. If a value is already marked as NaN, it doesn't get interpolated.
In pandas, .resample(...).interpolate(...) fills in existing NaNs.

I think this is a bug in pandas, since the behavior is inconsistent with other resample methods like ffill():
>>> s.reindex_like(slike).resample('1D').ffill()
time
2016-01-01     NaN
2016-01-02     0.0
2016-01-03     1.0
2016-01-04     2.0
2016-01-05     3.0
2016-01-06     NaN
2016-01-07     4.0
2016-01-08     5.0
2016-01-09     6.0
2016-01-10     7.0
2016-01-11     8.0
2016-01-12     9.0
2016-01-13    10.0
2016-01-14     NaN
2016-01-15     NaN
Freq: D, dtype: float32

More generally: resample() exists for resampling existing values, not filling in missing values. If you want to fill in values that are already NaN, you should use one of the existing filling methods (e.g., fillna() or interpolate()). Or you can drop this filling values with .dropna().
(This does suggest that xarray could use a direct DataArray.interpolate() method.)
Another example:
>>> s.reindex_like(slike).resample('12H').ffill()
time
2016-01-01 00:00:00     NaN
2016-01-01 12:00:00     NaN
2016-01-02 00:00:00     0.0
2016-01-02 12:00:00     0.0
2016-01-03 00:00:00     1.0
2016-01-03 12:00:00     1.0
2016-01-04 00:00:00     2.0
2016-01-04 12:00:00     2.0
2016-01-05 00:00:00     3.0
2016-01-05 12:00:00     3.0
2016-01-06 00:00:00     NaN
2016-01-06 12:00:00     NaN
2016-01-07 00:00:00     4.0
2016-01-07 12:00:00     4.0
2016-01-08 00:00:00     5.0
2016-01-08 12:00:00     5.0
2016-01-09 00:00:00     6.0
2016-01-09 12:00:00     6.0
2016-01-10 00:00:00     7.0
2016-01-10 12:00:00     7.0
2016-01-11 00:00:00     8.0
2016-01-11 12:00:00     8.0
2016-01-12 00:00:00     9.0
2016-01-12 12:00:00     9.0
2016-01-13 00:00:00    10.0
2016-01-13 12:00:00    10.0
2016-01-14 00:00:00     NaN
2016-01-14 12:00:00     NaN
2016-01-15 00:00:00     NaN
Freq: 12H, dtype: float32

It is useful that pandas's upsampling is only repeating values within the previously valid range. Otherwise it is likely to interpolate over true data gaps.
As another use-case: suppose we have a temperature dataset with 3 hourly measurements, and we want to upsample it to 1 hour resolution. Occasionally, measurements are missing for day(s) at a time, which we mark with missing values (suppose the server running the model ran out of disk space). It is useful to be able to resample to a higher resolution without entirely unrealistic interpolation over data gaps.

Thanks @shoyer. I always appreciated this feature in Pandas so I'm bummed to see it may not have been intentional. I need a xarray interpolate method that fills NaNs so I'll give that a go. I suspect it will be a widely used feature for dealing with missing data.

Let's see where the pandas discussion ends up. If xarray had a method for interpolating to fill missing values, achieving your desired result would be as a simple as chaining another interpolate call, e.g., .resample('1D').interpolate().interpolate_na() or .interpolate_na().resample('1D').interpolate().

Thanks for documenting this @jhamman. I think all the logic is in .resample(...).interpolate() to build out true interpolation or really imputation/infilling. I can jump in if there's any confusion in the code.

Thanks for posting this @jhamman.  It's really helping me understand what is going on with my data when I use xarray.  My understanding of Pandas is that it should not by default be interpolating - however I am downsampling and this is stated for upsampling (in Python for Data Analysis).


Module 1, Activity 1: nyc_trips csv and parquet tables differ

I ran the module 1 csv to parquet exercise without any issues. The crawlers were successful, schemas matched, and no exceptions. However, I checked the row counts and see the parquet table has 10 rows less that the source csv table.
spark.sql("select count(*) from nyc_trips_parquet").show()
+--------+
|count(1)|
+--------+
|25955596|
+--------+

spark.sql("select count(*) from nyc_trips_csv").show()
+--------+
|count(1)|
+--------+
|25955606|
+--------+

I re-ran the entire workshop from scratch and had the same results. I'll compare later and figure out the difference, but just wanted to note this in case anyone has already analyzed it.

I double checked with Athena which says they match:
SELECT count() FROM "nyc_trips"."nyc_trips_csv";
25955596
SELECT count() FROM "nyc_trips"."nyc_trips_parquet";
25955596
So, the question is why does spark sql show 10 more rows in nyc_trips_csv

Checking for unmatched rows via Athena and Glue spark.sql results in no rows:
SELECT vendor_name, trip_pickup_datetime, trip_dropoff_datetime FROM "nyc_trips"."nyc_trips_csv" EXCEPT SELECT vendor_name, trip_pickup_datetime, trip_dropoff_datetime FROM "nyc_trips"."nyc_trips_parquet"
SELECT vendor_name, trip_pickup_datetime, trip_dropoff_datetime FROM "nyc_trips"."nyc_trips_parquet" EXCEPT SELECT vendor_name, trip_pickup_datetime, trip_dropoff_datetime FROM "nyc_trips"."nyc_trips_csv"

The reason for the discrepancy is because the spark sql query is including the csv header row for each of the 10 input files. By filtering header rows, I get the same counts:
spark.sql("select count(*) from nyc_trips_csv").show()
25955606

spark.sql("select count(*) from nyc_trips_csv where vendor_name != 'vendor_name'").show()
25955596

spark.sql("select count(*) from nyc_trips_parquet").show()
25955596

Reference

Likewise, using the DataFrame header option:
df_csv = spark.read.csv("s3://ebo-glue-labs-1108-724598390090/data/nyc_trips_csv/")
df_csv.count()
25955606

df_csv = spark.read.option("header","true").csv("s3://ebo-glue-labs-1108-724598390090/data/nyc_trips_csv/")
df_csv.count()
25955596

yeah by default head is false, so the workshop should have the header turned on in the statement.


test-branch/execute-job/quoted-authorization/99f6747f-c834-46f1-b4d1-3d5a0d2da703

PR for testing d9c2cbde5a1a2f3d9f3c550b312f7955589be054

/canonical/self-hosted-runners/run-workflows d9c2cbde5a1a2f3d9f3c550b312f7955589be054


Dependency  "ocramius/package-versions": "^1.4" breaking php version requirements in 4.0.3

The dependency "ocramius/package-versions": "^1.4" introduces a bug with PHP 7.3 and 7.2. The package  "ocramius/package-versions" added PHP 7.4 as minimum required version in its 1.6 release. It's also using the new feature typed properties which results in
PHP Parse error:  syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST) in ...../vendor/ocramius/package-versions/src/PackageVersions/Installer.php on line 34
Unfortantly the 1.6.0 release of the package was not a major version change which should have been made as it's not backwards compatible. To keep the requirement of "php": ">=7.2" valid "ocramius/package-versions" must be constraint to be 1.5 as maximum.

Hey @oneserv-heuser

Unfortantly the 1.6.0 release of the package was not a major version change which should have been made as it's not backwards compatible.

Actually, @ocramius can bump the minimum required PHP version in a minor version of its package. This is OK because when you run composer update, Composer will never install a package that is not compatible with your version of PHP. The Composer solver will take care of this for you.
See https://www.doctrine-project.org/2017/07/25/php-7.1-requirement-and-composer.html
If you are having this issue, it is either that:

you are running composer update on a dev machine that does not have the same PHP version as your production environment. In this case, I would recommend either to use Docker or to enforce the PHP version in your composer.json (see https://andy-carter.com/blog/composer-php-platform)
you are using the --ignore-platform-reqs flag in Composer (which you should definitely avoid)

To keep the requirement of "php": ">=7.2" valid "ocramius/package-versions" must be constraint to be 1.5 as maximum.

This would actually introduce a lot of problems. Other packages having the same dependency could target other versions of this package. If another package has a dependency on ocramius/package-versions:^1.6, we would not be able to install it. That would introduce useless conflicts with other packages (which we definitely want to avoid!)

@moufmouf Well, must've been a weird bug then. Anyway, thanks for your reply!


Experiment with alternative HMM implementation

Requires https://github.com/tskit-dev/tsinfer/pull/959
Haven't tried this out at scale yet, but setting a threshold of mu**k to detect sequences with k mismatches seems much more intuitive to me than all the other fiddling around we had to do.

This seems to be working very well. Here's how we perform on a particular day:
ms-match (1/0)100%|████████████████████████████████████████████████████████████████████| 1.34k/1.34k [00:58, 23.1it/s]
ms-match (1/0)100%|████████████████████████████████████████████████████████████████████████| 454/454 [01:02, 7.25it/s]
ms-match (1/0)100%|████████████████████████████████████████████████████████████████████████| 143/143 [02:07, 1.12it/s]
ms-match (1/0)100%|██████████████████████████████████████████████████████████████████████| 82.0/82.0 [02:29, 1.82s/it]

On 2020-12-27 we match against a ts with nodes=231163;samples=209704;mutations=137794.
We run the first batch of 1339  matches with a likelihood threshold of 1 (i.e., we only consider two possible likelihoods, 0 and 1 essentially). Those run in 1 minute, and a rate of 23 matches per second. We find 885 matches with either 0 or 1 mismatches, which we accept as final.
We then run the second batch of the remaining 454 with a likelihood threshold of 0.125 (the arbitrarily chosen mu). This runs in one minute at a rate of 7 matches per second. We find 311 final matches with 1 or 2 mismatches.
We then run the third batch of the remaining 143 with a likelihood threshold of 0.01562 (mu**2). This runs in 2 minutes at a rate of 1.1 matches per second. We find 61 final matches found at with 2 or 3 mismatches.
We then run the final batch of 82 sequences at the highest possible precision (likelihood threshold of 1e-200). This runs in 2.5 minutes at a rate of 1.82 seconds per match.
The output is very similar to other recent inferences, although with substantially fewer recombinants (only 8 as of the end of 2020). I'm hoping this is because the final high-precision pass is actually finding 3 mutation paths that earlier versions working at a lower precision weren't.
Conclusions:

The general strategy of running the HMM multiple times at different likelihood thresholds is working really well and substantially speeds things up
The third-pass to pick up 2 and 3 mutation matches probably isn't worth it as we end up redoing most of the matches again anyway, and we can just do 0, 1 and final pass.
We can probably set a higher threshold for the final pass if we think about it a bit harder. However, some of the sequences we're interested in will have a very low likelihood indeed, and it is important that we get good matches for them.


Add AspNetCore runtime install on Linux

I had installed the dotnet dotnet-runtime and dotnet-sdk packages on Artix Linux and installing gave me an dotnet no frameworks were found error, I then installed aspnet-runtime package and it worked.

dotnet-sdk-7.0 is all you should need - it implicitly includes all other components (unless there's something weird about the packages on Artix?)

There seems to be the same issue on endavouros - I needed to install the aspnet-runtime package to run.


Implement basic wallet interface

This issue describes the basic wallet interface which we should implement for the testnet. It could evolve into a reference wallet implementation, though, at this stage it is rather simplistic.
The interface defines 4 methods:

receive_asset
send_asset
auth_tx
set_pub_key

The first two of the above methods should probably be an interface on their own, and we should recommend that most accounts implement these methods.
The goals is to provide a wallet with the following capabilities

The wallet is controlled by a single key. The signature scheme is assumed to be Falcon.

However, sending assets to the wallet does not require knowing which signature scheme is used by the recipient.

The user can send, receive, and exchange assets stored in the wallet with other users. All operations (including receiving assets) must be authenticated by the account owner.
The user can update the public key associated with the account as long as they are still in possession of the current private key.

Not supported in this implementation:

Multi-sig wallets.
Ability to create notes with multiple assets.

The implementation probably could go into Miden lib, maybe under miden::wallets::simple namespace - but there could be other options as well.
Interface method description
Below, we provide high-level details about each of the interface methods.
receive_asset method
The purpose of this method is to add a single asset to an account's vault. Pseudo-code for this method could look like so:
receive_asset(asset)
    self.add_asset(asset)
end

In the above, add_asset is a kernel procedure miden::sat::account::add_asset described in https://github.com/0xPolygonMiden/miden-base/issues/3#issuecomment-1374401652.
Note: this method does not increment account nonce. The nonce will be incremented in auth_tx method described below. Thus, receiving assets requires authentication.
send_asset method
The purpose of this method is to create a note which sends a single asset to the specified recipient. Pseudo-code for this method could look like so:
send_asset(asset, recipient)
    self.remove_asset(asset)
    tx.create_note(recipient, asset)
end

In the above, remove_asset is a kernel procedure miden::sat::account::remove_asset and create_note is a kernel procedure miden::sat::tx::create_note, both described in https://github.com/0xPolygonMiden/miden-base/issues/3#issuecomment-1374401652.
recipient is a partial hash of the created note computed outside the VM as hash(hash(hash(serial_num), script_hash), input_hash). This allows computing note hash as hash(recipient, vault_hash) where the vault_hash can be computed inside the VM based on the specified asset.
Note: this method also does not increment account nonce. The nonce will be incremented in auth_tx method described below. Thus, sending assets requires authentication.
auth_tx method
The purpose of this method is to authenticate a transaction. For the purposes of this method we make the following assumptions:

Public key of the account is stored in account storage at index 0.
To authenticate a transaction we sign hash(account_id || account_nonce || input_note_hash || output_note_hash) using Falcon signature scheme.

Pseudo-code for this method could look like so:
auth_tx()
    # compute the message to sign
    let account_id = self.get_id()
    let account_nonce = self.get_nonce()
    let input_notes_hash = tx.get_input_notes_hash()
    let output_notes_hash = tx.get_output_notes_hash()
    let m = hash(account_id, account_nonce, input_notes_hash, output_notes_hash)

    # get public key from account storage and verify signature
    let pub_key = self.get_item(0)
    falcon::verify_sig(pub_key, m)

    # increment account nonce
    self.increment_nonce()
end

It is assumed that the signature for falcon::verify_sig procedure will be provided non-deterministically via the advice provider. Thus, the above procedure can succeed only if the prover has a valid Falcon signature over hash(account_id || account_nonce || input_note_hash || output_note_hash) for the public key stored in the account.
All procedures invoked as a part of this method, except for falcon::verify_sig have equivalent kernel procedures described in https://github.com/0xPolygonMiden/miden-base/issues/3#issuecomment-1374401652. We assume that falcon::verify_sig is a part of Miden standard library.
Open question: should the signed message be different? For example, maybe we should include hash of the entire account state (initial and final) into the message hash as well?
set_pub_key method
The purpose of this method is to rotate an account's public key (i.e., replace the current key with a new value). For the purposes of this method we make the following assumptions:

Public key of the account is stored in account storage at index 0.
To authenticate the update we sign hash(account_id || account_nonce || old_key || new_key) using Falcon signature scheme.

Pseudo-code for this method could look like so:
set_pub_key(new_key)
    # compute message to sign
    let account_id = self.get_id()
    let account_nonce = self.get_nonce()
    let old_key = self.get_item(0)
    let m = hash(account_id, account_nonce, old_key, new_key)

    # verify signature
    falcon::verify_sig(old_key, m)

    # update the key at storage location 0 to a new value
    self.set_item(0, new_key)

    # increment account nonce
    self.increment_nonce()
end

It is assumed that the signature for falcon::verify_sig procedure will be provided non-deterministically via the advice provider. Thus, the above procedure can succeed only if the prover has a valid Falcon signature over hash(account_id || account_nonce || old_key || new_key) for the public key stored in the account.
All procedures invoked as a part of this method, except for falcon::verify_sig have equivalent kernel procedures described in https://github.com/0xPolygonMiden/miden-base/issues/3#issuecomment-1374401652. We assume that falcon::verify_sig is a part of Miden standard library.
Usage examples
Examples of using the above interface are described below.
Receiving funds
To receive funds into an account we'd need a note which invokes receive_asset method. Script for this note could look something like this (this is actually identical to P2ID script):
note_script()
  let target_account_id = self.get_input(0)
  assert(account.get_id() == target_account_id)
  for asset in self.get_assets()
    account.receive_asset(asset)
  end
end

The above script assumes that the recipient account ID is specified via note inputs.
In addition to the note, transaction consuming it would need to have a tx script which invokes auth_tx method like so:
tx_script()
  account.auth_tx()
end

To execute this transaction, the user will need to provide a signature over hash(account_id || account_nonce || input_note_hash || output_note_hash) against the public key stored in the account.
Sending funds
To send funds from an account we'd need to create a transaction which invokes send_asset method as a part of its tx script. Tx script for such a transaction could look like so:
tx_script()
  account.send_asset(<asset1>, <recipient1>)
  account.send_asset(<asset2>, <recipient2>)
  account.auth_tx()
end

To execute this transaction, the user will need to provide a signature over hash(account_id || account_nonce || input_note_hash || output_note_hash) against the public key stored in the account.
Swapping assets
We can also combine receive_asset and send_asset methods to execute an atomic swap. A script for a note involved in the swap could look like so:
note_script()
  let target_account_id = self.get_input(0)
  assert(account.get_id() == target_account_id)

  account.receive_asset(<asset1>)
  account.send_asset(<asset2>, <recipient>)
end

In the above case, asset1, asset2, and recipient are hardcoded into the note script. Anyone consuming this note will add asset1 into their account and will have to create carrying asset2 addressed to the specified recipient.
To consume this note in a transaction, the transaction will also need to include a tx script which looks something like this:
tx_script()
  account.auth_tx()
end

To execute this transaction, the user will need to provide a signature over hash(account_id || account_nonce || input_note_hash || output_note_hash) against the public key stored in the account.
Updating public key
To update public key of an account, we'd need to create a transaction which invokes set_pub_key method as a part of its tx script. Tx script for such a transaction could look like so:
tx_script()
  account.set_pub_key(<new_key>)
end

To execute this transaction, the user will need to provide a signature over hash(account_id || account_nonce || old_key || new_key) against the public key stored in the account prior to the update.

This is a super cool description for a basic wallet interface. Step by step, it takes shape.
Client-side proving
When I send a transaction, I would expect to be able to switch modes to either prove the transaction locally or via the sequencer. That would be cool to add even to the basic wallet since this is our USP. What do you think?
Upgrading the public key
What would be the use case to change your public key? From a first read, it doesn't sound that this needs to be part of the basic wallet interface. Maybe we can drop this feature for the MVP "Alice can send an asset to Bob"?

What happens if the asset was removed but no note was created because my computer crashed?

For this case, the proof would not have been generated, so effectively the complete transaction was aborted.

What would be the use case to change your public key? From a first read, it doesn't sound that this needs to be part of the basic wallet interface. Maybe we can drop this feature for the MVP "Alice can send an asset to Bob"?

What about we add tags to track issue priorities, for example one tag per planned release? Then we can have one issue per feature and keep track of the when to work on something.

Questions:
receive_asset method
How will the kernel validate that:

The asset given as an argument is valid, i.e. there is a non-nullified note backing it
The asset is consumed only once?

I'm asking because I expect the kernel to receive the complete note to verify the nullifier DB.
send_asset method
I have the feeling this API is too error prone, it seems totally doable to have a call to remove_asset without a call to create_note.
auth_tx method
This is more of a note, auth_tx requires that the user computes the Falcon signature prior to running the transaction, that means they must know all the notes' hashes. Meaning that create_note has to be deterministic. It seems an important enough property that we should document it.

Open question: should the signed message be different? For example, maybe we should include hash of the entire account state (initial and final) into the message hash as well?

I have the feeling the state hash can replace the nonce, the same way that block's have prev_hash this creates ordering. It is still unclear to me how transactions are ordered though. (The nonce is useful there to determine how many transactions are missing to complete the queue, i.e. if account transaction nonce is 5 and the operator receives transaction 1000, they may as well just drop it, this is not really doable with a simple hash).
Sending funds
I think that authentication should be done as early as possible. Otherwise for network executed transactions the operator would have to run the transaction script until the end to determine if auth_tx succeeds. I think of auth_tx as opening the account context to be used.
Swapping assets
IMO, as it is, the script does not represent an atomic swap. That script has a account.receive_asset(<asset1>), which means the sender had created a note already. And then the receiver is creating the paying note account.send_asset(<asset2>, <recipient>). AFAICT it is missing a mechanism to only unlock the receive_asset if the send_asset is created. To do that the order would have to be different and the receive_asset script note would have to be executed and verify the corresponding send note was created.
The above interaction happens as follows. Alice creates a atomic swap with Bob and sends it to him, Bob receives the note for the atomic swap and run a transaction creating another note to complete the swap and consuming Alice's note, then sends its not to Alice.
What Alice's note must do is: 1. ensure that Bob created a valid note (right asset, amount, etc) 2. ensure the nonce bob chose is not nullified, AFAIU the nonce is open to be chosen by any participant, so this would require Alice's note to request data form the kernel (?) 3. and that she can consume that note. That means that Alice's note script must have access to the complete note produce from Bob, including the script, and that has been agreed prior to the atomic swap.
I have a different proposal. Both Alice and Bob can send a hash(secret) to each other, then concurrently both produce note's that are bound to the revealing of both secrets, and send the notes to one another. Then both Alice and Bob can verify the validity of the notes, if anything is incorrect (i.e. the script is not standard, meaning someone else could be paid, the note is nullified, etc.). They just don't reveal the secret. Otherwise both reveal the secret, and only then both notes are atomically unlocked.

What would be the use case to change your public key? From a first read, it doesn't sound that this needs to be part of the basic wallet interface. Maybe we can drop this feature for the MVP "Alice can send an asset to Bob"?

Agreed - we don't need this for MVP and we should remove it. It is useful to think though this though because it has implications on how auth_tx is defined. For example, as is described now, there is bug in the interface which would allow an operator to steal user funds if the user tries to rotate their key.

Got it, but this is only the case if we run everything in the VM, right? But as a user, I should have the possibility to create a note or execute a transaction outside the VM, so not creating a proof, but still create a note and send it to the sequencer to be proven.

Transactions are always executed on the VM. In case when a transaction is executed by the operator, the user still needs to create a transaction, but can then send the transaction to the operator. The operator then has to fully execute the transaction. Regardless of who executes the transaction, it is always executed fully (from start to finish) and thus the entire transaction script is atomic.

How will the kernel validate that:

The asset given as an argument is valid, i.e. there is a non-nullified note backing it
The asset is consumed only once?

As described in #14, tx kernel will need to check that a note consumed in the transaction was created (i.e., is a part of Note DB), but it does not check whether it was already consumed. The check that the note is not a part of the nullifier DB happens later. Specifically, this check will be done by the block producer when aggregating transactions into a block.
So, it is possible to create (and prove) a transaction which double-spends funds. But such a transaction will not be included into a block because it will fail to pass the check agains the Nullifier DB.

I have the feeling this API is too error prone, it seems totally doable to have a call to remove_asset without a call to create_note.

I'm not sure I understand this point. The API exposes a single send_asset method and as a part of that method we have to execute both remove_asset and create_note - we can execute one or the other. Thus, send_asset operation is atomic. How would it allow users to call remove_asset individually?

This is more of a note, auth_tx requires that the user computes the Falcon signature prior to running the transaction, that means they must know all the notes' hashes. Meaning that create_note has to be deterministic. It seems an important enough property that we should document it.

The signature can be computed during transaction execution (i.e., the verify_sig procedure could request the signature from the advice provider) - so, we don't need to do it for all possible note hashes a transaction can produce. But a transaction can produce only a limited set of note hashes. These hashes would be either hard-coded in the transaction MAST or be a part of one of note inputs.

I have the feeling the state hash can replace the nonce, the same way that block's have prev_hash this creates ordering. It is still unclear to me how transactions are ordered though. (The nonce is useful there to determine how many transactions are missing to complete the queue, i.e. if account transaction nonce is 5 and the operator receives transaction 1000, they may as well just drop it, this is not really doable with a simple hash).

The purpose of the nonce is authentication, not ordering. Basically, the rule is: if the state of the account changes, the nonce must be incremented by 1. If the state of the account changes but the nonce is not incremented the tx kernel will fail (i.e., transaction cannot be proven).
This approach allows us to separate state updates from authentication. Specifically: many methods can change the state of the account without incrementing the nonce (i.e., send_asset, receive_asset) and then there could be one method which checks the signature and increments the nonce (i.e., auth_tx). So, if we call any of the methods which change the state, we must also call the method which increments the nonce (and calling this method requires producing a valid signature).

I think that authentication should be done as early as possible. Otherwise for network executed transactions the operator would have to run the transaction script until the end to determine if auth_tx succeeds. I think of auth_tx as opening the account context to be used.

We can't run authentication before output notes have been created because we don't know what to sign yet. If we sign only the initial account state, a malicious operator would be able to re-purpose the signature for a different transaction and steal funds from the account.

IMO, as it is, the script does not represent an atomic swap. That script has a account.receive_asset(<asset1>), which means the sender had created a note already. And then the receiver is creating the paying note account.send_asset(<asset2>, <recipient>).

Let me explain how this works more concretely. Let's say we have Alice and Bob and Alice wants to exchange 1 ETH for 100 MATIC with Bob. Alice executes a transaction against her own account which creates a note with 1 ETH in its vault and has the following script (this is a slightly modified version of the script in the original post):
note_script()
  assert(account.get_id() == <bob's account_id>)

  account.receive_asset(<asset: 1 ETH>)
  account.send_asset(<asset: 100 MATIC>, <alice recipient>)
end

This script can be executed by Bob only (because of the assert) and since the script must be executed atomically, the result of the script is creation of a new note which hashes to hash(<asset: 100 MATIC>, <alice recipient>). This means that the note will have 100 MATIC in its vault. The script of this note much be a part of the pre-image of <alice recipient>. Since Alice is the one who computes <alice recipient> in the first place, she controls what the script is, and she could set it to something like this:
note_script()
    assert(account.get_id() == <alice's account_id>)
    account.receive_asset(<asset: 100 MATIC>)
end

The above script can be executed by Alice only and it deposits 100 MATIC into her account.
So, we have:

Alice creates a note with 1 ETH which can be consumed only by Bob.
When this note is consumed, it deposits 1 ETH into Bob's account and forces Bob to create a note which contains 100 MATIC and can be consumed only by Alice.
Alice consumes the note created by Bob.

The above is an atomic swap split over 2 separate transactions.

Agreed - we don't need this for MVP and we should remove it. It is useful to think though this though because it has implications on how auth_tx is defined. For example, as is described now, there is bug in the interface which would allow an operator to steal user funds if the user tries to rotate their key.

I don't see the bug yet. I must be here, right?
set_pub_key(new_key)
    # compute message to sign
    let account_id = self.get_id()
    let account_nonce = self.get_nonce()
    let old_key = self.get_item(0)
    let m = hash(account_id, account_nonce, old_key, new_key)

    # verify signature
    falcon::verify_sig(old_key, m)

    # update the key at storage location 0 to a new value
    self.set_item(0, new_key)

    # increment account nonce
    self.increment_nonce()
end

So the operator could change it to a different key? Because you don't verify the signature, including the new_key. And then the operator could add a key which he holds the priv_key as well and steal the money.
Is that it?

The issue here is that if the user creates a transaction which calls set_pub_key the signature covers only account_id, account_nonce, old_key, and new_key. Thus, the operator could create a different transaction which creates output notes and use the same signature for that transaction. The malicious transaction could have the following tx script:
tx_script()
  account.send_asset(<asset>, <recipient>)
  account.set_pub_key(<new_key>)
end

Signature needed for this transaction would be exactly the same as the signature needed for the original transaction with tx script set to:
tx_script()
  account.set_pub_key(<new_key>)
end

The way to fix it is to make auth_tx the only way to authenticate the transaction and also to make it more restrictive (i.e., include input/output state of the account into the signature).

The check that the note is not a part of the nullifier DB happens later. Specifically, this check will be done by the block producer when aggregating transactions into a block.

Oh, okay. I suppose this is a consequence of parallelism? Because otherwise the nullifier DB would become a synchronization point?

I'm not sure I understand this point. The API exposes a single send_asset method and as a part of that method we have to execute both remove_asset and create_note - we can execute one or the other. Thus, send_asset operation is atomic. How would it allow users to call remove_asset individually?

I was thinking of the user of the API, i.e. another wallet developer.

The signature can be computed during transaction execution (i.e., the verify_sig procedure could request the signature from the advice provider) - so, we don't need to do it for all possible note hashes a transaction can produce. But a transaction can produce only a limited set of note hashes. These hashes would be either hard-coded in the transaction MAST or be a part of one of note inputs.

Yes, what I meant is that later, when the transaction is being added to a block, the signature has to stay valid.

We can't run authentication before output notes have been created because we don't know what to sign yet. If we sign only the initial account state, a malicious operator would be able to re-purpose the signature for a different transaction and steal funds from the account.

I wasn't saying that we should change what is signed, just when a signature is verified by the sequencer. IOW, the local user can't and doesn't need to run the verification code, so it runs the transaction to get the data and sign it. But then the sequencer should verify the signature first, so that they don't have to execute a transaction that had an invalid signature to begin with (it may still be the case the signature is valid, but that signed data is invalid, but that we can't fix)

Alice executes a transaction against her own account which creates a note with 1 ETH in its vault and has the following script [...] This script can be executed by Bob only (because of the assert) and since the script must be executed atomically, the result of the script is creation of a new note which hashes to hash(<asset: 100 MATIC>, ).

Oh, I didn't understand that was a note script, I thought it was a transaction script. But this raises a question: How can someone tell if a note is safe to execute? From what I see we can only execute whitelisted / audited. If Alice can produce a note that changes my account when I run it, and the notes can send assets from my account.

The issue here is that if the user creates a transaction which calls set_pub_key the signature covers only account_id, account_nonce, old_key, and new_key

Ah, I hand't understood this. So set_pub_key doesn't run auth_tx? It seems we should always use auth_tx, no?

Oh, okay. I suppose this is a consequence of parallelism? Because otherwise the nullifier DB would become a synchronization point?

All 3 state DBs (account, note, and nullifier) are synchronization points - so, we'll need to think how to support parallelism there. The good thing is that the updates should be relatively simple - i.e., update a leaf of a tree to a new value.

I was thinking of the user of the API, i.e. another wallet developer.

Ah - for other developers we provide full flexibility. I don't think we should constrain things at this level as it will also limit the types of interesting things people will be able to do.

I wasn't saying that we should change what is signed, just when a signature is verified by the sequencer. IOW, the local user can't and doesn't need to run the verification code, so it runs the transaction to get the data and sign it. But then the sequencer should verify the signature first, so that they don't have to execute a transaction that has an invalid signature to begin with (it may still be the case the signature is valid, but that signed data is invalid, but that we can't fix)

I think this is definitely the right strategy for well-known scripts as we know ahead of time the structure of the code to be executed - but I'm not sure this will work for arbitrary scripts as we just don't know what to do. Specifically:

We don't know where the public key should come from.
We don't know what signature scheme is to be used.
We don't know what is the message that needs to be signed.

Oh, I didn't understand that was a note script, I thought it was a transaction script. But this raises a question: How can someone tell if a note is safe to execute? From what I see we can only execute whitelisted / audited. If Alice can produce a note that changes my account when I run it, and the notes can send assets from my account.

In the model I described above, the note is safe to execute because it requires your signature. There is a UX aspect to this of course. Like ideally the user would be prompted by something like: "This transaction adds 1 ETH to your account and sends out 100 MATIC. Would you like to execute it?" And only if the user signs, the transaction will be executed.

Ah, I hand't understood this. So set_pub_key doesn't run auth_tx? It seems we should always use auth_tx, no?

Yep, that's what I'm leaning toward as well. But this would be a part of the wallet interface, rather than something we dictate at the protocol level.

One potential issue with the auth_tx described above is that in a context of a multi-sig account it may result in a poor UX. Specifically, once the transaction has been defined, we'd need to collect multiple signatures and that may take time. Since the message the signers need to sign includes the exact account state, no other transactions on the account will be possible (or rather any transaction executed on the account would invalidate the multi-sig transaction we are trying to get signatures for).
This is not really relevant in the context of the basic wall, but it may impact how we want to handle nonces, and thus, may be good to think through this now.
To fix the UX problem, we could do the following:

Exclude initial and final account states from the message to be signed, but include the new nonce.
Modify nonce validation rule, which is currently defined as: if an account's state changes, its nonce must be incremented by 1.

The second item above is perhaps the one which requires the most attention. One option is to change the rule to the following: if an account's state changes, new nonce must be different from the old nonce.
We do however want to avoid nonce re-use - so, this new rule would probably mean that instead of setting the nonce deterministically, the user would need to set it to some random value. This, in turn, implies that we want the nonce value to be pretty big - so, it should probably be bigger than a single field element. Assuming we use 64-bit account IDs, we can easily increase the size of the nonce to 3 field elements (192 bits) - and I think this should be sufficient.
Putting everything together, the new auth_tx would look something like this:
auth_tx(new_nonce)
    # update the nonce
    self.set_nonce(new_nonce)

    # get account ID and commitments to consumed and created notes
    let acct_id = self.get_id()
    let input_notes_hash = tx.get_input_notes_hash()
    let output_notes_hash = tx.get_output_notes_hash()

    # compute the message to be signed
    let m = hash(acct_id, new_nonce, input_notes_hash, output_notes_hash)

    # get public key from account storage and verify signature
    let pub_key = self.get_item(0)
    falcon::verify_sig(pub_key, m)
end

Signature generated to satisfy the above procedure binds the transactions to a given account and nonce, as well as to the set of consumed and produced notes. But it does not bind the account to its initial and final state, and thus, other transactions can be executed between the time this transaction is created and executed.

Can you help me better understand this model please @bobbinth because I don't see how it prevents replay attacks.  Lets consider the scenario where a user executes two transactions with two different nonces. Neither consume any notes (no nullifiers produced) and they both produce output notes.  What is stopping an adversary from iteratively replaying these two transactions in a loop and draining the account.  The only condition here is that the nonce changes?  I believe we will need some form of data structure (probably a SMT) that keeps track of already used nonces for an account.  The block production kernel would then be responsible for ensuring that transactions included in blocks do not have an already used nonce.  Maybe I'm missing something?

@frisitano - you are correct. The scheme I proposed would in fact have this problem.
Storing nonces in a sparse Merkle tree would work - but I wonder if the overhead this imposes will be too big (especially for on-chain accounts where all used nonces would need to be stored on-chain).
An alternative could be the following scheme: we still require that nonces increase monotonically, but we don't require that they increase by exactly one.
This way, for transactions which require some off-chain interaction, the user can increase the nonce by 1000 (or 1000000), and while they wait for this transaction to be signed etc. they can still issue other transactions against their account. It does add some extra bookkeeping requirements to the wallet software, but I don't think it should be too much.

@bobbinth Yeah I agree, the SMT is undesirable.  It wouldn't need to be particularly deep and could be pruned aggressively but if possible I think it should be avoided.
Yeah I think your suggestion sounds reasonable.  I think it would be good to flesh out the execution flow for a single scenario which requires this functionality (maybe the multi-sig example you suggested).  It's still not completely clear to me.

I will detail some information that is relevant to this issue.
Kernel Procedures
Account
To access and modify an accounts state we expose some user facing kernel procedures.  Accessors can be invoked from any context but modifiers can only be invoked from within the account context (i.e. an account procedure).  The procedures are defined here: https://github.com/0xPolygonMiden/miden-base/blob/main/miden-lib/asm/sat/account.masm
We currently do not have methods to add or remove assets from the account vault as we are waiting on the tiered merkle tree implementation.  The storage procedures have been defined and as such they can be used to set a storage slot to hold the falcon public key and also fetch it for transaction authorisation.
Notes
We also have methods for accessing information related to the note currently being consumed here: https://github.com/0xPolygonMiden/miden-base/blob/main/miden-lib/asm/sat/note.masm.
The create note procedure can be found here: https://github.com/0xPolygonMiden/miden-base/blob/main/miden-lib/asm/sat/tx.masm#L68
Rust objects and feeding the virtual machine
We define our rollup objects in the objects rust crate here - https://github.com/0xPolygonMiden/miden-base/tree/main/objects.  Of particular interest will be the Account type defined here - https://github.com/0xPolygonMiden/miden-base/blob/main/objects/src/accounts/mod.rs#L36-L58.  The rust objects are fed into the virtual machine using the advice provider, more specifically the implementation of the ToAdviceInputs trait on the object of interest (see an example for the account type here: https://github.com/0xPolygonMiden/miden-base/blob/main/objects/src/accounts/mod.rs#L144-L175).
An example of constructing an account object, including code, storage, vault and nonce is seen here: https://github.com/0xPolygonMiden/miden-base/blob/main/objects/src/mock.rs#L142-L227.
The transaction pipeline
A transaction is the entry point for the miden rollup. A transaction is composed of an account, a reference block, a set of notes and a transaction script.  We have provided machinery to construct and execute transaction in the miden-tx crate here:  https://github.com/0xPolygonMiden/miden-base/tree/main/miden-tx.  Of particular interest will be the TransactionExecutor which is responsible for sourcing data, compiling transactions and executing them. See here: https://github.com/0xPolygonMiden/miden-base/blob/main/miden-tx/src/executor/mod.rs#L85-L129

Nice writeup, thanks, @frisitano. Super helpful.

We currently do not have methods to add or remove assets from the account vault as we are waiting on the tiered merkle tree implementation.

Then I will hold this issue and start with another one.

Yes, using RPO is sufficient. For Falcon, this hasn't been merged yet (see https://github.com/0xPolygonMiden/miden-vm/pull/1000).
Btw, I wonder if we should split this interface into two separate modules. For example:

miden::std::wallets::basic - this module will contain receive_asset and send_asset procedures.
miden::std::eoa::basic - this module will contain auth_tx procedure. Or maybe we call it auth_tx_falcon to indicate that the signature used is Falcon signature (and in the future we could add auth_tx_ecdsa etc.).

What exactly do we need to authenticate? Only when we create a note and remove an asset from our account.

This depends on whether account is on-chain or off-chain. Specifically:

For on-chain accounts we need to authenticate only transitions which are "important" - e.g., moving assets out of the account (e.g., only when we do send_note).
For off-chain accounts we need to authenticate all state transitions (e.g., for both receive_asset and send_note). This ensure that only account owner can update the state (and thus should always have the latest state).


Detect and auto-fix git clone URLs in project config

If someone accidentally enters the clone URL
https://github.com/owner/repo.git

instead of the standard repo one
https://github.com/owner/repo

the sidekick should be nice and fix it.

:tada: This issue has been resolved in version 6.2.0 :tada:
The release is available on:

GitHub release
v6.2.0

Your semantic-release bot :package::rocket:


object detection multiprocessing graph load fail

The single image test works.
This appears to be a video capture issue.
The object detection is producing a graph.LoadTensor error
[INFO/ForkPoolWorker-1] process shutting down
[DEBUG/ForkPoolWorker-1] running all "atexit" finalizers with priority >= 0
[DEBUG/ForkPoolWorker-1] running the remaining "atexit" finalizers
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 103, in worker
    initializer(*initargs)
  File "py_examples/object_detection_app.py", line 113, in worker
    graph.LoadTensor(resize(frame/255.0,dim,1)[:,:,(2,1,0)].astype(np.float16), 'user object')
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'
[INFO/ForkPoolWorker-1] process exiting with exitcode 1
[DEBUG/MainProcess] cleaning up worker 0
[Level 5/MainProcess] finalizer calling <built-in function close> with args (18,) and kwargs {}
[DEBUG/MainProcess] added worker
[DEBUG/ForkPoolWorker-3] Queue._after_fork()
[DEBUG/ForkPoolWorker-3] Queue._after_fork()
[INFO/ForkPoolWorker-3] child process calling self.run()

@apiszcz look at https://github.com/gudovskiy/yoloNCS/issues/1

Solution for this issue was to uninstall opencv-python with pip3
Working!
Thank you for all the work on YoloNCS
https://stackoverflow.com/questions/42325764/python-opencv-videocapture-differences-between-python-2-7-to-python-3-5

1down voteaccepted
I had exactly the same problem. OpenCV was installed from the sources. The difference between Python 2 and 3 environment was, that for Python 3 opencv-python was additionally installed via pip3. Apip3 uninstall opencv-pythonsolved it in my case.

@apiszcz thank you.

Finally, I solved this problem by the following two step:

Following this web mentioned OPTION 2, install OPENCV by source code.
Replaced the compiled .so dynamic library with the one that comes with opencv-python python3 package:
sudo cp <path to opencv source repo>/build/lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so /usr/local/lib/python3.5/dist-packages/cv2/cv2.cpython-35m-x86_64-linux-gnu.so
Hope it can help you.

@zsy372901 I'm getting this error
Traceback (most recent call last):
File "py_examples/object_detection_app.py", line 1, in
import os,time,cv2,argparse,multiprocessing
File "/usr/local/lib/python3.5/dist-packages/cv2/init.py", line 9, in
from .cv2 import *
ImportError: libavcodec-7625dabe.so.58.6.103: cannot open shared object file: No such file or directory

Hey, I followed the link to install opencv from source. While executing sample examples, it is working well for C++ and Java example but it shows "No module named cv2" for the Python example. Can anyone please help me with this.
Thanks!

the opencv installation does not appear to be in your path for PYTHON.
PYTHONLIB is one way to add it.
On Thu, Dec 28, 2017 at 1:51 AM, Mashrin Srivastava <
<EMAIL_ADDRESS>wrote:

Hey, I followed the link to install opencv from source. While executing
sample examples, it is working well for C++ and Java example but it shows
"No module named cv2" for the Python example. Can anyone please help me
with this.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354239226,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXVTWaHq3zRwi56ReXY2sPKexso5AZMks5tEzqMgaJpZM4QJ5Md
.

@apiszcz thanks a lot for your reply.
I copied the cv2.so and cv2.py file from /usr/local/lib/python2.7/dist-packages to /usr/local/lib/python2.7/site-packages and added-
export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages to ~/.bashrc and now it is getting imported in python2.
But I am still not being able to run it for python3 and when I repeated the same step to add the cv2.so and cv2.py to the python3.6/site-packages and again adding a PYTHON path for it similarly, in bashrc.
But now, on importing cv2 in python3, I am getting a Segmentation fault error.
Let me know if you have any suggestions regarding this.
Thanks!

After a reboot, I am getting this error while importing cv2 using python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined symbol: PyCObject_Type

Those error messages that you are using Python 2.7 you need to install
python 3.5
On Dec 28, 2017 11:25 AM, "Mashrin Srivastava"<EMAIL_ADDRESS>wrote:

After a reboot, I am getting this error while importing cv2 using python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined
symbol: PyCObject_Type
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354314919,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXVTbJ0Qb1h-xNG0zj-rKm7WXPuoz3Dks5tE8D6gaJpZM4QJ5Md
.

I already have python 3.5 as well as python 3.6 installed.

On Thu, Dec 28, 2017 at 10:03 PM, apiszcz<EMAIL_ADDRESS>wrote:

Those error messages that you are using Python 2.7 you need to install
python 3.5
On Dec 28, 2017 11:25 AM, "Mashrin Srivastava"<EMAIL_ADDRESS>wrote:

After a reboot, I am getting this error while importing cv2 using python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so:
undefined
symbol: PyCObject_Type
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354314919,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABXVTbJ0Qb1h-xNG0zj-
rKm7WXPuoz3Dks5tE8D6gaJpZM4QJ5Md>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354316240,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQFh3t4vCQovvfkPyJz-aifz7PZTyYc2ks5tE8LtgaJpZM4QJ5Md
.

Search path is incorrect
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
On Thu, Dec 28, 2017 at 1:39 PM, Mashrin Srivastava <
<EMAIL_ADDRESS>wrote:

I already have python 3.5 as well as python 3.6 installed.

On Thu, Dec 28, 2017 at 10:03 PM, apiszcz<EMAIL_ADDRESS>wrote:

Those error messages that you are using Python 2.7 you need to install
python 3.5
On Dec 28, 2017 11:25 AM, "Mashrin Srivastava"<EMAIL_ADDRESS>wrote:

After a reboot, I am getting this error while importing cv2 using
python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7,
in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so:
undefined
symbol: PyCObject_Type
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354314919
,
or mute the thread
<https://github.com/notifications/unsubscribe-
auth/ABXVTbJ0Qb1h-xNG0zj-
rKm7WXPuoz3Dks5tE8D6gaJpZM4QJ5Md>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354316240,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AQFh3t4vCQovvfkPyJz-
aifz7PZTyYc2ks5tE8LtgaJpZM4QJ5Md>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354334414,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXVTWRjsgKrZWLKBU8cOn45LEn26caHks5tE-BIgaJpZM4QJ5Md
.

I have already made the changes in bashrc.
I just copied the cv2.so and cv2.py from python 2.7 to python 3.5 and
python 3.6
And then added those path to the bashrc as well.
Now, when I try to run it, I am just getting segmentation fault (Nothing
else except this)
On 29-Dec-2017 12:11 AM, "apiszcz"<EMAIL_ADDRESS>wrote:

Search path is incorrect
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
On Thu, Dec 28, 2017 at 1:39 PM, Mashrin Srivastava <
<EMAIL_ADDRESS>wrote:

I already have python 3.5 as well as python 3.6 installed.

On Thu, Dec 28, 2017 at 10:03 PM, apiszcz<EMAIL_ADDRESS>wrote:

Those error messages that you are using Python 2.7 you need to install
python 3.5
On Dec 28, 2017 11:25 AM, "Mashrin Srivastava" <
<EMAIL_ADDRESS>wrote:

After a reboot, I am getting this error while importing cv2 using
python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7,
in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so:
undefined
symbol: PyCObject_Type
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-
354314919
,
or mute the thread
<https://github.com/notifications/unsubscribe-
auth/ABXVTbJ0Qb1h-xNG0zj-
rKm7WXPuoz3Dks5tE8D6gaJpZM4QJ5Md>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354316240
,
or mute the thread
<https://github.com/notifications/unsubscribe-
auth/AQFh3t4vCQovvfkPyJz-
aifz7PZTyYc2ks5tE8LtgaJpZM4QJ5Md>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354334414,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
ABXVTWRjsgKrZWLKBU8cOn45LEn26caHks5tE-BIgaJpZM4QJ5Md>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354334781,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQFh3isy02gP6WMJFMy6pvkQK16b-DU3ks5tE-DugaJpZM4QJ5Md
.

Do you think the existence of both python 3.5 and 3.6 can be a reason for
this?
On 29-Dec-2017 12:15 AM, "MASHRIN M SRIVASTAVA 14BCE1031" <
<EMAIL_ADDRESS>wrote:
I have already made the changes in bashrc.
I just copied the cv2.so and cv2.py from python 2.7 to python 3.5 and
python 3.6
And then added those path to the bashrc as well.
Now, when I try to run it, I am just getting segmentation fault (Nothing
else except this)
On 29-Dec-2017 12:11 AM, "apiszcz"<EMAIL_ADDRESS>wrote:

Search path is incorrect
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
On Thu, Dec 28, 2017 at 1:39 PM, Mashrin Srivastava <
<EMAIL_ADDRESS>wrote:

I already have python 3.5 as well as python 3.6 installed.

On Thu, Dec 28, 2017 at 10:03 PM, apiszcz<EMAIL_ADDRESS>wrote:

Those error messages that you are using Python 2.7 you need to install
python 3.5
On Dec 28, 2017 11:25 AM, "Mashrin Srivastava" <
<EMAIL_ADDRESS>wrote:

After a reboot, I am getting this error while importing cv2 using
python3
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7,
in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so:
undefined
symbol: PyCObject_Type
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-
354314919
,
or mute the thread
<https://github.com/notifications/unsubscribe-
auth/ABXVTbJ0Qb1h-xNG0zj-
rKm7WXPuoz3Dks5tE8D6gaJpZM4QJ5Md>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354316240
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AQFh3t4vCQovvfkPyJz-
aifz7PZTyYc2ks5tE8LtgaJpZM4QJ5Md>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354334414,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABXVTWRjs
gKrZWLKBU8cOn45LEn26caHks5tE-BIgaJpZM4QJ5Md>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gudovskiy/yoloNCS/issues/4#issuecomment-354334781,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQFh3isy02gP6WMJFMy6pvkQK16b-DU3ks5tE-DugaJpZM4QJ5Md
.

mashrin@mashrin:~$ python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

import cv2
quit()
mashrin@mashrin:~$ python3
Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import cv2
Segmentation fault (core dumped)
mashrin@mashrin:~$ python3.5
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
import cv2
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined symbol: PyCObject_Type
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/init.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in
import apt
File "/usr/lib/python3/dist-packages/apt/init.py", line 23, in
import apt_pkg
ImportError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/cv2/init.py", line 7, in
from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined symbol: PyCObject_Type

quit()

Output when importing using Python, Python 3.6 and 3.5:
(Also, bashrc now doesn't have any PYTHONPATH to opencv except for Python2.
mashrin@mashrin:~$ python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cv2
>>> quit()
mashrin@mashrin:~$ python3
Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Segmentation fault (core dumped)
mashrin@mashrin:~$ python3.5
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/cv2/__init__.py", line 7, in <module>
    from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined symbol: PyCObject_Type
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ImportError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/cv2/__init__.py", line 7, in <module>
    from . import cv2
ImportError: /usr/local/lib/python2.7/site-packages/cv2/cv2.so: undefined symbol: PyCObject_Type
>>> quit()

@zsy372901 on performing step 2 as mentioned by you, I am getting this.
cp: cannot stat '/home/mashrin/OpenCV/build/lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so': No such file or directory
The entire installation was completed without any error and I am even able to import cv2 using python2 after coping the cv2.so and cv2.py to /usr/local/lib/python2.7/site-packages
But, I am unable to import cv2 using python3. Also, I can't find cv2.cpython-35m-x86_64-linux-gnu.so anywhere. Kindly suggest.


uris must be created with the uri() function (not concat)

if the property is named "*uri" or "*url" then the value must not start with "[concat("
Skip parameters, but include vars
Also, there are properties that are arrays that we need to check - so if the property type is an array, check each element.

@bmoore-msft 👍


how to use

hi is there a tutorial that explains how to use it?

You can read the How to run or Standalone Executable
Please be careful in running them as it will modify your filesystem


Custom ID in add method

Hi, reading the documentation it looks like the only way to add a document to a collection with a custom "coordinates" key is the GeoCollectionReference.add() method, but in my application i need to set a custom document id, and the add() method doesn't provide the option of a custom id. Now if i use a GeoDocumentReference and use the set() method i lose the ability to set a custom "coordinates" key. It would be pretty useful to add an option to set a custom id in the add() method :)

Also, if i try to create the document with the set method and no custom "coordinates" key, i get this error
Unhandled error Error: Invalid GeoFirestore document: no valid key exists     at findCoordinatesKey (/user_code/node_modules/geofirestore/dist/index.cjs.js:308:15)     at encodeSetDocument (/user_code/node_modules/geofirestore/dist/index.cjs.js:242:27)     at GeoDocumentReference.set (/user_code/node_modules/geofirestore/dist/index.cjs.js:970:35)     at Promise (/user_code/lib/index.js:98:17)     at Object.<anonymous> (/user_code/lib/index.js:97:16)     at next (native)     at /user_code/lib/index.js:7:71     at __awaiter (/user_code/lib/index.js:3:12)     at exports.create_user.functions.https.onCall (/user_code/lib/index.js:87:65)     at /user_code/node_modules/firebase-functions/lib/providers/https.js:330:32

The set document method has a second argument of a setOptions where you can declare your custom key => https://geofirestore.com/interfaces/geofirestoretypes.geofirestoretypes.setoptions.html


chore: bdd-spring-1574192942 to 0.0.1

chore: Promote bdd-spring-1574192942 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Fullscreen mode

Summary

Added a settings box that allows the app to always be opened in full screen, if desired.
Added a CLI arg to force the app to be opened in full screen

If we don't like the specific config setting then we can just stick with the CLI arg.
CLI Example:
open release/mac/Mattermost.app --args --fullscreen true
open release/mac/Mattermost.app --args -f true

Ticket Link
Fixes https://github.com/mattermost/desktop/issues/1298
https://mattermost.atlassian.net/browse/MM-25484
Checklist

[X] Added or updated unit tests (required for all new features)
[X] Has UI changes
[X] executed npm run lint:js for proper code formatting

Device Information
This PR was tested on: Mac Monterey
Screenshots

Release Note
Added a new CLI parameter to open the application in full screen.
Additional setting/preference to always open the desktop application in full screen.

The files .circleci/config.yml, scripts/release.sh, scripts/release.sh are in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers

/update-branch

@esethna Pinging to check the wording on the settings page.

@coltoneshaw Also looks like a test is failing after merging in master.

@devinbinnie - Okay, I fixed the original test. It looks like it's now throwing an error on the below which I'm a bit confused about. The only change I made to this test was to read the Config.startInFullscreen. Any idea what broke it?
 FAIL  src/main/windows/mainWindow.test.js
  ● main/windows/mainWindow › createMainWindow › should set window size using bounds read from file

    expect(jest.fn()).toHaveBeenCalledWith(...expected)

    Expected: ObjectContaining {"fullscreen": undefined, "height": 700, "maximized": false, "width": 1280, "x": 400, "y": 300}
    Received: {"backgroundColor": "#fff", "frame": false, "fullscreen": false, "fullscreenable": true, "height": 700, "maximized": false, "minHeight": 240, "minWidth": 700, "paintWhenInitiallyHidden": true, "show": false, "title": undefined, "titleBarStyle": "hidden", "trafficLightPosition": {"x": 12, "y": 12}, "webPreferences": {"disableBlinkFeatures": "Auxclick", "nativeWindowOpen": true, "preload": undefined, "spellcheck": true}, "width": 1280, "x": 400, "y": 300}

    Number of calls: 1

      109 |         it('should set window size using bounds read from file', () => {
      110 |             createMainWindow({});
    > 111 |             expect(BrowserWindow).toHaveBeenCalledWith(expect.objectContaining({
          |                                   ^
      112 |                 x: 400,
      113 |                 y: 300,
      114 |                 width: 1280,

      at Object.<anonymous> (src/main/windows/mainWindow.test.js:111:35)

@devinbinnie - Okay, I fixed the original test. It looks like it's now throwing an error on the below which I'm a bit confused about. The only change I made to this test was to read the Config.startInFullscreen. Any idea what broke it?
 FAIL  src/main/windows/mainWindow.test.js
  ● main/windows/mainWindow › createMainWindow › should set window size using bounds read from file

    expect(jest.fn()).toHaveBeenCalledWith(...expected)

    Expected: ObjectContaining {"fullscreen": undefined, "height": 700, "maximized": false, "width": 1280, "x": 400, "y": 300}
    Received: {"backgroundColor": "#fff", "frame": false, "fullscreen": false, "fullscreenable": true, "height": 700, "maximized": false, "minHeight": 240, "minWidth": 700, "paintWhenInitiallyHidden": true, "show": false, "title": undefined, "titleBarStyle": "hidden", "trafficLightPosition": {"x": 12, "y": 12}, "webPreferences": {"disableBlinkFeatures": "Auxclick", "nativeWindowOpen": true, "preload": undefined, "spellcheck": true}, "width": 1280, "x": 400, "y": 300}

    Number of calls: 1

      109 |         it('should set window size using bounds read from file', () => {
      110 |             createMainWindow({});
    > 111 |             expect(BrowserWindow).toHaveBeenCalledWith(expect.objectContaining({
          |                                   ^
      112 |                 x: 400,
      113 |                 y: 300,
      114 |                 width: 1280,

      at Object.<anonymous> (src/main/windows/mainWindow.test.js:111:35)

Your function isFullScreen defaults to false, the object is expecting undefined.

@devinbinnie - Finally fixed this. Sorry for the delay, I haven't had a chance to look at this again until now.

@Willyfrog - Thanks for your changes! That makes total sense. Just commited them.

/update-branch


Run pylintrc for stashcp

We should run pylintrc on stashcp as part of the Travis-CI build; fail things if we aren't sufficiently happy.

Working on it in #33

@djw8605 - can you open a separate ticket to actually fail the build if there are concerns from pylint?  We might have to add a project-private pylintrc in order to make it ignore some of the more annoying code formatting items.


#35, fix mBank_set_tnr - it matches tnr with newlines

Coverage remained the same at 100.0% when pulling 309bcc87a168424fd2145e4e972b1a4f927bf336 on sq9mev:hotfix/35_mbank_breaklines into aac47dfeed4ab5230b99f337eb25cadd171194b6 on WoLpH:develop.


[Feature] Automatically scroll to the correct page when value is changed externally

Is your feature request related to a problem? Please describe.
Currently I'm implementing a two-way binded scroll feature where the user can scroll a list of calendar events and it will select the correct date on the calendar widget.
However I encountered an issue when changing the value, the new date can be in a different month or even year than before, and so the calendar doesn't scroll to the newly selected month, and the user is viewing irrelevant month.
Describe the feature you'd like
When changing the date value externally I would expect the calendar widget would automatically scroll to the relevant month and year.
Use cases
Two way binded scrolling as describe above.

Hi @HappyMakadiyaS, are you on CalendarDatePicker2Mode.scroll mode? Now you can achieve this by providing your own scrollViewController

Hi @theideasaler,
Not working with scroll mode. Working with the regular single date picker.
If you can also add a PageController property it would be good too.
Best implementation would be to include the controller property and also a boolean that decides whether to auto animate to the current value page.

This should be achievable now with the latest release.
More details can be found at https://github.com/theideasaler/calendar_date_picker2/tree/main/example

Thank you so much @theideasaler, but it doesn't seem to work at all..
I've set animateToDisplayedMonthDate to true, tried changing the value externally, still won't auto scroll to the correct month page.

Works like magic, thanks king!


Regenerate Azure AI Search with latest Swagger

Regenerate Azure AI Search with the latest Swagger available in azure-rest-api-specs. The SHAs for searchindex.json and searchservice.json are the following:
https://raw.githubusercontent.com/Azure/azure-rest-api-specs/4b7fbd8b842b509a0330f20260821dd844328dff/specification/search/data-plane/Azure.Search/preview/2024-09-01-preview/searchindex.json
https://raw.githubusercontent.com/Azure/azure-rest-api-specs/4b7fbd8b842b509a0330f20260821dd844328dff/specification/search/data-plane/Azure.Search/preview/2024-09-01-preview/searchservice.json
In addition to regenerating, as part of the 2024-07-01 service version SDK release there were Swagger transform added to rename and edit how code generates in the SDK. These Swagger modifications need to be added into the 2024-09-01-Preview generation so that the beta release for the new preview service version is backwards compatible with the last GA SDK release.

[ ] Regenerate searchindex.json and searchservice.json using the included Swagger links.
[ ] Ensure that the APIs modified with Swagger transforms are consistent with the last GA release.

Done.


Augment | TREC-6 DataBunch → Compare with Related Works

[x] Prepare TREC-6 DataBunch
[x] Experiment
[x] Compare with Related Works

For TREC the batch size must be smaller than other bigger datasets, otherwise it won't reach the state-of-the-art accuracy.


LanguageService should expand TypeScriptCompile elements

If you have the following in your project file:
<ItemGroup>
    <TypeScriptLibrary>myLib.d.ts</TypeScriptLibrary>
</ItemGroup>

<ItemGroup>
    <TypeScriptCompile Include="@(TypeScriptLibrary)" />
</ItemGroup>

The files in the TypeScriptLibrary items are not picked up.

Seems like no one else noticed


chore(workspace): add version info script

Fix

3690986

Chore

replace craco to vite.

:tada: This PR is included in version 1.0.0-alpha.24 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-alpha.21 :tada:
The release is available on npm package (@alpha dist-tag)
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-alpha.15 :tada:
The release is available on npm package (@alpha dist-tag)
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-alpha.1 :tada:
The release is available on npm package (@alpha dist-tag)
Your semantic-release bot :package::rocket:


bugfix: fixed incorrect bytestring encoding PlutusData

Fix #268
Summary
pycardano incorrectly encodes bytes longer than 64 bytes for PlutusData. Currently, a bytes element is encoded the same regardless of length, but if the length is larger than 64 bytes it should be broken up into 64 byte chunks per the following spec:
https://developers.cardano.org/docs/get-started/cardano-serialization-lib/transaction-metadata/#metadata-limitations
This PR fixes this issue by creating a dummy class to catch PlutusData objects during serialization and properly breaks up bytes values longer than 64 bytes in length.
If further explanation is needed, I can provide examples. This PR provides a unit test to verify the expected output is obtained with a long bytes input.

Is there a reason to introduce MetadatIndefiniteList? I don't see the benefit over using IndefiniteList directly

Removed the dummy class.
All unit tests pass, including the specific test created to test a long bytestring.
My specific use case works.
I'm happy to do any additional work required to finish this off. I didn't see any contributing doc, so I don't know how versioning is handled.

Codecov Report

Merging #269 (6dbdd4f) into main (0a95536) will increase coverage by 0.02%.
The diff coverage is 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
+ Coverage   85.14%   85.17%   +0.02%
==========================================
  Files          26       26
  Lines        2995     3001       +6
  Branches      719      721       +2
==========================================
+ Hits         2550     2556       +6
  Misses        335      335
  Partials      110      110

Files
Coverage Δ

pycardano/serialization.py
85.38% <100.00%> (+0.29%)
:arrow_up:

Alright, this was annoying. I made a new dummy class to wrap bytes. That seemed to properly wrap the PlutusData tests so that the unit tests pass (with a modification to the ground truth CBOR for one test), but wrecked about 20 other unit tests that did raw dictionary/list comparisons to primitives. I was able to get around that by modifying the comparator for the dummy class.
There's still one failing unit test that has to do with generating the script hash. I didn't have a chance to dig into it. Maybe you can take a look and more easily see what might be the issues. Once that is resolved, this should be good.

Alright, as best as I can tell, the last failing test was caused by serializing the COST_MODEL. If dictionaries are serialized to bytes, then the hash should change with the new way data is serialized to cbor.

Should be good to go now.
Now that I know all your qa stuff, it should be easier for me to contribute. I'm dedicated to building stuff on Cardano in Python, so I'll try to contribute as I find things. Like #273


[Bug]: Importing history with transforms doesn't work

Package Version
3.0.12
Flutter Version
3.22.2
Platforms
iOS
How to reproduce?
Steps to reproduce:

Open editor
Crop/Rotate the image
Save & export history (doesn't matter if it's toMap, toFile or toJson).
Open editor again with the same photo passing the previously exported history to StateHistoryConfigs

Expected:
Editor starts up with the image cropped/rotated.
Actual:
None of the previous transforms are restored.
------------
Issue is reproducible in the example project once we start updating the exported map.
Other properties like emojis etc. work as expected. Pretty sure the issue may be related to how the library maps the exported/imported json's.
Logs (optional)
Package Version
3.0.12

Flutter Version
3.22.2

Platforms
iOS

Example code (optional)
No response
Device Model (optional)
iPhone 15

Thanks for reporting this issue.
I have updated some code which should resolve this issue in version 3.0.13. Please let me know if you still have this problem after you update.


Number "lines to cover" Clover vs Jacoco

Our team has switched from Jacoco to Clover to ingest coverage reports into Sonarqube 8.9.2 Community Edition.
I have performed scans over two different projects with exactly the same code. One project I am using Clover and the project I am using Jacoco.
I have read in some forums that there might be a discrepancy on how the stastistics are calculated but I find weird that the overall "Lines to cover" under Jacoco are around 1k and under Clover are around 250 lines.
Checking the Measures tab under Sonarqube and running through the different packages I find that same java package reports 292 lines to cover in Jacoco based project while it reports only 62 lines to cover under the Clover project.
If I click on the top right "hamburguer" menu the "File measures" option. the number of lines, lines of code and comment lines are exactly the same but on the Coverage column is where the discrepancy shows.
Any idea? @Tony-Proum ?

Hi,
I would try to use the clover site (e.g: mvn clover:site) to see if the difference is also present on the clover report. The plugin here is just used to upload the clover report to sonarqube, the calculation have to be made before with clover itself. Could you check if the report is different when generated locally ?
I know that with maven, it exists some commands to aggregate reports when the project contains several modules etc, maybe does this issue related to the configuration of the clover reporting ?

No worries, I have just added your name to the conversation so the notification is triggered.
The maven reports are generated like this:
mvn clean clover:setup verify clover:aggregate clover:clover
Should I remove the aggregate flag? I can´t find in the html report for the particular .java file I was checking (randomly picked) and I can´t see where those "Lines to cover" are shown.

I think that this may be the point : if the file seems to be randomly picked by clover, it may not appear in the report uploaded in sonarqube. The repo here is responsible of the last point, but as far as I can remember, the configuration may be a bit tricky when the project directory tree is not conventional. Maybe the project you are instrumenting is needing more configuration ? For specific questions on clover, you may have more competent help here : https://github.com/openclover/clover
Also, I can try to help debugging even if there is a long time since I used clover. Is your project open source ? or maybe is your code publicly available ?

@Tony-Proum A few things:

the project is neither open source nor publicly available.
When I say randomly picked, I mean I just chose myself randomly to compare with Jacoco any java file. Apparently both Jacoco and Clover and scanning the coverage of all java files perfectly.
The question here is for a given java file, Jacoco and Clover are reporting different numbers, I am copying here:

I attach screenshots of what I mean. Is the right side where it says Coverage where number differs, uncovered lines and lines to cover
Clover: https://imgur.com/a/o6hnKDN
Jacoco: https://imgur.com/a/MAkNj68

the site should be generated in your target/clover/site directory (if you're using maven) also, I think that you should open an issue or talk with the clover team directly https://github.com/openclover/clover
AFAIK, the percentage calculated and uploaded by the plugin is extracted from the xml files generated by this project itself : https://github.com/openclover/clover

From the sonarqube forum, they say it means "Executable Lines"

the site should be generated in your target/clover/site directory (if you're using maven) also, I think that you should open an issue or talk with the clover team directly https://github.com/openclover/clover AFAIK, the percentage calculated and uploaded by the plugin is extracted from the xml files generated by this project itself : https://github.com/openclover/clover

Not sure if you have read my last answer @Tony-Proum

I just seen your last comment, could you try to generate the clover site in order to see what's in here ?
In my test cases the generated site and the coverage uploaded by the plugin seems equivalent so I cannot reproduce your issue.
In order to generate the site you can use this sample configuration :
<build>
   <pluginManagement>
       ... ( Your plugins here ) ...
       <plugin>
           <groupId>org.openclover</groupId>
           <artifactId>clover-maven-plugin</artifactId>
           <version>4.3.1</version>
           <configuration>
             <generateHtml>true</generateHtml>
             <generateXml>true</generateXml>
           </configuration>
       </plugin>
     </plugins>
   </pluginManagement>
 </build>```

then run
´´´
./mvnw clean clover:setup test clover:aggregate clover:clover
´´´
then, in the `target/site/clover` directory you should find the generated html site

I would like to be sure that the report generated is not errored

Thanks @Tony-Proum ! you pointed me to the right direction as you made me to review the whole thing.
I was using maven but my colleagues added this in the pom.xml:
<reportDescriptor>clover-report.xml</reportDescriptor>
that part I didn´t notice. Under the clover-report.xml you could find:
		<clover-report> 		  <current outfile="MyReport.xml" title="Coverage" summary="true"> 			<format srclevel="false" type="xml"/> 			<testsources dir="src/test" includes="/**/*.java"/> 			<testresults dir="target/surefire-reports" includes="TEST-*.xml"/> 		</current> 		</clover-report>
HTML and PDF reports were also added.
So now, I have commented out the clover-report.xml file and ingested the target\site\clover\clover.xml file and now the numbers are pretty similar:
Uncovered Lines 55
Lines to Cover 260
Uncovered Conditions 26
Conditions to Cover 78
while in Jacoco were:
Uncovered Lines 67
Lines to Cover 292
Uncovered Conditions 28
Conditions to Cover 83
Before with Jacoco I was using also mutationt tests with:
mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML
How do I combine clover and mutation tests now?  Mutation: removed call to com_atlassian_clover/CoverageRecorder::inc
which seem related to clover
I am using now this:
mvn clean clover:setup verify clover:aggregate clover:clover org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML

Oh happy to know that I may have helped 😁
I've done a search, and it seems like openclover does not support mutation coverage : https://openclover.org/doc/manual/latest/general--comparison-of-code-coverage-tools.html (so does jacoco) maybe should you exclude the mutation testing thing from the mvn clover command (exclude from the verify phase or use a more specific phase)
Or if you want to had this coverage too, maybe can you open a feature request here : https://github.com/openclover/clover

Oh happy to know that I may have helped 😁
I've done a search, and it seems like openclover does not support mutation coverage : https://openclover.org/doc/manual/latest/general--comparison-of-code-coverage-tools.html (so does jacoco) maybe should you exclude the mutation testing thing from the mvn clover command (exclude from the verify phase or use a more specific phase)
Or if you want to had this coverage too, maybe can you open a feature request here : https://github.com/openclover/clover

Thanks for pointing that out. How do I exclude the mutation thing? Sorry, I am pretty new to all this. Thanks for being so helpful

Thanks @Tony-Proum !
In the end I have run first the mutation tests and then without cleaning I run this:
mvn clover:setup verify clover:aggregate clover:clover
Also I now integrate the clover report like this with sonar:
-D"sonar.clover.reportPath=.\MyProject\target\site\clover\clover.xml"
and still be able to integrate dependency checks and mutation reports that were run before clover
It´s true now I run tests twice (maybe I can avoid it, what do you think?) but mutations reports are run without giving me so many vulnerabilities that were related to clover.
Anyway, would it make sense not to run the tests when calculating the clover coverage? Is it necessary?
Thanks again!

Thanks @Tony-Proum !
If I run
mvn clean clover:setup test clover:aggregate clover:clover org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML
the sonar project appears empty. Maybe am I missing something?
Thanks again!

@mgonzcast I think that it miss the sonar:sonar part of the maven command (this part is responsible of uploading the sonar/clover report to sonarQube)

Sorry, I use the sonar scanner instead of calling it from maven. I know it´s strange but I run the sonar scanner manually, this is not part of automated step.
This is exactly the steps that I do:
cd .\myProject
mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML
dependency-check\bin\dependency-check.bat" --out . -f="XML" -f="HTML" --scan .
cd ..
sonar-scanner-<IP_ADDRESS>1-windows\bin\sonar-scanner.bat" -D"sonar.projectKey=Gestor2CNegro-clover" -D"sonar.sources=." -D"sonar.host.url=http://localhost:9000" -D"sonar.login=111111111111111111111111111111" -D"sonar.dependencyCheck.xmlReportPath=.\myProject\dependency-check-report.xml" -D"sonar.clover.reportPath=.\myProject\target\site\clover\clover.xml" -D"dc5.mutationAnalysis.pitest.sensor.reports.directory=.\myProject\target\pit-reports\202202011234"

😄 oh ok, nothing wrong to run it this way, happy to see that it now works as expected !


Add instructions/ section for how users should interact with the site/ access the catalogue data  as .csv or .json

at the very least, a link to say data can be downloaded as .json from https://opendata.scot/datasets.json

Done and closing task. commit
(wrongly linked to the wrong issues board. habit, sorry.)


Server execution error: The server timed out waiting for the request to be processed. This may be due to high server load. Please try your request again.

Trying to execute "which gas station should I pick" and i keep getting: The server timed out waiting for the request to be processed. This may be due to high server load. Please try your request again.
I use Firefox for developers, i will try another browser as well.
EDIT: same timeout error in Chrome as well.
In firefox developer tab i see :
10:22:46.342 The Ably CDN URL you are using is deprecated because it does not contain a version number. Please see https://support.ably.io/solution/articles/3000063698
ably.min.js:26:9
10:22:51.794 Loading failed for the

Servers executing code are down.

Confirmed, I have the same issue (as earlier this week)...

Two days ago was the same thing with servers. Maybe because the academic year has started at schools...

Having the same issue here

Same thing here two days ago and today.

Is it possible to share processing power? I have a server at home that's powered 24/7, which is idle most of the time running OpenSuse. I would gladly share it if it helps balance the load!

I have the problem too. It's the same problem as the day before yesterday.

CW is down.
tmp solution: use preview.codewars.com
Works for me