Reduce test fixtures size to speed up tests

Current behavior
travis-ci takes > 15 minutes to run all tests. One reason is that the tests rely on a fixture database that's > 500MB in size. The fixture database is basically a zipped up testnet blockchain folder. This big size is the result of having testcases deployed on the regular TestNet, which then have many unnecessary blocks added to the fixtures in between different tests created on that network.
We believe that the fixture size can be easily reduced to < 50 MB by recreating all tests on a special private TestNet. This testnet should be based on neo-privatenet-docker
The image should only be spun up when adding new tests that requires events or interactions on a full blockchain (e.g. smart contract storage tests) to keep the database to the minimal size necessary.
TODO:

[ ] create an inventory of all tests to be ported to the private testnet (hint: look at what derives from: [BlockchainFixtureTestCase](https://github.com/CityOfZion/neo-python/blob/master/neo/Utils/BlockchainFixtureTestCase.
[ ] migrate all tests to a new private testnet docker image (based on neo-privatenet-docker)
[ ] make testnet docker image publicly available
[ ] create new fixture file, get it on CoZ AWS (ask @localhuman), update all test cases.

Hi ixje, I would be happy to help you with this issue. Shall I start with the creation of the inventory? Question: how exactly do you want to migrate all tests using docker? A concrete example would help here as I have basic knowledge of docker. Thanks.

Hi @dauTT that would be great!
To explain a bit more; almost all of the current tests are created on the official TestNet. Which means that there are smart contracts deployed on the official TestNet that have been interacted with in the past and we then confirm with our tests that we still get the same results.
We want to re-create those smart contracts (as the sources are likely lost @localhuman ?), but then deploy them on a new private network created with the neo-private-docker image. Have a look at https://github.com/CityOfZion/neo-privatenet-docker to see what that is about.

So the first step would be to try re-create the smart contracts belonging to the specific tests that are currently in neo-python. You can test and validate your new contracts either on the testnet or using a neo-private-docker.
once that is done we create a new private network where we deploy all smart contracts needed for the tests, do the required invocations (=interactions with the contracts) and then create a snapshot of the docker image to keep it small (because if you keep it running, it will keep generating blocks in the background and the blockchain will then unnecessarily increase in size).

I hope that makes sense. Let me know if I can clarify anything else.

Thanks ixje for the clarification. I believe I understand now what to do. I will proceed with migrating only one test (e.g,  neo-python/neo/SmartContract/tests/test_smart_contract2.py) and then after cross checking with you again I will migrate all the remaining test cases.

I'll be away for 4 days so I can't give you any support. If you want you can reach out to others like metachris and localhuman on Discord: https://discordapp.com/invite/b8QNXwD/

Hi ixje,
as @localhuman mentioned on Discord, he doesn't have the source code of the smart contracts used in the neo-python BlockchainFixture test cases. So he suggested that it may be a good idea to modify the test cases and use any smart contract in https://github.com/CityOfZion/neo-boa/tree/master/boa_test/example. I did that for a test case and was able to migrated to the private neo-private-docker. I will share the result of my test with you most likely tomorrow.

@dauTT Hello, I am wondering if you would like some help with this issue. Not sure how to collaborate in GitHub (I am new to GH) but I'd love to if that is possible.

sure, just drop me a message on Discord (same name as here)

@jseagrave21 , you most welcome to join. We will defintely need help here. :)
@ixje : I have create an image of the neo-privnet on the docker.hub:
docker pull  dautt/neo-privnet-unittest:v0.0.0
On this privnet I have deploy the smart contract (sample2.py) as described by localhum in his guideline. You can search for the contract by using this command:
contract search test
I have tested the image with neo-python cc4da934d3afc2ad46e1deb98ce638f0afd157bc (tag: v0.7.3). If we take that particular version and replace the 2 files:

https://github.com/CityOfZion/neo-python/blob/master/neo/SmartContract/tests/test_smart_contract.py
https://github.com/CityOfZion/neo-python/blob/master/neo/Utils/BlockchainFixtureTestCase.py

with the corresponding one in my fork:

https://github.com/dauTT/neo-python/blob/move-unittest-to-privnet-478/neo/SmartContract/tests/test_smart_contract.py
https://github.com/dauTT/neo-python/blob/move-unittest-to-privnet-478/neo/Utils/BlockchainFixtureTestCase.py

we see that the test cases in  test_smart_contract.py for the block 9369 will pass by running the following commands:

run the privnet:
docker run --rm -d --name neo-privnet-unittest  -p 20333-20336:20333-20336/tcp -p 30333-30336:30333-30336/tcp dautt/neo-privnet-unittest:v0.0.0
activate the virtual environment
python prompt.py -p
After the local blockchain is in sync with the privnet, exit from the prompt.
python3 -m unittest /neo/SmartContract/tests/test_smart_contract.py

Currently the latest neo-python version and the neo-privnet image are not compatible. So my branch move-unittest-to-privnet-478 is also not working properly. Do you know the reason for this discrepancy?
I will try to fix this issue as well, but meanwhile you can already review whatever has been done so far and let us know if we are in right direction. Thanks.
(Below, in the attachment, we find more information about the test case:)
POC test case.txt

@dauTT Could you specify exactly what you mean by "the latest neo-python version and the neo-privnet image are not compatible"? What errors/failures/warnings are you receiving?

After talking with @ixje, I think I understand how to proceed. I created a branch for this project but we will have to document our changes on this issue because there is no way we can think of to all edit the same code at the same time.
We will need to verify every test manually on the @dauTT's privnet before we can create the final snapshot.
I started to work on the next test: test_smart_contract2.py located here: https://github.com/jseagrave21/neo-python/blob/16133d5236f9e4c7218bf6a5e56480d16eaf8762/neo/SmartContract/tests/test_smart_contract2.py
Unfortunately, when I ran it I get this error:
ERROR: test_b_invocation (neo.SmartContract.tests.test_smart_contract2.SmartContractTest2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/Neo/neo-python-travis/neo/SmartContract/tests/test_smart_contract2.py", line 26, in test_b_invocation
    result = self._blockchain.Persist(block)
  File "/mnt/c/users/jseag/Neo/neo-python-travis/neo/Implementations/Blockchains/LevelDB/TestLevelDBBlockchain.py", line 71, in Persist
    if prevTx.outputs[input.PrevIndex].AssetId.ToBytes() == Blockchain.SystemShare().Hash.ToBytes():
AttributeError: 'NoneType' object has no attribute 'outputs'

I am not sure why I am receiving this error.
I will keep working.
I am also trying to think about a way to keep track of tests we have "updated". Maybe a Google Doc? What do you guys think? @ixje @dauTT

@dauTT looks good what you have so far! At the end of the road we'll have to keep most of your deletions from BlockchainFixtureTestCase.py, but for now it's a good approach for testing.
I followed all your instructions (nice a clear 👍) but I do not face any errors
(venv) Eriks-Air:neo-python erik$ python -m unittest neo/SmartContract/tests/test_smart_contract.py
[I 180801 20:39:28 LevelDBBlockchain:115] Created Blockchain DB at /Users/erik/.neopython/Chains/privnet
...
----------------------------------------------------------------------
Ran 3 tests in 0.408s

OK

The only difference is that I took the latest master 0.7.5 and applied your edits, where you run 0.7.3.

A document could work, or you post whatever test file you're going to work on next here. It's probably best to discuss it on Discord or alike though. Looking forward to your contributions 👍

I guess we can proceed to migrate the remaining test cases:

test_LevelDBBlockchain.py
test_leveldb.py
test_interop_blockchain.py
test_rest_api.py
test_json_invoke_rpc_api.py
test_json_rpc_api.py
test_smart_contract.py (closed)
test_smart_contract3.py
test_smart_contract2.py (@jseagrave21  is working on it)

There are not so many. So if you don't mind I can track them here. Please let me know if I have missed anything.
If we need to deploy more contracts on the privnet docker let me know. I can do it for you.

Hi @jseagrave21 , are you interested on working on the following test modules as well?

test_json_invoke_rpc_api.py
test_json_rpc_api.py
test_smart_contract3.py

Sure thing. I am stuck on test_smart_contract2.py. Would you mind taking a look at it? I will temporarily move on to test_smart_contract3.py

Okay, I will look at it maybe tomorrow. Thanks! :)

@dauTT I completed the update for test_smart_contract2.py
https://github.com/jseagrave21/neo-python/blob/patch-5/neo/SmartContract/tests/test_smart_contract2.py

@dauTT For test_smart_contract3, I think it is referencing a block which deploys a smart contract. I asked @ixje if there was a way to search the blocks in the privatenet for one which deploys a SC and he couldn't think of one. I am wondering if you think this test will require us to write a test SC and deploy it on the privatenet then test it with an updated version of test_smart_conract3?

Hi @jseagrave21 ,
I have already deploy a smart contract on privnet. It is related to block 9369. For more information you can look at the attachment that I have posted above (POC test case.txt). With respect to the a new SM contract deployment, If it make your life easy, just do it on your privnet for testing. But I will first try to work with what we already have. If you decide to use a new smart contract, remember that once I will deploy on the privnet to create the new image, your test cases won't work anymore and you will need to fix them again. So Unfortunately we need to do double work but if you have figure out how to fix them once, then the second time is quicker.

Oh that is great! I will see if I can work with that.

@dauTT @ixje okay test_smart_contract3 is done I think. Please review to make sure it is still doing what we want it to do.
https://github.com/jseagrave21/neo-python/blob/patch-5/neo/SmartContract/tests/test_smart_contract3.py

Still working on test_rest_api.py. It take a little bit longer as the unit test migration force me to get deeper into the understanding NEP5 standard and transfering token between different accounts. I have already deploy similar contract on the privnet and trying to migrate additional tests.
@jseagrave21 , could you please give an update from your side? Thanks.

@dauTT I am having similar issues. I have been struggling with understanding what the tests are trying to do and what the error codes mean. If you are having to deploy additional contacts, will you share an updated privnet for me to use? Also maybe some changes in the tests that have resulted in success?

Hi @jseagrave21 ,
I have created a new image:
docker pull dautt/neo-privnet-unittest:v0.0.4
So far I have deployed a couple of SM on the privnet. You can find them on the location
root/.neopython/UnitTest-SM of the running docker container.
I have modified the test module, test_rest_api.py as follow:

test_rest_api2_Dau.txt
BlockchainFixtureTestCase_Dau.txt

In order for you to run the test case you will need the following fixtures which are basically create from the above docker image:
fixtures_v8.tar.gz
notif_fixtures_v8.tar.gz
Relevant transactions used in the test case have been recorded in the following attached file:
test relevant transactions.txt
I am very close to finish the migration of the test_rest_api.py. Hopefully, It will done one or two day.
Let me know if you need further help.

@dauTT Awesome! I am digging in!

@dauTT I am tying to test my version of test_json_invoke_rpc_api.py but when I do I get this error from the BlockchainFixtureTestCase_Dau
ERROR: setUpClass (neo.api.JSONRPC.test_json_invoke_rpc_api.JsonRpcInvokeApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/Utils/BlockchainFixtureTestCase_Dau.py", line 35, in setUpClass
    response.raise_for_status()
  File "/mnt/c/users/jseag/Neo/neo-python-travis/venv/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/fixtures_v8.tar.gz```

Am I doing something wrong or do you need to give me access to the new fixture file?

Hi @jseagrave21 ,
if you look how the test cases are setup in test_rest_api2_Dau.py or test_rest_api2.py,
    def setUpClass(cls):

        super(NotificationDBTestCase, cls).setUpClass()

        if not os.path.exists(cls.N_FIXTURE_FILENAME):
            logzero.logger.info(
                "downloading fixture notification database from %s. this may take a while" % cls.N_FIXTURE_REMOTE_LOC)

            response = requests.get(cls.N_FIXTURE_REMOTE_LOC, stream=True)

            response.raise_for_status()
            with open(cls.N_FIXTURE_FILENAME, 'wb+') as handle:
                for block in response.iter_content(1024):
                    handle.write(block)

you can avoid to download the fixtures from AWS by putting the fixture archives here :
os.path.join(settings.DATA_DIR_PATH, 'docker-privnet/Chains/')
From my understanding, after we have migrated all the test cases to the privnet, we will create new  fixture archives and ask @localhuman to upload them to AWS.  Until then we have to do a workaround.

@dauTT Okay, I am almost done with test_json_invoke_rpc_api.py but I am running into an issue with test_invoke_5. I think it is because I am trying to use the wrong script for the invokescript. Where does the invokescript come from? Is it something I need to generate? Specifically, I am referring to here

Thanks @jseagrave21 , I will review your test cases soon and give you feedback.
@ixje , I need your help here to understand this strange behavior in test 9 of this module, test_rest_api.py
def test_9_by_tx(self):
        mock_req = requestMock(path=b'/tx/0x4c927a7f365cb842ea3576eae474a89183c9e43970a8509b23570a86cb4f5121')
        res = self.app.get_by_tx(mock_req, '0x4c927a7f365cb842ea3576eae474a89183c9e43970a8509b23570a86cb4f5121')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1)
        results = jsn['results']
        self.assertEqual(len(results), 1)

the transaction tx = 4c927a7f365cb842ea3576eae474a89183c9e43970a8509b23570a86cb4f5121 is included in block 455553. You can see more information about this test case here:

http://testnotifications.neeeo.org/v1/transaction/4c927a7f365cb842ea3576eae474a89183c9e43970a8509b23570a86cb4f5121

http://testnotifications.neeeo.org/v1/notifications/block/455553

If you look at the get_by_tx method you will see that the notification block (block_notifications) is actually stored at
height -1 = 455553-1 = 455552
def get_by_tx(self, request, tx_hash):
        request.setHeader('Content-Type', 'application/json')

        bc = Blockchain.Default()  # type: Blockchain
        notifications = []
        try:
            hash = UInt256.ParseString(tx_hash)
            tx, height = bc.GetTransaction(hash)
            print (tx, ' height =', height)
            if not tx:
                return self.format_message("Could not find transaction for hash %s" % (tx_hash))
            block_notifications = self.notif.get_by_block(height - 1)
            print ("block_notifications:", block_notifications)
            for n in block_notifications:
                if n.tx_hash == tx.Hash:
                    notifications.append(n)
        except Exception as e:
            logger.info("Could not get tx with hash %s because %s " % (tx_hash, e))
            return self.format_message("Could not get tx with hash %s because %s " % (tx_hash, e))

        return self.format_notifications(request, notifications)

In particular we can notice that the test transaction is stored at block 455553 but the corresponding notifications block is at 455552
If I use my fixtures and consider the following transaction which happened at block 9583 :
neo> wallet tkn_send b9fbcff6e50fd381160b822207231233dd3c56c2 AK2nJJpJr6o664CWJK
i1QRXjqeic2zRp8y AXpNr3SDfLXbPHNdqxYeHK5cYpKMHZxMZ9 10000
Used 2.807 Gas

-----------------------------------------------------------
Will transfer 10000.00000000 NXT2 from AK2nJJpJr6o664CWJKi1QRXjqeic2zRp8y to AXpNr3SDfLXbPHNdqxYeHK5cYpKMHZxMZ9
Transfer fee: 0.0001
-------------------------------------------------------------

[Password]> ***
[I 180809 23:50:52 Transaction:615] Verifying transaction: b'a2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8'
Relayed Tx: a2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8

and use it in test 9:
def test_9_by_tx(self):
        mock_req = requestMock(path=b'/tx/0xa2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8')
        res = self.app.get_by_tx(mock_req, '0xa2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1)
        results = jsn['results']
        self.assertEqual(len(results), 1)

**the test fail because the block_notifactions ( related to Tx: a2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8 ) are stored at 9583 instead of 9582 = 9583 -1. So in this particular both the the block of the test transaction and the corresponding notifications block have the same height number, 9583 **
{'type': 'SmartContract.Runtime.Notify',
'contract': '0xb9fbcff6e50fd381160b822207231233dd3c56c2',
'block': 9583,
'tx': '0xa2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8',
'notify_type': 'transfer',
'addr_to': 'AXpNr3SDfLXbPHNdqxYeHK5cYpKMHZxMZ9',
'addr_from': 'AK2nJJpJr6o664CWJKi1QRXjqeic2zRp8y', 'amount':<PHONE_NUMBER>000}

Do you have any explanation for this strange behavior?
(more information about the the tx = a2a37fd2ab7048d70d51eaa8af2815e0e542400329b05a34274771174180a7e8 can be found in the fixture archives or in the docker image: docker pull dautt/neo-privnet-unittest:v0.0.4)

Very good question and no idea. I've asked the author & maintainer if they can explain why that -1 block index is there in the first place. Once I understand that I can actually think about or look into why it's different now.

Thanks @ixje  for solving this puzzle. :+1:
I have cross-check again using the latest notification file:
"NotificationBootstrapFile":"https://s3.us-east-2.amazonaws.com/cityofzion/bootstrap_testnet/TestNotif1413xxx.tar.gz"
Indeed the  notification block of the transaction, 0x4c927a7f365cb842ea3576eae474a89183c9e43970a8509b23570a86cb4f5121 ,
is 455553 as we would have expected. So nothing to worry and we can move on with the migration of the unit tests.
@jseagrave21, I will try review everything by the end of this week.

@jseagrave21 , I have  reviewed the following:

test_json_invoke_rpc_api.py (Good. We still need to fix test_invoke_5)
test_json_rpc_api.py (Still some work to do)

----------------------------------------------------------------------
Ran 48 tests in 18.827s

FAILED (failures=15, errors=1)

In regard to your question related to test_get_block_int, I still need to investigate further.

test_smart_contract3.py (Good. :+1: )
test_smart_contract2.py (Good :+1: )

@ixje ,
In order for me to migrate this test case,
def test_pagination_for_addr_results(self):
        mock_req = requestMock(path=b'/addr/AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM')
        res = self.app.get_by_addr(mock_req, 'AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1027)
        results = jsn['results']
        self.assertEqual(len(results), 500)
        self.assertEqual(jsn['total_pages'], 3)

        mock_req = requestMock(path=b'/addr/AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM?page=1')
        res = self.app.get_by_addr(mock_req, 'AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1027)
        results = jsn['results']
        self.assertEqual(len(results), 500)

        mock_req = requestMock(path=b'/addr/AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM?page=2')
        res = self.app.get_by_addr(mock_req, 'AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1027)
        results = jsn['results']
        self.assertEqual(len(results), 500)

        mock_req = requestMock(path=b'/addr/AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM?page=3')
        res = self.app.get_by_addr(mock_req, 'AFmseVrdL9f9oyCzZefL9tG6UbvhPbdYzM')
        jsn = json.loads(res)
        self.assertEqual(jsn['total'], 1027)
        results = jsn['results']
        self.assertEqual(len(results), 27)

I will need to to create more then 1000 transfers. I am doing that automatically taking for granted that issue #558 will be fixed. In case issue # 558 is not a bug but a feature, I won't be able to generate the test case automatically. In that case we will need to do a lot of manual work if we want to preserve the test case. :(

@dauTT Thank you for the review. When I asked @localhuman about the invokeScript he sent me here but I am not sure how to implement this to build the script in order to retrieve the information to satisfy test_invoke_5 as written. Could you take a look and maybe we can discuss on Discord (in real time). It should be simple but I don't understand enough to make it work.

@jseagrave21 , I will have a look at test_invoke_5 in the upcoming days. If needed we can arrange a video call.
From my side, I manage to fully migrate test_rest_api.py to the privnet.
Please use the new update fixtures archives and image:
fixtures_v8.tar.gz
notif_fixtures_v8.tar.gz
docker pull dautt/neo-privnet-unittest:v0.0.18
Relevant transactions added in the new image are to be found in the following attachment:
test relevant transactions 3.txt
@ixje, we are getting close to the completion of this task.
To summarize the next steps:

Dau will look at test_invoke_5
@jseagrave21 will complete the migration of test_json_rpc_api.py
Dau will integrate @jseagrave21  work and make sure that all test cases are running correctly
@ixje  will do the final review
ask @localhuman  to upload the new fixture archives to AWS.
documentation
close the task

@dauTT I like your plan. My only question at this point is the issue I saw with test_get_block_int. When I query block 10 I return block 9 as indicated in my previous comment. Have you investigated this? Do I need to create a separate issue from the REST API -1 issue?

I did some more testing and I think the right script for test_invoke_5 is 0d444601ee23a8882f6fce312d5a0b59ea02f323f7f391e6ef3341669f411cbe. However, this does not return the expected values (i.e. bytearray(b'NEX Template V2'), bytearray(b'NXT2'), 8). This its raw return before it is sent to ContractParameter.FromJson:
'stack': [{'type': 'ByteArray', 'value': '444601ee23a8882f6fce312d5a'}, {'type': 'ByteArray', 'value': '59ea02f323f7f391e6ef33'}, {'type': 'ByteArray', 'value': '669f411cbe'}]

Hopefully this helps.

Thanks @jseagrave21  for the hint! :+1:
test_invoke_5 fixed:
def test_invoke_5(self):
        test_script = "00046e616d6567c2563cdd3312230722820b1681d30fe5f6cffbb9000673796d626f6c67c2563cdd3312230722820b1681d30fe5f6cffbb90008646563696d616c7367c2563cdd3312230722820b1681d30fe5f6cffbb9"

        req = self._gen_rpc_req("invokescript", params=[test_script])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(res['result']['state'], VMStateStr(VMState.HALT + VMState.BREAK))

        results = []
        for p in res['result']['stack']:
            results.append(ContractParameter.FromJson(p))

        self.assertEqual(len(results), 3)
        self.assertEqual(results[0].Value, bytearray(b'NEX Template V2'))
        self.assertEqual(results[1].Value, bytearray(b'NXT2'))
        self.assertEqual(results[2].Value, 8)

Wow. I spent a loong time trying to figure out how to do that! Good work @dauTT ! I am continuing to work on my remaining test but I have a feeling I am not going to be able to complete all of it. I will get as far as I can and then turn it over to you to look at.

@jseagrave21 ,
I looked at the test case test_get_block_int in your branch:
def test_get_block_int(self):
        req = self._gen_rpc_req("getblock", params=[10, 1])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(res['result']['index'], 9)
        self.assertEqual(res['result']['hash'], '0x7c5b4c8a70336bf68e8679be7c9a2a15f85c0f6d0e14389019dcc3edfab2bb4b')
        self.assertEqual(res['result']['confirmations'], 9598)
        self.assertEqual(res['result']['nextblockhash'], '0x7c5b4c8a70336bf68e8679be7c9a2a15f85c0f6d0e14389019dcc3edfab2bb4b')

This test case is not correct because we expect index = 10 instead of 9. However the test case pass why? the reason why it pass has nothing to do with #554. In general, at the present moment, when running our test cases by default it takes the setting of the TEST environment. So, the all privnet blockchain get the extra genesis block of TESTand therefore the chain is off of one. This is not correct. To fix this issue we need to activate the setting of privnet before we run the test cases. So please use this new update file going forward:
BlockchainFixtureTestCase_Dau.py.txt
Fortunately this change should not impact our previous testing. So, the new update test_get_block_int
will look like this:
def test_get_block_int(self):
        req = self._gen_rpc_req("getblock", params=[10, 1])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        print(res)
        self.assertEqual(res['result']['index'], 10)
        self.assertEqual(res['result']['hash'], '0xd69e7a1f62225a35fed91ca578f33447d93fa0fd2b2f662b957e19c38c1dab1e')
        self.assertEqual(res['result']['confirmations'], 12201)
        self.assertEqual(res['result']['nextblockhash'], '0x2b1c78633dae7ab81f64362e0828153079a17b018d779d0406491f84c27b086f')

I notice that the following assetion is no useful for us anymore:
self.assertEqual(res['result']['confirmations'], 12201)
If we look at the method, get_tx_output, behind the confirmations attribute we see that it is calculated as follow:
jsn['confirmations'] = Blockchain.Default().Height - header.Index + 1

This is not a good check in my opinion because every time we add a new block to the privnet fixtures, this confirmations number will change and therefore the assertion will fail. @ixje , are you fine to remove the above assertion from test_get_block_int?

@dauTT Perfect! That answers my question exactly. I am glad they were unrelated issues! I am still working but I haven't had much time this week because of school. Let me know if you decide to handle the remaining portion of test_json_rpc_api.py yourself.

@jseagrave21  no worry. I understand as I have also limited capacity due to work. So keep your pace. 👍

@dauTT For  test_account_state_not_existing_yet, I need an empty wallet. Is there an empty wallet on your testcase?

@jseagrave21 ,
the existing test case seems to work because this account, AHozf8x8GmyLnNv8ikQcPKgRHQTbFi46u2 does not exist on the privnet image (dautt/neo-privnet-unittest:v0.0.18)
def test_account_state_not_existing_yet(self):
        addr_str = 'AHozf8x8GmyLnNv8ikQcPKgRHQTbFi46u2'
        req = self._gen_rpc_req("getaccountstate", params=[addr_str])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(res['result']['balances'], {})
        self.assertEqual(res['result']['script_hash'], addr_str)

So we don't need to create an empty wallet.

@dauTT okay, I didn't know if that accomplished the intent of the test. Thank you for looking at it!

I see so much effort and perseverance, keep it up! 💪
If there's anything I can do or clarify let me know. I might be a slow in responding as my online time is a bit on and off the last week and probably next week.

@dauTT I think I have migrated as much as I can figure out. Here is my latest test:
https://github.com/jseagrave21/neo-python/blob/patch-5/neo/api/JSONRPC/test_json_rpc_api.py
Here are the failures/error I am still getting:
======================================================================
ERROR: test_send_raw_tx (neo.api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 425, in test_send_raw_tx
    self.assertEqual(res['result'], False)
KeyError: 'result'

======================================================================
FAIL: test_get_storage_item (neo.api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 342, in test_get_storage_item
    self.assertEqual(res['result'], '00f938dbba05')
AssertionError: None != '00f938dbba05'

======================================================================
FAIL: test_get_storage_item2 (neo.api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 352, in test_get_storage_item2
    self.assertEqual(res['result'], '003a0d7914b200')
AssertionError: None != '003a0d7914b200'

======================================================================
FAIL: test_get_storage_item_bad_contract_hash (neo.api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 376, in test_get_storage_item_bad_contract_hash
    self.assertTrue('error' in res)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 48 tests in 9.474s

FAILED (failures=3, errors=1)

I am not sure if your contract has storage and if it does, how to pull out the required info. And, I don't know how to send a valid raw tx. cc @ixje

Thanks @jseagrave21  for your hard work!  :+1:
I will have a look at the remaining test cases tomorrow.

Hi @ixje ,
there are few test cases left. This one is really tough:
    def test_get_storage_item(self):
        contract_hash = '16f1559c3c27d66d087bef936804105457617c8a'
        storage_key = binascii.hexlify(b'totalSupply').decode('utf-8')
        req = self._gen_rpc_req("getstorage", params=[contract_hash, storage_key])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(res['result'], '00f938dbba05')
        actual_val = int.from_bytes(binascii.unhexlify(res['result'].encode('utf-8')), 'little')
        self.assertEqual(actual_val,<PHONE_NUMBER>000)

I have try to replicate the test case on the privnet by running this transaction:
neo> testinvoke b9fbcff6e50fd381160b822207231233dd3c56c2 totalSupply []
Used 0.32 Gas

-------------------------------------------------------------------------------------------------------------------------------------
Test invoke successful
Total operations: 347
Results [{'type': 'ByteArray', 'value': '00a031a95fe300'}]
Invoke TX GAS cost: 0.0
Invoke TX fee: 0.0001
-------------------------------------------------------------------------------------------------------------------------------------

Enter your password to continue and invoke on the network

[password]> **********
[I 180830 23:01:20 Transaction:618] Verifying transaction: b'bad4face9c794900569f861a0386ccea1c3d99fda5c3602ab9d213d981524fe2'
Relayed Tx: bad4face9c794900569f861a0386ccea1c3d99fda5c3602ab9d213d981524fe2

However the transaction does not produce the desire
storage_key = binascii.hexlify(b'totalSupply').decode('utf-8')

If I look at all the key in my privnet:
def test_storage():
    from neo.Implementations.Blockchains.LevelDB.DBCollection import DBCollection
    from neo.Implementations.Blockchains.LevelDB.DBPrefix import DBPrefix
    from neo.Core.State.StorageItem import StorageItem
    import binascii
    blockchain = LevelDBBlockchain("/home/dau/.neopython/docker-privnet/Chains/backup/fixtures/test_chain")

    Blockchain.RegisterBlockchain(blockchain)
    _db = Blockchain.Default()._db
    sn=_db.snapshot()
    storages = DBCollection(_db, sn, DBPrefix.ST_Storage, StorageItem)
    for k in storages.Keys:
        value =storages.TryGet(k).Value
        print(k, "\n", value, "\n new k, v \n")

Output:
new k, v

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9H\x03;X\xefT|\xbfT\xc8\xee/r\xa4-[`<\x00\xaf'
 b'\x00B\xe4\xb52'

 new k, v

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9in_circulation'
 b'\x00\xa01\xa9_\xe3\x00'

 new k, v

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9initialized'
 b'\x01'

they are ending either  with no "human readable" suffix or with suffix  in  ['in_circulation', 'initialized']. Do you have any hint how I can generate a transaction with storage key ending with suffix totalSupply?.
I believe the answer of this question may have to do with the ico_template:
NEP5_METHODS = ['name', 'symbol', 'decimals', 'totalSupply', 'balanceOf', 'transfer', 'transferFrom', 'approve', 'allowance']

@dauTT Nice to see it coming together!

@dauTT super nice! I'm excited to go through all this, ~14 MB sounds amazing 🥇
As for the 2 tests, yes let's keep those in. It's a small adjustment to make whenever new tests are added, and still useful to quickly verify proper loading of the right blockchain fixtures. I might be able to go through them on Wednesday, otherwise no sooner then Friday (just to give an indication)

test cases
@dauTT the tests in your link look good to me , great job (@jseagrave21 too)👌
You probably want to run make lint before submitting the PR because I think i see some spaces that violate the automatically enforced code style checker.
fixture testnet data
The collection of smart contracts in the zip also looks fine. The readme can use a little bit of work. I know what you did and i can make sense of most data in the readme, but i think a first time reader might not or have trouble following. I think a structure along the following lines with a bit more words would be great
Intro
- neo-python fixtures are based of a private unit test net <link>
- it currently contains the following smart contracts ...
- a default wallet with the following credentials is in place...

How to add new fixture tests
- pull & run private net (howto...)
- use wallet with the following credentials as needed
- add test, pr, ask for image update

For instructions like
rm -rf /home/dau/.neopython/Chains/privnet

try to make them generic / applicable to all readers i.e.
rm -rf ~/.neopython/Chains/privnet

file placing for PR
As the BlockChainFixtureTestCase resides under the directory neo/Utils I think the right place to put the contents of UnitTest-SM.zip could be something like neo/Utils/fixtures-testnet/
The fixture files should have been uploaded to CoZ AWS by now so you should be good to PR and have travis pass (if not then i'll restart the build once they're up, no worries). I'll have to discuss how we want to port/backup the docker image but we can at least proceed for now.

S3 Files:
https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/fixtures_v8.tar.gz
https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/notif_fixtures_v8.tar.gz
https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/UnitTest_SM_v8.zip
Thanks for all the hard work on this, I can't imagine yet how much more enjoyable testing will be 🥇

@dauTT let me know if you need any help with anything else before submitting the PR. Or if you would like me to review anything. I am glad I could help and would be happy to in any way I can. Great work!

Thanks @jseagrave21 for the help.
I am a little bit confuse about submitting the PR.
@ixje / @jseagrave21, Shall I submit a PR to the Development branch or to the Master branch?
(If it is the Development branch, we need perhaps to update the test cases again before submitting a PR)
If I run, one by one, the test modules which we have migrated to the privnet, they are all passing using the latest version of neo-python  v0.7.8. However when I run this command:
python -m unittest discover neo
Some of them are failing and there are too many failures and errors.
Ran 353 tests in 86.778s
FAILED (failures=17, errors=11)

It seems there are still come conflicts which need to resolve. @jseagrave21, if you have time you can also look at this and let me know what is your opinion. Thanks!

@dauTT We are supposed to submit the PR to the Development Branch.

Can you link to or share the report so we can see what the errors are? Maybe a Google Doc?

Hi @jseagrave21 ,
I have this log file: log.txt,
My first impression is that when we run all the tests together, the fixtures are coming not from the privnet but from the testnet. Need to investigate further...

@dauTT Here are the results from my tests:
Log1.txt
I noticed in test_json_rpc_api.py that the same issue where getblock returns the wrong block has returned.

@dauTT When I was building my environment, I created a fresh pull of the CoZ Development branch and then modified all the files you mention here by copy and pasting from your neo-python repository "latest...." branch. I did not download anything else because I was assuming that since the files have been uploaded to AWS, then we should be able to pull down from there.
I also didn't do anything with the contents of UnitTest-SM.zip. Please let me know if I need to take further action to create an accurate environment.

Also, since several of the errors involve UserWalletTestCase, should that have been updated?

@dauTT I think option 1 is best. But, even if we go ahead with the PR, they will not approve it until we have a passing build.

That is why @ixje has delayed some of the other PRs. Because some are dependent on the WalletFixtureTestCase.

@dauTT I don't think it will be too hard to fix the errors but I am concerned about the errors we have already fixed that I mentioned earlier (getblock returning 9 when 10 was queried)

@jseagrave21 ,
in regards to the conflict of fixtures that I was talking above it is related to the fact that some other test cases (e.g, VerifiableTestCase) which are not related to the BlockchainFixtureTestCase class are using the location
os.path.join(settings.DATA_DIR_PATH, 'fixtures/test_chain')

For some strange reason the  setup and teardown of some test cases didn't work properly and in the end the fixtures are mixed up. The mixing of fixtures causes errors we have already fixed (e.g, getblock returning 9 when 10 was queried). This can be resolved by tweaking BlockchainFixtureTestCase but it will better to fix the problem at the source.
To my knowledge the following is the updated list of test cases to migrate to the privnet. These test cases depend on BlockchainFixtureTestCase and WalletFixtureTestCase:

test_withdraw.py
test_LevelDBBlockchain.py (Done, potentially to update)
test_leveldb.py (Done, potentially to update)
test_user_wallet.py
test_create_user_wallet.py
test_interop_blockchain.py (Done, potentially to update)
test_rest_api.py (Done, potentially to update)
test_json_invoke_rpc_api.py (Done, potentially to update)
test_json_rpc_api.py (Done, potentially to update)
test_token_commands.py
test_send_command.py
test_wallet_commands.py
test_claim_command.py
test_node_leader.py
test_smart_contract.py (Done, potentially to update)
test_payable.py
test_breakpoints.py
test_storage_find.py
test_smart_contract3.py (Done, potentially to update)
test_smart_contract2.py (Done, potentially to update)
test_gas_costs.py
test_migrate_destroy.py

So there are still some work to do. @jseagrave21 , shall I assign half of the remaining test cases to me and half to you?

@dauTT okay that sounds good!

Is there anything that we need help with? Like the tweaking of BlockchainFixtureTestCase or fixing it at the source? So we can have three people (including @ixje) working on it?

Let us involve @ixje , for difficult task as we will need his help of course. :smile:
I have assigned the first items to me an the remaining to you. Thanks!

I've asked those with merging rights to not merge any PRs that rely on or inherit from BlockchainFixtureTestCase until we get this one PR'ed. What else can i do for now? @dauTT are you on discord? If not, joining could make communicating a bit faster/easier at this stage

@dauTT okay, I will start at the bottom and work my way up

@dauTT So, test_claim_command.py completely fails and I think I know why. The WalletFixtureTestCase references wallets which do not exist in your fixture. You need to add the wallets you have been using to the WalletFixtureTestCase.
I think the easiest way to do this would be to open np-prompt and create a wallet labeled testwallet.db3 with the password testpassword and then import the wif for the address AK2nJJpJr6o664CWJKi1QRXjqeic2zRp8y and delete the other (original) address. Then you could replace the testwallet.db3 which already resided in fixtures/. Then create another wallet labeled testwallet2.db3 with the password testwallet and import the wif for the address AXpNr3SDfLXbPHNdqxYeHK5cYpKMHZxMZ9 and delete the other (orginal) address. Then you could replace the testwallet2.db3 in fixtures/. I am not sure if you have a third address you were using. If not, you should make a new wallet to replace testwallet3.db3 and send some assets to it. If you do this, then we shouldn't have to completely change all code for the class UserWalletTestCase(BlockchainFixtureTestCase) But we will have to change a lot. @ixje Could you double check what I am recommending to see if it makes sense? Essentially we are rebuilding the WalletFixtureTestCase.

Yes I think you need to 'rebuild' the wallets in WalletFixtureTestCase. These wallets will rely on blockchain data coming from fixtures_v7.tar.gz which is unlikely to match the new v8 blockchain data.
ps: with 'rebuild' i don't mean the command, but making sure the wallet addresses in the wallet actually exists in the blockchain data of the v8 fixtures. Including those addresses having funds, transactions and whatever is needed for the tests.

Thanks @jseagrave21 , @ixje. I will do that.

@dauTT sounds good. I would love to help with this part but I don't have the WIFs for any of the addresses used and I am not sure of all the addresses which are active.
As soon as the WalletFixtureTestCase is migrated, please post a readme with the addresses and their associated wallets and passwords and hashes (Look at test_claim_command.py to see what kind of information is referenced when a UserWalletTestCase is set up). The first wallet should have a watch_address set so the second wallet addresses. We might need to update these fixtures to me the demands of particular tests but that will at least get us rolling.
Afterwards, I will start over to make sure the UserWalletTestCase in each test references the correct info and also passes.

@dauTT I like what you proposed and I think I see what you are getting at. However, I think we can minimize the number of new wallets needed. Here is my counter-proposal.
testing.setup.proposal2.docx

@ixje / @jseagrave21 ,
you can find the most updated files here: link
I have modifed the following files:
        modified:   fixtures/testwallet.db3 (autogenerated, no need for review)
	modified:   neo/Implementations/Blockchains/LevelDB/test_LevelDBBlockchain.py (okay)
	modified:   neo/Implementations/Blockchains/LevelDB/tests/test_leveldb.py (okay)
	modified:   neo/Implementations/Wallets/peewee/test_create_user_wallet.py
	modified:   neo/Implementations/Wallets/peewee/test_user_wallet.py
	modified:   neo/Network/test_node_leader.py
	modified:   neo/Prompt/Commands/tests/test_claim_command.py (not okay, to investigate why in the fixture we have negative unavailable_bonus!)
	modified:   neo/Prompt/Commands/tests/test_send_command.py
	modified:   neo/Prompt/Commands/tests/test_token_commands.py
	modified:   neo/Prompt/Commands/tests/test_wallet_commands.py
	modified:   neo/Settings.py (okay)
	modified:   neo/SmartContract/tests/StorageTest.avm (autogenerated, no need for review)
	modified:   neo/SmartContract/tests/test_breakpoints.py
	modified:   neo/SmartContract/tests/test_gas_costs.py
	modified:   neo/SmartContract/tests/test_migrate_destroy.py
	modified:   neo/SmartContract/tests/test_smart_contract.py  (okay)
	modified:   neo/SmartContract/tests/test_smart_contract2.py  (okay)
	modified:   neo/SmartContract/tests/test_smart_contract3.py  (okay)
	modified:   neo/Utils/BlockchainFixtureTestCase.py (okay)
	modified:   neo/Utils/VerifiableTestCase.py
	modified:   neo/Utils/WalletFixtureTestCase.py
	modified:   neo/VM/tests/test_interop_blockchain.py (okay)
	modified:   neo/api/JSONRPC/test_json_invoke_rpc_api.py (okay)
	modified:   neo/api/JSONRPC/test_json_rpc_api.py (@ixje, @jsagrave to enrich the test cases as new ones were added in the meantime: test_getbalance_invalid_params, test_validate_address, test_getbalance_invalid_params, test_valid_multirequest, test_multirequest_with_1_invalid_request)
	modified:   neo/api/REST/RestApi.py (okay)
	modified:   neo/api/REST/test_rest_api.py (okay)
	deleted:    neo/contrib/nex/test_withdraw.py (okay, @ixje confirmed)

and added the following wallets:
fixtures/neo-test-coz.wallet (pwd=coz)
fixtures/neo.test1-bc.wallet (pwd=1234567890)
fixtures/neo-test2-bc.wallet (pwd=1234567890)
fixtures/neo-test1-w.wallet (pwd=1234567890)
fixtures/neo-test2-w.wallet (pwd=1234567890)
fixtures/neo-test3-w.wallet (pwd=1234567890)

Latest image created: docker pull dautt/neo-privnet-unittest:v0.0.22
Fixtures to upload to AWS:
fixtures_v8.tar.gz
notif_fixtures_v8.tar.gz
(both fixture files are in total about 7MB)
All the test cases have been tested using neo-python v0.7.9-dev from the development branch. They all pass if we execute one test at the time by running these commands:
python -m unittest neo/Implementations/Blockchains/LevelDB/test_LevelDBBlockchain.py
python -m unittest neo/Implementations/Blockchains/LevelDB/tests/test_leveldb.py
python -m unittest neo/Implementations/Wallets/peewee/test_user_wallet.py
python -m unittest neo/Implementations/Wallets/peewee/test_create_user_wallet.py
python -m unittest neo/VM/tests/test_interop_blockchain.py
python -m unittest neo/api/REST/test_rest_api.py
python -m unittest neo/api/JSONRPC/test_json_invoke_rpc_api.py
python -m unittest neo/api/JSONRPC/test_json_rpc_api.py
python -m unittest neo/Prompt/Commands/tests/test_send_command.py
python -m unittest neo/Prompt/Commands/tests/test_wallet_commands.py
python -m unittest neo/Prompt/Commands/tests/test_claim_command.py
python -m unittest neo/Network/test_node_leader.py
python -m unittest neo/SmartContract/tests/test_smart_contract.py
python -m unittest neo/SmartContract/tests/test_payable.py
python -m unittest neo/SmartContract/tests/test_breakpoints.py
python -m unittest neo/SmartContract/tests/test_storage_find.py
python -m unittest neo/SmartContract/tests/test_smart_contract3.py
python -m unittest neo/SmartContract/tests/test_smart_contract2.py
python -m unittest neo/SmartContract/tests/test_gas_costs.py
python -m unittest neo/SmartContract/tests/test_migrate_destroy.py

Hovever if we run this command:
python -m unittest discover neo

we get the following unexpected result:
======================================================================
ERROR: test_initial_setup (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 62, in test_initial_setup
    self.assertTrue(GetBlockchain().GetBlock(0).Hash.To0xString(), '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
AttributeError: 'NoneType' object has no attribute 'Hash'

======================================================================
FAIL: test_add_header (Implementations.Blockchains.LevelDB.tests.test_initial_db.LevelDBTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/Implementations/Blockchains/LevelDB/tests/test_initial_db.py", line 54, in test_add_header
    self.assertEqual(self._blockchain.CurrentHeaderHash, b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf')
AssertionError: b'b3181718ef6167105b70920e4a8fbbd0a0a56aacf460d70e10ba6fa1668f1fef' != b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf'

======================================================================
FAIL: test_block_non_verbose (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 239, in test_block_non_verbose
    self.assertEqual(block.Index, 2003)
AssertionError: 2002 != 2003

======================================================================
FAIL: test_get_block_int (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 182, in test_get_block_int
    self.assertEqual(GetBlockchain().GetBlock(1).ToJson()['hash'], '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273')
AssertionError: '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099' != '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273'
- 0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099
+ 0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273

======================================================================
FAIL: test_get_block_sysfee (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 214, in test_get_block_sysfee
    self.assertEqual(res['result'], 1560)
AssertionError: 1070 != 1560

======================================================================
FAIL: test_getblockhash (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 104, in test_getblockhash
    self.assertEqual(expected_blockhash, res["result"])
AssertionError: '0x049db9f55ac45201c128d1a40d0ef9d4bdc58db97d47d985ce8d66511a1ef9eb' != '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273'
- 0x049db9f55ac45201c128d1a40d0ef9d4bdc58db97d47d985ce8d66511a1ef9eb
+ 0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273

======================================================================
FAIL: test_getrawtx_1 (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 322, in test_getrawtx_1
    self.assertEqual(res['blockhash'], '0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417')
AssertionError: '0x0e9c0a7e88aeaf217654107041986f4491b447590cfc3d060304cef1b8536ea0' != '0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417'
- 0x0e9c0a7e88aeaf217654107041986f4491b447590cfc3d060304cef1b8536ea0
+ 0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417

======================================================================
FAIL: test_prompt_open_wallet (bin.test_prompt.PromptTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/bin/test_prompt.py", line 26, in test_prompt_open_wallet
    self.assertIn(checktext, text)
AssertionError: 'Opened' not found in "open wallet fixtures/testwallet.db3\r\ntestpassword\r\n\x1b[32m[I 180915 21:17:39 LevelDBBlockchain:114]\x1b[39m Created Blockchain DB at /home/dau/.neopython/Chains/SC234 \r\n\x1b[32m[I 180915 21:17:46 NotificationDB:73]\x1b[39m Created Notification DB At /home/dau/.neopython/Chains/Test_Notif \r\n\x1b[0m\x1b[?7h\x1b[0;38;5;4mNEO\x1b[0;38;5;10m cli. Type \x1b[0;38;5;197m'help' \x1b[0;38;5;10mto get started\x1b[0m\r\r\n\x1b[0m\r\n\r\n\x1b[?1l\x1b[6n\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[Jneo>  \x1b[0m\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n                                                                               \r\x1b[79C \r\x1b[7A\x1b[5C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[5Dneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[40C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[40D\x1b[0m\x1b[Jneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l\x1b[?1l\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[24C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[24D\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l"

----------------------------------------------------------------------
Ran 347 tests in 68.850s

FAILED (failures=7, errors=1)

I believe the major reason for the above failures, is related to this test case that I have added:
ERROR: test_initial_setup (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 62, in test_initial_setup
    self.assertTrue(GetBlockchain().GetBlock(0).Hash.To0xString(), '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
AttributeError: 'NoneType' object has no attribute 'Hash'

I tried to fix it but unsuccessfully. So I need help here. For some unknown reason the Genesis block is replaced by None when we run all the tests together, and therefore the final Blockchain is corrupted. (Strangely this behavior does not happen when we run the test one by one). If we can fix this one, most of the cases will be fix by themselves.
So how can we divide the remaining work?
I thought as follow, but please feel free to do as you wish. :smile:

@ixje can perhaps fix the issue related to this command python -m unittest discover neo
@ixje  can review the test cases which depend only WalletFixtureTestCase. The one which depend on BlockChainFixtureTestCase have been already reviewed by @ixje  and threfore there is no need to review them again. I have labeled them as okay.
@jseagrave21 can update the test case depending only BlockChainFixtureTestCase. In particular  test_json_rpc_api.py and help @ixje  perhaps we the test case test_claim_command.py

@dauTT Here is the migrated test_json_rpc_api.py including the new tests for sendtoaddress since @localhuman just merged that PR.
https://github.com/jseagrave21/neo-python/blob/DauTT-branch/neo/api/JSONRPC/test_json_rpc_api.py
Additionally, please note that the my "DauTT-branch" is completely up to date with your "latest2" branch, and it is also up-to-date with the latest CoZ Development branch. So, you should be able to merge my branch into your branch if you'd like.

For the other failures in test_json_rpc_api.py when running make test there must be something wrong with the way that discover neo is being implemented.
For example, here is one of the failures:
FAIL: test_block_non_verbose (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 241, in test_block_non_verbose
    self.assertEqual(block.Index, 2003)
AssertionError: 2002 != 2003

Yet, the block being queried in test_json_rpc_api.py is 2003:
def test_block_non_verbose(self):
        req = self._gen_rpc_req("getblock", params=[2003, 0])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertIsNotNone(res['result'])

        # we should be able to instantiate a matching block with the result
        output = binascii.unhexlify(res['result'])
        block = Helper.AsSerializableWithType(output, 'neo.Core.Block.Block')
        self.assertEqual(block.Index, 2003)
        self.assertEqual(len(block.Transactions), 1)

So, the discover neo is off by one block.

@dauTT, I am not sure about how to address this issue though perhaps it is related to #538 ?

@dauTT I reran make test several times. I changed test_intial_setup to this:
def test_initial_setup(self):
        expected_blockhash = '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099'
        req = self._gen_rpc_req("getblockhash", params=[0])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(expected_blockhash, res["result"])

Instead of the expected result, I consistently get 0xb3181718ef6167105b70920e4a8fbbd0a0a56aacf460d70e10ba6fa1668f1fef which I cannot find when I query with block in np-prompt. So, it looks like block 0 is being replaced by a different block at some point during the discover neo test.

@dauTT

I solved the unittest discovery difference. You'll have to merge this PR: https://github.com/CityOfZion/neo-python/pull/616

Having this call is good https://github.com/dauTT/neo-python/blob/latest2-move-unittest-to-privnet-478/neo/Settings.py#L232-L234
But we do not want to copy the settings from the privnet protocol file. We should create a new one and check that in (e.g. protocol.unittest-net.json). It can be a copy of the privnet one (content wise), but they cannot be the same file. Reason being is that people should be free to change their privatenet protocol to whatever matches their needs, however they might touch e.g. the validators or magic and that will mess up the test cases.

this test will still fail
https://github.com/CityOfZion/neo-python/blob/951de3a902a398ce6b178481e837ce96014affbd/neo/Implementations/Blockchains/LevelDB/tests/test_initial_db.py#L45-L56

You'll have to update the setUpClass()to add:
def setUpClass(self):
        settings.setup_fixturenet()
        Blockchain.DeregisterBlockchain()

and update the test to the following:
    def test_add_header(self):
        hexdata = binascii.unhexlify(self.block_one_raw)
        block_one = Helper.AsSerializableWithType(hexdata, 'neo.Core.Block.Block')

        if settings.MAGIC == 56753:
            self.assertEqual(self._blockchain.CurrentHeaderHash, b'996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
        else:
            self.assertEqual(self._blockchain.CurrentHeaderHash, b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf')

        self.assertEqual(self._blockchain.HeaderHeight, 0)

        self._blockchain.AddBlock(block_one)
        self.assertEqual(self._blockchain.HeaderHeight, 1)

that will fix the errors and actually make the test do what the testname suggests 😂

Thanks @ixje / @jseagrave21 , great team work!!! :smile:
I have created a clean branch with all the PR (611, 616) merged together along with the update file test_json_rpc_api from @jseagrave21 branch. Here is the link
Fixtures to upload to AWS:
notif_fixtures_v8.tar.gz
fixtures_v8.tar.gz
Tested the code by running:
make lint
make test

Everything pass except this one:
======================================================================
FAIL: test_prompt_open_wallet (bin.test_prompt.PromptTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/bin/test_prompt.py", line 26, in test_prompt_open_wallet
    self.assertIn(checktext, text)
AssertionError: 'Opened' not found in "open wallet fixtures/testwallet.db3\r\ntestpassword\r\n\x1b[32m[I 180919 00:31:55 LevelDBBlockchain:113]\x1b[39m Created Blockchain DB at /home/dau/.neopython/Chains/SC234 \r\n\x1b[32m[I 180919 00:31:58 NotificationDB:73]\x1b[39m Created Notification DB At /home/dau/.neopython/Chains/Test_Notif \r\n\x1b[0m\x1b[?7h\x1b[0;38;5;4mNEO\x1b[0;38;5;10m cli. Type \x1b[0;38;5;197m'help' \x1b[0;38;5;10mto get started\x1b[0m\r\r\n\x1b[0m\r\n\r\n\x1b[?1l\x1b[6n\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[Jneo>  \x1b[0m\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n                                                                               \r\x1b[79C \r\x1b[7A\x1b[5C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[5Dneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[40C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[40D\x1b[0m\x1b[Jneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l\x1b[?1l\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[24C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[24D\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l"

----------------------------------------------------------------------
Ran 352 tests in 71.690s

FAILED (failures=1)

This was also failing before we did any changes. It has nothing to do with BlockChainFixtureTestCase nor WalletFixtureTestCase. I am not sure if this is just happening on my local machine or not.
@jseagrave21 , I believe you know very well this test. Can you please have a look at it? Thanks. :smile:

@dauTT okay let me take a look

@dauTT since we didn't change the test and testwallet.db3 is still accessible, I suspect it is just the time out. I added 5 sec to the test here. Feel free to add time if you need to.
https://github.com/jseagrave21/neo-python/blob/DauTT-branch/neo/bin/test_prompt.py

Yes, your right, increasing the time out help. So all the test cases are fine. :+1:


Add possibility to ignore damage value for autocraft

Issue description:
Add the possibility to ignore damage value for autocraft
Version (make sure you are on the latest version before reporting):

Minecraft: 1.12.2
Refined Storage: 1.6.8

Does this issue occur on a server? [yes/no] yes

No


Parameter names must be unique for nested state URLs

Descendant states cannot share parameter names. Whether or not this is intended to be supported is unclear.
For example, if I
$state.go("parent", { id: 1 });
$state.go("parent.child", { id: 2 };

I'll end up at /2/2.
I would like for this feature to be supported. I enjoy the documented feature of passing my object as the key. It's common for my objects to use the parameter name "id" as the key. It would be convenient to be able to:
var model = {
    parent: { id: 1 },
    child: { id: 2 }
};
$state.go("parent", parent);
$state.go("parent.child", child);

However in this scenario nested states lose state information.
As one example, the sref "inheritParams" call (line 3405, release 0.2.13) merges parents into a single object without qualifying merged parameter names with the origin state name. The code runs, but uses the child id for all parent state ID tokens, resulting for example in the URL #/43/43/43.
See: http://plnkr.co/edit/PgCgtqAgNgqQngC8a3i0?p=preview

As I was expanding on this, I realize it makes no sense. The feature is reasonable; more assistance in dehydrating and rehydrating state objects. However, I recognize that because parameters for multiple ancestors can be provided simultaneously and they are currently assumed to be flat, there is no non-breaking way to support this as I imagined above.
The closest would be some to/from transform.
I'm closing this as it is a much larger topic than I intended.


refactor: de-duplicate Resource.get_export_headers() and export_resource()

Small refactor as discussed on #1846

coverage: 100.0%. remained the same
when pulling ba8085b674e4956cdfbf31f4b7aaa23b2bfb5d6f on PetrDlouhy:pr/refactor_get_fields
into 92a7f8fe5594c485143467ddef3c4240c042c27a on django-import-export:main.


cmd/go: using golang.org/x/* from mainland China is awkward with modules

What version of Go are you using (go version)?

[jukay@mac-pro]~/log$ go version
go version go1.11 darwin/amd64
[jukay@mac-pro]~/log$

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
[jukay@mac-pro]~/log$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jukay/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jukay/Code/Go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jukay/log/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build712144984=/tmp/go-build -gno-record-gcc-switches -fno-common"
[jukay@mac-pro]~/log$

What did you do?
just compile this project outside GOPATH
[jukay@mac-pro]~/log$ go build

What did you expect to see?
[jukay@mac-pro]~/log$ time go build

real	0m0.316s
user	0m0.195s
sys	0m0.232s

What did you see instead?
[jukay@mac-pro]~/log$ time go build

real	1m0.316s
user	0m0.195s
sys	0m0.232s

Unable to understand such a simple code, it took a minute to compile it.

when i usego build -v
[jukay@mac-pro]~/log$ go build -v
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout

If the cache isn't warm, the first build may take longer if it needs to download and build the dependencies. Have you tried running time go build a second time? It should take a few seconds at most.

@mvdan I run time go build many times, it cast much time  every time.

Is your GOCACHE directory /Users/jukay/Library/Caches/go-build writeable? Does it contain anything? What does go build -x show?

Also, note that your go build -v above shows HTTP errors. Are the build succeeding or failing? If the build fails because it can't download some modules or dependencies, they will of course not be cached.

Doesn't china block URLs with x in them? Someone mentioned it on go-nuts.

@mvdan go build alway success, it just very slow。i can go mod vendor,it create vendor directory,and download all dependencies. this is my source code
https://github.com/hellojukay/vgo-issue

Hi @hellojukay - to help us investigate the issue faster, it would be great if you can answer the questions asked above. Especially -
What does your GOCACHE directory contain ?
What is the output of go build -x -v ?
Thank you.

@agnivade yes ,yestoday i try to upgrade my  golang version to 1.11.2, this is output of  go build -x -v
[jukay@mac-pro]~/log$ go build -x -v
WORK=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build601851662
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile main=/Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
packagefile github.com/sirupsen/logrus=/Users/jukay/Library/Caches/go-build/e7/e7275b538c9195808fa163ca3c56c5d63281712d3d135165cce1aac98cc02bf8-d
packagefile runtime=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime.a
packagefile bufio=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bufio.a
packagefile bytes=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bytes.a
packagefile encoding/json=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/json.a
packagefile fmt=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/fmt.a
packagefile golang.org/x/crypto/ssh/terminal=/Users/jukay/Library/Caches/go-build/0e/0ebfbc4e93e20cd471ded2ecdc73841dc36970144e7957043cdc75f65e0099f8-d
packagefile io=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/io.a
packagefile log=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/log.a
packagefile os=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/os.a
packagefile reflect=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/reflect.a
packagefile sort=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sort.a
packagefile strings=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strings.a
packagefile sync=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync.a
packagefile sync/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync/atomic.a
packagefile time=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/time.a
packagefile internal/bytealg=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/bytealg.a
packagefile internal/cpu=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/cpu.a
packagefile runtime/internal/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/sys.a
packagefile errors=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/errors.a
packagefile unicode/utf8=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf8.a
packagefile unicode=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode.a
packagefile encoding=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding.a
packagefile encoding/base64=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/base64.a
packagefile math=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math.a
packagefile strconv=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strconv.a
packagefile unicode/utf16=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf16.a
packagefile golang.org/x/sys/unix=/Users/jukay/Library/Caches/go-build/7f/7f9284a7bb944dc9533f22fbc3ba192e51145ab0792f17b55a151eb6411981f5-d
packagefile internal/poll=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/poll.a
packagefile internal/syscall/unix=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/syscall/unix.a
packagefile internal/testlog=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/testlog.a
packagefile syscall=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/syscall.a
packagefile internal/race=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/race.a
packagefile encoding/binary=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/binary.a
packagefile math/bits=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math/bits.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=7l5ZuOc-Ol-x9AyQCky7/WlX-HbvTd8yVTtnfYvzf/J45TO0-nMm2qI1TaHJKq/7l5ZuOc-Ol-x9AyQCky7 -extld=clang /Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out main
rm -r $WORK/b001/

This is my GOCACHE directory
[jukay@mac-pro]~/log$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jukay/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jukay/Code/Go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jukay/log/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build984680892=/tmp/go-build -gno-record-gcc-switches -fno-common"
[jukay@mac-pro]~/log$ cd /Users/jukay/Library/Caches/go-build
[jukay@mac-pro]~/Library/Caches/go-build$ ls
00       14       28       3c       50       64       78       8c       README   b3       c7       db       ef
01       15       29       3d       51       65       79       8d       a0       b4       c8       dc       f0
02       16       2a       3e       52       66       7a       8e       a1       b5       c9       dd       f1
03       17       2b       3f       53       67       7b       8f       a2       b6       ca       de       f2
04       18       2c       40       54       68       7c       90       a3       b7       cb       df       f3
05       19       2d       41       55       69       7d       91       a4       b8       cc       e0       f4
06       1a       2e       42       56       6a       7e       92       a5       b9       cd       e1       f5
07       1b       2f       43       57       6b       7f       93       a6       ba       ce       e2       f6
08       1c       30       44       58       6c       80       94       a7       bb       cf       e3       f7
09       1d       31       45       59       6d       81       95       a8       bc       d0       e4       f8
0a       1e       32       46       5a       6e       82       96       a9       bd       d1       e5       f9
0b       1f       33       47       5b       6f       83       97       aa       be       d2       e6       fa
0c       20       34       48       5c       70       84       98       ab       bf       d3       e7       fb
0d       21       35       49       5d       71       85       99       ac       c0       d4       e8       fc
0e       22       36       4a       5e       72       86       9a       ad       c1       d5       e9       fd
0f       23       37       4b       5f       73       87       9b       ae       c2       d6       ea       fe
10       24       38       4c       60       74       88       9c       af       c3       d7       eb       ff
11       25       39       4d       61       75       89       9d       b0       c4       d8       ec       log.txt
12       26       3a       4e       62       76       8a       9e       b1       c5       d9       ed       trim.txt
13       27       3b       4f       63       77       8b       9f       b2       c6       da       ee
[jukay@mac-pro]~/Library/Caches/go-build$

There seems to be various oddities-

The build tries to connect to https://golang.org/x/crypto and https://golang.org/x/crypto everytime.

Your go.mod seems to replace

replace golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33 => github.com/golang/sys v0.0.0-20180905080454-ebe1bf3edb33

Yet, it is trying to connect to golang.org.
/cc @myitcv @bcmills for further insight.
And FYI - I think you mean go when you say vgo, because module functionality is integrated into the go toolchain.

@agnivade Not only on mac, but also on centos7, compiler will try to connect golang. org until connection timeout
[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/vgo-issue/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build194230206=/tmp/go-build -gno-record-gcc-switches"

[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# time go build -x -v
WORK=/tmp/go-build973675453
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp <IP_ADDRESS>:443: i/o timeout

real	1m0.108s
user	0m0.068s
sys	0m0.041s

A couple of observations

If the replacements need to be systematic (as they appear to be, potentially for reasons of certain URLs being blocked) then some sort of proxy is going to ultimately be a cleaner solution
The go tool checking golang.org regardless of the replace is likely related to https://github.com/golang/go/issues/26241

The issue with fetching golang.org is likely #27859.

@hellojukay, could you see if you can reproduce the issue with a go toolchain built from source¹ at HEAD? I'd like to figure out whether this is a duplicate of #27859 or if there is something more to it.
¹ See https://golang.org/doc/install/source for instructions.

Note another instance of effectively the same problem in #28716.
Stepping back for one second, if golang.org remains blocked in China (ref https://blog.golang.org/hello-china), then I wonder whether we need to think of a different approach here.
Even though in Go 1.12 https://github.com/golang/go/issues/26241 will give us comprehensive support for the concept of "wherever you see X use Y", it strikes me that users from mainland China will always need to add a replace statements for any of the golang.org/x/* modules, in every one of their projects where they are a (transitive) dependency.
This seems less than ideal. I would guess that this would impact a significant portion of users' projects.
Is there anything we can do here in the go tool itself?
cc @andybons

@myitcv I think this is a bug of the code of go modules, If I set up replace for golang.org, then it should not connect to golang.org。Isn't it? wait for me , i am building go compiler from the master branch golang 。

If I set up replace for golang.org, then it should not connect to golang.org

Correct. As @bcmills mentioned, the solution for https://github.com/golang/go/issues/27859 should fix the issue as far as go build is concerned. https://github.com/golang/go/issues/26241 is the more general solution, however.

@hellojukay I'm going to re-open this issue if that's ok. I've also retitled it to better reflect the question I asked @andybons and @bcmills in https://github.com/golang/go/issues/28652#issuecomment-437693560

@myitcv I build go from master branch source, it works well, build very fast。Thanks !!!

@myitcv I am using goproxy.io to resolve this problem .
but as you say we use golang.org/x/* packages always download zip from github.com/golang/*,
go get  was blocked. so we always can not use dep govendor to manage our project . so sadly !
or if I have a vps outside of china and has shadowsocks-server on this.
we can export https_proxy=http://<IP_ADDRESS>:1080 to go get . but it was too slowly . and there was so much people that don't has vps . them does not know what was happened to golang.org if them was new gopher .
so I hope there was a better way to solve this problem which site has been blcoked from our area.

Thanks for the further details @clearcodecn

so I hope there was a better way to solve this problem which site has been blcoked from our area.
or using domain alias

Yes indeed, that was the motivation behind my question.

@hellojukay is it ok if we keep this issue open to track the question I asked in https://github.com/golang/go/issues/28652#issuecomment-437693560?

@myitcv Well, it depends on you.

Thank you @hellojukay - I'll keep this open for @andybons to comment.

I try add replace golang.org/x/net/trace v0.0.0-20181129055619-fae4c4e3ad76 => github.com/golang/net/trace v0.0.0-20181129055619-fae4c4e3ad76 //indirect
also not work.

Hi @ScottHuangZL - As mentioned above in this comment, this is fixed in the master branch.
So it won't work until you try the master branch. Please let us know if you still face the issue if you build from master.

@agnivade
I try below, and finally work!
go mod edit<EMAIL_ADDRESS>go mod edit -replace=golang.org/x/net@v0.0.0=github.com/golang/net@latest
But the "go build" take a long time to complete build.   ==> I have concern for the long build time even it finally success complete, how to resolve it?? Thanks.
If I try "go build -x -v", it will be failed with below message:
Fetching https://golang.org/x/net?go-get=1
https fetch failed: Get https://golang.org/x/net?go-get=1: dial tcp <IP_ADDRESS>:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
It means every build actually it try fetch from online again for x/net. Why? thanks.

It means every build actually it try fetch from online again for x/net. Why?

That's #27859.

a. I have to try find out the version. Can we let it blank or use * to mean match any version?

You can already leave the source of a replacement blank to match any version.
That behavior is tested explicitly here:
https://github.com/golang/go/blob/42e8b9c3a430ccc4a03b2994f7dcbde630410e1e/src/cmd/go/testdata/script/mod_replace.txt#L12
The reason you have to specify any version in the first place is #26241.

@ScottHuangZL build golang compiler master source, it works.

@hellojukay @agnivade No plan to build go from master source branch. Will wait until the go1.11.3 version ready.
@agnivade @bcmills  May I know when go.1.11.3 release? thanks.

That fix (#27859) is not going to be backported. If you don't want to build from master, please wait for the 1.12 beta release which should be out soon (within this week).

@agnivade sounds good, thanks.

@panjiang wanted a wildcard replace shortcut so that one could replace all the golang.org/x repos to their github.com equivalent in one-shot. https://github.com/golang/go/issues/29076#issuecomment-443657989

A quick way:
go mod tidy
go mod vendor
go build -mod=vendor
Then the build speed still fast!!!

I just landed the fix for #26241, which should remove the need for the go mod edit -require step (you can skip straight to the go mod edit -replace and omit the arbitrary v0.0.0 version).

go module is a nightmare in China. Even if I have a very good VPN, go get -u will never complete in my life.
go mod edit won't work because you need to have the correct go.mod at first. It's like "which came first: the chicken or the egg?".
Please kill me I am sad and mad.

@sgon00 https://gocenter.io/ can help you.

@hellojukay Thanks a lot for letting me know gocenter.io. I have't tried it yet and will try it later. Sorry that I completely collapsed yesterday. I was trying go get -u simplePackage inside module directory from 11:00am to 03:00am without a luck. It can never complete successfully. But running the same command outside the module directory, it can be completed quickly without any problems. This is just crazy.

linux
GOPROXY=https://mirrors.aliyun.com/goproxy/ go build
or win
env GOPROXY=https://mirrors.aliyun.com/goproxy/ go build
or use https://goproxy.io

Given goproxy.io and other proxies with high availability in China, is there anything left to do here?

@oiooj @aofei


KullbackLeibler All numba methods

There are a few problems with the numba implementation. Below is a list of comments using the latest major release of numba=0.58 and llvmlit=0.41 (Sept2023).

njit is better in my opinion, specially when you want to add more on the decorator, dtypes, fastmath etc
in the case of parallel=True it is better to default to half of threads, using set_num_threads for every jitted function
Also, atm, the numba call method will raise the following warning

The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "../../../../../../tmp/ipykernel_2184/1742547634.py", line 40:
<source missing, REPL/exec in use?>

  warnings.warn(errors.NumbaPerformanceWarning(msg,

Also running parallel_diagnostics(level=4)
================================================================================
 Parallel Accelerator Optimizing:  Function kl_div_old_cil,
/tmp/ipykernel_2184/1742547634.py (40)
================================================================================
No source available
--------------------------------- Fusing loops ---------------------------------
Attempting fusion of parallel loops (combines loops with similar properties)...
----------------------------- Before Optimisation ------------------------------
--------------------------------------------------------------------------------
------------------------------ After Optimisation ------------------------------
Parallel structure is already optimal.
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

---------------------------Loop invariant code motion---------------------------
Allocation hoisting:
No allocation hoisting found

Instruction hoisting:
No instruction hoisting found
--------------------------------------------------------------------------------

I believed this is because, flatis inside theprangeiteration and every iter aflatiteris returned. In the case of thegradientmethod with or withoutmask` the kernel crashes.

Using ravel() instead of flat is better in my opinion. In CIL, as_array() does not return a new copy. Also, ravel will not return a new copy, just a view, since it is C_CONTIGUOUS. In SIRF,  as_array() will return a new copy but ravel will not return a new copy for the same reason.  Also, ravel seems to be faster.

I am not sure if mask is the right way to run KL methods when we encounter division by zero. First of all mask as an array is never used. Only the indices that mask[i]>0. In practice, we assume that A is a positive operator, so a mask could be A*1. However, it is possible that under strange geometries, this may not be true. Maybe is better to update out at indices where x + eta>0 in the case of gradient.

When numpy is used inside jitted functions,  fastmath=True is suggested, i.e., np.sqrt

Some benchmarks
N, M, K = 500,500,500
ig = ImageGeometry(N,M,K)

Call
%timeit -r 10 -n 10 kl_div_call_ravel(xx_np.ravel(), b_np.ravel(), eta_np.ravel())
58.2 ms ± 3.69 ms per loop (mean ± std. dev. of 10 runs, 10 loops each)

%timeit -r 10 -n 10 kl_div_call_flat(xx_np, b_np, eta_np)
1.28 s ± 12.9 ms per loop (mean ± std. dev. of 10 runs, 10 loops each)

Gradient
%timeit -r 20 -n 20 kl_gradient(xx_np, b_np, out1_np, eta_np)
39.2 ms ± 559 µs per loop (mean ± std. dev. of 20 runs, 20 loops each)

%timeit -r 20 -n 20 kl_gradient_ravel(xx_np.ravel(), b_np.ravel(), out2_np.ravel(), eta_np.ravel())
39.7 ms ± 715 µs per loop (mean ± std. dev. of 20 runs, 20 loops each)

Gradient(Mask)
%timeit -r 20 -n 20 kl_gradient_mask(xx_np, b_np, out1_np, eta_np, mask_np)
38.7 ms ± 460 µs per loop (mean ± std. dev. of 20 runs, 20 loops each)

%timeit -r 20 -n 20 kl_gradient_ravel_mask(xx_np.ravel(), b_np.ravel(), out2_np.ravel(), eta_np.ravel(), pos_ind)
The slowest run took 9.17 times longer than the fastest. This could mean that an intermediate result is being cached.
57.6 µs ± 55.7 µs per loop (mean ± std. dev. of 20 runs, 20 loops each)


Dark Mode Support

Although a cosmetic feature, implementing a dark mode would be beneficial for the eyes of the poor souls spending many hours on DefectDojo.

I think it's a good idea, I want to try to work on it.

Hello Everybody,
is anyone already working on this task?
Best Regards,
Fábio Sbano

Preview

Hello everybody,
unfortunately the bootstrap is a very old version which makes converting the theme to dark very hard
it would be better if we had an upgrade from sb-admin-2 to the newer bootstrap version (https://getbootstrap.com/docs/5.3/customize/color-modes/)
Is there anything in the darkmode roadmap for this year?
Another alternative would be to replace sb-admin-2 to use the bootstrap 5.x native components
Best Regards,
Fábio Sbano


ci: add flags on codecov report

Description:
Commit Message
type: commit message

see the guidelines for commit messages.
Changelog Entry
version: <log entry>

Checklist:

[x] I have performed a self-review of my own code
[ ] I have documented my code in accordance with the documentation guidelines
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have run the unit tests
[x] I only have one commit (if not, squash them into one commit).
[ ] I have a descriptive commit message that adheres to the commit message guidelines

Please review the CONTRIBUTING.md file for detailed contributing guidelines.

Codecov Report

Merging #356 (eaf9f0a) into main (af0d4d2) will increase coverage by 3.46%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #356      +/-   ##
============================================
+ Coverage     76.55%   80.01%   +3.46%
  Complexity      421      421
============================================
  Files           112       27      -85
  Lines         12538     2037   -10501
  Branches        284      284
============================================
- Hits           9599     1630    -7969
+ Misses         2798      266    -2532
  Partials        141      141

Flag
Coverage Δ

ibc
82.17% <ø> (?)

lightclients-tendermint
74.12% <ø> (?)

xcall
76.62% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 85 files with indirect coverage changes


[Reveal] Does not work using foundation.min.js

How can we reproduce this bug?

use foundation.min.js instead of foundation.js
use main Reveal example

What did you expect to happen?
Reveal should open a modal
What happened instead?
Absolutely nothing, not even an error call

Duplicate of https://github.com/zurb/foundation-sites/issues/7046

When did you start the project? We had an issue with mangled code on the .min version that we patched a fix for yesterday with the 6.0.1 release. From the browser console, type Foundation.version to see if you're up to date.

If you're using a copy of Foundation from the customizer, do not use the .min file for the js. There's been an issue getting it to update to v6.0.1. If you are using bower/npm to install, you should be able to update and switch back to the .min

I started the project today and downloaded it directly from the foundation website.
Sincerely,
Joshua Deaton

On Nov 20, 2015, at 12:55 PM, Chris Oyler<EMAIL_ADDRESS>wrote:
If you're using a copy of Foundation from the customizer, do not use the .min file for the js. There's been an issue getting it to update to v6.0.1. If you are using bower/npm to install, you should be able to update and switch back to the .min
—
Reply to this email directly or view it on GitHub https://github.com/zurb/foundation-sites/issues/7068#issuecomment-158493637.

We do apologize, there's an issue with how our node service handles versioned releases and is not yet giving updated .min files. Please use the unminified version for now.

https://github.com/zurb/foundation-sites/issues/7088


Microsoft.Insights/components@2020-02-02  - CustomMetricsOptedInType field is not documented

First reported here: https://github.com/Azure/bicep/issues/784#issuecomment-949653327
Under the ApplicationInsightsComponentProperties definition, there is no definition for the CustomMetricsOptedInType field:
https://github.com/Azure/azure-rest-api-specs/blob/88e7838a09868a51de3894114355c75929847a46/specification/applicationinsights/resource-manager/Microsoft.Insights/stable/2020-02-02/components_API.json#L465

@anthony-c-martin There's been a while... any news?

@anthony-c-martin There's been a while... any news?

The next step would be for the service team (@azmonapplicationinsights) to respond to this issue - I don't have any information other than what's present here.

ping


Are multi-line template expressions possible?

Dear community,
I have the following template expression inside a variables block:
${{ if and(eq(variables['Build.Reason'], 'IndividualCI'), startsWith(variables['Build.SourceBranchName'], 'test_')) }}
Which I would love to be able to write like this:
variables:
  - name: version
       ${{ if
             and(
                     eq(variables['Build.Reason'], 'IndividualCI'),
                     startsWith(variables['Build.SourceBranchName'], 'test_')
                    )
       }}
         value: test
...

but after reading the documentation I was unable to work out how to do this.
Alternatively, I would be happy to instead write:
variables:
  - name: newCommits
     value: $[ eq(variables['Build.Reason'], 'IndividualCI') ]
  - name: isTestBranch
     value: $[ startsWith(variables['Build.SourceBranchName'], 'test_') ]
  - name: ${{ if and(variables.newCommits, variables.isTestBranch) }}

But I've had issues accessing variables.newCommits and variables.isTestBranch inside the templates. This doesn't fail per se, but does not give the correct behaviour (I think this is because the execution order of $[...] vs ${...} maybe?).
Are either of these two methods possible? Many thanks.

Hi @LachlanMarnham, multi-line template expressions are not supported. And, yes, there is some difference in expanding variables in Template expressions and Runtime expressions (Please see more details here)
In your case, I think you can try this:
variables:
  ${{ if eq(variables['Build.Reason'], 'IndividualCI') }}:
    newCommits: True
  ${{ if startsWith(variables['Build.SourceBranchName'], 'test_') }}:
    isTestBranch: True
  ${{ if and(eq(variables['newCommits'], True), eq(variables['isTestBranch'], True)) }}:
    version: Test
  ${{ if not(and(eq(variables['newCommits'], True), eq(variables['isTestBranch'], True))) }}:
    version: Prod

@LachlanMarnham I'm closing this since it looks resolved - please feel free to ask any further questions.

Yaml has multiline keys support, those should be wrapped with ?:
This works for me in my templates:
  - stage: Sign
    ? ${{ if or(
        in(parameters.BuildType, 'nuspec'),
        and(
          not(in(variables['Build.SourceBranchName'], 'master', 'main')),
          not(startsWith(variables['Build.SourceBranch'], 'refs/heads/support/')),
          not(startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'))
        )
      ) }}
    : displayName: Sign - Skipped
    ${{ else }}:
      displayName: Sign
    jobs:
      - ? ${{ if or(
            in(parameters.BuildType, 'nuspec'),
            and(
              not(in(variables['Build.SourceBranchName'], 'master', 'main')),
              not(startsWith(variables['Build.SourceBranch'], 'refs/heads/support/')),
              not(startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'))
            )
          ) }}
        : - job: Skipped
            steps:
              - checkout: none
      - ${{ else }}:
          - template: sign.yml


Find files outside of working directory

Thank you for this helpful tool :heart_eyes_cat:
It fails to check files specified using the CLI though, while reading from STDIN works as expected:
bash-3.2$ echo 'Our developers are great guys.' > /tmp/alex.txt
bash-3.2$ alex /tmp/alex.txt

bash-3.2$ cat /tmp/alex.txt | alex

<stdin>
  1:26-1:30  warning  `guys` may be insensitive, use `people`, `persons`, `folks` instead

⚠ 1 warning
bash-3.2$ node -v
v5.1.0
bash-3.2$ alex --version
1.6.0

Thanks for reporting, @c089!

Thanks, that was fast :) Not sure this is my bug though, because it also fails from the same cwd, i.e. cd /tmp && alex ./alex.txt.

...but 1.6.1 fixed both :) Thanks a lot.

Hmm, well, weird. Glad it’s fixed though :smile: