[Bug] On Duplicate Values It shows Failed

Describe the bug
I tried to login using<EMAIL_ADDRESS>but I was unable to login as It showed an Error toast.
To Reproduce
Steps to reproduce the behavior:

Go to 'SignUp'
SignUp using<EMAIL_ADDRESS>See error

Expected behavior
It should say that the user already Exists if it finds a duplicate value trying to signUp.
Screenshots

@Yashgabani845 Please assign it to me.

@inkerton it is reserved for admin if you have any alternaive for admin then tell it  otherwise show error that it has been taken

If you are thinking of deploying the project you will need google auth and all else You cannot have<EMAIL_ADDRESS>as your admin details I would suggest add another role as 'Admin' or 'JobSeeker' or 'Employer'. Based on which you can differentiate . for now you can have a check if email = process.env.ADMIN_EMAIL. then assign role admin else by default jobseeker and if the user signs in via employee portal them employer.
It will make things more easy in the future also If you wish to add distinguished features for employers and job seeker.
@Yashgabani845

On duplicate emails maybe give a forgot password option so the user can relogin again.

It might be a person who is trying to signup again .

@inkerton it should not be done on signup handle this error and show message that user is already exist
(we do not have to assume that the person is owner of that particualer e mail id )
forget password may harm the account security

@inkerton if you need more info ask me or if you feel it can not be implemented then i can close this issue
give me update

I have thought of It I can make an API to check whether the email exists and if yes then We can show toast notifications . Their is no toast notification in the whole project if successfully loggedin or any error while logging in also their is nothing stated as to which values are compulsor and which are not if user leaves a cmpusory value empty it fails It should say to the user that Education cannot be empty of something . I can add toast notifications if You find this idea feasible.
For success

for warning

for error

I closed it by mistake let me know If you need notifications alarms or checks.

@inkerton notification is ok


can i do inference using CPU

Getting error while running cityscapes_evl.py on Cityscape Data set
when i run with "--gpu 0" , it shows "RuntimeError: CUDA error: out of memory", because of my 2GB GPU. and then, i run without "--cuda " or "--gpu" ,it also looks like "RuntimeError: CUDA error: out of memory",so how can i do test or eval using CPU, thx!

@jiaobingle U can refer to the Pytorch official example, and learn how to run the model on the cpu.


No Improvements to training speed with GPU (partial GPU usage?!)

I am trying to train my model on a GPU instead of a CPU on an AWS p2.xlarge instance from my Jupyter Notebook. I am using tensorflow-gpu backend (only tensorflow-gpu was installed and mentioned in requirements.txt and not tensorflow).
I am not seeing any speed improvements when training models on these instances compared to using a CPU, infact I am getting training speeds per epoch that is almost same as I am getting on my 4-core  laptop CPU (p2.xlarge also has 4 vCPUs with a Tesla K80 GPU). I am not sure if i need to do some changes to my code to accommodate faster/parallel processing that GPU can offer. I am pasting below my code for my model:
model = Sequential()
model.add(recurrent.LSTM(64, input_shape=(X_np.shape[1], X_np.shape[2]),
                        return_sequences=True))
model.add(recurrent.LSTM(64, return_sequences = False))
model.add(core.Dropout(0.1))
model.add(core.Dense(3, activation='softmax'))
model.compile(loss = 'categorical_crossentropy', optimizer = 'rmsprop', metrics=['accuracy'])

model.fit(X_np, y_np, epochs=100, validation_split=0.25)

Also interestingly the GPU seems to be utilizing between 50%-60% of its processing power and almost all of its memory every time I check for GPU status using nvidia-smi (but both fall to 0% and 1MiB respectively when not training):
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.81                 Driver Version: 384.81                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   47C    P0    73W / 149W |  10919MiB / 11439MiB |     52%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1665      C   ...ubuntu/aDash/MLenv/bin/python 10906MiB |
+-----------------------------------------------------------------------------+

Also if you's like to see my logs about using the GPU from Jupyter Notebook:
[I 04:21:59.390 NotebookApp] Kernel started: c17bc4d1-fa15-4b0e-b5f0-87f90e56bf65
[I 04:22:02.241 NotebookApp] Adapting to protocol v5.1 for kernel c17bc4d1-fa15-4b0e-b5f0-87f90e56bf65
2017-11-30 04:22:32.403981: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-11-30 04:22:33.653681: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-11-30 04:22:33.654041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:1e.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2017-11-30 04:22:33.654070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2017-11-30 04:22:34.014329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7
2017-11-30 04:22:34.015339: I tensorflow/core/common_runtime/direct_session.cc:299] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7

2017-11-30 04:23:22.426895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)

Please suggest what could be the problem. Thanks a ton for looking at this anyways!

Seems to run just like it should on a GPU. How much time is it taking exactly for a epoch?

@teja-bogireddy takes 18 seconds for an epoch for dataset of dim: (340075) on GPU same as on CPU

If this is your first program on that GPU. Then you must have missed something in gpu-integrated installation of tensorflow. Try installing it again[purge before installing]. I have been in this situation. Hope this works!

Yes it is my first try on the GPU. I am using https://aws.amazon.com/marketplace/pp/B077GCZ4GR AMI specifically ami-9e745afb with tensorflow_p36 virtualenv

@teja-bogireddy Although one other thing that I am not sure perfectly relates is I was getting python error in my notebook: error while loading shared libraries: libcublas.so.8.0 when importing Keras and figured that it was not found the correct path to the libraries and exported environment variable export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64 and that solved this error and the notebook was then running but only with the training speed that I earlier mentioned.


add three plugins of xinghuo bif

This pr will add three plugins of Xinghuo BIF

solidity compiler
solidity udapp
javascript udapp

Here is the official website of Xinghuo BIF.
https://bitfactory.cn/gywm.html
The organization behind it is CAICT. Here is the website of CAICT.
http://www.caict.ac.cn/english/about/202004/t20200413_279063.html
Here is the tips videos

https://www.bilibili.com/video/BV1Ni4y1h7Yw
https://www.bilibili.com/video/BV1uQ4y1g7CH

@ryestew @yann300 Can you check this PR first? We are about to pulish an article for these plugins. We can't do that until this PR is merged.

@drafish thanks for the PR and your work!
Could you give us more information about these plugins:

solidity compiler is this the same as the classic one in Remix? what are the differences?
same question for solidity udapp.
javascript udapp: what is the purpose of this one?

Thanks very much!

solidity compiler is almost the same as the classic one in Remix. The only difference between them is the compiler. Xinghuo BIF made some changes based on the original solidity, so the original compiler won't work, we have to use their special compiler.
solidity udapp is almost the same as the classic one in Remix too.  But the classic one is for Ethereum. We can't deploy contract to Xinghuo BIF in the classic one. That's why this one is here.
js udapp is for deploying js contract to Xinghuo BIF. JS is another contract language that Xinghuo BIF supports.

@yann300

solidity compiler is almost the same as the classic one in Remix. The only difference between them is the compiler. Xinghuo BIF made some changes based on the original solidity, so the original compiler won't work, we have to use their special compiler.
solidity udapp is almost the same as the classic one in Remix too.  But the classic one is for Ethereum. We can't deploy contract to Xinghuo BIF in the classic one. That's why this one is here.
js udapp is for deploying js contract to Xinghuo BIF. JS is another contract language that Xinghuo BIF supports.

@yann300

Thanks for the answer, could you send me some references where I can learn more about these differences?

This is the documentation that explain the difference between xinghuo solidity and ethereum solidity -- https://bif-doc.readthedocs.io/zh-cn/2.0.0/contract/Solidity合约介绍.html#soliditysolidity
This is the documentation for the JS contract -- https://bif-doc.readthedocs.io/zh-cn/2.0.0/contract/Javascript合约介绍.html
These documentations are chinese, you need to do a quick translation. @yann300

Can you guys merge it as soon as possible? @yann300 @ryestew


[build-407] Mount shared secret and configmap list config path into shared driver node

Part of https://github.com/openshift/csi-driver-shared-resource-operator/pull/53
For review and suggestions:
/assign @coreydaley
/assign @gabemontero
/assign @jkhelil

Can you spin up a cluster in cluster-bot with this pull request and make sure that it behaves like we need it to?

Can you spin up a cluster in cluster-bot with this pull request and make sure that it behaves like we need it to?

Yup I need to do that.

ok @coreydaley I have created cluster with this change and I see all the files residing under the configmap path which means we can use --config multiple times and then read the data from the file.
$ oc -n openshift-cluster-csi-drivers exec -i -t shared-resource-csi-driver-node-4bm6c -c hostpath -- /bin/bash
[root@shared-resource-csi-driver-node-4bm6c /]# ls -a /var/run/configmaps/config/
.  ..  ..2022_10_11_06_53_31.3090059889  ..data  config.yaml  sharedconfigmap-list.yaml  sharedsecret-list.yaml

I will do changes in the resource repo and then it should work fine. However this change looks good to me.

bump @coreydaley

/approved
/lgtm

/approve

/label docs-approved
/label px-approved
/label qe-approved


Support EIP-6963

This will be done once wagmi v2 is out

I assume this will be same task as https://github.com/snapshot-labs/sx-monorepo/issues/125.


Fix span compression for redis et. al.

What does this pull request do?
@LucaWintergerst pointed out that span compression doesn't work with redis. As it turns out, there were two bugs that caused this:

after we implemented auto-calculation of span destination resource fields, these were no longer available in span compression. Moving the auto-calculation to run before span destination fixed this (e2c9c8f)
redis spans, along with a bunch of other services, were not marked as leaf spans

To ensure that we don't regress on this again, I added a bunch of service-specific span compression tests.

:broken_heart: Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-10T14:50:08.354+0000

Duration: 32 min 5 sec

Commit: 6da59990968b09cf7572ab35594ff75c563ee769

Test stats :test_tube:

Test
Results

Failed
6

Passed
10316

Skipped
9483

Total
19805

Test errors
Expand to view the tests failures
Initializing / Test / Python-python-3.6-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.set'
  'Client.get_many'
  'Client.get'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7f08bf9b2080>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.set'
E         'Client.get_many'
E         'Client.get'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Initializing / Test / Python-python-3.10-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.set'
  'Client.get'
  'Client.get_many'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7f08bd92d180>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.set'
E         'Client.get'
E         'Client.get_many'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Initializing / Test / Python-python-3.8-3 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.get'
  'Client.get_many'
  'Client.set'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7fc5720b8550>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.get'
E         'Client.get_many'
E         'Client.set'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Initializing / Test / Python-python-3.9-3 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.get'
  'Client.get_many'
  'Client.set'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7ffa7f0eb130>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.get'
E         'Client.get_many'
E         'Client.set'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Initializing / Test / Python-python-3.7-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.set'
  'Client.get_many'
  'Client.get'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7f4fb65fe290>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.set'
E         'Client.get_many'
E         'Client.get'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Initializing / Test / Python-pypy-3-0 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

Expand to view the error details
 AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
  Extra items in the right set:
  'Client.set'
  'Client.get'
  'Client.get_many'
  Full diff:
    {
  -  'Client.get',...

  ...Full output truncated (8 lines hidden), use '-vv' to show

Expand to view the stacktrace
 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x000000000ac65fa0>

    @pytest.mark.integrationtest
    def test_pymemcache_hash_client(instrument, elasticapm_client):
        elasticapm_client.begin_transaction("transaction.test")
        host = os.environ.get("MEMCACHED_HOST", "localhost")
        with capture_span("test_pymemcache", "test"):
            conn = pymemcache.client.hash.HashClient([(host, 11211)])  # can't use mc_conn here due to different init
            conn.set("mykey", "a")
            assert b"a" == conn.get("mykey")
            assert {"mykey": b"a"} == conn.get_many(["mykey", "myotherkey"])
        elasticapm_client.end_transaction("BillingView")

        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])

        expected_signatures = {
            "test_pymemcache",
            "HashClient.set",
            "HashClient.get",
            "HashClient.get_many",
            "Client.set",
            "Client.get",
            "Client.get_many",
        }

>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'HashClient....t_pymemcache'} == {'Client.get'...ent.set', ...}
E         Extra items in the right set:
E         'Client.set'
E         'Client.get'
E         'Client.get_many'
E         Full diff:
E           {
E         -  'Client.get',...
E
E         ...Full output truncated (8 lines hidden), use '-vv' to show

tests/instrumentation/pymemcache_tests.py:154: AssertionError

Steps errors
Expand to view the steps failures

Show only the first 10 steps failures

Shell Script

Took 0 min 17 sec . View more details here
Description: ./tests/scripts/docker/run_tests.sh python-3.9 pymemcache-newest

Shell Script

Took 0 min 19 sec . View more details here
Description: ./tests/scripts/docker/run_tests.sh python-3.10 pymemcache-newest

Shell Script

Took 0 min 18 sec . View more details here
Description: ./tests/scripts/docker/run_tests.sh python-3.10 pymemcache-newest

Shell Script

Took 1 min 8 sec . View more details here
Description: ./tests/scripts/docker/run_tests.sh pypy-3 pymemcache-newest

Shell Script

Took 1 min 8 sec . View more details here
Description: ./tests/scripts/docker/run_tests.sh pypy-3 pymemcache-newest

Restore files previously stashed

Took 0 min 0 sec . View more details here
Description: coverage-python-3.7-sanic-newest

Restore files previously stashed

Took 0 min 0 sec . View more details here
Description: coverage-python-3.8-sanic-newest

Restore files previously stashed

Took 0 min 0 sec . View more details here
Description: coverage-python-3.9-sanic-newest

Archive the artifacts

Took 0 min 0 sec . View more details here
Description: [2022-01-10T15:22:09.949Z] Archiving artifacts
Python-python-3.6-pymemcache-newest tests failed : h

Error signal

Took 0 min 0 sec . View more details here
Description: Python-python-3.6-pymemcache-newest tests failed : hudson.AbortException: script returned exit code 2

:bug: Flaky test report
:grey_exclamation: There are test failures but not known flaky tests.
Expand to view the summary

Genuine test errors
:broken_heart: There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / Test / Python-python-3.6-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests
Name: Initializing / Test / Python-python-3.10-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests
Name: Initializing / Test / Python-python-3.8-3 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests
Name: Initializing / Test / Python-python-3.9-3 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests
Name: Initializing / Test / Python-python-3.7-2 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests
Name: Initializing / Test / Python-pypy-3-0 / test_pymemcache_hash_client – tests.instrumentation.pymemcache_tests

:robot: GitHub comments
To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

/test linters : Run the Python linters only.

/test full : Run the full matrix of tests.

/test benchmark : Run the APM Agent Python benchmarks tests.

run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)


Template out known_users.csv, optionally add groups

lineinfile isn't idempotent, template out known_users, and allow operator to add groups to static password file

ci check this

@whereismyjetpack can you sign the cla?

yes sir, working on it... sorry for the delay


Upgrade to chrome 43 (atom/electron)

Hello,
Can you please consider updating atom/electron module due to an issue with Chrome prior to 43? Check issues atom/electron#1955 and atom/electron#1478 for more detail.
Thank you,
Bruno

Thanks for opening this! This is definitely going to happen though, but only when it's ready and only when several show-stopping bugs have been fixed, and no sooner, unfortunately. You can check out https://github.com/atom/atom/pull/6877 for some context.


0167 - Two Sum II - Input array is sorted

Description of the Problem
Given an array of integers that is already sorted in ascending order, find two numbers such that they add up to a specific target number.
The function twoSum should return indices of the two numbers such that they add up to the target, where index1 must be less than index2.
Code
class Solution:
    def twoSum(self, numbers: List[int], target: int) -> List[int]:
        for x in numbers:
            if x > target:
                break
            else:
                if target - x == x:
                    return (numbers.index(x)+1,numbers.index(x)+2)
                else:
                    if target - x in numbers:
                        return(numbers.index(x)+1,numbers.index(target - x)+1)

Link To The LeetCode Problem
https://leetcode.com/problems/two-sum-ii-input-array-is-sorted/

I can do this. Assign to me


test annotate_from_gff works on gff3

Use the fasta and gff3 files from #378, test that you can correctly apply annotations to the sequence directly, and to the sequence when it's a part of a Alignment.

we need more extensive demo files. Truncate to cover following cases:

object has multiple parents, in which case separate instances are created on each parent
need at least gene, transcript, cds, exons

Here's the flybase link ftp://ftp.flybase.org/genomes/dmel/current/gff/dmel-all-filtered-r6.30.gff.gz.


c++ async client unexpectly timeout if I don't call CQ::Next()

What version of gRPC and what language are you using?
v1.21.x c++
What operating system (Linux, Windows,...) and version?
windows 10pro
What runtime / compiler are you using (e.g. python version or version of gcc)
vs2015 Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
What did you do?
1> start a very simple c++ sync server, like echo server.
2> start an async c++ client, it has one associated CQ, entrusting 2 requests to the CQ, setting the timeout value T as 10 milliseconds,  and then do the following:
uint32_t _counter = 0;
while (cq_->Next(&got_tag, &ok)) {

    std::cout << "before counter:" << _counter++ << std::endl;

    std::this_thread::sleep_for(std::chrono::seconds(2));

    GPR_ASSERT(ok);
    if (!call->status.ok()) {
        std::cout << call->status.error_code() << ",msg:" << call->status.error_message();
        GPR_ASSERT(false);
    }
}

What did you expect to see?
output this:
before counter:0
before counter:1
What did you see instead?
only output :
before counter:0
before counter:1
and crashed at GPR_ASSERT(false) with a timeout error msg.
Anything else we should know about your project / environment?
When I set the timeout value T from 10 milliseconds to 5 seconds(> 2seconds), the issue disappeared.
It seems that, even if the server has returned the result to the async client within the timeout value, grpc will still report a timeout error just because of you don't fetch the result out from the CQ by calling cq_->Next(), if this is true(which I can't believe), then it must be a VERY BAD DESIGN.
Am I doing something wrong?

I think this is fine. The whole client->server->client cycle times out.
What's your suggestion for a better behavior here?

Thank you for your question. The gRPC Async API indeed relies on the application to call CQ->Next() for performing internal work. This is very much the intended design.

@AspirinSJL
I think it's better to make CQ can automatically detecting timeout event, recording it and tell the application when it coming to fetch the result.   The bad thing about the current design is : if I have heavy workload(e.g., it will take hundreds of milliseconds for each response to deal with) for the threads which are polling the CQ on the client side, it will certainly get a change that when the result coming back, the threads are busying at doing other jobs, and when are free to fetch the next job, they will found that the job has already been time out, even though they are actually not in reality .
The current behavior is quite implicit, cost me a lot to find out.

@a11r  I know. But not everything is best to be driven by application.  I think the time out detection job is one of the exceptions.

Regarding https://github.com/grpc/grpc/issues/19573#issuecomment-510168980, I think in the case you described, the response actually timed out due to lack of client resource. I feel timeout can describe any case as long as the result is not properly handled within a period, not just the network part.

alright, it makes sense in that way.


Feat: Implementing Install() for github-reposcaffolding-golang plugin

Signed-off-by: Daniel Hu<EMAIL_ADDRESS>Summary
Implementing Install() for github-reposcaffolding-golang plugin

/cc @IronCore864 @lfbdev Please take a review at this pr.


Should be able to execute specifications with compatible versions of plugins

Expected behavior
Should be able to execute specifications with compatible versions of plugins
Actual behavior
Getting error Failed to start gauge API: Compatibility error. Incompatible version. Minimum support version a.b.c is higher than current version x.y.z
Steps to replicate

Install Gauge gauge 1.0.1 with js plugin 2.3.1
Create a gauge-js project
Downgrade Gauge version to 0.9.8
gauge run specs on the project

It recognizes that a compatible version is not available and installs js version 2.3.0 which is compatible with 0.9.8. However the execution phase fails with the following error

Compatible version of plugin js not found. Installing plugin js...
........
[grpc] Success: ".../.gauge/plugins/js/2.3.0/node_modules/grpc/src/node/extension_binary/node-v64-darwin-x64-unknown/grpc_node.node" is installed via remote
added 169 packages from 148 contributors and audited 684 packages in 4.917s
found 12 vulnerabilities (10 low, 2 moderate)
  run `npm audit fix` to fix them, or `npm audit` for details
Successfully installed plugin 'js'.
Error ----------------------------------

Failed to start gauge API: Compatibility error. Incompatible version. Minimum support version 1.0.0 is higher than current version 0.9.8

Get Support ----------------------------
    Docs:          https://docs.gauge.org
    Bugs:          https://github.com/getgauge/gauge/issues
    Chat:          https://gitter.im/getgauge/chat

Your Environment Information -----------
    darwin, 0.9.8, c23df9f
    html-report (4.0.5), java (0.6.8), js (2.3.1), screenshot (0.0.1)

Blocks #1157

Though this is not related to npm installer directly. This fix is required for npm local installations to work seamlessly. This issue will occur when user downloads different Gauge binary per project and they have plugins that are incompatible with both Gauge binaries.

Closing this as an old issue. This may be fixed in the newer releases.
Will reopen if it still needs fixing.


Unable to run in Testing environment

React-simple-keyboard version
Using version:  3.5.30

EDIT: Updated to version 3.5.32, however, the issue remains

Testing environment:

VItest: ^0.25.8
Vite: ^4.0.1
@testing-library/jest-dom: ^5.16.5
@testing-library/react: 13.4.0

Description
The Keyboard component works well during local serve and production builds. However, when running in a test environment, the keyboard component produces the following error message:
Warning: React.jsx: type is invalid -- expected a string (for built-in components) or a class/function (for composite components) but got: object.
Check your code at index.tsx:224.

Steps i've attempted

Removed the createPortal - Same result
Tried to test input with keyboard in isolation - Same result
Tried to test only Keyboard - Same result
Tried modifying import to import * as Keyboard from "react-simple-keyboard - Same result

Screenshots
Console error message

Import of component

Rendering component

@dewald-els Could you share a test repository for the issue you're seeing? That would help me to debug this issue. Thanks! Will reopen when that is provided.

@hodgef sure thing! Thanks for checking it out. I'll get a test repo ready tomorrow and share a link. I did fork the repo to try and isolate the issue, but didn't have much success :) Might be something with my implementation. I'll send the repo soon! 👍

@hodgef I've added you as a collaborator to a project that includes the keyboard test with Vite. Made it private, let me know if you have any trouble accessing it 👍

@dewald-els How do you run the tests? There seems to be no scripts in the package.json :\

@dewald-els How do you run the tests (and the project in general)? There seems to be no scripts in the package.json :\

Oh! It's a monorepo, I made it as close as possible to the actual project environment. You can run
npx nx test keyboard-test

np! Try using this as the import. It worked for me:
import { KeyboardReact as Keyboard, SimpleKeyboard } from 'react-simple-keyboard';

@hodgef I know the issue is close, but damn! you are amazing. How did i miss that! haha thanks so much for the quick feedback.


Address cargo clippy warnings

The following Cargo clippy warnings and warnings should be addressed.

cargo clippy --workspace @ e6d7e7b6218ba87d13162d71f2db58b348908071
    Checking ssi v0.4.0 (/home/cel/src/ssi)
warning: associated function is never used: `as_struct_mut`
   --> src/eip712.rs:240:8
    |
240 |     fn as_struct_mut(&mut self) -> Option<&mut HashMap<StructName, EIP712Value>> {
    |        ^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: unneeded `return` statement
   --> src/did_resolve.rs:892:5
    |
892 |     return (res_meta, result.did_document, result.did_document_metadata);
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove `return`: `(res_meta, result.did_document, result.did_document_metadata)`
    |
    = note: `#[warn(clippy::needless_return)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_return

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> src/did_resolve.rs:927:51
    |
927 |             percent_encoding::utf8_percent_encode(&did, percent_encoding::CONTROLS).to_string();
    |                                                   ^^^^ help: change this to: `did`
    |
    = note: `#[warn(clippy::needless_borrow)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> src/did_resolve.rs:953:28
    |
953 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = note: `#[warn(clippy::to_string_in_format_args)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `to_string` applied to a type that implements `Display` in `format!` args
    --> src/did_resolve.rs:1102:28
     |
1102 |                         err.to_string()
     |                            ^^^^^^^^^^^^ help: remove this
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: this `.into_iter()` call is equivalent to `.iter()` and will not consume the `Vec`
    --> src/eip712.rs:1038:40
     |
1038 |                 let mut values = array.into_iter();
     |                                        ^^^^^^^^^ help: call directly: `iter`
     |
     = note: `#[warn(clippy::into_iter_on_ref)]` on by default
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#into_iter_on_ref

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> src/jws.rs:175:48
    |
175 |                 let hash = crate::hash::sha256(&data)?;
    |                                                ^^^^^ help: change this to: `data`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
   --> src/revocation.rs:277:9
    |
277 | /         match revocation_list_credential.validate() {
278 | |             Err(e) => {
279 | |                 return result.with_error(format!("Invalid list credential: {}", e));
280 | |             }
281 | |             Ok(()) => {}
282 | |         }
    | |_________^
    |
    = note: `#[warn(clippy::single_match)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_match
help: try this
    |
277 ~         if let Err(e) = revocation_list_credential.validate() {
278 +             return result.with_error(format!("Invalid list credential: {}", e));
279 +         }
    |

warning: `ssi` (lib) generated 8 warnings
    Checking did-onion v0.1.2 (/home/cel/src/ssi/did-onion)
    Checking did-sol v0.0.1 (/home/cel/src/ssi/did-sol)
    Checking did-tz v0.1.2 (/home/cel/src/ssi/did-tezos)
    Checking did-method-key v0.1.3 (/home/cel/src/ssi/did-key)
    Checking did-pkh v0.1.1 (/home/cel/src/ssi/did-pkh)
    Checking did-webkey v0.1.1 (/home/cel/src/ssi/did-webkey)
    Checking did-web v0.1.2 (/home/cel/src/ssi/did-web)
warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-onion/src/lib.rs:119:28
    |
119 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = note: `#[warn(clippy::to_string_in_format_args)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-onion/src/lib.rs:132:28
    |
132 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-onion/src/lib.rs:149:28
    |
149 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `did-onion` (lib) generated 3 warnings
    Checking did-ion v0.1.0 (/home/cel/src/ssi/did-ion)
warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-webkey/src/lib.rs:242:28
    |
242 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = note: `#[warn(clippy::to_string_in_format_args)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

    Checking did-ethr v0.1.1 (/home/cel/src/ssi/did-ethr)
warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-webkey/src/lib.rs:259:28
    |
259 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

    Checking ssi-vc-test v0.1.0 (/home/cel/src/ssi/vc-test)
warning: `did-webkey` (lib) generated 2 warnings
warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-web/src/lib.rs:118:28
    |
118 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = note: `#[warn(clippy::to_string_in_format_args)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-web/src/lib.rs:135:28
    |
135 |                         err.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `did-web` (lib) generated 2 warnings
warning: this `impl` can be derived
  --> did-tezos/src/lib.rs:36:1
   |
36 | / impl Default for DIDTz {
37 | |     fn default() -> Self {
38 | |         Self { tzkt_url: None }
39 | |     }
40 | | }
   | |_^
   |
   = note: `#[warn(clippy::derivable_impls)]` on by default
   = help: try annotating `DIDTz` with `#[derive(Default)]`
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#derivable_impls

warning: `to_string` applied to a type that implements `Display` in `format!` args
   --> did-tezos/src/lib.rs:428:28
    |
428 |                     address.to_string()
    |                            ^^^^^^^^^^^^ help: remove this
    |
    = note: `#[warn(clippy::to_string_in_format_args)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

warning: `did-tz` (lib) generated 2 warnings
    Checking ssi-did-test v0.1.0 (/home/cel/src/ssi/did-test)
warning: unused imports: `StatusCode`, `header`
 --> did-ion/src/sidetree.rs:5:15
  |
5 | use reqwest::{header, Client, StatusCode};
  |               ^^^^^^          ^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: field is never read: `type`
   --> did-ion/src/sidetree.rs:517:5
    |
517 |     r#type: Option<String>,
    |     ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default
note: `PartiallyVerifiedCreateOperation` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
   --> did-ion/src/sidetree.rs:514:10
    |
514 | #[derive(Debug, Clone)]
    |          ^^^^^  ^^^^^
    = note: this warning originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)

warning: field is never read: `anchor_origin`
   --> did-ion/src/sidetree.rs:519:5
    |
519 |     anchor_origin: Option<String>,
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: `PartiallyVerifiedCreateOperation` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
   --> did-ion/src/sidetree.rs:514:10
    |
514 | #[derive(Debug, Clone)]
    |          ^^^^^  ^^^^^
    = note: this warning originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)

warning: field is never read: `signed_anchor_origin`
   --> did-ion/src/sidetree.rs:542:5
    |
542 |     signed_anchor_origin: Option<String>,
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: `PartiallyVerifiedRecoverOperation` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
   --> did-ion/src/sidetree.rs:536:10
    |
536 | #[derive(Debug, Clone)]
    |          ^^^^^  ^^^^^
    = note: this warning originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:318:54
    |
318 |         let update_reveal_value = Self::reveal_value(&canonicalized_update_pk.as_bytes());
    |                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: change this to: `canonicalized_update_pk.as_bytes()`
    |
    = note: `#[warn(clippy::needless_borrow)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:326:37
    |
326 |             Self::commitment_scheme(&new_update_pk).context("Generate new update commitment")?;
    |                                     ^^^^^^^^^^^^^^ help: change this to: `new_update_pk`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:375:55
    |
375 |         let recover_reveal_value = Self::reveal_value(&canonicalized_recovery_pk.as_bytes());
    |                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: change this to: `canonicalized_recovery_pk.as_bytes()`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:377:37
    |
377 |             Self::commitment_scheme(&new_update_pk).context("Generate new update commitment")?;
    |                                     ^^^^^^^^^^^^^^ help: change this to: `new_update_pk`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:379:37
    |
379 |             Self::commitment_scheme(&new_recovery_pk).context("Generate new update commitment")?;
    |                                     ^^^^^^^^^^^^^^^^ help: change this to: `new_recovery_pk`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:451:55
    |
451 |         let recover_reveal_value = Self::reveal_value(&canonicalized_recovery_pk.as_bytes());
    |                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: change this to: `canonicalized_recovery_pk.as_bytes()`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: large size difference between variants
   --> did-test/src/main.rs:109:5
    |
109 | /     Resolve {
110 | |         did_document: Option<Document>,
111 | |         did_resolution_metadata: ResolutionMetadata,
112 | |         did_document_metadata: DocumentMetadata,
113 | |     },
    | |_____^ this variant is 592 bytes
    |
    = note: `#[warn(clippy::large_enum_variant)]` on by default
note: and the second-largest variant is 216 bytes:
   --> did-test/src/main.rs:121:5
    |
121 | /     Dereference {
122 | |         dereferencing_metadata: DereferencingMetadata,
123 | |         content_stream: String,
124 | |         content_metadata: ContentMetadata,
125 | |     },
    | |_____^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant
help: consider boxing the large fields to reduce the total size of the enum
    |
110 |         did_document: Box<Option<Document>>,
    |                       ~~~~~~~~~~~~~~~~~~~~~

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:625:49
    |
625 |     let computed_reveal_value = S::reveal_value(&commitment_value);
    |                                                 ^^^^^^^^^^^^^^^^^ help: change this to: `commitment_value`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:628:30
    |
628 |         S::commitment_scheme(&pk).context("Unable to compute public key commitment")?;
    |                              ^^^ help: change this to: `pk`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:673:21
    |
673 |                     &update_commitment,
    |                     ^^^^^^^^^^^^^^^^^^ help: change this to: `update_commitment`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:683:21
    |
683 |                     &recovery_commitment,
    |                     ^^^^^^^^^^^^^^^^^^^^ help: change this to: `recovery_commitment`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> did-ion/src/sidetree.rs:703:21
    |
703 |                     &recovery_commitment,
    |                     ^^^^^^^^^^^^^^^^^^^^ help: change this to: `recovery_commitment`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: for loop over a single element
   --> did-test/src/main.rs:284:5
    |
284 | /     for did in &["did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid"] {
285 | |         let did_vector = did_method_vector(&resolver, did).await;
286 | |         did_vectors.insert(did.to_string(), did_vector);
287 | |     }
    | |_____^
    |
    = note: `#[warn(clippy::single_element_loop)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
284 ~     {
285 +         let did = &"did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid";
286 +         let did_vector = did_method_vector(&resolver, did).await;
287 +         did_vectors.insert(did.to_string(), did_vector);
288 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:341:5
    |
341 | /     for did in &["did:webkey:ssh:demo.spruceid.com:2021:07:14:keys"] {
342 | |         let did_vector = did_method_vector(&resolver, did).await;
343 | |         did_vectors.insert(did.to_string(), did_vector);
344 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
341 ~     {
342 +         let did = &"did:webkey:ssh:demo.spruceid.com:2021:07:14:keys";
343 +         let did_vector = did_method_vector(&resolver, did).await;
344 +         did_vectors.insert(did.to_string(), did_vector);
345 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:511:5
    |
511 | /     for did in &["did:key:z6MkpTHR8VNsBxYAAWHut2Geadd9jSwuBV8xRoAnwWsdvktH"] {
512 | |         report
513 | |             .resolve_representation(
514 | |                 &did_method_key::DIDKey,
...   |
518 | |             .await;
519 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
511 ~     {
512 +         let did = &"did:key:z6MkpTHR8VNsBxYAAWHut2Geadd9jSwuBV8xRoAnwWsdvktH";
513 +         report
514 +             .resolve_representation(
515 +                 &did_method_key::DIDKey,
516 +                 did,
  ...

warning: for loop over a single element
   --> did-test/src/main.rs:543:5
    |
543 | /     for did in &["did:web:identity.foundation"] {
544 | |         report
545 | |             .resolve_representation(&did_web::DIDWeb, did, &ResolutionInputMetadata::default())
546 | |             .await;
547 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
543 ~     {
544 +         let did = &"did:web:identity.foundation";
545 +         report
546 +             .resolve_representation(&did_web::DIDWeb, did, &ResolutionInputMetadata::default())
547 +             .await;
548 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:574:5
    |
574 | /     for did in &["did:tz:tz1YwA1FwpgLtc1G8DKbbZ6e6PTb1dQMRn5x"] {
575 | |         report
576 | |             .resolve_representation(&did_tz, did, &ResolutionInputMetadata::default())
577 | |             .await;
578 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
574 ~     {
575 +         let did = &"did:tz:tz1YwA1FwpgLtc1G8DKbbZ6e6PTb1dQMRn5x";
576 +         report
577 +             .resolve_representation(&did_tz, did, &ResolutionInputMetadata::default())
578 +             .await;
579 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:594:5
    |
594 | /     for did in &["did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid"] {
595 | |         report
596 | |             .resolve(&resolver, did, &ResolutionInputMetadata::default())
597 | |             .await;
598 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
594 ~     {
595 +         let did = &"did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid";
596 +         report
597 +             .resolve(&resolver, did, &ResolutionInputMetadata::default())
598 +             .await;
599 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:600:5
    |
600 | /     for did in &["did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid"] {
601 | |         report
602 | |             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
603 | |             .await;
604 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
600 ~     {
601 +         let did = &"did:onion:fscst5exmlmr262byztwz4kzhggjlzumvc2ndvgytzoucr2tkgxf7mid";
602 +         report
603 +             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
604 +             .await;
605 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:620:5
    |
620 | /     for did in &["did:pkh:doge:DH5yaieqoZN36fDVciNyRueRGvGLR3mr7L"] {
621 | |         report
622 | |             .resolve(&resolver, did, &ResolutionInputMetadata::default())
623 | |             .await;
624 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
620 ~     {
621 +         let did = &"did:pkh:doge:DH5yaieqoZN36fDVciNyRueRGvGLR3mr7L";
622 +         report
623 +             .resolve(&resolver, did, &ResolutionInputMetadata::default())
624 +             .await;
625 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:626:5
    |
626 | /     for did in &["did:pkh:doge:DH5yaieqoZN36fDVciNyRueRGvGLR3mr7L"] {
627 | |         report
628 | |             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
629 | |             .await;
630 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
626 ~     {
627 +         let did = &"did:pkh:doge:DH5yaieqoZN36fDVciNyRueRGvGLR3mr7L";
628 +         report
629 +             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
630 +             .await;
631 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:646:5
    |
646 | /     for did in &["did:webkey:ssh:demo.spruceid.com:2021:07:14:keys"] {
647 | |         report
648 | |             .resolve(&resolver, did, &ResolutionInputMetadata::default())
649 | |             .await;
650 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
646 ~     {
647 +         let did = &"did:webkey:ssh:demo.spruceid.com:2021:07:14:keys";
648 +         report
649 +             .resolve(&resolver, did, &ResolutionInputMetadata::default())
650 +             .await;
651 +     }
    |

warning: for loop over a single element
   --> did-test/src/main.rs:652:5
    |
652 | /     for did in &["did:webkey:ssh:demo.spruceid.com:2021:07:14:keys"] {
653 | |         report
654 | |             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
655 | |             .await;
656 | |     }
    | |_____^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_element_loop
help: try
    |
652 ~     {
653 +         let did = &"did:webkey:ssh:demo.spruceid.com:2021:07:14:keys";
654 +         report
655 +             .resolve_representation(&resolver, did, &ResolutionInputMetadata::default())
656 +             .await;
657 +     }
    |

warning: `ssi-did-test` (bin "ssi-did-test") generated 12 warnings
warning: this expression creates a reference which is immediately dereferenced by the compiler
    --> did-ion/src/sidetree.rs:1412:53
     |
1412 |             .map(|url| HTTPSidetreeDIDResolver::new(&url));
     |                                                     ^^^^ help: change this to: `url`
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

error: this loop never actually loops
    --> did-ion/src/sidetree.rs:1596:9
     |
1596 | /         for opt in options.keys() {
1597 | |             return Err(DIDMethodError::OptionNotSupported {
1598 | |                 operation: "create",
1599 | |                 option: opt.clone(),
1600 | |             });
1601 | |         }
     | |_________^
     |
     = note: `#[deny(clippy::never_loop)]` on by default
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#never_loop
help: if you need the first element of the iterator, try writing
     |
1596 |         if let Some(opt) = options.keys().next() {
     |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

error: this loop never actually loops
    --> did-ion/src/sidetree.rs:1676:9
     |
1676 | /         for opt in options.keys() {
1677 | |             return Err(DIDMethodError::OptionNotSupported {
1678 | |                 operation: "update",
1679 | |                 option: opt.clone(),
1680 | |             });
1681 | |         }
     | |_________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#never_loop
help: if you need the first element of the iterator, try writing
     |
1676 |         if let Some(opt) = options.keys().next() {
     |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

error: this loop never actually loops
    --> did-ion/src/sidetree.rs:1708:9
     |
1708 | /         for opt in options.keys() {
1709 | |             return Err(DIDMethodError::OptionNotSupported {
1710 | |                 operation: "recover",
1711 | |                 option: opt.clone(),
1712 | |             });
1713 | |         }
     | |_________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#never_loop
help: if you need the first element of the iterator, try writing
     |
1708 |         if let Some(opt) = options.keys().next() {
     |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

error: this loop never actually loops
    --> did-ion/src/sidetree.rs:1739:9
     |
1739 | /         for opt in options.keys() {
1740 | |             return Err(DIDMethodError::OptionNotSupported {
1741 | |                 operation: "deactivate",
1742 | |                 option: opt.clone(),
1743 | |             });
1744 | |         }
     | |_________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#never_loop
help: if you need the first element of the iterator, try writing
     |
1739 |         if let Some(opt) = options.keys().next() {
     |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

warning: this expression creates a reference which is immediately dereferenced by the compiler
  --> did-ion/src/lib.rs:20:26
   |
20 |         if !is_secp256k1(&key) {
   |                          ^^^^ help: change this to: `key`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

warning: `did-ion` (lib) generated 17 warnings
error: could not compile `did-ion` due to 4 previous errors; 17 warnings emitted

I believe this is closed by #467


Merge Custom Publisher Statement #3097 to main

After #3094, #3146 and #3291 are done, we can then merge #3097 to main

PR with the feature branch for 3097 and company: https://github.com/sul-dlss/happy-heron/pull/3271

the feature branch is merged.  will pull now-unblocked backlog tickets to the ready column on the zenhub board...


feat: implement getter and setter for CUTBUFFER (For example, interaction with the system clipboard)

I'm considering wrapping the CUTBUFFER into getter and setter functions. This approach might minimize the extent of changes required in the existing code while also allowing other users to override these two functions to fulfill their own needs, such as interacting with the system clipboard.
For example:
# CUTBUFFER getter
function zvm_get_cutbuffer() {
  pbpaste
}

# CUTBUFFER setter
function zvm_set_cutbuffer() {
  echo -n $1 | pbcopy
}

Close #216

@jeffreytse I hope to receive your advice and response.


Random color picking function

I feel, if chalk can expose a random function which will pickup any random color and print it that will be really useful.
Any thoughts?

Useful for what? The bar for including new things in Chalk is very high. You need to be convincing.

I have one use case which I need it for.
I have some logs which is not specific to any color, I just want it to be colored.
In this case, I can just use chalk.random and it will pickup any random color configured in chalk.
Right now this is the only use case I have.

Why not write this function yourself? In the decade I have been writing colored console applications Ive never needed this once.

Looks like its more my usecase then a general one.
I guess writing my own function would be a better option

You could use https://github.com/sindresorhus/randoma#randomcolorsaturation and pass it to chalk.hex.


⚠ Bofrost.lu has degraded performance

In 098f860, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1718 ms

Resolved: Bofrost.lu performance has improved in d4211f7 after 6 minutes.


Add safety checks when handling Android input

It is possible that input comes before the engine is fully initialized.
This fixes the crashes that ocurred when that happens.

Cherry picked into 3.0.3


added the ability to use custom colors by extending the default color…

Twix currently only recognizes the default colors, so if I use a custom color that I've defined in the tailwind.config.js file, it will not be merged.  This code change allows a developer to extend the colors that Twix uses by adding a list in their config.exs file: `config :twix, colors: ["primary"]
Example of current behavior:
tw "text-primary-500 text-green-200"
"text-primary-500 text-green-200"
Example of new behavior:
tw "text-primary-500 text-green-200"
"text-green-200"

Any update on the status of this? Evaluating using Twix in our app, we do have customer colours so this seems like it would be necessary.


Allow annotation to root nodes

Curators should be able to create annotations to the root nodes of each aspect of the ontology.
molecular_function (GO:0003674)
biological_process (GO:0008150)
cellular_component (GO:0005575)

For root node, select the more option button (ellipsis button "...") then choose from the drop down menu

was fixed https://github.com/geneontology/simple-annoton-editor/commit/8ac08dfa33aedc0fd755ac6c99cc21a66a920e19 and https://github.com/geneontology/simple-annoton-editor/commit/0e39aca41ac125a901b12b87dd7d2e1f7693f5e9


Hyphen at line break removed

In the first pubmed manuscript, a number of times 'α2-integrin' is at a line break, e.g.:
"was mediated through the inhibition of expression of α2-
integrin (1,2). Integrins are receptors that mediate attachment"
In the output it becomes:
"was mediated through the inhibition of expression of α 2integrin..."
(The space is another issue https://github.com/kermitt2/grobid/issues/179)
In some cases it may be desirable to remove the hyphen. Not in this case. Probably never when there is a number?

Actually there is another not so clear word hyphenation example:
"Mean PK parameters CL, V, and F were calculated by non-
compartmental analysis. The tumor growth experiments in"
Becomes:
"Mean PK parameters CL, V, and F were calculated by noncompartmental analysis...."
In the nxml file it is annotated as 'non-compartmental'. I believe both versions are valid. So I would probably not treat that as a bug. (But I thought it was worth mentioning anyhow)

Thanks for the issue!
Dehyphenization is tricky ;). I was aware of these issues, but it's very useful to have a dedicated issue for that and discuss.
So far what is implemented now is very simplistic - it does not require dictionaries and resources, but it produces the errors you are mentioning. Basically, when there is an hyphen at the end of the line, we always dehyphenize.
We could introduce additional rules to improve it, like looking if we have numbers in the tokens before concatenating it, and/or to have a language-specific list of prefix (like anti-, non-, post-). We might always have some rare errors/exceptions. Adding rules manually is endless and not really in the spirit of GROBID...
Another approach could be to use a Machine Learning based text cleaner/normalizer, for instance like seq2seq, but then the problem is to have enough training data. The advantage would be maybe to tackle at the same time other text cleaning problems like diacritics combinations, invalid spacing, etc.

Intuitively I was going to claim that it isn't common that words are broken across lines. Scanning through the same PDF the evidence shows it is quite common actually. One approach could also be to look at other examples within the same document. I checked a few examples (around 7), the only one I couldn't find another time so far was 'NON-MEM'. If it was for me, then I would like to have the option to include an element for the hyphens at line boundaries. Then I could do some post processing.
Would you be happy to use the PMC dataset as training data?
Otherwise that seems to be training data that should be fairly easily extracted from any PDF with XML / text data (at least in English). Maybe even without PDF.

On 05/04/17 13:16, Daniel Ecer wrote:

Intuitively I was going to claim that it isn't common that words are broken across lines. Scanning through the same PDF the evidence shows it is quite common actually.

That's one of the multiple differences between MSWord and LaTeX: by default, Word does not "hyphenize" words (and did not propose an option to do so before v2010), whereas LaTeX always had an option to do so (at least manually - to my knowledge, dating back to the 90s). Since most of the scientific papers are written with LaTeX, you'll indeed encounter many word hyphenation :)
--
Guillaume MULLER, PhD
Presans c/o REMIX COWORKING - L'APPART
57 rue de Turbigo
75003 Paris
France
http://www.presans.com
http://feeds.feedburner.com/OYI/fr

As a complement, here is a blog post from one founder of Authorea about the ratio of scientific papers written with LaTeX (~18% of all articles, but, as expected, largely dominating in a couple of domains).
https://www.authorea.com/users/3/articles/107393-how-many-scholarly-articles-are-written-in-latex/_show_article

We are also running into this issue. Example PDF: http://ecp.acponline.org/sepoct01/kent.pdf
There are a lot of hyphens in the text at the end of sentences, breaking words:

E.g. "modeling techniques" becomes "modeling tech- niques".

@borkdude thank you for reporting these errors - I think the best would be to have a more robust dehyphenization process (it works not that bad normally...).
The problem with outputing the End Of Line in the final TEI result are:

sometimes EOL are pure garbage in some PDF (like one word per line), and often there are much more EOL in the actual PDF stream than what we see, so it might not be so useful for post-processing (GROBID on the other hand has all the coordinates information in each token to improve a dehyphenation)

the principle of the TEI is to give the logical structure of the document, abstracting from any presentation information. So this would require to write some sort of debug alternative TEI output, and everybody would want different information out for post-processing and it would be a pain to develop and maintain I think. So the best imho is to try to get the dehyphanization as good as possibly in GROBID, because dehyphenation is really part of its  job.

@lfoppiano hello Luca, would you have some time to look at these dehyphenization errors? my bad excuse: you're the last one who has modified it :D :D ?

FYI I'm checking on the pdf kent.pdf:

the text coming from the abstract (BiblioItem.getAbstract()) contains missing break lines which make the dehypenisation failing (according to the naive assumption that a hypen + breakline (whatever form) = dehypenisation). See the raw text before the call to dehypenise():

CONTEXT. A meta-analysis found that primary percutaneous transluminal coronary angioplasty (PTCA) was more effective than thrombolytic therapy in reducing mor-tality from acute myocardial infarction. However, fewer than 20% of U.S. hospitals have facilities to perform PTCA and many clinicians must choose between immedi-ate thrombolytic therapy and delayed PTCA. COUNT. The number of minutes of PTCA-related delay that would nullify its bene-fits. CALCULATION. For 10 published randomized trials, we calculated the following: PTCA-related delay = median "door-to-balloon" time -median "door-to-needle" time Survival benefit = 30-day mortality after thrombolytic therapy -30-day mortality after PTCA The relationship between delay and benefit was assessed with linear regression. RESULTS. The reported PTCA-related delay ranged from 7 to 59 minutes, while the absolute survival benefit ranged from -2.2% (favoring thrombolytic therapy) to 7.4% (favoring PTCA). Across trials, the survival benefit decreased as the PTCA-related delay increased: For each additional 10-minute delay, the benefit was predicted to decrease 1.7% (P< 0.001). Linear regression showed that at a PTCA-related delay of 50 minutes, PTCA and thrombolytic therapy yielded equivalent reductions in mor-tality . CONCLUSIONS. In clinical trials with short PTCA-related delays, PTCA produced better outcomes, while trials with longer delays favored thrombolytic therapy. A more precise estimate of the time interval to equipoise between the two therapies needs to be modeled with patient-level data. At experienced cardiac centers, PTCA is probably still preferable, even with delays longer than 50 minutes.

In the fulltext the dehypenisation is not applied (it was removed, perhaps because it wasn't working so well?).

What I would do is:

apply dephypenisation using LayoutToken and not text (method taking text can always tokenized on the fly)
review (and migrate using the Clusteror) how the abstract is extracted (perhaps this should be another task?) because at first glance, it looks like some breakline are lost

yes that's why we apply in this case another dehyphenization method (dehyphenizeHard()) which is not excepting a break line. It explains why mor- tality is correctly dehyphenized as mortality in the abstratc of the pdf example.
The problem with header model is that it is very old and is not working with LayoutToken. The best would be to update this complete model and put it in line with the other models, but that's quite a lot of work.

Ah it's my mistake it seems, but I don't remember exactly why I removed it. Probably I wanted to have it explicitly called where it is relevant (even in the full text, in some field, like formula, we don't want to dehyphenized, but still normalize the text), via TextUtilities.dehyphenize() in the appropriate fields in TEIFormatter.java.

About the dephypenisation, the current method using LayoutToken is not working well. Dephypenisation using text is much better for the moment because more flexible with the spaces around, it's why this one was used. The method using LayoutToken should be reviewed/extended I think.
Be careful that dehyphenize must be called only in certain fields where we are sure to have only text, performing it at clusteror level does not seem the right moment, because we still not know what is exactly the type of the current labelled segment.

yes that's why we apply in this case another dehyphenization method (dehyphenizeHard()) which is not excepting a break line. It explains why mor- tality is correctly dehyphenized as mortality in the abstratc of the pdf example.
The problem with header model is that it is very old and is not working with LayoutToken. The best would be to update this complete model and put it in line with the other models, but that's quite a lot of work.

the issue with mor- tality is before the dehypenisation since the \n is lost. First reason of why my suggestion to work directly on the layout tokens.

Ah it's my mistake it seems, but I don't remember exactly why I removed it. Probably I wanted to have it explicitly called where it is relevant (even in the full text, in some field, like formula, we don't want to dehyphenized, but still normalize the text), via TextUtilities.dehyphenize() in the appropriate fields in TEIFormatter.java.

Yes, indeed it has to be applied only to text

About the dephypenisation, the current method using LayoutToken is not working well. Dephypenisation using text is much better for the moment because more flexible with the spaces around, it's why this one was used. The method using LayoutToken should be reviewed/extended I think.

The current dehypenisation method using layout tokens is not complete. I would aim to merge the three methods and produce a single one using layout tokens and having the possibility to have more aggressive approach.

Be careful that dehyphenize must be called only in certain fields where we are sure to have only text, performing it at clusteror level does not seem the right moment, because we still not know what is exactly the type of the current labelled segment.

The idea was to use the clusteror to extract, and apply the dehypenisation after the text is recomposed, not at the same moment.

OK I see, you were talking about the abstract for the clusteror. As I said the old-fashioned Header model is not using LayoutToken for decoding the CRF results, it follows a different logic where the EOL are (voluntarily) not preserved - they were actually used to represent two discontinuous segments for the same field, for instance for keyword or author fields... so the different dehyphenization method (which works fine in the kent.pdf example).
It would be necessary to rewrite entirely the method HeaderParser.resultExtraction() (with clusteror for decoding CRF results) and pay attention to some other stuff in BiblioItem (there is a special hack to propagate LayoutToken for authors, in order to make bounding boxes for authors present in the TEI - we would need to find a way to generalize that, in order to keep the layout tokens for any fields for creating corresponding bounding boxes).
For me it was a different task, issue #136, to have every aspects updated at the same time - which is why I mentioned that it is quite a lot of work (and also why it is still an open issue since one and half year ;) ). Then the textual fields extracted from the header would be aligned with all the other models, and ready to use the common dehyphenization method.

OK so I will focus on the dehypenise() method using LayoutTokens and we could have a version getting text and tokenizing it under the hood. I will see whether to merge also with the aggressive version or not.

I've implemented something to fix the dehypenisation. I'm sure it will require a couple of iterations.
Could someone test it, focusing only on the body (not the abstract)?

I've ran the pubmed end 2 end evaluation.
======= Header metadata =======

Evaluation on 1942 random PDF files out of 1943 PDF (ratio 1.0).

======= Strict Matching ======= (exact matches)

===== Field-level results =====

label                accuracy     precision    recall       f1

abstract             82.02        15.67        14.45        15.04
authors              93.12        69.59        67.61        68.58
first_author         97.99        93.65        90.56        92.08
keywords             93.28        69.29        56.27        62.1
title                93.34        71.44        68.68        70.03

all fields           91.95        64.1         59.86        61.91   (micro average)
                     91.95        63.93        59.51        61.57   (macro average)

======== Soft Matching ======== (ignoring punctuation, case and space characters mismatches)

===== Field-level results =====

label                accuracy     precision    recall       f1

abstract             88.33        48.38        44.61        46.42
authors              93.22        70.06        68.08        69.06
first_author         98.07        94.03        90.92        92.45
keywords             94.08        75.8         61.57        67.95
title                94.66        77.92        74.91        76.39

all fields           93.67        73.34        68.49        70.83   (micro average)
                     93.67        73.24        68.02        70.45   (macro average)

==== Levenshtein Matching ===== (Minimum Levenshtein distance at 0.8)

===== Field-level results =====

label                accuracy     precision    recall       f1

abstract             94.67        81.26        74.92        77.96
authors              95.84        82.75        80.4         81.56
first_author         98.08        94.08        90.97        92.5
keywords             95.7         89.02        72.3         79.79
title                96.11        84.99        81.71        83.32

all fields           96.08        86.26        80.56        83.31   (micro average)
                     96.08        86.42        80.06        83.03   (macro average)

= Ratcliff/Obershelp Matching = (Minimum Ratcliff/Obershelp similarity at 0.95)

===== Field-level results =====

label                accuracy     precision    recall       f1

abstract             93.63        75.87        69.95        72.79
authors              94.05        74.1         72           73.03
first_author         97.99        93.65        90.56        92.08
keywords             95.15        84.46        68.6         75.71
title                95.3         81.03        77.9         79.43

all fields           95.22        81.66        76.26        78.87   (micro average)
                     95.22        81.82        75.8         78.61   (macro average)

===== Instance-level results =====

Total expected instances:       1941
Total correct instances:        146 (strict)
Total correct instances:        437 (soft)
Total correct instances:        874 (Levenshtein)
Total correct instances:        710 (ObservedRatcliffObershelp)

Instance-level recall:  7.52    (strict)
Instance-level recall:  22.51   (soft)
Instance-level recall:  45.03   (Levenshtein)
Instance-level recall:  36.58   (RatcliffObershelp)

======= Citation metadata =======

Evaluation on 1942 random PDF files out of 1943 PDF (ratio 1.0).

======= Strict Matching ======= (exact matches)

===== Field-level results =====

label                accuracy     precision    recall       f1

authors              97.24        81.29        70.09        75.27
date                 98.82        92.2         77.57        84.25
first_author         98.29        88.91        76.55        82.27
inTitle              95.86        71.08        66.85        68.9
issue                99.37        83.63        77.86        80.64
page                 98.47        92.11        78.65        84.85
title                96.79        77.43        69.43        73.21
volume               99.05        94.81        82.16        88.03

all fields           97.98        85.15        74.56        79.51   (micro average)
                     97.98        85.18        74.89        79.68   (macro average)

======== Soft Matching ======== (ignoring punctuation, case and space characters mismatches)

===== Field-level results =====

label                accuracy     precision    recall       f1

authors              97.32        81.85        70.58        75.8
date                 98.82        92.2         77.57        84.25
first_author         98.3         89.03        76.65        82.38
inTitle              97.37        81.62        76.76        79.12
issue                99.37        83.63        77.86        80.64
page                 98.47        92.11        78.65        84.85
title                98.31        88.56        79.41        83.74
volume               99.05        94.81        82.16        88.03

all fields           98.37        88.33        77.34        82.47   (micro average)
                     98.37        87.98        77.45        82.35   (macro average)

==== Levenshtein Matching ===== (Minimum Levenshtein distance at 0.8)

===== Field-level results =====

label                accuracy     precision    recall       f1

authors              98.09        87.39        75.35        80.93
date                 98.82        92.2         77.57        84.25
first_author         98.33        89.21        76.8         82.54
inTitle              97.52        82.68        77.76        80.15
issue                99.37        83.63        77.86        80.64
page                 98.47        92.11        78.65        84.85
title                98.66        91.16        81.73        86.19
volume               99.05        94.81        82.16        88.03

all fields           98.54        89.65        78.5         83.7    (micro average)
                     98.54        89.15        78.49        83.45   (macro average)

= Ratcliff/Obershelp Matching = (Minimum Ratcliff/Obershelp similarity at 0.95)

===== Field-level results =====

label                accuracy     precision    recall       f1

authors              97.65        84.27        72.66        78.04
date                 98.82        92.2         77.57        84.25
first_author         98.29        88.93        76.56        82.28
inTitle              97.17        80.23        75.46        77.77
issue                99.37        83.63        77.86        80.64
page                 98.47        92.11        78.65        84.85
title                98.54        90.26        80.93        85.34
volume               99.05        94.81        82.16        88.03

all fields           98.42        88.69        77.66        82.81   (micro average)
                     98.42        88.3         77.73        82.65   (macro average)

===== Instance-level results =====

Total expected instances:               89789
Total extracted instances:              86507
Total correct instances:                35610 (strict)
Total correct instances:                46360 (soft)
Total correct instances:                50433 (Levenshtein)
Total correct instances:                47359 (RatcliffObershelp)

Instance-level precision:       41.16 (strict)
Instance-level precision:       53.59 (soft)
Instance-level precision:       58.3 (Levenshtein)
Instance-level precision:       54.75 (RatcliffObershelp)

Instance-level recall:  39.66   (strict)
Instance-level recall:  51.63   (soft)
Instance-level recall:  56.17   (Levenshtein)
Instance-level recall:  52.74   (RatcliffObershelp)

Instance-level f-score: 40.4 (strict)
Instance-level f-score: 52.59 (soft)
Instance-level f-score: 57.21 (Levenshtein)
Instance-level f-score: 53.73 (RatcliffObershelp)

Matching 1 :    62566

Matching 2 :    3384

Matching 3 :    2786

Matching 4 :    665

Total matches : 69401

======= Fulltext structures =======

Evaluation on 1942 random PDF files out of 1943 PDF (ratio 1.0).

======= Strict Matching ======= (exact matches)

===== Field-level results =====

label                accuracy     precision    recall       f1

figure_title         96.55        28.3         23.14        25.46
reference_citation   57.14        55.99        52.72        54.31
reference_figure     94.58        61.04        61.1         61.07
reference_table      99.08        82.87        82.21        82.54
section_title        94.47        74.91        66.88        70.67
table_title          97.44        7.91         8.22         8.06

all fields           89.88        58.19        54.69        56.38   (micro average)
                     89.88        51.84        49.05        50.35   (macro average)

======== Soft Matching ======== (ignoring punctuation, case and space characters mismatches)

===== Field-level results =====

label                accuracy     precision    recall       f1

figure_title         98.4         74.09        60.57        66.65
reference_citation   59.53        60.14        56.63        58.33
reference_figure     94.53        62.06        62.12        62.09
reference_table      99.08        83.39        82.73        83.06
section_title        95.08        79.14        70.65        74.65
table_title          97.57        15.51        16.13        15.81

all fields           90.7         63.24        59.44        61.28   (micro average)
                     90.7         62.39        58.14        60.1    (macro average)

====================================================================================

@kermitt2 do you see any differences (hopefully a little improvement) with the previuos e2e measures?

There are differences, in particular I see a loss in citation metadatas and improvement on abstract. However the only way to be sure is to run it on the same architecture with and without the fixes (in case you have a branch). It depends also if you use consolidation or not.

I've checked based on the first comment and with 0.5.6 the hypens are safe :-)


Suggestion: Subfolder organization menu

The addition of an options menu that allows the user to sort downloaded files into new folders automatically. For example it could sort all the files by %rating% into separate folders after downloading:

downloads/rating_safe/
downloads/rating_questionable/
downloads/rating_explicit/

Or you could do more than one sort e.g. "%rating%%ext%" would mean:

downloads/rating_safe/png/
downloads/rating_safe/jpg/
downloads/rating_safe/gif/

This would be a great feature as you have to use advanced software or do this manually at the moment.

You can already do this, just set your filename to %rating%/%ext%/.... Each / will generate one additional level of sub-folders. And if you want to rename existing files, there's a tool in the "Tools" menu to do so (but it's slow).

Oh I see, sorry about that. The description in my filename options menu is missing so I have never been able to click the link at the bottom and have just been reading the tokens off the wiki. I think the reason it's bugged for me might be because of a custom windows theme i'm using so I never issued a bug report on it. The question mark button also does nothing for me.
Here is a picture if you are curious:

Thanks!


Initial commit.

Are these meant to be generic messages, or Jackal messages?

Generic messages.

Should we use these? https://github.com/clearpathrobotics/cpr_common_core/tree/foxy-devel/cpr_common_core_msgs


ARROW-420: Align DATE type with Java implementation

Things to do:

[ ] Update parquet-cpp
[ ] Add conversion from datetime.date to DateArray
[ ] Add python unit test

Why would we need more than int32_t for date?

See the ticket https://issues.apache.org/jira/browse/ARROW-413
The way DATE is implemented in Java is: "Here's a millisecond timestamp but please only look at the days".

Ready for review. As parquet-cpp builts without changes, I'd first like to get this here merged and would then add in another iteration support for DATE in parquet-cpp.


i18n example

I added i18next which is a library for handling internationalization.
Each language has its own json file in the locales folder. For example english is in locales/en/translation.json
i18n looks for a file of this name in a directory matching the name of the language. You can even define something more specific, for example en-US for US english. If that doesn't exist, it will fall back to 'en'. The naming convention follows ISO standard naming for languages.
In order to load this file, it uses XHR and requires cross domain requests, so when you are editing on local, you will need to host on localhost in order for it to work. I used an app called CodeKit to do this locally since I didn't see anything for setting up a dev environment inside this git.
For the purposes of this example I just added the needed libraries from cdn links, and only on the app.html page. Really this should be defined in a way so that it is shared across all the pages to reduce duplication.
I added some translations for the navigation and header title so you could see how this system works.
I made a new file for i18n that will talk to i18n directly. To activate it via jquery for a specific page, I call this:
<script> 		i18nManager(function() { 			$('body').localize(); 		}) 	</script>
To add a string to an element, add data-i18n="key" to the html element, where key is the key path in the translations.json.
You can choose where on the page to apply localization too. We want the whole page, so I set it to body. This passes this function to the i18n library and runs those changes whenever the language changes. You could also run this code inside the i18n js file, and it would apply to the body tag of any page that includes this js file.
i18next also detects the language from the browser on load, and uses that by default if translations exist.
For more info on i18next: https://www.i18next.com/
I am also using a jquery plugin for i18next which lets you have the attributes on html tags.

We are thinking to use vue.js since one of developers thought this method requires hardcoding for every language, is it true?

I think using Vue.js is a great idea. This app is in desperate need of a front-end framework. But even if you're using vue.js, you still will need an i18n plugin. But the you won't need to apply the js to each page, and you can reduce duplication. Highly recommend using a front-end framework. i18next supports vue js https://github.com/panter/vue-i18next

can this be run on github static page? or do we need a server to run locally?
Amil says requirement right now is that it can run on github pages.

My understanding is that this should still work with github pages, since it supports CORS. You will might need to tweak the load path. In your local environment you will need a server for development, as opposed to just opening the HTML file.

I see ! Hmm it does complicate our local workflow a bit, but I guess if it's only option, it's fine. We'll probably replace to vue js within a week or two, seems like.

it is very easy to create a local server of a folder. If you can't include it in your workflow, that is a larger issue. When you decide to use vue.js you will without a doubt need a more proper setup that includes a local development environment. I think it is important to invest in making these changes sooner rather than later.

yeah! if I merge this now, will it just work on the website?
for the time being this can be a good change.
and then later we can do the more proper refactoring

It should work, but I haven't tested it on github pages so would have to try it and make sure that the loadPath works. If you want to try it out first, we can put these changes in an alternate html file and push that to see how it works.

Okay I'm asking Tracy to try, for me though I'll probably call it a night
soon
2020年7月15日(水) 午後9:55 Canyon Boak<EMAIL_ADDRESS>
It should work, but I haven't tested it on github pages so would have to
try it and make sure that the loadPath works. If you want to try it out
first, we can put these changes in an alternate html file and push that to
see how it works.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/virufy/virufy.github.io/pull/83#issuecomment-659157950,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALVU7S2ZUXDS5GZ6LACQYTR32B3TANCNFSM4O2G4DKQ
.

Sounds good. I did some googling previously and had found other examples of people making cross-domain requests on github pages, which is why I think it should be possible. Like I said though, you should test it, because I haven't tried this on github pages so I can't say for sure without trying. Its possible that the sub-domain will result in it needing a different loadPath.

merged as https://github.com/virufy/virufy.github.io/blob/master/app-translation.html !!

https://virufy.org/app-translation