[Model] Add Phi-2 LoRA support

FILL IN THE PR DESCRIPTION HERE
FIX #4141

#4141

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

 PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

 After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
 After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
 After the review, the reviewer will put an  action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
 Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You
 Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

Test
from vllm import LLM
from vllm import SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM("/data/LLM-model/phi-2", enable_lora=True)

sql_lora_path = "/data/PEFT-LoRA/phi-2/phi-2-universal-NER"

prompts = [
    "<|im_start|>human\nText: Mit Patel here from India<|im_end|>\n<|im_start|>gpt\nI've read this text.<|im_end|>\n<|im_start|>human\nwhat is a name of the person in the text?<|im_end|>\n<|im_start|>gpt:\n",
]

sampling_params = SamplingParams(
    temperature=0.8, top_p=0.95, max_tokens=64, stop="<|im_end|>"
)

outputs = llm.generate(
    prompts, sampling_params, lora_request=LoRARequest("phi_adapter", 1, sql_lora_path)
)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Output
INFO 05-17 21:22:23 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/data/LLM-model/phi-2', speculative_config=None, tokenizer='/data/LLM-model/phi-2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/data/LLM-model/phi-2)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 05-17 21:22:23 cpu_executor.py:112] float16 is not supported on CPU, casting to bfloat16.
WARNING 05-17 21:22:23 cpu_executor.py:115] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 05-17 21:22:23 cpu_executor.py:142] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
WARNING 05-17 21:22:23 punica.py:14] punica LoRA kernels require a GPU to run. But you are using the CPU version vLLM
INFO 05-17 21:22:24 selector.py:52] Using Torch SDPA backend.
INFO 05-17 21:22:35 cpu_executor.py:71] # CPU blocks: 819
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [09:09<00:00, 549.94s/it, Generation Speed: 0.02 toks/s]
Prompt: "<|im_start|>human\nText: Mit Patel here from India<|im_end|>\n<|im_start|>gpt\nI've read this text.<|im_end|>\n<|im_start|>human\nwhat is a name of the person in the text?<|im_end|>\n<|im_start|>gpt:\n", Generated text: 'Mit Patel\n'

OK, but I haven't tested this on punica kernel yet, and only tested it on the cpu kernel proposed in another PR: #4830.  I will test it in GPU environment later.
It seems that Phi-2 has a different vocab_size compared to existing punica dimension in bgmv_config.h, I wonder how we can determine the punica dimension in bgmv_config.h.

cc @Yard1 for the question above (who's more familiar with punica kernel)

Well, this works on RTX3080 Ti with punica kernel so we don't need to modify the punica code anymore :)
INFO 05-18 01:19:13 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/data/LLM-model/phi-2', speculative_config=None, tokenizer='/data/LLM-model/phi-2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/data/LLM-model/phi-2)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 05-18 01:19:14 selector.py:37] Using FlashAttention-2 backend.
INFO 05-18 01:19:16 model_runner.py:145] Loading model weights took 5.1933 GB
INFO 05-18 01:19:18 gpu_executor.py:83] # GPU blocks: 948, # CPU blocks: 819
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.72it/s, Generation Speed: 27.19 toks/s]
Prompt: "<|im_start|>human\nText: Mit Patel here from India<|im_end|>\n<|im_start|>gpt\nI've read this text.<|im_end|>\n<|im_start|>human\nwhat is a name of the person in the text?<|im_end|>\n<|im_start|>gpt:\n", Generated text: 'Mit Patel\n'

I will add the remaining test later.

sounds great! It is awesome we just need to change supported modules specification

Emmm, it's strange that the CI worker OOM during running test_phi.py in lora-test-3:
=========================== short test summary info ============================
FAILED lora/test_phi.py::test_phi2_lora - RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
FAILED lora/test_punica.py::test_lora_a_extra_shapes[11806239938951-1-256-float16] - RuntimeError: CUDA generator expects graph capture to be underway, but the current stream is not capturing.
FAILED lora/test_punica.py::test_lora_a_extra_shapes[11806239938951-1-1024-float16] - RuntimeError: CUDA generator expects graph capture to be underway, but the current stream is not capturing.

Is there any way to solve this?

@Isotr0py try reducing max_loras in the test to 2

It seems that enforce_eager=True can solve the OOM problem, but should we use this in test CI?

@Isotr0py we can keep it but can you add a comment whenever you use enforce_eager in test to clarify why we set it there?

yeah technically enforce_eager=False case should be already tested

I have added a comment to mark that enforce_eager=True is used for test_phi2 to reduce memory usage for CI test.

FAILED lora/test_phi.py::test_phi2_lora - ValueError: Head size 80 is not supported by FlashAttention. Supported head sizes are: [32, 64, 96, 128, 160, 192, 224, 256].

<br class="Apple-interchange-newline">

looks like the test is failing with this error.

@rkooo567 I have created #4944 to fix this, and LoRA-test-3 passed in the fix PR.
However, the lora-long-context-distributed test failed in that PR, can you have a look at that?

The failure looks kind of unrelated... let me retry the test


sanity.nuxtjs.org is 404'ing

Any of the documentation pages are currently showing 404 Not Found error messages.

The site works if I'm viewing incognito but only in Safari.  404's in non-incognito Safari, Chrome (both incognito and not), Firefox (both modes).

Actually the trick is reloading the page after navigating. The 404'ing in the console when the content is fetched during the soft navigation.


Spanner log query and read elapsed times

fixes #1211
log elapsed times for spanner query and read.

Codecov Report

Merging #1251 into master will decrease coverage by 0.08%.
The diff coverage is 40.9%.

@@             Coverage Diff              @@
##             master    #1251      +/-   ##
============================================
- Coverage     67.45%   67.37%   -0.09%
  Complexity     1314     1314
============================================
  Files           186      186
  Lines          5387     5398      +11
  Branches        547      551       +4
============================================
+ Hits           3634     3637       +3
- Misses         1521     1527       +6
- Partials        232      234       +2

Flag
Coverage Δ
Complexity Δ

#integration
?
?

#unittests
67.37% <40.9%> (-0.09%)
1314 <4> (ø)

Impacted Files
Coverage Δ
Complexity Δ

...k/cloud/gcp/data/spanner/core/SpannerTemplate.java
67.94% <40.9%> (-2.26%)
58 <4> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05fb735...407fdba. Read the comment docs.

@dmitry-s PTAL


Add deathlog button to better statistics screen

Better Statistics Screen

While I appreciate that this is technically missing functionality, it would most likely make much more sense to be implemented on their side instead of DeathLog directly - you might want to consider opening an issue with them instead. Further, while the button is no doubt useful sometimes, it's not required as you can always open the screen using the provided keybind instead
Cheers

Why? Better Statistics Screen is unrelated to deathlog. Them implementing support for tens or hundreds of mods doesn't make sense. It should be deathlog that adds support IMHO

DeathLog also isn't related to Better Statistics Screen. Why should we have to add support for mods that literally entirely replace vanilla screens?

And what if another mod comes around? Are we also supposed to modify the Better Statistics Screen Plus and Overhauled Statistics Screen 2024 edition? Tom's Amazing Statisics Viewer?
If a mod modifies vanilla behavior, it also becomes that mod's responsiblity to ensure other mods don't break. Either way this is a pointless discussion as DeathLog is still perfectly usable through the keybind

Like any other keybind it should be present in the Controls screen


Is there a function to compare two hash values from one original password?

In my web application, i need a feature to check two hash values just from one original key.
now i can use  'bcrypt.compareSync('B4c0//', hash)' , but what i want is 'bcrypt.compareFromOneOrigin(hash1, hash2)', hash1&hash2 are from one origin with the same one number, maybe 10 or 12..like this-> 'var salt = bcrypt.genSaltSync(10);'
does it have a method like this? 3Q~

I don't understand what this is asking but I think you will have better luck on stack overflow or other support sites. This is an issue tracker better suited to bugs founds in the module, we don't usually review issues for support.


Want to get clarity on how to give the input for zero-shot NER.

I have few questions:-

Here are the list of prompts given in the paper for NER:-
a. Please list all entity words in the text that fit the category. Output format is "type1: word1; type2: word2".
b. Please find all the entity words associated with the category in the given text. Output format is "type1: word1; type2: word2".
c. Please tell me all the entity words in the text that belong to a given category. Output format is "type1: word1; type2: word2".
d. Please list all entity words in the text that fit the category. Output format is word1, word2.
e. Given options, please tell me the categories of all the listed entity words. Output format is "type1: word1; type2: word2".
So, are these prompts dataset specific? And if not, so does that mean that all of these prompts will give best results on all of the datasets? And if it is dataset specific, then can you please throw a light on this like which prompt to be used given the dataset specifics.

What is the exact format of input to the model for zero-shot NER. Let's say if I want to use prompt p1 = "Please list all entity words in the text that fit the category. Output format is "type1: word1; type2: word2"." and the sentence s = "My name is alexa and I live in washigton" and options  O = "location, person, organization, miscellaneous". So input would be """p1 + \n + 'Option: ' + O + \n + 'Text: '+s""". Right?

Hi,

Hi,

These prompts are not dataset specific, their performance gap is insignificant. We will add some ablation study experiments to show the model performance under different prompts settings.
Almost right! Plz add "\nAnswer:" at the end of your input.

Xiao Wang


Fix syntax

Fix syntax. I think "Outside of the Drupal root" should be "Outside of Drupal root. Could be?

Thanks :)

I think this is outdated now, as the block this was pertaining to no longer exists.


Add install help guide to contributing.md

Users are prompted to view this guide before raising an issue.
Putting it here as well as in the wiki may help more people to see the info needed to debug their install.

:confused: not sure why the build is failing here?

On Travis CI some tests are failing because of Connection reset by peer or couldn't connect to host while connecting to http://llvm.org/apt/llvm-snapshot.gpg.key
In other words, some problem with the Internet connectivity on Travis. It's not okay, but it happens and we should just ignore it.
On AppVeyor there was one job that couldn't even install Node.


CD: add linux musl builds

A statically linked version of bkt is useful as glibc (or new enough version of glibc) may not be available.

Thanks! CI looks good so this is fine by me.


Support multiple libraries and executables

The frontend only picks up one artefact instead of extracting all libraries and executables from a crate.

I can't reproduce, can you link to a workspace example where this is not working?

Looks like this fixed itself


Release artifacts for release v1.3.0

Updates the release version of IAM controller to v1.3.0

/ok-to-test


AKS Provisioner: Cloud Credentials

Similar to https://github.com/rancher/rancher/issues/31009, we need to look at using cloud credentials with the AKS Provisioner.

This one is just the integration to the cluster object, as the azure cloud credentials themselves already exist.

This one is just the integration to the cluster object, as the azure cloud credentials themselves already exist.

Need to add the TenantID still along with some other bits of code.

After discussion we agreed to not add tenantId to Cloud Credentials, because it might brake rancher machine for RKE.

Reopening, we are looking at adding again.

QA cases to test:

that it doesn't break existing setups without a tenant
fresh clusters with and without a tenant
not sure this is valid use case, but see if you can convert existing cloud-credential cluster to one with a tenant

We also need to test plain old azure node driver with and without tenant, and upgrading that from 2.5

QA cases to test:

that it doesn't break existing setups without a tenant

Does not, as AKSv1 don't use stored CC

fresh clusters with and without a tenant

This is not possible, as cc created without tenantID does not show up in list of CCs while provisioning AKSv2 cluster

not sure this is valid use case, but see if you can convert existing cloud-credential cluster to one with a tenant

No possible. V1 cluster does not use stored CC


obcpbft breaks under stress

Hi, I was doing some performance tests with obcpbft enabled. I have 4 peers, vp0-vp3, running on 4 z/VM zLinux guests. Security is disabled. I used jmeter to do the tests. The elements of the jmeter thread group are:

deploy (only once)
invoke
query

The deployed the chaincode is chaincode_example02 used in the github docs with the following REST call:
{
    "type": "GOLANG",
    "chaincodeID":{
        "path":"github.com/openblockchain/obc-peer/openchain/example/chaincode/chaincode_example02"
    },
    "ctorMsg": {
        "function":"init",
        "args":["a", "100", "b", "200"]
    }
}

The invoke REST call is:
{
  "chaincodeSpec":{
      "type": "GOLANG",
      "chaincodeID":{
      "name":"bb540edfc1ee2ac0f5e2ec6000677f4cd1c6728046d5e32dede7fea11a42f86a6943b76a8f9154f4792032551ed320871ff7b7076047e4184292e01e3421889c"
      },
      "ctorMsg":{
          "function":"invoke",
          "args":["a", "b", "10"]
      }
  }
}

And the query REST call is:
{
  "chaincodeSpec":{
      "type": "GOLANG",
      "chaincodeID":{
      "name":"bb540edfc1ee2ac0f5e2ec6000677f4cd1c6728046d5e32dede7fea11a42f86a6943b76a8f9154f4792032551ed320871ff7b7076047e4184292e01e3421889c"
      },
      "ctorMsg":{
          "function":"query",
          "args":["b"]
      }
  }
}

I set the loop to 5000 times. All the REST calls were sent to vp0. At some point, vp0 goes into some kind of error state and continuously print out the following in the log forever, even after the client has stopped sending requests:
15:19:38.228 [consensus/statetransfer] StateTransferState -> ERRO 1c58c name:"vp0"  could not retrieve state as recent as 0 from any of specified peers
15:19:38.314 [consensus/statetransfer] StateTransferState -> WARN 1c58d name:"vp0"  in tryOverPeers loop trying name:"vp1"  : name:"vp0"  received a corrupt delta from name:"vp1"  : Error unmarashaling size: unexpected EOF
15:19:38.374 [consensus/statetransfer] StateTransferState -> WARN 1c58e name:"vp0"  in tryOverPeers loop trying name:"vp2"  : name:"vp0"  received a corrupt delta from name:"vp2"  : Error unmarashaling size: unexpected EOF
15:19:38.453 [consensus/statetransfer] StateTransferState -> WARN 1c58f name:"vp0"  in tryOverPeers loop trying name:"vp3"  : name:"vp0"  received a corrupt delta from name:"vp3"  : Error unmarashaling size: unexpected EOF

On vp1-3, the log shows something like the following:
15:02:12.782 [consensus/obcpbft] pbftCore -> INFO 1c2f Replica 2 sending view-change, v:13, h:5950, |C|:1, |P|:1, |Q|:1
15:02:12.782 [consensus/obcpbft] pbftCore -> INFO 1c30 Replica 2 received view-change from replica 2, v:13, h:5950, |C|:1, |P|:1, |Q|:1
15:02:12.782 [consensus/obcpbft] pbftCore -> INFO 1c31 Replica 2 received view-change from replica 3, v:13, h:5950, |C|:1, |P|:1, |Q|:1
15:02:12.782 [consensus/obcpbft] pbftCore -> INFO 1c32 Replica 2 received view-change from replica 1, v:13, h:5950, |C|:1, |P|:1, |Q|:1
15:02:28.782 [consensus/obcpbft] pbftCore -> INFO 1c33 Replica 2 view change timer expired, sending view change
15:02:28.783 [consensus/obcpbft] pbftCore -> INFO 1c34 Replica 2 sending view-change, v:14, h:5950, |C|:1, |P|:1, |Q|:1
15:02:28.783 [consensus/obcpbft] pbftCore -> INFO 1c35 Replica 2 received view-change from replica 2, v:14, h:5950, |C|:1, |P|:1, |Q|:1
15:02:28.783 [consensus/obcpbft] pbftCore -> INFO 1c36 Replica 2 received view-change from replica 3, v:14, h:5950, |C|:1, |P|:1, |Q|:1
15:02:28.783 [consensus/obcpbft] pbftCore -> INFO 1c37 Replica 2 received view-change from replica 1, v:14, h:5950, |C|:1, |P|:1, |Q|:1

This looks to be the same as #680 ?
Which commit are you running ?

Commit tag d0dd6cc1393c224f3c87f31c52aff232a03027ea on Feb. 18.

@gongsu832 Yes, the bug you are seeing was fixed a few commits after the level you are running at.  Please see the discussion of issue #680 which @tuand27613 referenced and attempt with a commit beyond the merge request which fixed that issue.

OK I'll try to rebase my fork to include that fix. Thanks!

I rebased to commit tag 3c5fbcdf8a288b7d5ed5aa0dae5d570472169e73 and did some tests. It appears to have fixed the problem. But I'm seeing another strange behavior which I don't know whether it's expected or not. Basically, after deploying the chaincode, I send in 10K invoke requests. I can see that on the primary validator, it prints out a lot of
14:53:49.496 [rest] ServerOpenchainREST -> INFO 1913 REST invoking chaincode...
14:53:49.499 [consensus/obcpbft] obcClassic -> INFO 1914 New consensus request received
14:53:49.500 [rest] ServerOpenchainREST -> INFO 1915 Successfuly invoked chainCode with txuuid (f60dc52d-b05c-4a91-91d2-42862240f3f1)
14:53:49.500 [consensus/obcpbft] pbftCore -> INFO 1916 Replica 0 executing/committing request for view=0/seqNo=1580 and digest gP7349B9SpAWtVTKVEJUMo8H1zPiGw8KmJZiPY/r0+lts0JyLRAqubeP5s3IlFmSzd7GXjO1TEZ2qsYVaTGkSQ==

And on the replicating validators, they are printing out a lot of
14:54:31.683 [consensus/obcpbft] pbftCore -> INFO 278b Replica 2 executing/committing request for view=0/seqNo=9889 and digest XX5X2ymalfqAmxkX9Vpl6LTPUoojesB6Ja90WSvFzumvmaiZcw3F4GQBUD1Xg+ajbyDBoB3KsvBlGgM5g5uySA==
After all the requests are done, all the validators print out some messages about sending/receiving view-change. Now I send in a second batch of 10K invoke requests and this is when things look a bit strange. On the primary validator, it prints out a lot of
14:56:44.384 [rest] ServerOpenchainREST -> INFO 111d2 REST invoking chaincode...
14:56:44.387 [consensus/obcpbft] obcClassic -> INFO 111d3 New consensus request received
14:56:44.387 [rest] ServerOpenchainREST -> INFO 111d4 Successfuly invoked chainCode with txuuid (9d2231cd-46e3-4d74-8360-91c2501b840b)

Note that it's not printing out the "Replica 0 executing/committing request" messages anymore. And on all the replicating validators, besides a few receiving view-change messages, they are not printing out the "Replica X executing/committing request" messages either. It looks as if the consensus protocol is not being conducted, only the primary validator is doing the invoke request.

It looks like this might be related to #699, if it is convenient for you to post the full logs, I could take a closer look to try to confirm.

@jyellick here are the logs for the all the validators. I did 3 batches of 10K invoke requests. In the vp0 log, the first started at 14:53:41.545, the second started at 14:55:59.957, and the third started at 15:24:40.573. You can see that only the first batch prints out "Replica X executing/committing request ...".
obc-peer.log.zip

@jyellick , #803 sounds similar ?

Preliminary analysis

vp3 seems to be processing a bit slow and starts rejecting prepare messages starting from 2061, because its low-mark is still at 2040; last exec 2047.  Soon after, vp3 discovers a stable checkpoint for 2070, but never manages to catch up with the other nodes.  I don't know what it means by

[33m14:53:52.000 [peer] Handler -> WARN 925 Did NOT send SyncBlocks message to channel for range: 2046 - 2046

  Eventually, state transfer complains about
  ```
[33m14:54:31.771 [consensus/statetransfer] StateTransferState -> WARN 9ec4 name:"vp3"  in tryOverPeers loop trying name:"vp2"  : name:"vp3"  had block sync request to name:"vp2"  time out
14:54:31.772 [consensus/statetransfer] StateTransferState -> ERRO 9ec5 name:"vp3"  recovered to an incorrect state at block number 9503, (318fb098127ab82300ba6dd67e36bff967c993751181fc0285fc271fd7d3a6286cee1774f5ae10ff1f1c7d274b220d654d2db06e26ad79e89751adc41d39eac0 d738f0c1bc3c31b3a0b9937d4ed6bc87b1456dfe606b14320513f5f1012032da1456473542ada9eafbe820141e5408af95e04313a9b8c3616dfafa4ca850ada7) retrying

All the time while state transfer seems to be going on, replica 3 keeps sending out view-change messages, with increasing view (and increasing reference checkpoint).

For some reason, replicas 1 and 2 expire their view change timer 2s after having executed 9889.  Their view-change message refers to 9880.
Even though replica 1 receives 3 view change messages for view=1 (and another from replica 3 for view=20), replica 1 never seems to send out a new-view message.

Actionables

Improve logging of state transfer progress (on level INFO).
Improve logging of view change timer reason (which outstanding request triggered this)
Improve logging on received consensus requests.

Points to investigate

Do we correctly set ourselves as out of state, and does the view-change system honor this?
Where do these extra view changes come from?  Do we clean up once state transfer has been successful?
Do we need to extend our high watermark level (relative to checkpoint period), so that a fresh caught up replica can still receive plenty of additional requests?

@gongsu832 can you please always update to latest master?  I see that the commit you retried is 20 days old, and many bugfixes have landed since then.

Trying to repeat the tests after updating to commit tag ...
First thing to report is that I'm seeing the same error messages flooding vp1-vp3 reported in #819.
The good news is that the invoke test no longer breaks obcpbft. Each batch of 10000 invokes results in action on all validators. The only thing I noticed is that occasionally you see a warning message like the following:
12:10:04.218 [consensus/obcpbft] pbftCore -> WARN 70f07 Replica 0 found view-change message incorrect
It happens on all 4 validators.
Now the bad news is that query no longer works. It gets a "ledger: resource not found" error.
12:44:23.678 [crypto] main -> INFO 001 Log level recognized 'info', set to INFO
12:44:23.685 [main] serve -> INFO 003 Security enabled status: false
12:44:23.685 [eventhub_producer] eventProcessor -> INFO 002 event processor started
12:44:23.685 [main] serve -> INFO 004 Privacy enabled status: false
12:44:23.717 [buckettree] initConfig -> INFO 005 configs passed during initialization = map[string]interface {}{"numBuckets":10009, "maxGroupingAtEachLevel":10}
12:44:23.717 [buckettree] initConfig -> INFO 006 Initializing bucket tree state implemetation with configurations &{maxGroupingAtEachLevel:10 lowestLevel:5 levelToNumBucketsMap:map[0:1 1:2 2:11 3:101 4:1001 5:10009] hashFunc:0x807818e0}
12:44:23.720 [chaincode] NewChaincodeSupport -> INFO 007 Chaincode support using peerAddress: <IP_ADDRESS>:30303
12:44:23.721 [rest] StartOpenchainRESTServer -> INFO 008 Initializing the REST service...
12:44:23.724 [main] serve -> INFO 009 Starting peer with id=name:"vp0" , network id=dev, address=<IP_ADDRESS>:30303, discovery.rootnode=, validator=true
12:44:23.724 [genesis] $nested0 -> INFO 00a Creating genesis block.
12:44:23.725 [genesis] $nested0 -> INFO 00b No genesis block chaincodes defined.
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 00c PBFT type = *obcpbft.obcClassic
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 00d PBFT Max number of validating peers (N) = 4
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 00e PBFT Max number of failing peers (f) = 1
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 00f PBFT byzantine flag = false
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 010 PBFT request timeout = 2s
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 011 PBFT view change timeout = 2s
12:44:31.315 [consensus/obcpbft] newPbftCore -> INFO 012 PBFT Checkpoint period (K) = 10
12:44:31.316 [consensus/obcpbft] newPbftCore -> INFO 013 PBFT Log multiplier = 2
12:44:31.316 [consensus/obcpbft] newPbftCore -> INFO 014 PBFT log size (L) = 20
12:44:54.850 [rest] ServerOpenchainREST -> INFO 015 REST deploying chaincode...
12:44:54.855 [container] writeGopathSrc -> INFO 016 rootDirectory = /opt/openchain/src
12:45:02.359 [consensus/obcpbft] obcClassic -> INFO 017 New consensus request received
12:45:05.077 [rest] ServerOpenchainREST -> INFO 018 Successfuly deployed chainCode: bb540edfc1ee2ac0f5e2ec6000677f4cd1c6728046d5e32dede7fea11a42f86a6943b76a8f9154f4792032551ed320871ff7b7076047e4184292e01e3421889c.
12:45:05.656 [consensus/obcpbft] pbftCore -> INFO 019 Replica 0 received view-change from replica 2, v:1, h:0, |C|:1, |P|:0, |Q|:1
12:45:05.932 [consensus/obcpbft] pbftCore -> INFO 01a Replica 0 received view-change from replica 3, v:1, h:0, |C|:1, |P|:0, |Q|:1
12:45:05.932 [consensus/obcpbft] pbftCore -> INFO 01b Replica 0 received f+1 view-change messages, triggering view-change to view 1
12:45:05.932 [consensus/obcpbft] pbftCore -> INFO 01c Replica 0 sending view-change, v:1, h:0, |C|:1, |P|:1, |Q|:1
12:45:05.932 [consensus/obcpbft] pbftCore -> INFO 01d Replica 0 received view-change from replica 0, v:1, h:0, |C|:1, |P|:1, |Q|:1
12:45:06.263 [consensus/obcpbft] pbftCore -> INFO 01e Replica 0 received view-change from replica 1, v:1, h:0, |C|:1, |P|:0, |Q|:1
12:45:07.932 [consensus/obcpbft] pbftCore -> INFO 01f Replica 0 view change timer expired, sending view change
12:45:07.933 [consensus/obcpbft] pbftCore -> INFO 020 Replica 0 sending view-change, v:2, h:0, |C|:1, |P|:1, |Q|:1
12:45:07.933 [consensus/obcpbft] pbftCore -> INFO 021 Replica 0 received view-change from replica 0, v:2, h:0, |C|:1, |P|:1, |Q|:1
12:46:13.881 [rest] ServerOpenchainREST -> INFO 022 REST querying chaincode...
12:46:13.885 [rest] ServerOpenchainREST -> ERRO 023 {"Error": "Querying Chaincode -- Error:Failed to launch chaincode spec(Could not get deployment transaction for bb540edfc1ee2ac0f5e2ec6000677f4cd1c6728046d5e32dede7fea11a42f86a6943b76a8f9154f4792032551ed320871ff7b7076047e4184292e01e3421889c - ledger: resource not found)"}

I used INFO instead of DEBUG log level. Because the periodic messages in DEBUG level make it hard to see things. If you need DEBUG level log, let me know.

Sorry meant to say commit tag 3ef73d319b01fe84c0c88973350a240fc490d82e above.

@gongsu832 ,  now that PR #833  is done, can you re-run ?

OK updated to latest commit tag f08ccce33b13f23eb1af97a43bf654aa48129794 and reran the test.
4 nodes, consensus=obcpbft, security=disabled.
After starting obc-peer on all 4 nodes, vp1-3 logs were being polluted by the message:
16:06:04.721 [peer] PeerImpl -> ERRO 1e9 Error handling message: Peer FSM failed while handling message (DISC_HELLO): current state: created, error: transition canceled with error: Error registering Handler: Duplicate Handler error: {name:"vp1"  <IP_ADDRESS>:30303 VALIDATOR }
vp0 didn't have this problem.
Deploy chaincode_example02 then send in 10K invoke requests (all to vp0). It appears that only vp0 is printing out messages indicating it's executing the request:
16:04:47.593 [rest] ServerOpenchainREST -> INFO 036ESC[0m REST invoking chaincode...
16:04:47.598 [consensus/obcpbft] obcClassic -> INFO 037ESC[0m New consensus request received
16:04:47.598 [rest] ServerOpenchainREST -> INFO 038ESC[0m Successfuly invoked chainCode with txuuid (6712b9ec-510d-428b-8fdf-7309b182ae7f)

None of vp1-3 printed out any messages indicating they are executing the request. All of them just periodically print out some view-change messages like the following:
16:05:35.640 [consensus/obcpbft] pbftCore -> INFO 19cESC[0m Replica 1 received view-change from replica 0, v:26, h:0, |C|:1, |P|:1, |Q|:1
16:05:35.651 [consensus/obcpbft] pbftCore -> INFO 19dESC[0m Replica 1 view change timer expired, sending view change
16:05:35.652 [consensus/obcpbft] pbftCore -> INFO 19eESC[0m Replica 1 sending view-change, v:26, h:0, |C|:1, |P|:0, |Q|:1
16:05:35.652 [consensus/obcpbft] pbftCore -> INFO 19fESC[0m Replica 1 received view-change from replica 1, v:26, h:0, |C|:1, |P|:0, |Q|:1
16:05:35.652 [consensus/obcpbft] pbftCore -> INFO 1a0ESC[0m Replica 1 received view-change from replica 1, v:26, h:0, |C|:1, |P|:0, |Q|:1
ESC[33m16:05:35.652 [consensus/obcpbft] pbftCore -> WARN 1a1ESC[0m Replica 1 found view-change message incorrect

Log files for all nodes attached.
obc-peer.log.zip

@gongsu832 ,  do your deploy/invoke/query  work and do you get correct values ?
vp0 is the only one sending the new consensus request received and successfully invoked ... because  it's the peer that's receiving the request. The others are receiving consensus messages which don't show up unless we have debug mode.

@tuand27613
OK, with log level set to debug, I see the replicas printing out messages for each invoke request like the following:
17:10:58.479 [consensus/obcpbft] pbftCore -> DEBU 31ee Replica 1 received request: Z0Hu3A1S8mUENmMLkCzk7d1RGo5rw/djLME/zEGuzYjfLsGZptC/YqbqkocknTkRtOaNW+mpOXw8d+8TAKFhUg==
Query still doesn't work. I'm getting the same "ledger: resource not found" message I reported above 14 days ago.
Since query doesn't work, I can't really tell whether invoke did the right thing or not but the invoke REST call is successful.

The Failed to launch chaincode spec(Could not get deployment transaction for ...... - ledger: resource not found) error says that we couldn't find the transaction based on its UUID ( @srderson, let me know if I'm all wet )
So @gongsu832 , could you append the debug logs again ?
Also, what does your query call look like ?

Here is the log with level=DEBUG. Note that the query request is close to the end after 10K invoke requests. Just search for "Failed".
vp0-obc-peer.log.zip
And here is the query:
{
  "chaincodeSpec":{
      "type": "GOLANG",
      "chaincodeID":{
          "name":"bb540edfc1ee2ac0f5e2ec6000677f4cd1c6728046d5e32dede7fea11a42f86a6943b76a8f9154f4792032551ed320871ff7b7076047e4184292e01e3421889c"
      },
      "ctorMsg":{
          "function":"query",
          "args":["b"]
      }
  }
}

@gongsu832 Thank you for the additional logs, it's now quite clear what the problem is.
Essentially, the many invokes are clogging message queue, causing PBFT to become very slow, which triggers a view change.  Before the view change can complete, the request timer triggers again and causes yet another view change.  Each time this occurs, the view change timer doubles.  By the time the request flood is through, the view change is set to expire in 2^32 seconds, or essentially never.
This leads to two actionable things from the consensus/PBFT side of things.  The request timer should not start when the current view is not active, this is a trivial one line change.  Secondly, the view change timeout should be capped to some more reasonable, and configurable value, say, 1 minute.  This should also be a fairly direct change.
The combination of these changes should ensure that once the flood of requests has abated, consensus progress will continue.  However, this is still a bit of a denial of service opportunity, and I'll likely need to talk with @jeffgarratt and others to see if there's a better way we can proceed to better tolerate floods of client activity.

@jyellick Thanks for looking into the problem and finding the potential cause. However, even without the 10K invoke requests, a query request after successful deploy also fails the same way, just like what I reported above 15 days ago. Here is the log with level=DEBUG (I tried query twice, both failed).
vp0-obc-peer.log.zip

@gongsu832 Yes, I see.  Looks like we are dealing with multiple bugs here, as the other is definitely a problem as well.  Could you collect DEBUG logs for all 4 validating peers? From your latest set of logs, I see missing replies to vp0 from the other vps which need to be investigated.

@jyellick Here they are.
obc-peer.log.zip

@jyellick Do you also need DEBUG logs for all 4 peers for the query after 10K invoke case? If so I'll redo the tests.

@jyellick could you explain the message queue clogging issue in more detail?

@gongsu832 Based on these new logs, it seems that there is network connectivity between vp0 and vp{1,2,3}, but no connectivity which does not include vp0.  This is unrelated to the consensus, but causes consensus to never be achieved (but in a very odd way which causes many view changes which makes it appear to be consensus related).  It seems unlikely to me that this is a firewall issue or some such, but could you make sure that there is network connectivity between some pair of vps, not including vp0? (So, for instance, can vp1 establish an ssh session to vp2?)
@jeffgarratt You can see the full logs above, but here is the most interesting bit to me:
[yellickj@jmobile logs]$  grep 'bi-directional' * | sed -e 's/address.*event/.../'
vp0-obc-peer.log:13:19:34.030 [peer] Handler -> DEBU 02f The Peer's bi-directional stream to ID:<name:"vp1" > ... DISC_HELLO
vp0-obc-peer.log:13:19:38.383 [peer] Handler -> DEBU 039 The Peer's bi-directional stream to ID:<name:"vp2" > ... DISC_HELLO
vp0-obc-peer.log:13:19:42.738 [peer] Handler -> DEBU 04d The Peer's bi-directional stream to ID:<name:"vp3" > ... DISC_HELLO
vp1-obc-peer.log:13:19:34.047 [peer] Handler -> DEBU 02f The Peer's bi-directional stream to ID:<name:"vp0" > ... DISC_HELLO
vp1-obc-peer.log:13:19:40.061 [peer] Handler -> DEBU 046 The Peer's bi-directional stream to ID:<name:"vp1" > ... DISC_HELLO
vp2-obc-peer.log:13:19:38.429 [peer] Handler -> DEBU 02f The Peer's bi-directional stream to ID:<name:"vp0" > ... DISC_HELLO
vp2-obc-peer.log:13:19:44.444 [peer] Handler -> DEBU 050 The Peer's bi-directional stream to ID:<name:"vp2" > ... DISC_HELLO
vp3-obc-peer.log:13:19:42.758 [peer] Handler -> DEBU 02f The Peer's bi-directional stream to ID:<name:"vp0" > ... DISC_HELLO
vp3-obc-peer.log:13:19:48.771 [peer] Handler -> DEBU 047 The Peer's bi-directional stream to ID:<name:"vp3" > ... DISC_HELLO
[yellickj@jmobile logs]$

Each peer is able to connect to vp0 and itself, but to no-one else.  I'm not entirely sure how to read the rest of the messages, but maybe you'll have some insight?
@corecode The queue clogging turns out to be more of a non-issue after some discussions with @jeffgarratt.  Some of the deadlock issues we'd had with Sieve and state transfer led me to believe the queues were less independent than they are.  Still, I think it is important that replicas do not initiate a view change from request timeout if their view is not currently active, this breaks the view-timeout doubling scheme, and can make the view change timer enormous.  If a byzantine replica could cause a denial of service for say, 60 seconds, successfully triggering 30 view changes via the 2 second request timer, then stop when the view made them the primary, that byzantine replica would have 2^30 seconds to complete the view change, which would be a problem.

@jyellick vp0-3 are 4 z/VM guests. Every guest has connectivity to every other guest. I just tried and I could ssh from every guest to every other guest.

@gongsu832 Thanks for checking, thought it seemed unlikely, but worth ruling out.  I think below may be the solution.
In your openchain.yml file, could you ensure that peer.address which is normally set to <IP_ADDRESS>:30303 is set to the IP of the machine (plus the :30303)? If you set this, you should also set peer.addressAutoDetect=false.
Alternatively, you could simply set peer.addressAutoDetect=true, and this might also work.
Finally, you can always set these values via the environment, with OPENCHAIN_PEER_ADDRESS and OPENCHAIN_PEER_ADDRESSAUTODETECT.

@jyellick I never touch openchain.yaml and always use env vars to override settings I want to change.
I set OPENCHAIN_PEER_ADDRESS to host_ip:30303 and query now works. Unfortunately, I'm not getting correct values for "a", "b" using chaincode_example02. After deploy, I send in alternate invoke and query, i.e., invoke, query, invoke, query, etc.
The chaincode is deployed with ["a", "100", "b", "200"]. Each Invoke is invoked with ["a", "b", "10"]. Query queries ["a"]. And here are the results after 25 invoke/query pairs:
1.  invoke: success
    query: 100
2.  invoke: success
    query: 80
3.  invoke: success
    query: 80
4.  invoke: success
    query: 70
5.  invoke: success
    query: 60
6.  invoke: success
    query: 50
7.  invoke: success
    query: 40
8.  invoke: success
    query: 30
9.  invoke: success
    query: 10
10. invoke: success
    query: 10
11. invoke: success
    query: 0
12. invoke: success
    query: -10
13. invoke: success
    query: -20
14. invoke: success
    query: -30
15. invoke: success
    query: -40
16. invoke: success
    query: -50
17. invoke: success
    query: -60
18. invoke: success
    query: -60
19. invoke: success
    query: -80
20. invoke: success
    query: -90
21. invoke: success
    query: -90
22. invoke: success
    query: -90
23. invoke: success
    query: -90
24. invoke: success
    query: -90
25. invoke: success
    query: -90

It looks like invoke is async? Since there are missing values like 90 and there are repeating values like 80. What's more troublesome though is that after "a" gets to -90, i.e., after being subtracted 19 times (100-19*10=-90), it stops decreasing (I tried 10K pairs of invoke/query and "a" also stops decreasing at -90). I look at the source code for chaincode_example02 and it doesn't appear to have any check to stop "a" from decreasing.
Here are the DEBUG logs for all 4 peers.
obc-peer.log.zip
Just for kicks, I deployed chaincode_example02 with ["a", "1000", "b", "2000"], and send in the same 1000 invoke/query pairs. "a" stops decreasing at 810, i.e., after being subtracted 19 times (1000-19*10=810).

@gongsu832 Yes, the chaincode invocations are asynchronous, the request is submitted to consensus to be ordered and executed, and if you are interested in the result, there are some other mechanisms you will need to use.  I think @muralisrini might be the right person to answer this question for you.
In regards to this magic 19 times and failure, this appears to because your blockchains are in an inconsistent state at startup.  What is happening, is that PBFT 'checkpoints' in order to garbage collect internal state, this means that enough replicas all agree on the state of the blockchain.  When the chains are not the same, no consensus can be achieved, and progress will stall once the log size (20 by default, here 1 deploy, and 19 invokes) is exceeded.  Presently, the PBFT implementation only supports starting from a clean state, ie no existing blockchain.  Looking at your logs, I see that your blockchains are not all of the same length, and may not contain the same content.  I suggest you delete the contents of your blockchains and try from a fresh genesis block, @srderson should be able to help you do this.

@jyellick I'm using jmeter to see the performance of invoke and query. If invoke is async, that makes interpreting the throughput number a bit difficult, i.e., if jmeter tells me the invoke throughput is 200/sec, then it simply means the system is able to accept the requests and send back OK 200/sec. But we have no idea how long it actually took to execute those invoke requests. The effective throughput could be much lower than 200/sec.
 I actually always start the nodes with clean states. But due to a bug in my script, I thought I was rm -rf /var/openchain/production/* but instead I was rm -rf /var/openchain/production/production/* DOH! Fixed my script and now the numbers appear to come out right. So that was good. Thanks.

@gongsu832 Glad at least the executions are working better for you now.
The system was definitely designed with transaction submission being purposefully asynchronous.  We are rapidly leaving my area of expertise (consensus), but I believe there is an eventing mechanism you can hook into.  @muralisrini will need to weigh in here though.

Also check with @corecode who's been through some of these measurement issues during his perf runs

@jyellick yeah thanks for helping identifying the problems. The need to set peer.address to IP address instead of <IP_ADDRESS> for query to work probably needs to be documented.
@tuand27613 thanks I can talk with @muralisrini and @corecode offline on the async throughput issue but it's not an urgent matter at this moment. The jmeter numbers at least give some ball-park estimation.
Next I'm going to turn on security+privacy and see what comes out. :-)

check out my perf branch for a commit that allows for closed loop testing.

@corecode thanks. Can you be a bit more specific? I'm not well versed in git/github. Where do I find your branch? I need to patch the code in order for it to work on zLinux so I'll have to do the same with your branch.
By "closed loop testing", do you mean you have to do the test in a specific way? Do you have doc in your branch describing how to do the test? Thanks.

@gongsu832 Can this issue be closed (as the original 'bug' is resolved)? If you'd like to open a new issue for your performance testing questions, if you tag me, I will tag it as a consensus question and assign it.

@jyellick Yes you can close it. I haven't heard from @corecode on where to get his branch and how to run the "closed loop testing" he mentioned. When I do and if I have issues, I'll open a new issue and tag you. Thanks again for the help.

@gongsu832 You can find @corecode's branches here
If I had to make an educated guess, I'd think you might want the branch closed-loop-rest, but @corecode is definitely the person to answer you here.


lib/bats: Extract lib/bats/helper-function

First part of #156. Though it's still a work in progress, the speedup in the tests with this new behavior added to the functions from lib/testing/environment is substantial enough that I'm eager to get it merged sooner than later. The other changes will be more extensive, so this will reduce the amount of noise in the pull requests as well.
Also, I finally grokked Bats enough to understand why processingBATS_PREVIOUS_STACK_TRACE is unnecessary. I've removed that processing everywhere, and added a thorough comment to restore_bats_shell_options.

Coverage increased (+0.02%) to 95.297% when pulling 629d5cd3f49c0b0c2e0254a29a106ede6fef3e60 on bats-function into 8b82085498b1e35694cf978a0c9952515dd88b54 on master.


Simulator: Fix keyboard input

Keyboard input was broken after Simulator CI pr. Thanks Ebadi for noticing it https://github.com/commaai/openpilot/pull/24274#issuecomment-1108531350.

Thanks @gijskoning   for quickly fixing it. I confirm that the keyboard is working now but I still don't see the camera stream in the openpilot window and modeld has not been started.
I don't think your changes caused this but I just wonder how come tests in #24274 actually don't catch this.

Are you sure you executed all the steps? Running the scripts in tools/sim folder: start_carla.sh, launch_openpilot.sh and bridge.py allows me to run the simulator easily.
You could also try to run bridge in low quality mode using ./bridge.py --low_quality.
If this doesn't fix your issue it is best to create a new issue with some more details.


elons-toys: minor tweaks needed

Description
Minor tweaks found on elons-toys concept exercise:

Typo on README.md file, line 56. "On" instead of "own":

"All the methods we have seen so far have a value receiver which means they on a copy of the value passed to the method, meaning that any modification done to the receiver inside the method is not visible to the caller."

Wrong return example on items 2 and 3. Instructions and tests require string type return:

car.DisplayBattery()
// Output: "Battery at 100%"

//and

car.DisplayDistance()
// Output: "Driven 0 meters"

Tests are sorted differently from the exercise order:

|       Exercise       |      |           Tests          |
| 1. Drive()           |      | 1. TestDrive()           |
| 2. DisplayDistance() |  --> | 2. TestCanFinish()       |
| 3. DisplayBattery()  |      | 3. TestDisplayDistance() |
| 4. CanFinish()       |      | 4. TestDisplayBattery()  |

Suggestions

Simple typo fix
Either remove the returning string type and require student to use fmt.Print() to display the message or change the examples to:

msg := car.DisplayBattery()
fmt.Print(msg)
// Output: "Battery at 100%"

Use exercise order on tests

These are good points.

I'll probably change the text to something simpler like "they will receive a copy...". I don't think "own" is clear language, as it is open to interpretation and in other languages it has a different meaning, i.e, in Rust variables can be owned by functions, but that is not related with copy (at least directly).
I would just change the example
Sound good.


Demo midi in and out

Hi, fantastic work on all of this so far, awesome stuff.
Here is a very small contribution, to demo midi in and out. MIDI out plays a square wave on a piezo buzzer connected to pins C0 and C1. MIDI in is sent when buttons connected to C2, C3, C4 and C5 are pressed.

Thank you so much for your work.  Can you make a video of it working?

Does this work in Windows?

Oh wow! I just saw your video and hackaday article!  Very cool!

Cheers, yes I briefly tested it on windows and it seems to work OK.


add binder link?

It's great that sphinx-nbexamples has links to download, view in nbviewer, etc.
Could we also have a link to open in binder?
In pangeo, we are creating a custom binder hub: https://github.com/jhamman/pangeo-binder
We would love it if the notebooks in our gallery could automatically launch directly into that binder.
This may be complicated to achieve, but I wanted to open the discussion.
cc @jhamman

Hey @rabernat and @jhamman! Sorry for being so unresponsive! This is just to inform you, that I just implemented binder links in sphinx-nbexamples (see the docs and PR https://github.com/Chilipp/sphinx-nbexamples/pull/11). It should be straight forward to let it point to the pangeo binder.

Fantastic, thanks @Chilipp!


Switch to pulumi-go-provider as a provider framework

Fixes #84
This PR switches the entire provider to use https://github.com/pulumi/pulumi-go-provider, improving the authoring and correctness experience and increasing the development velocity for adding new resources.
This RP relies on https://github.com/pulumi/pulumi-go-provider/pull/210 (not yet merged) and will not be fully backwards compatible with the current version until https://github.com/pulumi/pulumi-go-provider/issues/193 is implemented.

Closing in favor of https://github.com/pulumi/pulumi-pulumiservice/pull/258.


Replace context hook with direct call

Description
Context providing hook (that can't be overwritten in the project) is replaced by a direct call to a specific TemplateLoader in the ContextHelper to use environmental variables.
Resolves #107  #72
PR Checklist

[x] Tests added
[x] Changelog updated

License Compliance is complaining and I think we can drop this one further on.
The change looks good to me - well done.


Authenticate requests to Exhibitor during upgrade

High Level Description
This teaches the upgrade script to authenticate Exhibitor requests if Exhibitor requires it.
Depends on #1528.
Related Issues

DCOS-15495 DC/OS upgrade script does not support exhibitor HTTP auth

Checklist for all PR's

[x] Included a test which will fail if code is reverted but test is not. If there is no test please explain here: Covered by upgrade integration test
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

@mesosphere-mergebot bump-ee

@mesosphere-mergebot bump-ee

@mesosphere-mergebot bump-ee

@amitaekbote @BenWhitehead This is ready for review. It's based on #1528, so you can skip reviewing the first commit.

@mesosphere-mergebot label ready for review

@mesosphere-mergebot label ship it

@branden Can we get a 1.9.1 branch backport of this too?

@mesosphere-mergebot label work in progress

changed the label till we figure out the branches

Closing this in favor of https://github.com/dcos/dcos/pull/1492.

@adam-mesos 1.9 backport here: https://github.com/dcos/dcos/pull/1559


spork stops running

When I hit spork or guard (spork) I have the following error:
rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:68:in require' .rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:68:in block (2 levels) in require'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:66:in each' .rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:66:in block in require'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:55:in each' .rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler/runtime.rb:55:in require'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/bundler-1.0.18/lib/bundler.rb:120:in require' .../config/application.rb:7:in <top (required)>'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/app_framework/rails.rb:48:in require' .rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/app_framework/rails.rb:48:in preload_rails'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/app_framework/rails.rb:7:in preload' .rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/test_framework.rb:134:in block in preload'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork.rb:62:in exec_prefork' .rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/test_framework.rb:120:in preload'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/run_strategy/forking.rb:25:in preload' .rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/runner.rb:74:in run'
.rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/lib/spork/runner.rb:10:in run' .rvm/gems/ruby-1.9.2-p180@rails31/gems/spork-0.9.0.rc9/bin/spork:10:in <top (required)>'
.rvm/gems/ruby-1.9.2-p180@rails31/bin/spork:19:in load' .rvm/gems/ruby-1.9.2-p180@rails31/bin/spork:19:in '
Which is caused by:
Bundler.require *Rails.groups(:assets => %w(development test))
in application.rb
if I comment the line above everything works fine. My questions are what is the problem? how can it be solved? is it a bug?
Best.

I'm not sure to understand well that error. Please can you give me your Guardfile & Gemfile and be more precise about what you are doing to get this (command line used, actions...). It could also be a error specific to Spork and not guard-spork.

I have installed spork and guard-spork in a recent created rails project. When I run spork i have the error I have posted. When i open the file application.rb and comment the line: Bundler.require *Rails.groups(:assets => %w(development test)),
spork and guard-spork starts working again. My gemfile:
source 'http://rubygems.org'
gem 'rails', '3.1.0'
Bundle edge Rails instead:
gem 'rails',     :git => 'git://github.com/rails/rails.git'
Database
gem 'sqlite3'
gem 'sqlite3-ruby', :require => 'sqlite3'
Inherited Resources
gem 'inherited_resources'
Responders
gem 'responders'
HasScope
gem 'has_scope'
Gems used only for assets and not required
in production environments by default.
group :assets do
gem 'haml-rails'
#gem 'sass-rails' in rails 3.1 by default
gem 'coffee-rails', '~> 3.1.0'
gem 'uglifier'
gem 'jquery-rails'
end
group :development do
Deploy with Capistrano
gem 'capistrano'
Capybara
gem 'capybara'
Oink
gem 'oink'
Annotate
gem 'annotate'
end
group :test do
Use unicorn as the web server
gem 'unicorn'
Sport -> tests server
gem 'spork', '~> 0.9.0.rc'
gem 'guard-spork'
gem 'autotest', '4.4.6'
gem 'autotest-rails-pure', '4.1.2'
gem 'autotest-fsevent', '0.2.4'
gem 'autotest-growl', '0.2.9'
To use debugger
gem 'ruby-debug19', :require => 'ruby-debug'
FactoryGirl as object mocking
gem 'factory_girl'
Rspec as test framework -> TDD
gem 'rspec'
gem 'rspec-rails', :require => 'rpsec'
gem 'guard-rspec'
Cucumber as scenarios tests framework -> BDD
gem 'cucumber'
gem 'cucumber-rails', :require => 'cucumber'
gem 'guard-cucumber'
Pickle -> Cucumber support
gem 'pickle'
Database
gem 'database_cleaner'
Migrations helper
gem 'migration_test_helper'
Routes test
gem 'webrat'
gem 'webmock'
Colors in test feedback
gem 'syntax'
end
Guard:
A sample Guardfile
More info at https://github.com/guard/guard#readme
guard 'spork', :cucumber_env => { 'RAILS_ENV' => 'test' }, :rspec_env => { 'RAILS_ENV' => 'test' } do
watch('config/application.rb')
watch('config/environment.rb')
watch(%r{^config/environments/.+.rb$})
watch(%r{^config/initializers/.+.rb$})
watch('Gemfile')
watch('Gemfile.lock')
watch('spec/spec_helper.rb')
watch('test/test_helper.rb')
end
guard 'cucumber', :cli => "--drb" do
watch(%r{^features/.+.feature$})
watch(%r{^features/support/.+$})          { 'features' }
watch(%r{^features/step_definitions/(.+)_steps.rb$}) { |m| Dir[File.join("**/#{m[1]}.feature")][0] || 'features' }
end
guard 'rspec', :version => 2, :cli => "--drb", :all_on_start => false, :all_after_pass => false do
watch(%r{^spec/.+_spec.rb$})
watch(%r{^lib/(.+).rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
watch('spec/spec_helper.rb')  { "spec" }
Rails example
watch(%r{^spec/.+_spec.rb$})
watch(%r{^app/(.+).rb$})                           { |m| "spec/#{m[1]}_spec.rb" }
watch(%r{^lib/(.+).rb$})                           { |m| "spec/lib/#{m[1]}spec.rb" }
watch(%r{^app/controllers/(.+)(controller).rb$})  { |m| ["spec/routing/#{m[1]}routing_spec.rb", "spec/#{m[2]}s/#{m[1]}#{m[2]}_spec.rb", "spec/acceptance/#{m[1]}_spec.rb"] }
watch(%r{^spec/support/(.+).rb$})                  { "spec" }
watch('spec/spec_helper.rb')                        { "spec" }
watch('config/routes.rb')                           { "spec/routing" }
watch('app/controllers/application_controller.rb')  { "spec/controllers" }
Capybara request specs
watch(%r{^app/views/(.+)/.*.(erb|haml)$})          { |m| "spec/requests/#{m[1]}_spec.rb" }
end

Ok so it isn't a guard-spork issue, but more a Spork issue. I think you have a conflict with a gem required by Bundler, try to update Bundler (1.1.rc just came out this week) and commented gem after gem in your Gemfile to see from which it comes.


Add Xfce Terminal to terminal emulators

Xfce Terminal works flawlessly on Wayland sessions. It's also easy to configure graphically despite offering dozens of different options.

This list/repository is intended to be for software that's wayland-native in some way. As far as I understand, Xfce is based on the VTE terminal, which has nothing wayland-native to it. I wasn't able to find source code for the xfce terminal though, so I'm open to feedback if my information on it is incorrect!

I'm not particularly informed on this, but IIRC, VTE is just a library that serves as the interface for GNOME Terminal. It has a few Wayland-specific patches, but since GNOME Terminal is just a GTK application anyways, it tends to run fine on Wayland without much help.
The same goes for Xfce Terminal - it has relatively few patches related to Wayland, but runs just fine without any Xwayland nonsense. In my view, it should hit any guidelines you may have for this repo. :)
By the way, you can see the Xfce Terminal source here: https://gitlab.xfce.org/apps/xfce4-terminal

Okay cool! Thank you for explaining that. In that case I think it's a great addition - thank you for submitting it!


Some released versions are not tagged in git

Such as 0.5.0 and 0.5.1

Oh, weird, it seems that I somehow failed to push them. Now they are in!

Thank you!


Bug report - Editor settings, error when set save on focus loss

Godot version
4.1.2 stable
System information
Linux - Ubuntu 22.04
Issue description
When set a configuration the project window close.
image
Steps to reproduce
1 - Open some project
2- Editor settings
3 - search for save
4 - set up the option "Save on focus loss"
Minimal reproduction project
Project with the tutorial assets Your First 2D game

Thanks for the report! Consolidating in https://github.com/godotengine/godot/issues/78558.


Adds the ability to export summary graph

Adds the ability to export a summary graph with a specific width to the given export_path.

Thanks! The ability to set the width seems great. I am less sure about the export_path. How does that improve over just calling pl.savefig after calling summary plot with the show=False option?

Well, I am not sure why, but even when I set show=False the plot gets printed, hence is not available to be saved afterward. At least that is what I've experienced working in a notebook.

Perhaps that is because you have an interactive mode set for the notebook?


Installation of f3dasm

I have faced errors during the installation of f3dasm, which made me dig into the installation a bit more in detail. The following suggestions wont necessarily fix the issue I am facing, but would definitely be a clean-up, and also make installation more straightforward for others.
Comments on current instructions: For Linux
Reg: Preferred way
I see that there are many (slightly more than usual) packages  in the f3dasm_environment.yml file, and probably these are packages that got installed as additional packages. If this is the case:

I recommend that the yaml file is exported as:
conda env export -n f3dasm_env --from-history -f f3dasm_environment.yml

Of course, this wouldn't list the pip installations. For that, do a regular export as:
conda env export -n f3dasm_env  -f f3dasm_environment_pip.yml

Manually update the f3dasm_environment.yml to have the pip installation list from f3dasm_environment_pip.yml

Reg: If things fail

Minor documentation error: Before pip install -e . the environment has to be activated. So,

conda activate f3dasm_env
pip install -e .

Importing f3dasm fails because setup.py doesn't install tensorflow as a dependency. I also see in that in setup.py there are some packages which the preferred way are installed through via conda .  It would be a good idea to separate out the pip installations and conda installations.

Furthermore, it might be worth defining the version of the packages in the setup.py, unless it is left out on purpose.

Reg: Installing all packages manually

Minor documentation error: Before pip install -e . the environment has to be activated. So,

conda activate f3dasm_env
pip install -e .

Suggested instruction: For Linux
If the preferred way and if things fail are updated as suggested above, the Installing all packages manually would become redundant. For if things fail,  I suggest that conda installations happens first, followed by installation through setup.py, which only installs packages that are additionally required (i.e. redundancies removed from setup.py).

Hey Hari, thanks for your comments
I'm completely changing the way f3dasm is installed
This is because it is too complicated and is not working consistently on different OS
I'll show you how during the next coding practices session


Add binaries for Mac ARM, Linux ARM, Windows

This should enable cross-compilation to build binaries for Mac ARM (Apple Silicon) and also enables builds for all supported Python versions on both Windows and aarch64 Linux.
Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

ARM builds will fail until conda-forge/cgal-cpp-feedstock#18 is merged.


Could admin password be required to Activate / Deactivate?

Hi there - this tool looks GREAT for helping control users being able to write to external disks, something I am looking to be able to control. On some workstations external drives still need to be accessible to read from, and Apple MDM doesn't seem to have ability to mount them as Read Only, only to disable completely.
So - is there any chance that Activate / Deactivate could require admin password to change to Deactivate? Thanks so much!

Duplicate of #20?


NTP stats getter from previously get_ntp_peers()

https://github.com/napalm-automation/napalm-base/pull/12

@dbarrosop I start with this one, cannot rebase. Anyway I will come back today with the implementation for the new get_ntp_peers() method and preparing the Travis.


Fix link to node selector documentation.

LGTM. You'll need to rebase onto a current master to fix shippable (go1.3 build was broken)

Actually, is that link supposed to work? I get redirected to the home page.
http://releases.k8s.io/HEAD/docs/user-guide/node-selection/README.md

Very weird. When you wrote that comment yesterday, I checked and you were right. But I clicked again just now and it went to the right place!

I concur. Readding LGTM.

One of the two shippable builds failed but it is #11857 so this is fine to merge.


Adding deferInitialization docs. Fixes #1304.

I will definitely need a review on the Japanese docs since I just ran the text through Google Translate 😅
This PR fixes #1304.

Thanks for the suggestions! I've added them and updated the PR. If there are no other suggestions for the English docs, then perhaps the Japanese docs could get a pass to try to go from "understandable" to "good" 😄 ?

@filmaj Thanks for revising the doc quickly! @wongjas will come up with JP translation parts today (I've already reviewed most of his translation)

Added the Japanese translation suggestions. One final check please @seratch! 🙇

@wongjas I might forget submitting your review - I cannot see your suggestions

Thanks for the suggestions! I've added and squashed them. @seratch or @wongjas, feel free to merge if you think the Japanese documentation is now ready 👍

v3.10 is released! 🎉
Rebased and merging this now.


Will this be an open source project?

There is a reimplementation available on ComyUI: https://github.com/logtd/ComfyUI-Fluxtapoz

太好了 非常感谢Great! Thank you very much.


Monit dies trying to revive Docker; master borked

The cluster I created on Friday (at head) was running over the weekend, and when I tried to talk to it this morning, I couldn't.
max@mac$ ./_output/local/bin/darwin/amd64/kubectl version
Client Version: version.Info{Major:"0", Minor:"15+", GitVersion:"v0.15.0-728-g64b4ba688bbb56-dirty", GitCommit:"64b4ba688bbb56c60e0eca5e3da4b414bc42d41a", GitTreeState:"dirty"}
Couldn't read version from server: Get https://<master ip>/version: dial tcp <master ip>:443: connection refused

SSH'ing into the master, I couldn't hit /validate
maxforbes@e2e-test-maxforbes-master:~$ curl localhost:8080/validate
curl: (7) Failed to connect to ::1: Network is unreachable

maxforbes@e2e-test-maxforbes-master:~$ curl <IP_ADDRESS>:8080/validate
curl: (7) couldn't connect to host

and Docker isn't running
maxforbes@e2e-test-maxforbes-master:~$ sudo docker ps -a
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

and neither is monit
maxforbes@e2e-test-maxforbes-master:~$ ps auxww | grep monit
1000     13515  0.0  0.1   7844  1972 pts/0    S+   17:57   0:00 grep monit

so what happened to monit?
maxforbes@e2e-test-maxforbes-master:~$ cat /var/log/monit.log
[UTC Apr 24 17:46:34] info     : monit: generated unique Monit id 07087e649065a353a1aeb843eee5e57d and stored to '/var/lib/monit/id'
[UTC Apr 24 17:46:34] info     : Starting monit daemon
[UTC Apr 24 17:46:34] error    : State file '/var/lib/monit/state': Unable to read magic
[UTC Apr 24 17:46:34] info     : 'system_e2e-test-maxforbes-master.c.mtestp1.internal' Monit started
[UTC Apr 24 17:46:35] info     : monit daemon with pid [4353] killed
[UTC Apr 24 17:46:35] info     : 'system_e2e-test-maxforbes-master.c.mtestp1.internal' Monit stopped
[UTC Apr 24 17:46:35] info     : Starting monit daemon
[UTC Apr 24 17:46:35] info     : 'system_e2e-test-maxforbes-master.c.mtestp1.internal' Monit started
[UTC Apr 24 17:48:35] error    : 'docker' process PID changed from 2011 to 5358
[UTC Apr 24 17:50:35] info     : 'docker' process PID has not changed since last cycle
[UTC Apr 25 08:46:58] error    : 'docker' failed protocol test [HTTP] at UNIX[/var/run/docker.sock] -- HTTP: Error receiving data -- Resource temporarily unavailable

[UTC Apr 25 08:46:59] info     : 'docker' trying to restart
[UTC Apr 25 08:46:59] info     : 'docker' stop: /etc/init.d/docker
[UTC Apr 25 08:49:23] info     : 'docker' start: /etc/init.d/docker
[UTC Apr 25 08:52:01] error    : 'docker' failed protocol test [HTTP] at UNIX[/var/run/docker.sock] -- HTTP: Error receiving data -- Resource temporarily unavailable

[UTC Apr 25 08:52:05] info     : 'docker' trying to restart
[UTC Apr 25 08:52:08] info     : 'docker' stop: /etc/init.d/docker
[UTC Apr 25 08:58:47] error    : 'docker' failed to stop
[UTC Apr 25 09:00:54] error    : 'docker' process is not running
[UTC Apr 25 09:00:55] info     : 'docker' trying to restart
[UTC Apr 25 09:00:55] info     : 'docker' start: /etc/init.d/docker
[UTC Apr 25 09:00:56] error    : Cannot fork a new process

@zmerlynn  found where this happens in monit. My questions are: "Why?" and "Should we try to debug this further?" @roberthbailey pointed out that a possible preventative measure is to have a cron job running that kicks monit if it's dead.
I left the cluster running, so if anyone would like me to run commands or would like to dig in more, please let me know.

I haven't thought much about this, but it may be related to Docker being slow at restarting?
https://github.com/docker/docker/issues/11181
As a side note, it seems like pretty poor design that monit dies if it fails to restart something.

@ArtfulCoder Could you please take a look at this? Thanks!

@a-robinson

When you started the cluster on Friday, could you connect to apiserver using kubectl ?
Did you see any errors in the startupscript.log file ?

cc @mbforbes

Yes, the cluster functioned on Friday; I was running tests against it. (If you look at the monit log I pasted above, it appears it was functioning for quite a while.)

I don't see any errors there, but I can dump it here if it would help (it's quite long, obviously).

1.No out-of-memory errors seen/logged around the time when monit failed to fork a child process.
2. kubelet seems to consume 400m of memory. not sure why it is so high
3. maybe the fork did fail because of insufficient memory, but this is a guess and no logs to substantiate it

Just as an explicit followup, there were at least 8 out-of-memory errors that happened on that instance (a master) from the logs in  /var/log/kern.log.1—is this normal? I can provide several different logs or VM access for interested parties...

Regardless of the cause of the problem (which I agree would be interesting to discover) it seems like we need a cron job or something to make sure that monit is always running. @ArtfulCoder can you look into how we can babysit our babysitter?

I will take a look this one today since there are several ones related to each other: #7815, #7795, ...


[Feature Request] Append to highlight note instead of overwriting.

Painfully realised that currently when importing highlights, the entire highlights page is generated fresh.
This means that any caret references, edits to highlight notes are broken / overwritten on each import.
This limits the usefulness of the plugin quite severely as the highlights page can only then ever be used as a visual reference to read and remind ones self (or copy and paste 😵), and therefore not truly integrated into the Obsidian / Zettelkasten note making workflow.
I think the simplest solution around this would be to simply append new highlights / notes to the highlights file instead of overwriting the whole file completely with a fresh import. From memory the Kindle highlights plugin works this way.
Is it realistic to look into something like this? Would be immensely appreciated.
I want to help, but my programming chops are probably not going to do much.
I'd be happy to update and extend the documentation if needs be though.

You're referencing https://github.com/hadynz/obsidian-kindle-plugin?
I would need to have a look on how they solved it and see.
But most likely I would need to keep a state somewhere of what has been exported and what not.

Would be amazing man! 🙏
I'll also try have a look.

Considering the chapter names are returned from the book (ie values that are not user defined), could they be used as references for an incoming import?
eg:

on import, reference the last highlight from the last chapter heading
check these against the incoming import (maybe the user deleted those highlights)
append the incoming highlight/note pairings that come after the latest highlight/note pairings to the file

not sure how to handle the else branch then in step 2. But will keep thinking.

Hi,
I found your plugin when doing my "let's see if there is something out there, before I start developing my own" check. I would like to contribute to your plugin if you allow it, rather than creating something similar myself. I came up with a naive implementation tonight that keeps edits including caret references, and have pushed my changes to my fork at https://github.com/rsletta/obsidian-kobo-highlights-import.

@rsletta yes please open a pull request! Contributions are always welcome.
I did not have a chance yet to have a good look at this, so your start will certainly help.

@tjex are you available to test and see if it fulfils your use case?
you can download the plugin from https://github.com/OGKevin/obsidian-kobo-highlights-import/releases/tag/1.6.0-beta.0 and replace .obsidian/plugins/obsidian-kobo-highlights-importer-plugin/main.js then restart the app.

hey. great that this got done! makes the plugin so much more valuable 🙏.
I tested:

Making a new highlight and note, that sit in between older and newer highlight+notes: works great (creates a new highlight and note in the markdown doc in the correct position.
edit a note of a highlight: updates the note in the markdown doc
deleting an annotation: the annotation does not get deleted from the markdown doc
changing the range of the highlighted text: the quote (highlighted text from book) does not update in the markdown doc

Features 1 and 2 are definitely the most useful. 3 and 4 would be nice to haves. I'm not sure how much more effort it would take to implement them.
Also, just on a readability / UX standpoint, the --> you can write text here blocks <-- should be replaced with something cleaner, like Local note: (this text won't be deleted on resync. And perhaps only in one spot after each highlight?
Perhaps this was already part of the plan, but just in case, it's worth pointing out for clarity!

Also, just on a readability / UX standpoint, the --> you can write text here blocks <-- should be replaced with something cleaner, like Local note: (this text won't be deleted on resync. And perhaps only in one spot after each highlight?

It looks like I forgot to push a change 🤦🏾
It’s supposed to be %% instead of —>, if not it will show in read mode, as in, %% is a comment marker.
Like shown here: https://github.com/OGKevin/obsidian-kobo-highlights-import/pull/107#issuecomment-1886984245
With the way it looks in the screenshot, do you still think the text needs to change due to UX?

Regarding deletion, if I understand the code correctly, the current implementation would remove everything within that ID marker if it goes missing from the database. What you’re describing, sounds like when you delete it via the e-reader, the e-reader might not actually delete the entry from the DB? I would need to play around with it and reproduce.
Do you mind creating a new issue so we can look into this?

I need to investigate your 4th point.
Based on the investigation that @rsletta did https://github.com/OGKevin/obsidian-kobo-highlights-import/pull/107#issuecomment-1506391674
And this: https://github.com/rsletta/obsidian-kobo-highlights-import/blob/6bc945f26436eaa74f677b585fca44b5e697f099/src/database/Highlight.test.ts#L488-L555
Updating the highlight should work if the ID’s are the same. I’ll see if I can reproduce your case with an actual end to end test case.
I’ll create a followup issue for this one 👌🏾

With the way it looks in the screenshot, do you still think the text needs to change due to UX?

I think wrapping in %% ... %% as you've done is good. Then at least in reader view, it's all neat.
Will make a new issue. Thanks so much :)


Consistently disable submit button on forms to prevent double submissions

See comment on issue #953

Can you tell me that in which file do you have written the code for submit button

This file contains all of the event handlers. Submit handlers start here (look for a class name on the button that corresponds with the function name): https://github.com/photo/frontend/blob/master/src/html/assets/themes/fabrizio1.0/javascripts/op/Handlers.js#L220
The dispatching happens here: https://github.com/photo/frontend/blob/master/src/html/assets/javascripts/openphoto-util.js#L150

the program is not running

bro plz provide package.json file

This was build long before the mess we now call npm and package.json :)
I'm not able to provide detailed troubleshooting for this. I would recommend using the vagrant image and associated instructions as they were repeatedly used and tested to bring up development environments.
Instructions can be found here: https://github.com/photo/developer


[QUESTION] Examples of GraphQL using async/await?

Description
How can I use GraphQL in conjunction with async/await syntax?
Any examples? I see using Starlette

from graphql.execution.executors.asyncio import AsyncioExecutor

app = Starlette()

app.add_route("/", GraphQLApp(schema=graphene.Schema(query=Query, executor_class = AsyncioExecutor)

But if app is already
app = FastAPI()
now it is
app = Starlette() ?
Now add_route got an unexpected keyword argument 'executor_class' if
app = FastAPI()

Try read it: https://fastapi.tiangolo.com/tutorial/graphql/

Thanks for the help @prostomarkeloff and @includeamin ! :clap: :cake:

@includeamin this goes outside of the dependency injection system though, right? Do you know a way to integrate the Depends() calls into this solution?


[windows] add LTS versions for windows EOL [fixes #1425]

changes
add LTS versions

reasons
already described (E),(W), and LTS version, but there is no eol of LTS versions in the current API.

others
ref: https://microsoft.fandom.com/wiki/Windows_10_version_history
ref: https://docs.microsoft.com/en-us/windows/release-health/release-information
ref: https://ja.wikipedia.org/wiki/Microsoft_Windows_10のバージョン履歴#cite_note-Windows_1507_CBB_Support_postponed-6

@captn3m0 maybe just maybe we might add .1  for each of  cycleShortHand so they (lts ones) will shown a little higher than related branches

I think we should use the current cycleShortHand as the releaseCycle instead for better API support (with some prefix/suffix for E/W/LTS).
Will take that up separately.

Sorry, I'm away from these, Thank you for the correction and release.


Connection to non-default host and port

Good day.
How can i connect to another host and port?
I tried with this code below, but MLAPI anyway using defaults "localhost" and "7777" port:
using UnityEngine;
using UnityEngine.UI;
using MLAPI;

public class UIController : MonoBehaviour {
    public Text host;
    public Text port;

    public void EventHost() {
        NetworkingManager.singleton.NetworkConfig.ConnectAddress = host.text.Trim();
        NetworkingManager.singleton.NetworkConfig.ConnectPort = int.Parse(port.text.Trim());
        NetworkingManager.singleton.StartHost();
    }

    public void EventClient() {
        NetworkingManager.singleton.NetworkConfig.ConnectAddress = host.text.Trim();
        NetworkingManager.singleton.NetworkConfig.ConnectPort = int.Parse(port.text.Trim());
        NetworkingManager.singleton.StartClient();
    }
}

Unity Version: 2018.3.1f2
MLAPI Version (Commit # or release version): 5.0.0

I realize this is very old but the port and ip are no longer accessible in the config. where should I set them.


add note about deprecated properties

This is the number one issue I've seen with mods on the workshop. It's probably worth calling it out.

Sure, thanks. Remind me to push it to the workshop later, since I'm not at home for a few days

i just pushed a new version


Dependency Updates

Is your feature request related to a problem? Please describe.
Update dependency versions with possible security vulnerabilities.
Describe the solution you'd like
Updating dependency versions

Please make a note of which dependences can't be upgraded (e.g. we're reliant on a X.Y and can't go to X.(Y+1) or there is no greater version).

We are unable to update the following dependencies:

elasticsearch:build-tools:7.10.2 -> This is the last open source version. Starting in Data Prepper 1.2, we are using OpenSearch libraries instead.
checkstyle:8.20 -> The build fails with this update. It appears to be at least partially because of the ElasticSearch build tools. Regardless, this dependency is only used a compile/build-time.


Improve performance of data processing season

Rabbit season

Failed Test
Failure Summary
Logs

[chromium] should show the FAQ page correctly.
The FAQ page was not loaded correctly, and the title and heading were not found.
Logs ↗

Flaky Test
Failure Summary
Logs

[chromium] should login with credentials correctly correctly.
The login request to the server failed with a 404 Not Found error.
Logs ↗

[chromium] should display error message for invalid login correctly.
The expected HTTP status code 401 was not received, instead a function was returned.
Logs ↗

[chromium] should return list of users correctly.
The API call to get the list of users failed with a 404 Not Found error.
Logs ↗

[chromium] should load the v1 homepage correctly.
The test timed out waiting for the expected text "Real SaaS App" and the "Homepage" heading to be visible on the page.
Logs ↗

... and 2 more

View Full Report ↗ ⋅ Docs


add AWS CLI v.2 toolset to have a possibility of AWS Code integration

In order to install AWS Toolkit for JetBrains the AWS AWS Command Line Interface should be installed as a prerequisite. So it is included in Dockerfile steps.

The piece of guide added for those who are trying to run the container on Windows hosted docker

This is excellent, thank you very much!
I'm uncertain if everyone needs AWS cli in their PyCharm, though maybe it's a nice addition. As an alternative, we could make the Dockerfile more extensible, so you could build an image based on this one that installs extra things like unzip and the awscli utilty. The downside is that you'll have to build the image yourself and rebuild it when there's a new version out. Apart from that, it's working all right, this is how I use IDEA pro at work. What are your thoughts on this?
The Windows instructions are perfect, I'd like to get those in anyway. 👌
Thanks a lot!

Thanks for your comment!
Actually by some reasons I get a 'Permission denied' error when I'm trying to build own Dockerfile based on your:
FROM rycus86/pycharm-pro

RUN apt-get update && apt-get install --no-install-recommends -y  unzip

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
  && unzip awscliv2.zip \
  && rm awscliv2.zip \
  && /opt/pycharm/aws/install

CMD [ "/opt/pycharm/bin/pycharm.sh" ]

E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
The command '/bin/sh -c apt-get update && apt-get install --no-install-recommends -y  unzip' returned a non-zero code: 100

Still investigating...

Ah yes, this is where extension points might work better. At that point the user is developer, that's why you get permission issues. Try this:
FROM ...
USER root
RUN ...your commands ...
USER developer

It works now. Thanks a lot.
So I reverted Dockerfile back to original. Let's update Readme only?
Thank you!

Excellent, thank you very much!


[Feature Request] Text diff checker

Duplicate with https://github.com/ObuchiYuki/DevToysMac/issues/15


Remove schema/schema

Remove Schema and SchemaBuilder
Remove GsonRepository, and comment the code using GsonRepository.

Out of scope
Implementation of serialization/deserialization

@yongpingchen LGTM


partners/mongodb: Improved search index commands

Hardens index commands with try/except for free clusters and optional waits for syncing and tests.
efriis These are the upgrades to the search index commands (CRUD) that I mentioned.

@efriis I'll discuss with my team how best to make tests non-trivial  in the morning and get back to you.

Ok. Added an additional exception to pytest.raises. That's sufficient to capture all cases that I want to test that shouldn't work.

@efriis If you have a spare minute, would you please merge this? It will make the Retrievers one cleaner.

@eyurtsev Hi Eugene, would you please take a look at this? Quick and easy, and it will really help me deal with merge conflicts on #23237 (or what will be our HybridSearchRetriever PR).

Thanks, @efriis . Appreciate the review and additional commits.


an simple program to vote

This program let vote between candidates that are listed in the candidate.txt

Hi @hanshulll.
I have add the vorgotton file. sorry.


Random effect

Is it possible to give a random effect?

Are able to explain a little more?
Are wanting to set a random effect when turning on?
Would something like this work?


Toggle not working in Safari 9.1

I am able to access hotel.dev. But on clicking the toggle, it doesn't work on Safari 9.1.
Here is the stack trace.

It looks like the fetch library is missing (from the stack trace above). It works in Chrome though.
Any known bugs here?

Thank you, just forgot to add fetch polyfill. Will fix it right now.

Awesome!

Should be good v0.4.10


Returned fit causes overflow error

Again, when trying to fit the compressor maps, I've gotten a number of returned fits with very large exponents. Sometimes, when these are plotted outside of log space, overflow errors occur. I'm not sure this can be avoided, and I don't think the data I'm using is too well conditioned, but it would be nice if there was someway to control how large the exponents were in the fit.
'''
w0.231 = 0.00187 * (u_1)-302 * (u_2)58.2
+ 3.75e-12 * (u_1)-2.67e+03 * (u_2)496
+ 0.326 * (u_1)-7.81 * (u_2)**0.962
+ 0.465 * (u_1)1.97 * (u_2)-0.525
'''

Yes! It's called regularization, and we should implement it as an option.
L2 regularization is trivial to implement. L1 requires more work, but is likely worth it for the better (sparsity inducing) performance. We should implement both.


Question about model tuning

Hi,
Thanks for making your enhancements to minGPT available. I am curious why your play_math model will deal with ndigit = 4 when anything above ndigit = 3 with stock minGPT results in SIGKILL.
Ultimately I am trying to train on much longer addition sequences, for the purpose of multiplication eventually. But it looks like whenever the sequence length > 4:
  File "/home/asdf/minGPT-tuned/play_math.py", line 108, in <module>
    train_dataset = AdditionDataset(ndigit=ndigit, split='train')
  File "/home/asdf/minGPT-tuned/play_math.py", line 80, in __init__
    perm = r.permutation(num)
  File "mtrand.pyx", line 4528, in numpy.random.mtrand.RandomState.permutation
MemoryError: Unable to allocate 74.5 GiB for an array with shape<PHONE_NUMBER>0,) and data type int64

From here:
perm = r.permutation(num)

Any thoughts on model tuning to support say 24-bit addition sequences?

ok by setting num = 100000000 I can train on arbitrarily larger ndigit sequence lengths. Training will take forever however lol

ok by setting num = 100000000 I can train on arbitrarily larger ndigit sequence lengths. Training will take forever however lol

Try a better DataLoader that dynamically generates the AdditionDataset (instead of building the full dataset in init).


Main application hangs when call mediaplayer stop

This only happens to my Samsung S4, Android v5.0.1. Other devices work fine.
Here's the log it says:
02-01 15:24:19.294 7259-7265/com.videoeditor.kruso.dev I/art: Thread[2,tid=7265,WaitingInMainSignalCatcherLoop,Thread*=0xaf90e400,peer=0x12c8b080,"Signal Catcher"]: reacting to signal 3 02-01 15:24:20.999 7259-7265/com.videoeditor.kruso.dev I/art: Wrote stack traces to '/data/anr/traces.txt'
I also pulled the traces.txt from my device:
traces.txt
After the hang and then force close the app, other video apps no longer able to open any video, till I restart my device.
I did abit of debug and notice the app hangs at net.protyposis.android.mediaplayer.Decoders.java release() method.
p/s: I'm using the master branch after the #81 fix

Do you get the same behavior with v4.3.2 before fix #81? Is it reliably reproducible? Does it always happen or only occasionally?
At first sight it looks like the lock in MediaPlayer.stop() that was fixed in #81 is stuck, but I guess it only waits for the PlaybackThread to finish, which itself seems to be stuck in MediaCodec.stop(). I've seen the video codec also crash system wide on other old Samsung devices which had to be rebooted to get it working again. This was because the decoders were not stopped correctly and after a few instances all resources were used up.

Do you get the same behavior with v4.3.2 before fix #81?
Yes, tried 4.3.2, and still the same.

Is it reliably reproducible?
Yes~ very often it happens

Does it always happen or only occasionally?
It happens occasionally, sometimes it works, but it fails very often

Will that happen if we reinit the decoders when it has not been released? We have a code where we open and close the video multiple times.

In addition, i noticed if i seek frame within the last second of a video, then play the video. The video will just stuck for few mins, then continue play to the end. This also happened in MediaPlayer-Extended app in Google Play in most of my devices. Is it the same issue too?

Yes it can happen when too many decoders are created and old ones are not fully destroyed. I'm afraid I won't be able to analyze or fix this issue without an actual affected device. I'll take a look if I have an S4 somewhere to test but I doubt that.
The second issue seems unrelated. Please test if it also happens when you set the seek mode to SeekMode.FAST_TO_CLOSEST_SYNC and if so, open a separate issue.

sure~ I'll check again on my code and see if it has incorrect flow.
For second issue, it seems ok when set to SeekMode.FAST_TO_CLOSEST_SYNC. Is there any way we can fix if we wanna use SeekMode.EXACT?

For first issue, I checked my code and noticed it will hang when it calls prepare video before setSurface. I updated my code to prepare video after setSurface, then everything is working again.

Great, so this issue is solved? Yes setSurface must always be called before prepare, but actually there should be an exception if the surface isn't set.
Regarding seeking, please open a separate issue and post the logcat output from a seek. It might be related to the GOP size of your video.

Yea I guess it's considered solved.
Yup ok. I'll create another for second issue.


Beautify code not working?

Hi all! When I press ctrl+shift+L I see "Uncaught TypeError: b is not a function"

So 15+ days and still not answer on this. Guys I'm getting the same error as well please advice...

One note: Everything is working in the browser using, so a fatal error in your own js can cause failures like this. So make sure to test all keybindings with a working bit of js, or none.

I get this issue too on Chrome 48.0.2564.10.
http://s.natalian.org/2015-11-21/1448114876_1246x1058.png
Annoyingly it works fine in Firefox 42.

"Guys"? Who are you referring to? If its me, its singular. Either way, I
didn't implement this (it was a community PR), so if it's broken my only
option is to remove support if no one wants to take a shot at fixing it.
Let me know what you think you'd like me to do.
On Sat, 21 Nov 2015 14:10 Kai Hendry<EMAIL_ADDRESS>wrote:

I get this issue too on Chrome 48.0.2564.10.
http://s.natalian.org/2015-11-21/1448114876_1246x1058.png
Annoyingly it works fine in Firefox 42.
—
Reply to this email directly or view it on GitHub
https://github.com/jsbin/jsbin/issues/2544#issuecomment-158645270.

Is there an un-minified jsbin I can use to debug with? Without setting up jsbin myself?

Drop a cookie that has the name and value of "debug=1"
On Sun, 22 Nov 2015 16:08 Kai Hendry<EMAIL_ADDRESS>wrote:

Is there an un-minified jsbin I can use to debug with? Without setting up
jsbin myself?
—
Reply to this email directly or view it on GitHub
https://github.com/jsbin/jsbin/issues/2544#issuecomment-158773644.


Fixed create bucket error not visible. Added Z-index to p-growl so that it can be visible over the create bucket panel

What type of PR is this?

bug fix

What this PR does / why we need it:
This PR adds the Z-index to P-growl. due to this buckets error are now visible too the user
Which issue(s) this PR fixes:
Fixes #623
Test Report Added?:

TESTED

Test Report:

Special notes for your reviewer:

#SODA-PR-ACCEPTED
@skdwriting @joseph-v Please review


Make the JS global object immutable

Prevent global variables and registered functions/objects from being reset in the JavaScript vms.

Having a test demonstrating that no new global variables can be created and ones which existed after module initialization are frozen would be nice.
Also does freezing disable writing to Map()?

This is a breaking change, probably worth having a very visible entry in the Changelog for it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: sesposito:x: zyroYou have signed the CLA already but the status is still pending? Let us recheck it.


Requests other than "get" is not working

When setting method:  'post and trying to make post request to a the route I get Cannot POST /foo. Looked in the docs and didn't find if any config is needed for this.

Oh, yeah... we actually have an old ticket open for that internally from before we even open-sourced react-server...
As a workaround you can actually just leave it method: "get" and it will accept POST requests. 😬
Thanks for reporting @mattiasewers.  Good to have that tracked here.

@gigabo ahh I see, cool. Another thing, how do I get the payload sent with the request? this.getRequest().getBody() returns undefined, although I get the data in the config object return from the request....weird

Ah, yeah.  We use the body-parser express middleware internally.  #249 is open to add that to the server set up by react-server-cli.  Should be a straightforward patch.

@gigabo ok I got it working, I could make a PR?

Nice! Please do!

Solving this issue and addressing the breaking changes when upgrading routr from 0.1.3 to >1.0 need to be addressed prior to the next release.

Fixed!


Build error (that doesn't break the build?) in Create-PInvokeTxtFile.ps1

From this AppVeyor build
PInvoke.Win32 -> C:\projects\pinvoke\bin\Release\Packages\PInvoke.Win<IP_ADDRESS>6-beta-ge68a656187.nupkg
 420  C:\projects\pinvoke\tools\Create-PInvokeTxtFile.ps1 : Cannot bind argument to
 421  parameter 'AssemblyPath' because it is an empty string.
 422  At line:1 char:68
 423  + & { & 'C:\projects\pinvoke\src\..\tools\Create-PInvokeTxtFile.ps1' '' }
 424  +                                                                    ~~
 425      + CategoryInfo          : InvalidData: (:) [Create-PInvokeTxtFile.ps1], Pa
 426     rameterBindingValidationException
 427      + FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAl
 428     lowed,Create-PInvokeTxtFile.ps1
 429
 430  CodeGeneration.Debugging -> C:\projects\pinvoke\src\CodeGeneration.Debugging\bin\Release\CodeGeneration.Debugging.exe

I wonder if this is because the specialized build scripts aren't prepared to skip processing of the code generation projects.

Two things here:

The build should fail if the powershell command fails to execute.
We need to improve the MSBuild condition for including the PInvoke.Extra.targets file in the build process.

First point.
I don't know why the Exec task can't see that error (maybe because powershell.exe does not set a proper exit code?). As a workaround I can set to true the property LogStandardErrorAsError  to catch the error.
Second point.
This is happening since we added the PInvoke.Win32 NuProj project and the error is caused due to an empty $(TargetPath) variable. So we should do something like:
Condition=" !$(MSBuildProjectName.Contains('Test')) AND $(TargetPath) != '' "

Instead of
Condition=" !$(MSBuildProjectName.Contains('Test')) AND !$(MSBuildProjectName.Contains('NuGet'))  "

to exclude any project without a $(TargetPath) (NuProj so far) regardless the name.
I wouldn't complicate more that condition to exclude the CodGen projects since they don't have any [DLLImport] decorated method and the increased time for processing those project is only on the build server (since we have the GeneratePInvokeTxt build parameter).