Getting typescript-angular work

Getting typescript-angular work.
What's done besides templates being modified:

distinguish different 'HTTP' auth schemes (basic vs bearer for now)
$ref for operation body
distinguish error responses from success
fixed schemas with more than one enum field
output directory command line argument

Many thanks for the PR!
Could you let me know if the changes to the templates are due to deficiencies in the templates themselves, or to work around bugs or incompatibilities between swagger-codegen and our adaptor/model?
It might be nice to split this into two PRs, one which touches the templates and one which does not, and maybe eliminate some whitespace changes (like the omitting of blank lines at the end of files).
Thanks again!

@MikeRalphson, unfortunately when I started trying this code, Angular integration didn't work at all. So most of the work is to make ends meet from both sides: templates and code. The only change that is kinda separate is adding output folder option.

Perhaps we could move the modified templates to a new directory structure, such as
/templates/clients/javascript/angular
This would also help provide a basis for openapi-codegen as a web API / service.
What I'm trying to say is that openapi-codegen is designed to work with unmodified upstream templates, so the ideal case would be to add all the model/adaptor code improvements here, but template improvements to swagger-codegen.
I appreciate that may not be possible or easy, especially as swagger-codegen is in the midst of its upgrade to OAS 3 support.
Thoughts?

Hmmm... I get your idea of using unmodified templates... In this case one should try integrate latest templates from upstream here (I might do this later if have time).
Regarding my usage of openapi-codegen in the project I lead... I think in this case it's better to have an option of using external config+templates for the generation.
What do you think?

There's a script for pulling in the latest upstream templates. I'll run and push in a short while.

I've updated the templates. There's a couple of test failures to address in configs go and lua, so Travis might show errors until this is fixed.

@MikeRalphson I've looked thru the templates from latest upstream. They are not good compatible with Open API 3. The most problematic section is POST/PUT payloads. In 2.0 (which are templates for) there are in body params, they are bodyParams in template; and in 3.0 there are requestBody sections in definition that do set whole body as a param. Also authorization definiutions of 3.0 are richer and can't be supported with templatyes we have in upstream. Pity

Hi, it's not clear to me whether you have looked at the upstream templates in the master branch, or the 3.0 branch (or both).
The templates in the 3.0 branch - although designed for OpenAPI 3.0 will not be a good match to our current adaptor/model.
The templates in the master branch (for swagger-codegen 2.4) should work fine - possibly with some additions to the adaptor/model.
When processing OpenAPI 3.0 we fake a body parameter from the requestBody, so this should not be a problem. Down-converting the security from OpenAPI 3.0 to 2.0-compatible does, as you say, involve some loss.
I would still like to break this PR down to see if we can accept some aspects of it, and maybe copy the amended templates into a new directory/config-name. But I can do that if you want.

I think in this case it's better to have an option of using external config+templates for the generation.

Maybe this is the best plan though?

@MikeRalphson I've created two more separate PRs from this one.
And I agree to have CLI option to provide custom config and templates

Custom output and template directory options have been implemented. Thanks for splitting the PR.


Phylogeny: selection not summarized in details pan

Selected few genomes from phylogenetic tree, the number of genomes selected is not summarized
in the details pan on the right.
-Maulik

Fixed in https://github.com/PATRIC3/p3_web/pull/328


[Github] Workflow for testing installation

This tests installation of revup with various versions of python.
Signed-off-by: Greg Balke<EMAIL_ADDRESS>Topic: github_workflow_python_install
Reviewers: jerry-skydio

Reviews in this chain:
└https://github.com/gbalke/revup/pull/1 [Github] Workflow for testing installation

#
head
base
diff
date
summary

0
ac24af0b
c6262ecc
diff
Sep 14 8:09 PM
1 file changed, 22 insertions(+)


CVE-2024-43800-serve-static

There is vulnerability in the latests scan for serve-static module.
└─┬<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>

This should not be an issue, based on the advisory (https://github.com/expressjs/serve-static/security/advisories/GHSA-cm22-4g7w-348p) this impact < 1.16.0 and >=2.0.0, <2.1.0.
<EMAIL_ADDRESS>includes<EMAIL_ADDRESS>so it is not affected.
I will close the issue, but feel free to reopen/comment if you need more help

Thanks, closed then.


test(ln): add tests for linking to cwd

Add tests for symlinking to ./ (this closes #363 if the behavior here is correct). If this behavior is not posix behavior, we should fix this instead.

@ariporad this is ready for review

@nfischer: Umm... I don't think I really understand this well enough to properly review it. It seems very confusing. I'll do some more research, and get back to you.

@ariporad this is related to how ln handles relative symbolic links (see #282). I think POSIX has somewhat confusing behavior, but at least it's consistent. This was a change we introduced in v0.6, I'm just adding a couple extra tests to make sure we don't get a regression.

@nfischer: Ok, LGTM!

LGTM!


Inconsistent CoCiP output with respect to cached ERA5 data

Description
The output of CoCiP varies (predicting different contrails) depending on the state of my cached ERA5 data.
Details

Version: 0.51.2
OS: Windows 11

Steps to Reproduce
To reproduce and evaluate this issue I ran the same code without any changes 13 times. Before the 1st, 6th, and 11th run, I cleared my pycontrails cache at C:\Users\Name\AppData\Local\pycontrails\pycontrails\Cache. These are the results:

Clear pycontrails cache
Re-run CoCiP
 Output: 3926 contrail points
Re-run CoCiP
 Output: 295 contrail points
Re-run CoCiP
 Output: 295 contrail points
Re-run CoCiP
 Output: 295 contrail points
Re-run CoCiP
 Output: 295 contrail points
Clear pycontrails cache
Re-run CoCiP
 Output: 1925 contrail points
Re-run CoCiP
 Output: 1359 contrail points
Re-run CoCiP
 Output: 1359 contrail points
Re-run CoCiP
 Output: 1359 contrail points
Re-run CoCiP
 Output: 1359 contrail points
Clear pycontrails cache
Re-run CoCiP
 Output: 0 contrail points
Re-run CoCiP
 Output: 4701 contrail points
Re-run CoCiP
 Output: 4701 contrail points

Additional Notes
The only files that can be found in the pycontrails cache are ERA5 files such as  20230801-00-era5pl0.25reanalysis.nc, likely indicating that this is a problem with the ERA5 cache rather than CoCiP.
I will look into it and update if I find anything. I am happy to share any files/code needed to reproduce this.

Initialising ERA5 with cachestore=None leads to more consistent results in that the CoCiP output changes every run (rather than only the first two runs).
I am not entirely familiar with ERA5/CoCiP, perhaps this is expected behaviour? But I am happy for the issue to be closed, using cachestore=None works for my purposes.

This seems like very strange behavior to me.... CoCiP is a deterministic model, so you should get the same results as long as you're running it on the same meteorology data.
Could you check what version of the netCDF4 package you're using? We had issues with the most recent release (see https://github.com/contrailcirrus/pycontrails/pull/204) and just updated pycontrails dependencies to require netCDF4<1.7.0. That will change will be released in v0.52.0, most likely tomorrow (6/17).
If you're using netCDF4 1.6.*, then sharing a MWE would be great!

I am currently using netCDF4 1.6.5 so that should be okay.
I have also tried a fresh install of python to see if there were any issues with my current environment, but this did not seem to have changed anything. For this fresh install I did the following:
conda create -n debug_cocip python=3.9
conda activate debug_cocip
pip install git+https://github.com/contrailcirrus/pycontrails.git
pip install netcdf4==1.6.5
pip install "pycontrails[ecmwf]"
pip install "pycontrails[zarr]"
pip install "pycontrails[sat]"
I have shared an MWE via email but don't worry about it too much! It does seem likely to be an issue with netCDF4/xarray rather than pycontrails.

Thanks for the MWE! Unfortunately I'm not able to reproduce your issue: I get the same value from print(len(contrails)) each time I run the notebook.
FWIW, I'm getting 36218 from print(len(contrails)). This is much larger than the values you listed, but I'm not sure whether the numbers you gave came from the same set of flights.
I'm going to close this issue, but I'll follow up by email and can always re-open it if we come across anything that needs fixing.

Thanks for testing out the MWE! I have been able to solve this now (also giving an output of 36218 consistently) by commenting out the following line, xr_kwargs.setdefault("lock", OPEN_WITH_LOCK) found on line 744 in open_dataset (link).
As per the xarray documentation, this seems allow xarray's default behaviour of "By default, appropriate locks are chosen to safely read and write files with the currently active dask scheduler".
I am not sure why this was only an issue on my machine (perhaps a Windows 11 issue?), but I hope that can be of some help.

Weird, but glad you got it working and thanks for noting the solution!


roles/permissions determine visibility of parts of UI

Do not merge

create role and permission models
{{aptible-ability}} component
update version of ember-data-hal-adapter
change test helper signIn to push a stubbed current user into the store and set on session

Still to do

[x] fix failing tests at travis-ci
[x] get changes to hal adapter merged
[x] tests for User.can, {{aptible-ability}}
[x] add authenticated route that preloads permissions/roles
[x] determine if we can avoid loading roles from stack permissions since the API gives their URLs already
[ ] PR changes to api to embed permissions in stacks

refs #42 #28

@bantic I'd like to see the requireAuthentication: true statements get dropped, but the rest is :+1: and please :shipit: when you are good.


[placeholders]: Add regex capturing group as placeholders

What this PR does / why we need it:
Adds regex capturing group as placeholders
Which issue this PR fixes:
fixes #255

Fixed the conflict and updated the comment.
P.S for GitHub: I didn't dismiss anything :angry: :sweat_smile:

/lgtm


Feature: Consolidate the vm_name and suffix into knobs that the user can adjust

This is a feature request to move the vm_name specified for each template under the user-variables. One reason why this is handy is so that the user can explicitly control every aspect of the format of the VM name. Right now this is explicitly defined for each builder which can result in inconsistencies or mistakes (Issue #214 is an example where the vm_name is wrong for one of the builders).

This is closed by PR #217.


add more to sharepoint-webhooks-using-azure-functions.md

Category

[ ] Question
[ ] Typo
[ ] Bug
[x] Enhancement / Suggestion
[x] Additional article idea

Expected or Desired Behavior
Several lines of code devoted to talking to Azure Storage Queue.  Can be simplified with default Azure Functions configuration in the Integrate tab.
Section detailing how to set up SharePoint to set up the WebHook is missing.  This can be run from client side javascript on SharePoint's side (need access_token to talk to SharePoint REST API).
some quick notes:

add Azure Queue to additional Output for the Function

Select the Storage Account

Add additional output parameter

result

@jansenbe can you have a look? Thanks!

I should sit down and learn how to commit a PR properly.  :-(
(I actually don't know how)

Closing out old issues.  I can't tell what the status of this actually is, or if it is even an issue.


[5.9] Package.swift: make package name consistent with repo name

Rest of SwiftPM packages provided by Apple follow a dash-case naming scheme consistent with repository name, usually with swift- prefix.
This shouldn't have an impact on SwiftSyntax clients. For example swift-format already uses package: "swift-syntax" argument in its Package.swift for specifying the dependency.

@swift-ci test

@swift-ci test

@swift-ci test windows

@swift-ci Please test Windows

@swift-ci please test windows

Would this require more approvals before merging?


不支持immortalwrt,显示登录页，登录进去就报错了，进不去

不支持immortalwrt,显示登录页，登录进去就报错了，进不去
immmortalwrt的luci是23.05的

english please sir. 😁😅😅

The author's code needs to include a branch similar to https://github.com/jerrykuku/luci-theme-argon/tree/18.06 to resolve this error.


Mini Pick grep and grep_live search .git folder as well

Contributing guidelines

[X] I have read CONTRIBUTING.md
[X] I have read CODE_OF_CONDUCT.md
[X] I have updated 'mini.nvim' to latest version

Module(s)
mini.pick
Description
Grep and grep live functionality will also search the .git/ folder, which isn't really useful. It also pollutes search results.
An effective solution could be passing --glob=.git/* to ripgrep, but there are other ways to prevent searching .git folders.
Neovim version
nightly
Steps to reproduce

nvim -nu minimal.lua
:Pick grep some string that's in the .git folder

Expected behavior
don't search .git folder
Actual behavior
searches .git folder

Yeah, indeed it does that and indeed not really useful. I'll think about it.

Ripgrep also supports an rc file which you can globally ignore certain files/filetypes or folders with: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#configuration-file
Example how I do it in .ripgreprc:
# Exclude directories.
--glob=!{.git,.svn,node_modules}

# Exclude file types.
--iglob=!*.{~sql}

Ripgrep also supports an rc file which you can globally ignore certain files/filetypes or folders with: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#configuration-file

Oooh... I like this approach a lot more than having it hard coded in default command. Thanks so much for this!
@BeauSLM, I think I am going to close this in favor of users setting up their local rg configs.


Potential bug in prompt tuning

If yes, can you add an example?
I currently see attention masks like this: [1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1]
@pacman100

Also, I think this is also going to mess up the position ids in the model. Especially alibi stuff in BLOOM

Hello @mayank31398, as mentioned in point 3 of https://github.com/huggingface/peft#caveats, generate  is not supported for Prompt Tuning/P-Tuning.
P_TUNING or PROMPT_TUNING doesn't support generate functionality of transformers bcause generate strictly requires input_ids/decoder_input_ids but P_TUNING/PROMPT_TUNING appends soft prompt embeddings to input_embeds to create new input_embeds to be given to the model. Therefore, generate doesn't support this yet.

It would be great if you could provide a minimal reproducible example for the bug in training for the Prompt Tuning for the CLM models

@pacman100 but I see the method generate implemented in PeftModelForCausalLM. Am I missing something?

This problem will only occurs if padding is on left side. I think right side padded models should be fine

Hello @mayank31398, I don't think it would be a problem as those pad tokens are unattended and have 0 in the attention mask. With position embeddings:

for models employing ALiBi, the relative positions of <prompt> and <input_text> is maintained just that the weightage is reduced as the distance is increased due to intermediate <prompt> tokens.
for models employing sinusoidal or frozen learned encodings, they are just lookup tables. While pre-training, no <pad> tokens are observed due to fully packing the context window, so adding <pad> tokens itself to the left side should already be leading to different position embeddings for input ids as they are now shifted by certain lengths to the right.  So, this is unexplored territory.
I did a quick visualization of attention masks of prefix-tuning which is highly performant and has huge performance gains over prompt-tuning. prefix-tuning incorporates separate prompt tokens to each layer unlike prompt-tuning which only incorporates it at the start. It also supports generate method. Below screenshot shows that during inference it is able to attend to the virtual token embeddings properly.

Um, but its still not correct right?
For models that have learnt positional embeddings/ sinusoidal?
For instance, the positional embeddings assigned in the case 110011 vs 001111
pos embedding is different for the last 2 tokens in both cases.
in the first case, it gets pos_5 and pos_6 and in the second, it gets pos_3, pos_4

But is this also done during training?
I only see it being done during generation. prepare_inputs_for_generation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from ..peft import get_peft_model, TaskType
from ..peft.tuners.prompt_tuning import PromptTuningConfig, PromptTuningInit

model_name = "bigscience/bloom-560m"

model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

peft_model = get_peft_model(
    model,
    PromptTuningConfig(
        task_type=TaskType.CAUSAL_LM,
        prompt_tuning_init=PromptTuningInit.TEXT,
        num_virtual_tokens=8,
        prompt_tuning_init_text="Classify the sentiment of the sentence:",
        tokenizer_name_or_path=model_name,
    )
)

# no padding
def construct_vanilla_inputs(prompt, input, output):
    x = tokenizer(prompt + input)
    input_ids = x["input_ids"]
    attention_mask = x["attention_mask"]
    labels = tokenizer(output)["input_ids"]

    in_len = len(input_ids)

    input_ids = input_ids + labels
    attention_mask = attention_mask + [1] * len(labels)
    labels = [-100] * in_len + labels
    # labels = input_ids

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels
    }

# left padding
def modify1(input_ids, attention_mask, labels, n=100):
    input_ids = list(range(n)) + input_ids
    attention_mask = [0] * n + attention_mask
    labels = [-100] * n + labels

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels
    }

# middle padding
def modify2(input_ids, attention_mask, labels, n=100):
    input_ids = input_ids[:5] + list(range(n)) + input_ids[5:]
    attention_mask = attention_mask[:5] + [0] * n + attention_mask[5:]
    labels = labels[:5] + [-100] * n + labels[5:]

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels
    }

# right padding
def modify3(input_ids, attention_mask, labels, n=100):
    input_ids = input_ids + list(range(n))
    attention_mask = attention_mask + [0] * n
    labels = labels + [-100] * n

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels
    }

# peft test
def construct_peft_inputs(input, output):
    x = tokenizer(input)
    input_ids = x["input_ids"]
    attention_mask = x["attention_mask"]
    labels = tokenizer(output)["input_ids"]

    in_len = len(input_ids)

    input_ids = input_ids + labels
    attention_mask = attention_mask + [1] * len(labels)
    labels = [-100] * in_len + labels

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels
    }

def f(x: dict):
    for i in x:
        x[i] = torch.tensor([x[i]])
    return x

prompt = "Classify the sentiment of the sentence:"
input = " The movie sucks\nSentiment:"
output = " negative</s>"

loss = model(**f(construct_vanilla_inputs(prompt, input, output))).loss
print("no pad")
print(loss)

loss = model(**f(modify1(**construct_vanilla_inputs(prompt, input, output)))).loss
print("left pad")
print(loss)

loss = model(**f(modify2(**construct_vanilla_inputs(prompt, input, output)))).loss
print("middle pad")
print(loss)

loss = model(**f(modify3(**construct_vanilla_inputs(prompt, input, output)))).loss
print("right pad")
print(loss)

loss = peft_model(**f(construct_peft_inputs(input, output))).loss
print("peft")
print(loss)

The output is:
no pad
tensor(6.1106)
left pad
tensor(6.1107)
middle pad
tensor(6.1107)
right pad
tensor(6.1106)
peft
tensor(6.1106, grad_fn=<NllLossBackward0>)

Looks correct to me :)

Still a minute difference in there. Would you suggest to ignore it?

Also, another question @pacman100:
you mentioned that generation currently doesn't work for prompt tuning.
are there plans to support it?

Also, feel free to close this :)

Let me know if I can add this as a unit test or something if you prefer :)

Hello @mayank31398, thank you for the example script and as per it things look fine.
wrt generate functionality support, this commit should enable it
https://github.com/huggingface/peft/pull/55/commits/bdeb3778d0a35d0fd66f87f884e6f8fe32ecf705
outputs = model.generate(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"], max_new_tokens=10, eos_token_id=3)

I am seeing below result for one of the raft task using bloom-560m

All contributions are welcome wrt tests, features, documentation and examples 🤗🚀✨.
Please raise PRs with the tests that you have planned.
Thank you for taking the time to try this repo. Keep the feedback incoming!

Thanks. Closing this issue


Add reducers and actions

In this pull request, these are the major changes:

Added redux toolkit to the react project.
Wrote book's actions and reducer (in /src/redux/books/books.js) using the ducks pattern.
Wrote categories's actions and reducer (in /src/redux/categories/categories.js) using the ducks pattern.
Configure the Redux Store (/src/redux/configureStore.js).
Implement the add and remove book functionality.
set up hooks to update the UI when there is a state change

Hello mckent05, I have updated the action types to follow the ducks pattern


Volume randomly changed to Read-only file system

We have a pod with 2 volume mounts, both for the same volume:
        volumeMounts:
        - mountPath: /var/files
          name: vol1
          subPath: files
        - mountPath: /tmp/sessions
          name: vol1
          subPath: sessions

and a pod with one volume mount:
        volumeMounts:
        - mountPath: /var/files
          name: vol1
          subPath: files

Last week on wednesday we updated to Longhorn 0.4.0, everything worked fine.
On friday we suddenly lost write rights to our sessions subPath, other was still working.
We just got the message Read-only file system (30). I couldnt change the file rights with chmod (ubuntu) with the same error message. The other volume had normal write rights.
Also the Pod wouldnt redeploy. So I removed the session mount and scaled it down to 0. Then everything started up again. I added the session-mount to the 2nd Pod and I had write rights again.
The Logs still spamming this:

I0311 09:19:11.406212 1 leaderelection.go:192] failed to acquire lease longhorn-system/external-attacher-leader-io-rancher-longhorn
I0311 09:19:19.329324 1 leaderelection.go:271] lock is held by csi-attacher-6fdc77c485-fjjdk and has not yet expired
I0311 09:19:19.329347 1 leaderelection.go:192] failed to acquire lease longhorn-system/external-attacher-leader-io-rancher-longhorn
I0311 09:19:28.092038 1 leaderelection.go:271] lock is held by csi-attacher-6fdc77c485-fjjdk and has not yet expired
I0311 09:19:28.092055 1 leaderelection.go:192] failed to acquire lease longhorn-system/external-attacher-leader-io-rancher-longhorn
I0311 09:19:34.462578       1 leaderelection.go:271] lock is held by csi-attacher-6fdc77c485-fjjdk and has not yet expired

Some Data:
Kubernetes:  v1.12.3
Rancher: v2.1.4
Nodes: 1
Longhorn: 0.4.0

Hi @XzenTorXz
Those error messages you saw are expected since attachers/provisioners are running in leader election mode.
Can you send me the support bundle? You can find my email on my profile page.

I get a 504 Gateway Time-out,  if I try to create a support bundle. Is there anyway i can increase the nginx timelimit ?
There is no crashed entry, this is the log from around the happening:
time="2019-03-08T14:43:49Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T14:46:16Z" level=debug msg="websocket: open" id=6465732517898813829 type=nodes
time="2019-03-08T14:46:16Z" level=debug msg="websocket: open" id=2146011462876558535 type=volumes
time="2019-03-08T14:46:16Z" level=debug msg="websocket: open" id=2340821296862669722 type=settings
time="2019-03-08T14:46:16Z" level=debug msg="websocket: open" id=4966883895305377330 type=events
time="2019-03-08T14:46:29Z" level=debug msg="websocket: open" id=982373595318644533 type=engineimages
time="2019-03-08T14:51:48Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:00:12Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=2146011462876558535 type=volumes
time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=2340821296862669722 type=settings
time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=6465732517898813829 type=nodes
time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=4966883895305377330 type=events
time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=982373595318644533 type=engineimages
time="2019-03-08T15:01:33Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:01:33Z" level=debug msg="Detaching volume pvc-6910751ffa3411e8 from node1"
time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8 due to conflict"
time="2019-03-08T15:01:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-e-02ff87fe\", UID:\"6952ced4-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933193\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-e-02ff87fe"
E0308 15:01:34.211227 1 engine_controller.go:542] failed to update engine pvc-6910751ffa3411e8-e-02ff87fe to stop monitoring: failed to reste engine status for : Operation cannot be fulfilled on engines.longhorn.rancher.io "pvc-6910751ffa3411e8-e-02ff87fe": the object has been modified; please apply your changes to the latest version and try again
time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
time="2019-03-08T15:01:34Z" level=debug msg="Stop monitoring pvc-6910751ffa3411e8-e-02ff87fe"
time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-ad49d8cf\", UID:\"696522c9-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933220\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-ad49d8cf"
time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-4fbabcfd\", UID:\"6968758d-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933221\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-4fbabcfd"
time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-181578bc\", UID:\"695e41cd-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933222\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-181578bc"
time="2019-03-08T15:01:39Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
time="2019-03-08T15:01:39Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-4fbabcfd due to conflict"
time="2019-03-08T15:01:40Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-181578bc due to conflict"
time="2019-03-08T15:01:40Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
time="2019-03-08T15:01:40Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:01:41Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:01:44Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:01:48Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:01:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8\", UID:\"6939df48-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933194\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-6910751ffa3411e8 has been detached"
time="2019-03-08T15:02:16Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:03:52Z" level=debug msg="Attaching volume pvc-6910751ffa3411e8 to node1"
time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-ad49d8cf\", UID:\"696522c9-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933607\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-ad49d8cf"
time="2019-03-08T15:03:53Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-181578bc\", UID:\"695e41cd-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933608\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-181578bc"
time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-4fbabcfd\", UID:\"6968758d-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933609\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-4fbabcfd"
time="2019-03-08T15:03:54Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:03:55Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-181578bc starts running, IP <IP_ADDRESS>"
time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-ad49d8cf starts running, IP <IP_ADDRESS>"
time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-4fbabcfd starts running, IP <IP_ADDRESS>"
time="2019-03-08T15:04:01Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-e-02ff87fe\", UID:\"6952ced4-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933689\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-e-02ff87fe"
time="2019-03-08T15:04:01Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
time="2019-03-08T15:04:02Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
time="2019-03-08T15:04:08Z" level=debug msg="Instance pvc-6910751ffa3411e8-e-02ff87fe starts running, IP <IP_ADDRESS>"
time="2019-03-08T15:04:08Z" level=debug msg="Start monitoring pvc-6910751ffa3411e8-e-02ff87fe"
time="2019-03-08T15:04:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8\", UID:\"6939df48-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933610\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-6910751ffa3411e8 has been attached to node1"
time="2019-03-08T15:04:48Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:05:16Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:14:48Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"
time="2019-03-08T15:16:16Z" level=debug msg="Requeue longhorn-system/node1 due to conflict"

I saw the following in the log:

time="2019-03-08T15:01:33Z" level=debug msg="Detaching volume pvc-6910751ffa3411e8 from node1"

It seems somehow volume was asked to be detached, then was attached again 2 minutes later.

time="2019-03-08T15:03:52Z" level=debug msg="Attaching volume pvc-6910751ffa3411e8 to node1"

If it's not asked to be detached in the UI, then CSI driver plugin should have some records of the request.
For the support bundle, see https://github.com/rancher/longhorn/issues/414#issuecomment-472131576 for the workaround.

I have these issues as well. At some point, it tells me the volume is read-only.
@yasker I did send you the bundle with latest errors.
Thx!

So, if volumes are in Faulted state, will there be any attempt to somehow repair this volume, reinitialize them? Some kind of self heal?

@botzill Can you file a new issue for volume fail into "faulted" state in your setup? Your issue involved a faulted volume, which is different from this issue.
After a volume became "faulted", you can use salvage command from UI to recover the volume.
Though it's unknown why your volume became faulted so frequently. Normally it's due to loss of network connectivity that results in Longhorn engine cannot talk to the replicas, so the engine has to mark them as faulted. Once all the replicas became faulted, engine itself has no active replica to use so it became faulted itself.
Normally DigitalOcean's network is fine and we've been developing on it for quite some time. I am not sure if it's because of their kubernetes cluster offering or something else.
We may want to put some network condition detection mechanism in place for the user to easily decide what's the root cause of the outage.
Followings are one basic way to monitor the network outage:
We can run three small pods per node, each on one node (by setting nodename in the pod), and make them ping Longhorn manager on the all the nodes. Then when network outage happens, we can take a look at those logs to see if it’s only Longhorn or the whole kubernetes network on the node goes down
Also, on each node, use screen to run ping towards the other two nodes, put output to a file and check the result when the outage happens
if only the test in Kubernetes pod failed, that’s kubernetes overlay network issue; if both failed, it’s a vendor VMnetwork issue

Hi @yasker and thx for this complete guide!
For now, I did upgrade k8s cluster on DO to latest version plus I upgraded to more powerful droplets. I did the same tests and now it works OK. Will see how it goes with this setup and let you know if any issues.
Thx!

I just sendet you the report bundle i created 6 days ago. Its 160mb so i put it on google drive.

@XzenTorXz I've received your log.
It's Kubernetes asked the volume pvc-6910751ffa3411e8 to be detached:

longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:00:59.444584041Z time="2019-03-08T15:00:59Z" level=info msg="NodeServer NodeUnpublishVolume req: volume_id:"pvc-6910751ffa3411e8" target_path:"/var/lib/kubelet/pods/9d794d85-41b2-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6910751ffa3411e8/mount" "
longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:00:59.476006107Z time="2019-03-08T15:00:59Z" level=debug msg="NodeUnpublishVolume: done pvc-6910751ffa3411e8"
longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:01:31.132849908Z time="2019-03-08T15:01:31Z" level=info msg="NodeServer NodeUnpublishVolume req: volume_id:"pvc-6910751ffa3411e8" target_path:"/var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6910751ffa3411e8/mount" "
longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:01:31.445714976Z time="2019-03-08T15:01:31Z" level=debug msg="NodeUnpublishVolume: done pvc-6910751ffa3411e8"
longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:01:33.810995861Z time="2019-03-08T15:01:33Z" level=info msg="ControllerServer ControllerUnpublishVolume req: volume_id:"pvc-6910751ffa3411e8" node_id:"node1" "

Then it asked it to be attached again:

longhorn-csi-plugin-26jrd/longhorn-csi-plugin.log:2019-03-08T15:03:52.054965314Z time="2019-03-08T15:03:52Z" level=info msg="ControllerServer ControllerPublishVolume req: volume_id:"pvc-6910751ffa3411e8" node_id:"node1" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_attributes:<key:"baseImage" value:"" > volume_attributes:<key:"fromBackup" value:"" > volume_attributes:<key:"numberOfReplicas" value:"3" > volume_attributes:<key:"staleReplicaTimeout" value:"30" > volume_attributes:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1544195145047-8081-rancher.io/longhorn" > "

Is this the volume we have trouble with? If Kubernetes initiated the request, it should be fine for the pod.

I'm sorry for not clearing this up. The volume that changed state to write only is pvc-6910751ffa3411e8
so this might be relevant in the lonhorn-manager.log
2019-03-08T15:00:21.925069099Z time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=2146011462876558535 type=volumes
2019-03-08T15:00:21.925108559Z time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=2340821296862669722 type=settings
2019-03-08T15:00:21.925115498Z time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=6465732517898813829 type=nodes
2019-03-08T15:00:21.925121133Z time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=4966883895305377330 type=events
2019-03-08T15:00:21.925126836Z time="2019-03-08T15:00:21Z" level=debug msg="websocket: close 1001 (going away)" id=982373595318644533 type=engineimages
2019-03-08T15:01:33.97144653Z time="2019-03-08T15:01:33Z" level=debug msg="Detaching volume pvc-6910751ffa3411e8 from node1"
2019-03-08T15:01:34.023209596Z time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8 due to conflict"
2019-03-08T15:01:34.039650015Z time="2019-03-08T15:01:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-e-02ff87fe\", UID:\"6952ced4-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933193\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-e-02ff87fe"
2019-03-08T15:01:34.211446307Z E0308 15:01:34.211227       1 engine_controller.go:542] failed to update engine pvc-6910751ffa3411e8-e-02ff87fe to stop monitoring: failed to reste engine status for : Operation cannot be fulfilled on engines.longhorn.rancher.io "pvc-6910751ffa3411e8-e-02ff87fe": the object has been modified; please apply your changes to the latest version and try again
2019-03-08T15:01:34.302605988Z time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
2019-03-08T15:01:34.305145046Z time="2019-03-08T15:01:34Z" level=debug msg="Stop monitoring pvc-6910751ffa3411e8-e-02ff87fe"
2019-03-08T15:01:34.377165971Z time="2019-03-08T15:01:34Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
2019-03-08T15:01:39.572412902Z time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-ad49d8cf\", UID:\"696522c9-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933220\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-ad49d8cf"
2019-03-08T15:01:39.682649568Z time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-4fbabcfd\", UID:\"6968758d-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933221\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-4fbabcfd"
2019-03-08T15:01:39.735768001Z time="2019-03-08T15:01:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-181578bc\", UID:\"695e41cd-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933222\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-6910751ffa3411e8-r-181578bc"
2019-03-08T15:01:39.950212817Z time="2019-03-08T15:01:39Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
2019-03-08T15:01:39.95278367Z time="2019-03-08T15:01:39Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-4fbabcfd due to conflict"
2019-03-08T15:01:40.127874877Z time="2019-03-08T15:01:40Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-181578bc due to conflict"
2019-03-08T15:01:40.216571792Z time="2019-03-08T15:01:40Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
2019-03-08T15:01:53.925989046Z time="2019-03-08T15:01:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8\", UID:\"6939df48-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933194\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-6910751ffa3411e8 has been detached"
2019-03-08T15:03:52.11252102Z time="2019-03-08T15:03:52Z" level=debug msg="Attaching volume pvc-6910751ffa3411e8 to node1"
2019-03-08T15:03:53.312547395Z time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-ad49d8cf\", UID:\"696522c9-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933607\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-ad49d8cf"
2019-03-08T15:03:53.465759809Z time="2019-03-08T15:03:53Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-r-ad49d8cf due to conflict"
2019-03-08T15:03:53.504720718Z time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-181578bc\", UID:\"695e41cd-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933608\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-181578bc"
2019-03-08T15:03:53.704921642Z time="2019-03-08T15:03:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-r-4fbabcfd\", UID:\"6968758d-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933609\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-r-4fbabcfd"
2019-03-08T15:04:01.309230333Z time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-181578bc starts running, IP <IP_ADDRESS>"
2019-03-08T15:04:01.473246138Z time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-ad49d8cf starts running, IP <IP_ADDRESS>"
2019-03-08T15:04:01.850848432Z time="2019-03-08T15:04:01Z" level=debug msg="Instance pvc-6910751ffa3411e8-r-4fbabcfd starts running, IP <IP_ADDRESS>"
2019-03-08T15:04:01.904948978Z time="2019-03-08T15:04:01Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-6910751ffa3411e8-e-02ff87fe\", UID:\"6952ced4-fa34-11e8-a6d1-c860005ec7a4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"12933689\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6910751ffa3411e8-e-02ff87fe"
2019-03-08T15:04:01.975536777Z time="2019-03-08T15:04:01Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
2019-03-08T15:04:02.007913604Z time="2019-03-08T15:04:02Z" level=debug msg="Requeue longhorn-system/pvc-6910751ffa3411e8-e-02ff87fe due to conflict"
2019-03-08T15:04:08.435730128Z time="2019-03-08T15:04:08Z" level=debug msg="Instance pvc-6910751ffa3411e8-e-02ff87fe starts running, IP <IP_ADDRESS>"
2019-03-08T15:04:08.684989158Z time="2019-03-08T15:04:08Z" level=debug msg="Start monitoring pvc-6910751ffa3411e8-e-02ff87fe"

The volume u mention, was bount to the 2nd pod, that i tried to take down so i can reatache the problem volume.

I just got really confused with the volumes, since the hash looks so simular.
I'm sorry for not clearing this up. The volume that changed state to write only is pvc-6911aebbfa3411e8.
The only relevant log i could find is in the csi-plugin log:
2019-03-08T14:40:08.005689084Z E0308 14:40:08.005520       1 mount_linux.go:151] Mount failed: exit status 32
2019-03-08T14:40:08.005717961Z Mounting command: mount
2019-03-08T14:40:08.005725122Z Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-6911aebbfa3411e8 /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005731609Z Output: mount: /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted or /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount busy
2019-03-08T14:40:08.005737783Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005742956Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/init-import/0
2019-03-08T14:40:08.005746872Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005750746Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/0
2019-03-08T14:40:08.005754482Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/1
2019-03-08T14:40:08.021707421Z E0308 14:40:08.009532       1 utils.go:100] GRPC error: rpc error: code = Internal desc = mount failed: exit status 32

then this is repeated.
The volume u mention, was bount to the 2nd pod, that i tried to take down so i can reatache the problem volume.

It's a race condition that result in two mount called at the same time:
2019-03-08T14:40:07.58922291Z time="2019-03-08T14:40:07Z" level=info msg="NodeServer NodePublishVolume req: volume_id:\"pvc-6910751ffa3411e8\" target_path:\"/var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6910751ffa3411e8/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volum
e_attributes:<key:\"baseImage\" value:\"\" > volume_attributes:<key:\"fromBackup\" value:\"\" > volume_attributes:<key:\"numberOfReplicas\" value:\"3\" > volume_attributes:<key:\"staleReplicaTimeout\" value:\"30\" > volume_attributes:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1544195145047-8081-rancher.io/longhorn\" > "
2019-03-08T14:40:07.603448852Z time="2019-03-08T14:40:07Z" level=info msg="NodeServer NodePublishVolume req: volume_id:\"pvc-6911aebbfa3411e8\" target_path:\"/var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volu
me_attributes:<key:\"baseImage\" value:\"\" > volume_attributes:<key:\"fromBackup\" value:\"\" > volume_attributes:<key:\"numberOfReplicas\" value:\"3\" > volume_attributes:<key:\"staleReplicaTimeout\" value:\"30\" > volume_attributes:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1544195145047-8081-rancher.io/longhorn\" > "
2019-03-08T14:40:07.99977213Z time="2019-03-08T14:40:07Z" level=debug msg="NodePublishVolume: done pvc-6910751ffa3411e8"
2019-03-08T14:40:08.005689084Z E0308 14:40:08.005520       1 mount_linux.go:151] Mount failed: exit status 32
2019-03-08T14:40:08.005717961Z Mounting command: mount
2019-03-08T14:40:08.005725122Z Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-6911aebbfa3411e8 /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005731609Z Output: mount: /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted or /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount busy
2019-03-08T14:40:08.005737783Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005742956Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/init-import/0
2019-03-08T14:40:08.005746872Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.005750746Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/0
2019-03-08T14:40:08.005754482Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/1
2019-03-08T14:40:08.005758309Z
2019-03-08T14:40:08.021707421Z E0308 14:40:08.009532       1 utils.go:100] GRPC error: rpc error: code = Internal desc = mount failed: exit status 32
2019-03-08T14:40:08.021733136Z Mounting command: mount
2019-03-08T14:40:08.021739995Z Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-6911aebbfa3411e8 /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.021757287Z Output: mount: /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted or /var/lib/kubelet/pods/10d2efe3-41b0-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount busy
2019-03-08T14:40:08.021764491Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.021771085Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/8bf9e7bc-fa36-11e8-a6d1-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/init-import/0
2019-03-08T14:40:08.02177817Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volumes/kubernetes.io~csi/pvc-6911aebbfa3411e8/mount
2019-03-08T14:40:08.021785069Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/0
2019-03-08T14:40:08.021791951Z        /dev/longhorn/pvc-6911aebbfa3411e8 is already mounted on /var/lib/kubelet/pods/81caa641-40cf-11e9-bdcf-c860005ec7a4/volume-subpaths/pvc-6911aebbfa3411e8/prod-impro/1
2019-03-08T14:40:08.021798623Z

According to the CSI spec, the mount call should be idempotent, so the second mount call to the same place shouldn't fail.
I suspect there is a bug in our code for detecting mounted or not. https://github.com/rancher/longhorn-manager/blob/master/csi/node_server.go#L37
Would investigate more.

it happened while the pod was allready running. Also one subpath of the volume was working correctly.
I hope this helps.

We observe the same behaviour with longhorn v0.6.1. Our prometheus pods that use longhorn volumes fail randomly because the mount becomes ReadOnly (not during mount time but while pod is running and the volume is already mounted). Some errors I collected form instance-manager-e instance that is running on the same node around the time that this happens:
[pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3] fatal error: concurrent map read and map write
receive_msg: fail to read magic version
response_process: Receive response returned error
lh_client_close_conn: Closing connection
lh_client_close_conn: Cancel request 44534 due to disconnection
tgtd: bs_longhorn_request(91) fail to write at 10db62000 for 4096
tgtd: bs_longhorn_request(150) io error 0x1ff5f80 2a -14 4096<PHONE_NUMBER>, Success
[pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3]

then we get a bunch of stack traces:
goroutine 506743 [running]:
runtime.throw(0xc52f20, 0x21)
        /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc422f1e4b0 sp=0xc422f1e490 pc=0x42ad11
runtime.mapaccess2(0xb587e0, 0xc4202c2f60, 0xc423037a00, 0xc423037a00, 0x10)
        /usr/local/go/src/runtime/hashmap.go:409 +0x225 fp=0xc422f1e4f8 sp=0xc422f1e4b0 pc=0x408b95
reflect.mapaccess(0xb587e0, 0xc4202c2f60, 0xc423037a00, 0xc4202c2f60)
        /usr/local/go/src/runtime/hashmap.go:1195 +0x3f fp=0xc422f1e530 sp=0xc422f1e4f8 pc=0x40ae4f
reflect.Value.MapIndex(0xb587e0, 0xc42054ac60, 0x195, 0xb18240, 0xc423037a00, 0x98, 0xb18240, 0xc4204428b0, 0x98)
        /usr/local/go/src/reflect/value.go:1081 +0x126 fp=0xc422f1e5b8 sp=0xc422f1e530 pc=0x4b5026
github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto.makeMapMarshaler.func2(0xc42054ac60, 0x1, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto/table_marshal.go:2321 +0x22a fp=0xc422f1e698 sp=0xc422f1e5b8 pc=0x79cbda
github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto.(*marshalInfo).size(0xc420214a80, 0xc42054ac60, 0x131c200)
        /go/src/github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto/table_marshal.go:183 +0xaa fp=0xc422f1e700 sp=0xc422f1e698 pc=0x77481a
github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto.(*InternalMessageInfo).Size(0x131c200, 0xcc2080, 0xc42054ac60, 0x7fc5b8599378)
        /go/src/github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto/table_marshal.go:125 +0x65 fp=0xc422f1e730 sp=0xc422f1e700 pc=0x774425
github.com/longhorn/longhorn-engine/controller/rpc/pb.(*BackupReplicaMap).XXX_Size(0xc42054ac60, 0xcc2080)
        /go/src/github.com/longhorn/longhorn-engine/controller/rpc/pb/controller.pb.go:958 +0x43 fp=0xc422f1e760 sp=0xc422f1e730 pc=0x91b0e3
github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto.(*Buffer).Marshal(0xc42054a548, 0xcc2080, 0xc42054ac60, 0xc420694b40, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/github.com/golang/protobuf/proto/table_marshal.go:2740 +0x8a fp=0xc422f1e830 sp=0xc422f1e760 pc=0x78759a
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/encoding/proto.marshal(0xbdab20, 0xc42054ac60, 0xc42054a540, 0x0, 0x0, 0x0, 0x0, 0xc420154080)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/encoding/proto/proto.go:59 +0xda fp=0xc422f1e880 sp=0xc422f1e830 pc=0x7f69fa
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/encoding/proto.codec.Marshal(0xbdab20, 0xc42054ac60, 0x453f60, 0xc4205c1080, 0x40e55d, 0xc4203aa000, 0xb749e0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/encoding/proto/proto.go:74 +0xaf fp=0xc422f1e900 sp=0xc422f1e880 pc=0x7f6b4f
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x13440b0, 0xbdab20, 0xc42054ac60, 0x5, 0xc42029a420, 0xc422f1ea10, 0x411439, 0xc42054ac60)
        <autogenerated>:1 +0x48 fp=0xc422f1e948 sp=0xc422f1e900 pc=0x7f73b8
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.encode(0x7fc5c9833210, 0x13440b0, 0xbdab20, 0xc42054ac60, 0x13440b0, 0xc42024a5a0, 0xc420330f00, 0x46e5be, 0x411439)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/rpc_util.go:543 +0x61 fp=0xc422f1e9d0 sp=0xc422f1e948 pc=0x833511
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.(*Server).sendResponse(0xc4204b2d80, 0xcc8180, 0xc42032f500, 0xc420142600, 0xbdab20, 0xc42054ac60, 0x0, 0x0, 0xc420352f20, 0x0, ...)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:837 +0x89 fp=0xc422f1eae8 sp=0xc422f1e9d0 pc=0x838569
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4204b2d80, 0xcc8180, 0xc42032f500, 0xc420142600, 0xc4203077d0, 0x12d66a8, 0x0, 0x0, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:1033 +0x5ad fp=0xc422f1ede0 sp=0xc422f1eae8 pc=0x83932d
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4204b2d80, 0xcc8180, 0xc42032f500, 0xc420142600, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:1278 +0xe02 fp=0xc422f1ef80 sp=0xc422f1ede0 pc=0x83ce72
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4203ee830, 0xc4204b2d80, 0xcc8180, 0xc42032f500, 0xc420142600)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:717 +0x9f fp=0xc422f1efb8 sp=0xc422f1ef80 pc=0x84865f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc422f1efc0 sp=0xc422f1efb8 pc=0x458051
created by github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:715 +0xa1

and after it all calms we constantly see:
goroutine 506744 [select]:
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc4202ec9c0, 0x1, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x133
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc4203edda0, 0x0, 0x0)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435[pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3]  +0x1a8
github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc42032f800)
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xda
created by github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
        /go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xf0f
[longhorn-instance-manager] time="2019-10-04T01:50:39Z" level=info msg="Process Manager: process pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3 error out, error msg: exit status 2"
[longhorn-instance-manager] time="2019-10-04T01:50:39Z" level=debug msg="Process update: pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3: state error: Error: exit status 2"
[longhorn-instance-manager] time="2019-10-04T01:50:39Z" level=debug msg="Process update: pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3: state error: Error: exit status 2"
[longhorn-instance-manager] time="2019-10-04T01:50:39Z" level=debug msg="Process update: pvc-69ef44b0-1b94-426b-a065-cb7f5094a8b8-e-0cc6736f-e79ab6f3: state error: Error: exit status 2"

and the volume stays in ReadOnly mode. I was able to recover by deleting the pod and re attaching volumes through the ui

remote stale

FYI: the stack trace issue was solved in #790 , it was a protobuf bug.

Hello,
I currently have the same or a very similar problem with Longhorn v1.2.4.
Several volumes suddenly go read-only. Only deleting and recreating the pods helps to make the volumes rw again.
In the manager log you can see that there are timeouts after 8s.
I have created a support bundle. Can someone take a look at it?


🧼 Some small cleanups

Remove a ts-ignore or two
Remove unused reference to node-forge
remove some logs

@ivanovSPvirtru This is code I think you added to add compatibility with file streams? It looks like the lstat path is dead code or at least that is what the static types indicate


Avoid calling http_head for non-HTTP URLs

Avoid calling http_head for non-HTTP URLs, by adding and else statement.
Currently, it makes an unnecessary HTTP call (which adds latency) for non-HTTP protocols, like FTP, S3,...
I discovered this while working in an unrelated issue.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Role-based provisioning does not work properly when defined in Camel case

Role-based provisioning does not work properly when the Identity Provider OutBound Provisioning Roles is defined in Camel Case.

Similar issues
https://github.com/wso2/product-is/issues/3689

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Distinguish run-time errors in user's code, from errors in our code.

This (incorrect) user code:

resulted in this message on the Elan console:

We really need to distinguish between errors that are from our code, and those that are resulting from running the user program. I realise this may be hard!

Duplicates #551


Models

Hi, thanks for releasing Samba! Are there any plans to release the pretrained models? Thanks!

Yep! It would be great to see the 3B and 7B or 8B models...

Releasing of the 3.8B insturction-tuned model is on the plan! We may also release smaller base models like Samba-421M and Samba-1.3B trained on SlimPajama (hopefully in a week or two). We currently don't have the plan to train a larger 7B model due to the shortage of GPUs. 🥲

Nice! Please release the base models and smaller models!

Releasing of the 3.8B insturction-tuned model is on the plan! We may also release smaller base models like Samba-421M and Samba-1.3B trained on SlimPajama (hopefully in a week or two). We currently don't have the plan to train a larger 7B model due to the shortage of GPUs. 🥲

This architecture looks great.
How much GPU time would be required to train a 7B and 14B model? In your opinion, would it be able to beat a Llama 3 70B transformer model in the benchmarks?

Samba-421M

I used Microsoft's Deberta-V3 models A LOT in different projects, because they are so small and I can run quick experiments with them at home. So I am really looking forward to a new small model :)

After the internal business review, we are sorry to hear that we can not release the Samba 421M and 1.3B models trained on SlimPajama. This is because the SlimPajama dataset contains the Books3 dataset which has copyright infringement.🥲 We will continue to push the releasing of Samba 1.7B and 3.8B models trained on the Phi datatsets.

Releasing of the 3.8B insturction-tuned model is on the plan! We may also release smaller base models like Samba-421M and Samba-1.3B trained on SlimPajama (hopefully in a week or two). We currently don't have the plan to train a larger 7B model due to the shortage of GPUs. 🥲

This architecture looks great. How much GPU time would be required to train a 7B and 14B model? In your opinion, would it be able to beat a Llama 3 70B transformer model in the benchmarks?

It depends on how many tokens we want to train it on. The Samba 3.8B model takes around the same amount of GPU time as the Phi3 models. I personally think it is definitely possible to beat Llama3 70B on benchmarks with better data mixtures customized for a 14B model.


Solution for developers to interact more in real-time

Currently we only have GitHub available and communicate via issues and pull requests. It might be beneficial to discuss some aspects such as features and who is going to work on what more interactively in a chat.
This issue can be used to discuss options and decide on which solution to use.

I've setup a Matrix instance (not completely working yet) which looks good to me.
The only "rule" I want to have expressed before we use that is, that we only use the chat for non public discussions that should not take place within the issues. Everything else must be done via issues to keep the development as transparent as possible.
Let me know what you think!

@kbabioch @gerw @BenPru
Feel free to register an account over at https://matrix.bouni.de/
I have the registration open for now but will disable it as soon as all of you have an account.
I think it should be possible to join with an existing matrix account on another instance but I have not tried that.

Update: I enabled Login via Github, that should give us Logins like @kbabioch:matrix.bouni.de for example

@kbabioch I've sent you an invite, you should be able to accept it and then enter the room


Wiremock stops responding after running for a while

I am running wiremock in standalone mode i have exposed 7 services each on different port to mock different external systems we interface with.  We use this server in our development and QA environment. After running for a while we found java wiremock stops. I am not sure what is the issue
java -Xms1024m -Xmx1024m -jar wiremock-standalone-with-log-2.8.0.jar --port 1111 --root-dir D:/qa_wiremock/qa > 1111.log
I have enabled the log and found the following error
wiremock_1111.log:wiremock.org.eclipse.jetty.io.EofException: null
wiremock_1111.log:      at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.writeAndTranslateExceptions(WireMockHandlerDispatchingServlet.java:165)
wiremock_1111.log:Caused by: java.io.IOException: Broken pipe
wiremock_1111.log:05:41:07.221 [qtp849596823-21 - /user/2LDBPHXJ] DEBUG wiremock.org.eclipse.jetty.io.WriteFlusher - write exception
wiremock_1111.log:wiremock.org.eclipse.jetty.io.EofException: null
We are running in a amazon m4 large (2 vcpu, 8gb mem)

If you're keeping it running for a long time you'll need to limit the request journal, otherwise it keeps growing indefinitely and consumes all the heap space.
Please post further questions to the mailing list.


rotation matrix convertion to quaternion

you code validate rotation matrix must be orthogonal and its determinant must be precisely 1., which is not the case for many real datset like 7scenes.
I wonder if it's possible for you to ease the limitation for this? We can set a relative big number for np.allclose and np.isclose to tolerate many datasets.
Thank you

I'm also having problems with default thresholds for tolerance for when a matrix is considered orthogonal. Due to numeric errors I'm unable to construct many quaternions from a rotation matrix because the np.allclose tolerances are too restrictive for my use case. Relaxing them or letting the user adjust the tolerance seems like a good idea to me.

Thanks for the feedback. A number of people have raised this, issue.
It will be fixed in a future release, coming soon!
See https://github.com/KieranWynn/pyquaternion/pull/44

Glad to hear that.


Plat 4140

removed restrict version of Alamofire

Approved, but there are build failures to look at.

AT's are failing as Scott mentioned. Must be addressed before this is merged.

This is building for me and passing unit tests (ones that are enabled)

tested before and after running 'pod update'
@climbonbelay Can you show me your testing process and what AT's are failing?

Jeremiah - We were referring to the builds Travis runs on all pull requests. Looks like they all passed when they were re-run yesterday.


Moving average ignored a value of inf/nan

Hi, me again.
"Moving average ignored a value of inf/nan "
issue happens again with multi-GPUs!
Would you please take a look at this issue once again?

(prev issue link : https://github.com/dbolya/yolact/issues/186)
Thanks!

What was your command used to run this? And are you on the latest master?

Yes, it was the latest one.
But to make it clear, I just cloned the repository to newly created folder and changed the coco DB location in the config file so that I can use my own coco DB without moving it to the 'data' folder.
After that, I did the following.
export CUDA_VISIBLE_DEVICES=0,1,2,3
python train.py --config=yolact_base_config --batch_size=32
and it failed once again.

Wow that's quite weird. This is the unmodified base config? I'll see if I can reproduce this on monday.
And just to sanity check, does it happen with 1 gpu?

As I mentioned above, I changed the coco DB location in the config file so that I can use my coco DB without moving it to the 'data' folder.

Except that, the other values have the default values.
I started training with the following command.
export CUDA_VISIBLE_DEVICES=0
python train.py --config=yolact_base_config --batch_size=8
I'll let you know what happens.

Quick related question!
When do you suppose it is trained enough?

You mean as to what point it the loss shouldn't be able to explode anymore? That should be around epoch 1-2, but evidently it can still explode on epoch 2 as you found. I'm currently testing 4 GPUs on yolact_base_config, I'll let you know if I can reproduce this.

I trained it roughly for a day and it's safe!

With second attempt, loss exploded with single GPU :<

Ok this is an issue, I was able to reproduce this on 4 gpus. I'm currently testing on 1 gpu, but dang everything was working fine with pascal, no idea why it's not working with COCO.
I'll take a deeper look as to why this is happening and see if I can fix it.

I hope all of those problems that occurred in single/multi GPU cases are originated from the same flaw.
Thanks and  plz,
You are my only hope T-T

And what about gradient clipping?

Ok sorry about the delay (thanksgiving and all). The single GPU test came back successful and reproduced the mAP in the paper, so I guess the issue may be somewhere else? Now that I'm back, I'll investigate this further.

OK Thanks~

An update: I figured out the source of the random infs during training that you see every now and again. Augmentation could actually output detections with 0 bbox area, hence why some things are divided by 0 and thus produce infinity. Since I ignore batches with inf loss, this wouldn't be the cause of the explosion, but it might be boxes with really small area instead that's causing this explosion (since I don't filter those out).
Right now I'm trying out discarding any box with fewer than than 4 pixels of width and 4 pixels of height. I haven't seen a single inf or nan so far, so this looks promising. I'll update you once / if it finishes training.

That's good news!
And well, it seems like the problem that is not related to the multiple GPUs..?

@dbolya Thanks for your work. Do you finish the training that discarding any box with fewer than than 4 pixels of width and 4 pixels of height ?

It's epoch 49/54 but that's enough for me to verify that it reached the same mAP as in the paper with 4 GPUs and batch size 28 (so 7 / gpu) with no loss explosion! I'll push the code now so you guys can test.

Pushed the change. Let me know if I borked anything!

It works fine at epoch 45/54.
Thank you!


[QUESTION/HELP] k3d script k3d-entrypoint-dns.sh might have a problem

Hi,
I use k3d version 5.6.3 with rootless podman, version 4.6.1, and I had no problems creating k3d clusters for the last months but now suddenly during cluster creation the startup of the k3d containers fail.
I found out that startup of software in container is with script k3d-entrypoint.sh which first searches for additional scripts k3d-entrypoint-*.sh and executes them.
The problem I have is in script k3d-entrypoint-dns.sh. First part of the problem is the line
set -o errexit

This will end script execution immediately if a command returns with exit code not equal 0.
The second part of my problem are the lines
grep -q "${docker_dns}" /etc/resolv.conf
grepstatus=$?

Here if ${docker_dns} is not found in /etc/resolv.conf the grep command terminates with 1. This case is handled in the next lines. But this next line is never executed due to set -o errexit and so the script terminates and as a result the complete startup terminates.
My question: Is that as expected or should the startup script k3d-entrypoint-dns.sh be modified to be able to handle the result of the grepcommand?
Btw, set -o errexit is set in nearly every kde-entrypoint-*.sh script. Maybe there are additional code lines with similar behavior?
Best regards
Ulrich.

Additional information.
In my last post I wrote that start of cluster members failed suddenly. In the meantime I found the reason for that. Cluster member startup only fails if I have the official installation installed on my system (installed using curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash). If I build from branch main everything works fine. There is a difference in file k3d-entrypoint-dns.sh between branch main and main-v5.
From branch main (working in my case)
.
# Update resolv.conf to use the Gateway IP if needed: this will also make CoreDNS use it via k3s' default `forward . /etc/resolv.conf` rule in the CoreDNS config
if grep -q "${docker_dns}" /etc/resolv.conf; then
  echo "[$(date -Iseconds)] [DNS Fix] > Replacing IP in /etc/resolv.conf ..."
.

From branch main-v5
.
# Update resolv.conf to use the Gateway IP if needed: this will also make CoreDNS use it via k3s' default `forward . /etc/resolv.conf` rule in the CoreDNS config
grep -q "${docker_dns}" /etc/resolv.conf
grepstatus=$?
if test $grepstatus -eq 0; then
.

In the working case (call of grep within "if") the exit code of the grep is handled by the if and so set -o errexit has no impact.
Please consider to at least merge the content of entrypoint-dns.sh from branch main to main-v5.
Best regards
Ulrich

this was fixed in #1441


Date & Time: demo popups are plain strange under FF

http://w2ui.com/web/demos/#!fields/fields-2
Happens with FF 47 but not with recent Chrome or Chromium.
First click on a field and the popup points to the right field.
Then start using TAB and Shift-TAB to navigate around.  The popup always points to the last field, not the field being edited.  When going from calendar to time fields, it renders time, but pointing to the calendar field.

When returning to a date field from a time field, all of the formating is lost in the rendering.

But visiting the next field renders properly (but still pointing to the wrong field).

Unsure if related, but this is a HiDPI display (210x215 dpi).  Oh yeah, and Ubuntu.

I can reproduce this in FF 47.0 with w2ui 1.4
However, it seems to be fixed in w2ui 1.5 - http://jsfiddle.net/kjzyh43k/

There will be no more updates in 1.4 branch and 1.5 will be out by the end of the summer. Since it is not a problem in 1.5.x, I am clothing it. If someone needs it, please fix in 1.4 branch. Thank you.


Logging to stdout in JSON

Added the zerolog package to log to stdout in a structured JSON format.
Output contains severity level (Debug, Info, Error, or Panic)
Also has timestamp, file name, function name (fn), and line number.

Worth noting that I actually also just rebased & merged this into the develop branch.


Honor #, @, and + tags position within todo item text

This could be nitpicky, but I like to use @context, #tag and +project attributes within the body text of my todo items. However the extension automatically moves any attributes to the end of the line.
For example, the todo item: Move +Project items to other #buckets would render as: Move items to other #buckets +Project  (which admittedly, while not perfect, is comprehensible).
But, since I like to use the @context attribute for colleagues' names, the item: Speak to @Context about +Project #issue. becomes; Speak to about #issue +Project @Context, which is a bit harder to parse than the original.
Is it possible to override this automatic attribute repositioning? It's probably a trivial change if you know CSS (or JS?) but unfortunately I don't.
If not, is it possible to have an override option added?

Yes, it should be the default to honor the position.
There are probably 3 different places in code that move the text with different logic: 1) Tree View 2) Webview 3) Editor hover / Tree View Hover.

It should work in 2.9.0.


(init): Server account password set

[ ] Ask for server password
[ ] Hash and overwrite db password
[ ] Store in env for system use
[ ] Confirm success

table: user_credentials


code-victor-now's file

I have created a branch, created a file and made a commit, and opened a pull request

trying again


upgrade louvain and scib

scIB now supports louvain 0.8 This removes our conflicting igraph deps which have caused flakes.

Closes #682


Richard vkcube fix

Fixed crash on exit for vkcube due to
    CVDisplayLinkRelease(displayLink);

Being called twice on shutdown. Removed redundant call that was added recently.

CI Vulkan-Tools build # 451 running.

CI Vulkan-Tools build # 451 passed.


Add option to pass index template for ES

as per: https://github.com/uken/fluent-plugin-elasticsearch#template_name
This will enable users to have more control over the indices created on elasticsearch by fluentd.

This patch will be merged after merged https://github.com/fluent/fluentd/pull/2685

@repeatedly updated my PR to use the syntax added in 1.8.0.


On "Public Distributions" page, create a seperate "Official FLDC Distributions" list

Please create a seperate list of distributions which is populated exclusively with the completed distributions of username FoldingCoin, which corresponds with email address<EMAIL_ADDRESS>

3 SP


The About window is a bit messy

In Godot 3.0 master, the About window shouldn't have two nested scrollbars, I find it too much for such a window. Also, it draws a rectangle going out of bounds:

... and  the version label is missing the git commit sha:

Fixed in the PR.
Focus rectangle issue is reported here: #9612
I'm not sure what happened with the upper-right text. It should be as the following screenshot (without hash in the current version):

The rectangle bug is still present:

Yes, #9612 is still open. Need to fix where the hightlight is drawn.