Decompressor returns -5 although decompresses everything or even more

Tried that library and trying to decompress compressed data decompressor returned -5. Later investigation shown that it decompressed everything and even more.
With other string result was 100% correct decompression but function returned -5 as well.
#include <stdio.h>
#include <string.h>
#include "lzav.h"

int main()
{
    char input[] = "Which witch is which? Which witch is which?";
    char c_buf[100] = {0}, d_buf[100] = {0};
    int c_len, d_len;

    printf("'%s'\n", input);
    printf("strlen=%d\n", strlen(input));

    ///static inline int lzav_compress_default( const void* const src, void* const dst, const int srcl, const int dstl );
    c_len = lzav_compress_default(input, c_buf, strlen(input), sizeof(c_buf));
    printf("c_len=%d\n", c_len);

    d_len = lzav_decompress(c_buf, d_buf, c_len, sizeof(d_buf));
    printf("d_len=%d\n", d_len);

    printf("'%s'\n", d_buf);
    printf("strlen=%d\n", strlen(d_buf));
    ;
    return 0;
}

$ cc -o lzav-1cd lzav-1cd.c; ./lzav-1cd
'Which witch is which? Which witch is which?'
strlen=43
c_len=39
d_len=-5
'Which witch is which? Which witch is which? Whic'
strlen=48

You have a bug in your code. The line
 d_len = lzav_decompress(c_buf, d_buf, c_len, sizeof(d_buf));

should really be
 d_len = lzav_decompress(c_buf, d_buf, c_len, strlen(input));

LZAV does not add a header, it produces a raw compressed stream. The uncompressed data size should be known: this also provides an additional safety check. Parameter description for lzav_decompress() specifically notes that "dstl" is an expected destination data length.

LZAV API is not equivalent to LZ4 nor Snappy. Expected decompressed length is usually always known in practice, because otherwise one can't allocate an in-memory buffer to hold all decompressed data.

You're right. For some reason I thought it was output buffer size.
That brings the question - it would be good to have some kind of 'end of steam' marker in case you don't know size of compressed steam. I know it's more like streaming feature, but still.

I understand your point, but in that case a safety check against real destination size would not be possible. LZ77 stream is really an unnumerated number of blocks, so expected destination length should be stored somewhere anyway. I'm not implementing a header in LZAV, because it's more useful to leave that part implementation-specific. For example, a custom header may store CRC value along with source and destination lengths. In my opinion, both LZ4 and Snappy have an unnecessary overhead in that respect, because a programmer has to store compressed data length anyway, independent of such in-memory compressor - it's more reasonable to store it with destination data length.


Resharding

@enisoc I want to fix more stuff (like finish the other Keyspace methods), but that's where I'm headed.
(not clear what should have an Impl or a Server yet, will simplify code as I find it...)

LGTM

OK I'm done. Now the topo implementations don't do any events. Good stopping point, I could fix a few more details but I'm getting tired.

Gonna merge and import to limit impact, I'll fix forward.


Outline-Client.AppImage on Ubuntu 18.04 Desktop Icon Missing?

Is this intended or I'm missing something?

Downloaded it from here: https://github.com/Jigsaw-Code/outline-client/releases

Full screenshot

Outline Manager App Image works fine.

Same issue on linux-v1.0.0-test1.
And it doesn't have an icon anymore on System Menu Bar.


Encoding png

How do I encode a png?
Every online byte array tool I've tried prints nothing to the display

It may not be the byte array tool, but the code you're using. Can you share a minimal example which reproduces the problem?
btw - you should use my image_to_c tool --> https://github.com/bitbank2/image_to_c

using this sketch, i get the panda on my display, i copy just the bits from online conversion and paste them after this: ``
#include <pgmspace.h> const uint8_t panda_png[] PROGMEM = {
into the panda_png.h file, but i only see a color changing background, no image
`
// This example renders a png file that is stored in a FLASH array
// using the PNGdec library (available via library manager).
// Include the PNG decoder library
#include <PNGdec.h>
#include "panda_png.h" // Image is stored here in an 8 bit array
PNG png; // PNG decoder inatance
#define MAX_IMAGE_WDITH 240 // Adjust for your images
int16_t xpos = 0;
int16_t ypos = 0;
// Include the TFT library https://github.com/Bodmer/TFT_eSPI
#include "SPI.h"
#include <TFT_eSPI.h>              // Hardware-specific library
TFT_eSPI tft = TFT_eSPI();         // Invoke custom library
//====================================================================================
//                                    Setup
//====================================================================================
void setup()
{
Serial.begin(115200);
Serial.println("\n\n Using the PNGdec library");
// Initialise the TFT
tft.begin();
tft.fillScreen(TFT_BLACK);
Serial.println("\r\nInitialisation done.");
}
//====================================================================================
//                                    Loop
//====================================================================================
void loop()
{
int16_t rc = png.openFLASH((uint8_t *)panda_png, sizeof(panda_png), pngDraw);
if (rc == PNG_SUCCESS) {
Serial.println("Successfully png file");
Serial.printf("image specs: (%d x %d), %d bpp, pixel type: %d\n", png.getWidth(), png.getHeight(), png.getBpp(), png.getPixelType());
tft.startWrite();
uint32_t dt = millis();
rc = png.decode(NULL, 0);
Serial.print(millis() - dt); Serial.println("ms");
tft.endWrite();
// png.close(); // not needed for memory->memory decode
}
delay(3000);
tft.fillScreen(random(0x10000));
}
//=========================================v==========================================
//                                      pngDraw
//====================================================================================
// This next function will be called during decoding of the png file to
// render each image line to the TFT.  If you use a different TFT library
// you will need to adapt this function to suit.
// Callback function to draw pixels to the display
void pngDraw(PNGDRAW *pDraw) {
uint16_t lineBuffer[MAX_IMAGE_WDITH];
png.getLineAsRGB565(pDraw, lineBuffer, PNG_RGB565_BIG_ENDIAN, 0xffffffff);
tft.pushImage(xpos, ypos + pDraw->y, pDraw->iWidth, 1, lineBuffer);
}`

What is the serial print output? Is png.openFLASH succeeding? Tell me what the return code is.
Perhaps you are trying to decode an interlaced PNG?

using this tool:
https://lvgl.io/tools/imageconverter
after copying the bits minus the headers into panda.h and uploading to teensy, serial says: Initialisation done.
nothing else,
if i copy the bits from your m5stickc_plus_test .h file, into the panda.h file
i get:
Initialisation done.
Successfully png file
image specs: (240 x 135), 8 bpp, pixel type: 2
22ms
and it displays that fine,
think i figured out the image_to_c terminal command
Lees-iMac:MacOS lee$ ./image_to_c 1.png > test.h
but im on MacOS 10.13 and get this error:
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
Referenced from: /Users/lee/Desktop/image_to_c-master/dist/MacOS/./image_to_c (which was built for Mac OS X 10.15)
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: ____chkstk_darwin
Referenced from: /Users/lee/Desktop/image_to_c-master/dist/MacOS/./image_to_c (which was built for Mac OS X 10.15)
will try on old windows machine

just rebuild it on linux/windows/macos from the command line (makefile)

Works great after running Makefile, thank you!


Bug: Inconsistent Python syntax highlighting

Description of the problem or steps to reproduce
Say, you have a statement that is a complex boolean expression:
v1 != "Dolor" or v2 != "Ipsum"
If the any part of the expression has a string ("Dolor", for example), then any comparision operators on that line coming after it will not have syntax highlighting:

Specifications

Commit hash: 225927b9
OS: Arch Linux 6.0.2-arch1-1
Terminal: Windows Terminal SSH connection, host has zsh

It sounds wierd but I think this is related to #2548.
It is also an operator after a string in the line.

This seems to be related to #1457.


[FEATURE REQUEST] Use namedtuple to represent sol, obj, beh, idx, meta

Description

Instead of returning a tuple of sol, obj, beh, idx, meta in archive methods like get_random_elite and elite_with_behavior, return a namedtuple called Elite. This will maintain backwards compatibility, but at the same time, people will no longer have to remember the order of items.

We should also check benchmarks to make sure the namedtuple is not slowing us down too much, as namedtuples are slower to construct than tuples.


Table Weather

Display the weather data of a certain place in a table

Developed simple homepage that shows the basic stats of New York


Package upgrades

Looks like these packages were upgraded two days ago. So I am closing this PR.
I will take a pull and start working on admin panel.


能不能集成AI助手

感谢您创建此问题！
然而，由于您没有使用英语，我们很难进行处理。如果可能，请您将此问题描述修改为英文。请您直接编辑此问题，不要提交回复或新问题。
请注意，机器翻译服务通常无法处理用于讨论软件问题所需的技术性语言，所以我们推荐您寻找一位熟悉相关技术的人来协助翻译。
如果您无法做到，没有关系。此问题已经被标记为需要翻译，等待社区成员志愿进行翻译。

Thanks for creating this issue!
Unfortunately, as the description is not in English, it's hard for us to work on. If possible, please edit the original issue to be in English.
In our experience, automatic translation services are not able to handle the technical language needed to effectively communicate about software issues, so we recommend finding a human translator who is familiar with our technology to help.
Otherwise, we have marked this as needing translation and a community member may be able to help out.

感谢您创建此问题！
然而，由于您没有使用英语，我们很难进行处理。如果可能，请您将此问题描述修改为英文。请您直接编辑此问题，不要提交回复或新问题。
请注意，机器翻译服务通常无法处理用于讨论软件问题所需的技术性语言，所以我们推荐您寻找一位熟悉相关技术的人来协助翻译。
如果您无法做到，没有关系。此问题已经被标记为需要翻译，等待社区成员志愿进行翻译。
Thanks for creating this issue!
Unfortunately, as the description is not in English, it's hard for us to work on. If possible, please edit the original issue to be in English.
In our experience, automatic translation services are not able to handle the technical language needed to effectively communicate about software issues, so we recommend finding a human translator who is familiar with our technology to help.
Otherwise, we have marked this as needing translation and a community member may be able to help out.

Issues: Can AI assistant be integrated?

@Metamotrix
据我目前所知，你可以使用 Github Copilot
As far as I know so far, you can use Github Copilot

Hey @joyceerhl, this issue might need further attention.
@Metamotrix, you can help us out by closing this issue if the problem no longer exists, or adding more information.

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


should fix windows 10 issue

I wasn't able to reproduce this on my Windows 10 VM, but this addresses some things I was able to glean from Joachim's screenshot.
Based on Joachim's screenshot, it seems that this line of code was generating the notification: https://github.com/kiteco/atom-plugin/blob/master/lib/notifications-center.js#L171, with arch being an empty string. The empty string case happens when the execution of the BAT file errors.
so what we did:

Instead of relying on the BAT file to determine the architecture, we rely on Node's os library.

cc @its-dhung @jansorg

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@76e6092). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master      #25   +/-   ##
=========================================
  Coverage          ?   76.87%
=========================================
  Files             ?       17
  Lines             ?      679
  Branches          ?        0
=========================================
  Hits              ?      522
  Misses            ?      157
  Partials          ?        0

Impacted Files
Coverage Δ

lib/support/windows.js
69.66% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76e6092...22b1d68. Read the comment docs.


Wrong registry entries SupemiumHTM

Installation creates filetype association entries SupemiumHTM (.HTM, .HTML, etc.) and HKEY_CLASSES_ROOT\SupemiumHTM key.
Instead of SupermiumHTM.

SupemiumHTM is not a typo. On Windows 8 and above, some registry key names used by the shell had an undocumented length limit imposed on them. "Supermium" was too long so it had to be shortened.

Ah, OK. Thanks. Those new Windows versions keep getting better and better...


[SPARK-31446][WEBUI][3.0] Make html elements for a paged table possible to have different id attribute.

What changes were proposed in this pull request?

This PR backports #28217 to branch-3.0, making each id attribute for page navigations in a page unique.
PagedTable#pageNavigation returns HTML elements representing a page navigation for a paged table.
In the current implementation, the method generates an id and it's used for id attribute for a set of elements for the page navigation.
But some pages have two page navigations so there are two set of elements where corresponding elements have the same id.
For example, there are two form-completedJob-table-page id in JobsPage.
Why are the changes needed?

Each id attribute should be unique in a page.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

I added a test case for pageNavigation extended.
I also manually tested that there were no warning messages for the uniqueness in JobsPage and JobPage.

Test build #121421 has started for PR 28246 at commit fe152e2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26105/
Test PASSed.

Thank you, @sarutak !

Test build #121421 has finished for PR 28246 at commit fe152e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121421/
Test PASSed.


E2E: go tag main SLSA3 config-ldflags assets tag

Repo: https://github.com/slsa-framework/example-package/tree/v20.0.18
Run: https://github.com/slsa-framework/example-package/actions/runs/2498486832
Workflow name: go tag main SLSA3 config-ldflags assets version
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.config-ldflags-assets-tag.slsa3.yml
Trigger: push
Branch: v20.0.18
Date: Tue Jun 14 22:26:46 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/v20.0.19
Run: https://github.com/slsa-framework/example-package/actions/runs/2498534014
Workflow name: go tag main SLSA3 config-ldflags assets tag
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.config-ldflags-assets-tag.slsa3.yml
Trigger: push
Branch: v20.0.19
Date: Tue Jun 14 22:38:17 UTC 2022
Tests are passing now. Closing this issue.

re-opening for testing


Module not found error

I am using IntelliJ 2019.2.3, I set my PATH_TO_FX to C:\Program Files\Java\javafx-sdk-13.0.1, my VM options to --module-path ${PATH_TO_FX} --add-modules javafx.controls,javafx.fxml, my library for javafx 13 has been added under project settings, and I am still getting the error message
"Error occurred during initialization of boot layer
java.lang.module.FindException: Module javafx.base not found"
I'm not sure if I am missing anything. Please help.

PATH_TO_FX should be pointing to the lib folder. In your case it should be C:\Program Files\Java\javafx-sdk-13.0.1\lib

I have the same bug When I search for my path in windows explorer it brings me to correct folders that contains javafx.base

@avacadoadam %PATH% and %PATH_TO_FX% are two different environmental variables; make sure you aren't getting confused between the two. If you want to be sure this is a bug, just hard code your module-path to your javafx-sdk-13.01\lib folder.
Below is a batch script with the hardcoded pathing. You can just copy and paste the below in a .bat file in the same directory containing HelloFX.java. Should run without any issues (assuming you have Java 13 installed).
javac --module-path "C:\Program Files\Java\javafx-sdk-13.0.1\lib" --add-modules javafx.controls HelloFX.java
java --module-path "C:\Program Files\Java\javafx-sdk-13.0.1\lib" --add-modules javafx.controls HelloFX

I had this same problem in Netbeans 11.2, even after having correctly set the VM options for Run in project properties to set the correct modules path.
The solution for me was to turn off 'Compile on Save', as mentioned in this reply to this closed issue
https://github.com/openjfx/openjfx-docs/issues/91#issuecomment-537354974
Might be good to add to the docs?

I am facing the same issue. I am using IDEA IntelliJ 2019.1.5 on Ubuntu 18.04. When I set PATH_TO_FX correctly to my local JFXdirectory/lib folder, I am facing the error below:
Error occurred during initialization of boot layer
java.lang.module.FindException: Module javafx.base not found

Facing the same issue in Windows10, Intellij IDEA version 2020.2
Output -
Error occurred during initialization of boot layer java.lang.module.FindException: Module javafx.base not found
Tried all ways, also added this folder to both PATH Variable and VM options javafx-sdk-11.0.2\lib
Still, no clue how to solve this.

Same problem as above. No clue what to do.

I am running JavaFX-15 with JDK-15 on Netbeans 12.0.
Go to https://openjfx.io/openjfx-docs/#install-javafx (follow the guide for "JavaFX and NetBeans -> Non-modular from IDE").
In Netbeans, turn off the "compile on save" option:
Project Properties -> Build -> Compiling ->> Compile on save (off)

This got rid of the "Error occurred during initialization of boot layer...  java.lang.module.FindException: Module javafx.controls not found".
Java FX-15 works beautifully.  I hope this helps someone...
===
To me, it seems like the nb-javac is causing the problem....
===

I've just installed JDK15 and was encountering this issue as well. (macOS 10.14.6)
I used the .bash_profile environment and had defined the $PATH_TO_FX via.
vim .bash_profile
export PATH_TO_FX=/<file path including lib> save and close
source .bash_profile load
Closed completely all terminal windows, rebooted terminal and ran echo,
echo $PATH_TO_FX, verified; and
resumed to compile and run the HelloFX file.
It may work for some, small oversight on my end

Very strange:
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac -version
javac 11.0.9
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ java -version
openjdk version "11.0.9" 2020-10-20
OpenJDK Runtime Environment (build 11.0.9+11-suse-lp<IP_ADDRESS>-x8664)
OpenJDK 64-Bit Server VM (build 11.0.9+11-suse-lp<IP_ADDRESS>-x8664, mixed mode)
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

It is compiling ..
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac --module-path ~/bin/javafx-jmods-11.0.2 --add-modules javafx.controls StageExamples.java
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

But not running ...
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ java --module-path ~/bin/javafx-jmods-11.0.2 --add-modules javafx.controls StageExamples
Error occurred during initialization of boot layer
java.lang.module.FindException: Module javafx.controls not found
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

Very strange:
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac -version
javac 11.0.9
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ java -version
openjdk version "11.0.9" 2020-10-20
OpenJDK Runtime Environment (build 11.0.9+11-suse-lp<IP_ADDRESS>-x8664)
OpenJDK 64-Bit Server VM (build 11.0.9+11-suse-lp<IP_ADDRESS>-x8664, mixed mode)
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

It is compiling ..
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac --module-path ~/bin/javafx-jmods-11.0.2 --add-modules javafx.controls StageExamples.java
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

But not running ...
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ java --module-path ~/bin/javafx-jmods-11.0.2 --add-modules javafx.controls StageExamples
Error occurred during initialization of boot layer
java.lang.module.FindException: Module javafx.controls not found
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$

You should only use the jmod files with jlink. Not with javac (although it happens to work, it isn't supported) or java.
You need to use the sdk if you want to locally compile and run.

You should only use the jmod files with jlink. Not with javac (although it happens to work, it isn't supported) or java.
You need to use the sdk if you want to locally compile and run.

No way with sdk ..
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac --module-path ~/bin/javafx-sdk-11.0.2 --add-modules javafx.controls StageExamples.java
error: module not found: javafx.controls
1 error

No way with sdk ..
gilberto.andrade@C430760:~/tmp/JAVAFX/stage$ javac --module-path ~/bin/javafx-sdk-11.0.2 --add-modules javafx.controls StageExamples.java
error: module not found: javafx.controls
1 error

That should be javafx-sdk-11.0.2/lib

That should be javafx-sdk-11.0.2/lib

That should be javafx-sdk-11.0.2/lib
Sorry, it was an oversight on my part.
Thank you, @kevinrushforth ! Finally it is running!

That should be javafx-sdk-11.0.2/lib
Sorry, it was an oversight on my part.
Thank you, @kevinrushforth ! Finally it is running!


Post-install on Parallels and VMWare doesn't support mouse

Apparently, the default install of FreeBSD doesn't have the following line in /boot/loader.conf which is needed for the mouse to work in those VMs:
ums_load="YES"

What do you think, should the Agora installer add that line if it isn't already present?

I personally tested the fix on Parallels (M1 Mac).
VMware forums also suggest this solution:
https://communities.vmware.com/t5/Fusion-22H2-Tech-Preview/FreeBSD-13-1-and-mouse-support/td-p/2914699


Bug - "White" color swatch is invisible

Because the color swatches don't have a border, white is currently invisible...
Editor with white color added, but not selected...

Selected...

How about a faint border on the swatches so you can see very faint colors?

Fixed with https://github.com/maxwellito/triangulart/pull/1


When upgrade Flutter SDK it's show me white screen in Android Phone

My pubspec.yaml file
`
version: 1.0.0+1
environment:
sdk: ">=2.7.0 <3.0.0"
dependencies:
flutter:
sdk: flutter
built_collection: ^5.1.0
built_value: ^8.0.6
cupertino_icons: ^1.0.3
google_fonts: ^2.1.0
font_awesome_flutter: ^9.1.0
country_code_picker: ^2.0.1
form_validator: ^0.1.7
regexpattern: ^1.0.0
giffy_dialog: ^1.8.0
qr_code_scanner: ^0.4.0
flutter_otp_timer: ^0.9.7
flutter_countdown_timer: ^4.1.0
group_radio_button: ^1.1.0
flutter_native_splash: ^1.2.0
top_sheet: ^0.0.4
persistent_bottom_nav_bar: ^4.0.2
sticky_and_expandable_list: ^0.3.1
percent_indicator: ^3.0.1
implicitly_animated_reorderable_list: ^0.4.0
intl: ^0.17.0
just_audio: ^0.7.5 #play audio files
flutter_video_info: ^1.2.0 #retrieve video metadata
video_thumbnail: ^0.3.3 #get a thumbnail from a file
collection: ^1.15.0
chat_ui_kit: ^1.2.3
json_serializable: ^4.1.3
json_annotation: ^4.0.1
mime_type: ^1.0.0
functional_widget_annotation: ^0.9.0
firebase_core: ^1.3.0
firebase_auth: ^1.4.1
cloud_firestore: ^2.2.2
firebase_storage: ^8.1.3
google_sign_in: ^5.0.4
timeago: ^3.1.0
latlng: ^0.1.0
shared_preferences: ^2.0.6
flutter_hooks: ^0.17.0
dev_dependencies:
flutter_test:
sdk: flutter
build: ^2.0.2
build_runner: ^2.0.4
built_value_generator: ^8.0.6
json_serializable: ^4.1.3
flutter:
uses-material-design: true
assets:
- assets/images/
- assets/avatars/
- assets/other/
- images/a_dot_ham.jpeg
`

Run flutter create bug.
Update the files as follows: ...
...

Expected results: run application show UI
Actual results: not show UI only showing White screen

I have upgrade Flutter SDK it gets the problem to run when I am run Android it shows me in Android phone only white screen Please check the error, How can I fixed this issue as soon as possible
Error ->
Launching lib/main.dart on Mi A3 in debug mode...
Running Gradle task 'assembleDebug'...
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app.apk...
E/flutter (19043): [ERROR:flutter/shell/common/shell.cc(103)] Dart Error: error: import of dart:mirrors is not supported in the current Dart runtime
E/flutter (19043): [ERROR:flutter/runtime/dart_isolate.cc(171)] Could not prepare isolate.
E/flutter (19043): [ERROR:flutter/runtime/runtime_controller.cc(415)] Could not create root isolate.
E/flutter (19043): [ERROR:flutter/shell/common/shell.cc(588)] Could not launch engine with configuration.

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(MaterialMainApp());
}

class MaterialMainApp extends StatefulWidget {
  const MaterialMainApp({Key key}) : super(key: key);

  @override
  _MaterialAppState createState() => _MaterialAppState();
}

class _MaterialAppState extends State<MaterialMainApp> {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(home: HomeView());
  }
}

(base) sonukumar@Sonus-MacBook-Air elearningmobile % flutter analyze
Analyzing elearningmobile...

   info • The declaration '_addKey' isn't referenced • lib/core/share_prefs.dart:3:14 • unused_element
   info • The declaration '_getKey' isn't referenced • lib/core/share_prefs.dart:8:16 • unused_element
   info • Unused import: 'package:elearningmobile/auth/firebase_user_provider.dart' • lib/main.dart:1:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/details/batch_details.dart' • lib/main.dart:2:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/home/home.dart' • lib/main.dart:3:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/login/login.dart' • lib/main.dart:4:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/screens.dart' • lib/main.dart:5:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/settings/setting_screen.dart' • lib/main.dart:6:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/splash/splashview.dart' • lib/main.dart:7:8 • unused_import
   info • The value of the field '_currentIndex' isn't used • lib/screens/details/batch_details.dart:24:7 • unused_field
   info • Unused import: 'package:build_runner/build_runner.dart' • lib/screens/home/chat_screen.dart:10:8 • unused_import
   info • Unused import: 'package:flutter/foundation.dart' • lib/screens/home/chat_screen.dart:11:8 • unused_import
   info • Name non-constant identifiers using lowerCamelCase • lib/screens/home/chat_screen.dart:296:8 • non_constant_identifier_names
   info • Unused import: 'package:elearningmobile/screens/home/nav/main_drawer.dart' • lib/screens/home/home.dart:2:8 • unused_import
   info • Unused import: 'package:elearningmobile/screens/settings/setting_screen.dart' • lib/screens/home/home.dart:5:8 • unused_import
   info • The value of the local variable 'size' isn't used • lib/screens/home/home_main.dart:24:9 • unused_local_variable
   info • Name types using UpperCamelCase • lib/screens/home/nav/main_drawer.dart:141:7 • camel_case_types
   info • Unused import: 'package:elearningmobile/screens/screens.dart' • lib/screens/home/profile/batch_profile.dart:2:8 • unused_import
   info • Avoid using braces in interpolation when not needed • lib/screens/home/profile/info_profile.dart:65:24 • unnecessary_brace_in_string_interps
   info • Unused import: 'package:elearningmobile/auth/email_auth.dart' • lib/screens/login/login.dart:3:8 • unused_import
   info • The declaration '_createRoute' isn't referenced • lib/screens/login/login.dart:318:9 • unused_element
   info • Unused import: 'package:intl/intl.dart' • lib/utils/time_utils.dart:5:8 • unused_import
   info • Name types using UpperCamelCase • lib/widgets/BottomBarHideView.dart:8:7 • camel_case_types
   info • Name types using UpperCamelCase • lib/widgets/bottom_sheet_dialog.dart:4:7 • camel_case_types
   info • Name types using UpperCamelCase • lib/widgets/center_dialog.dart:6:7 • camel_case_types
   info • 'RaisedButton' is deprecated and shouldn't be used. Use ElevatedButton instead. See the migration guide in flutter.dev/go/material-button-migration-guide). This
          feature was deprecated after v1.26.0-18.0.pre. • lib/widgets/flutter_flow_widgets.dart:65:16 • deprecated_member_use
   info • 'RaisedButton.icon' is deprecated and shouldn't be used. Use ElevatedButton instead. See the migration guide in flutter.dev/go/material-button-migration-guide).
          This feature was deprecated after v1.26.0-18.0.pre. • lib/widgets/flutter_flow_widgets.dart:65:16 • deprecated_member_use
   info • 'RaisedButton' is deprecated and shouldn't be used. Use ElevatedButton instead. See the migration guide in flutter.dev/go/material-button-migration-guide). This
          feature was deprecated after v1.26.0-18.0.pre. • lib/widgets/flutter_flow_widgets.dart:95:14 • deprecated_member_use
   info • Name types using UpperCamelCase • lib/widgets/model_dialog.dart:4:7 • camel_case_types
   info • The declaration '__modelDialog' isn't referenced • lib/widgets/model_dialog.dart:4:7 • unused_element

30 issues found. (ran in 7.8s)

/Users/sonukumar/Library/Android/sdk/flutter/bin/flutter doctor --verbose
[✓] Flutter (Channel beta, 2.2.2, on macOS 11.4 20F71 darwin-x64, locale en-IN)
    • Flutter version 2.2.2 at /Users/sonukumar/Library/Android/sdk/flutter
    • Framework revision d79295af24 (3 days ago), 2021-06-11 08:56:01 -0700
    • Engine revision 91c9fc8fe0
    • Dart version 2.13.3

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/sonukumar/Library/Android/sdk
    • Platform android-30, build-tools 30.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5, Build version 12E262
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2.3)
    • Android Studio at /Applications/Android Old/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_112-release-b06)

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.1.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.23.0

[✓] Connected device (3 available)
    • Mi A3 (mobile)  • b82ae69342ee • android-arm64  • Android 11 (API 30)
    • macOS (desktop) • macos        • darwin-x64     • macOS 11.4 20F71 darwin-x64
    • Chrome (web)    • chrome       • web-javascript • Google Chrome 91.0.4472.101

• No issues found!
Process finished with exit code 0

@skbhati199
It's failing due to :
E/flutter (19043): [ERROR:flutter/shell/common/shell.cc(103)] Dart Error: error: import of dart:mirrors is not supported in the current Dart runtime
I see that you are using json_annotation: ^4.0.1 as well as json_serializable: ^4.1.3. Can you remove json_serializable: ^4.1.3 and try again to see if it helps to resolve your issue ?
It could be due to one of these packages may not be compatible with flutter due to mirror package.

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.

Sure, get from master code, downgrade version, some issue build package, now fixed, thank you


Use new schema

This lays the groundwork to run services-advisor using any set of services, for example using Greek services published by an Amani instance cc @arvandtiv @PeaceGeeks
There's still some work to be done:

using a different mapbox account
testing against amani

@tuckerbuchy got everything working with the new schema, nice work.
@Jareechang @amyngai if you get the latest version of the code, you'll need to run this or nothing will appear! `cp src/site-specific-config.js.dist src/site-specific-config.js


Client generator throwing multiple errors

Describe the bug
On generating a new client from an .edmx file I am getting multiple missing type errors although I have installed the generator as local dependency.
The files that were generated in the outputdir does seem to be correct though.
To Reproduce
npx generate-odata-client --inputDir service-specs/ --outputDir generatedService --clearOutputDir
Expected behavior
No errors should ne thrown
Used Versions:

node version via node -v: 18.1.0
npm version via npm -v: 8.8.0
SAP Cloud SDK version you used as dependency: "@sap-cloud-sdk/generator": "^2.7.1"

Hey @szymarcus, thanks for reaching out to us.
Since the errors seem to be compilation errors, this has most likely to do with your tsconfig.json.
However, since you appear to use typescript anyways, you can set the --generateJs false flag.

Hey @tomfrenken thanks a lot for your quick answer. I now disabled the js generation and just compile the ts files myself within the build step.

Glad it worked 👍


change: always use internal image ref for IAR validation, if possible (#2067)

Ref: #2067
We want to make sure, that the "local" image (and thus the local signature) is always preferred for IAR verification.
Also, we had the keychain derived from local auth lingering around there which was unused and is now gone.
Checklist

[x] The title of this PR would make a good line in Acorn's Release Note's Changelog
[x] The title of this PR ends with a link to the main issue being address in parentheses, like: This is a title (#1216). Here's an example
[x] All relevant issues are referenced in the PR description. NOTE: don't use GitHub keywords that auto-close issues
[x] Commits follow contributing guidance
[ ] Automated tests added to cover the changes. If tests couldn't be added, an explanation is provided in the Verification and Testing section
[x] Changes to user-facing functionality, API, CLI, and upgrade impacts are clearly called out in PR description
[x] PR has at least two approvals before merging (or a reasonable exception, like it's just a docs change)

The failing test brought up the issue that I initially discussed in yesterday's standup - I made some adjustments accordingly and added more tests around it.


Delay after adding thing or action

There seems to be a delay in updating the complete list of things and actions. As a result, I'm unable to fetch a newly added thing or action. The Weaviate instance that I was working with did not contain more than 5 things.

Add a new thing or action
Request all things or actions, right after receiving a 200 response status on step 1.

The recently added thing or action is not in the list of all things and actions. However, it is there when I try the request a few seconds later. I would expect my recently added thing or action to be in the request for all things or actions immediately after a successul response.

This is to be expected with the janusgraph connector for all requests other than getting a thing by its uuid ID. This is because the only index we have in Cassandra itself is for UUIDs. All other indices are coming from Elasticsearch and Elasticsearch needs some time to build the indices. This time however can be configured, so it might help to reduce it, in cases where the playground is used. According to the Elasticsearch docs it defaults to 1s:
index.refresh_interval
    How often to perform a refresh operation, which makes recent changes to the index visible to search. Defaults to 1s. Can be set to -1 to disable refresh.

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings
This is something where SeMI's solution engineering team can guide a potential user. They will probably want a very small delay when adding things interactively through the UI, but will want a very big delay when adding things in bulk (to reduce the overhead of constant re-indexing).
We could try setting it to a very small timespan, but I don't think we'll get atomic indexing.

Closing. While the eventual consistency of Elasticsearch is something that is central to our stack and can't be changed, we introduced #980 which makes the above much less noticeable.


I'd like to share some information to use pedeps inside a github action

See my post here in Code::Blocks forum:
Re: build bot in the github, I see one nice project
The final goal is to use this tool to create a standalone C::B packages inside the github action. Thanks.

I was planning to meet the maker of Code::Blocks (who lives about a 1 hour drive from me) to discuss how winlibs and Code::Blocks can come together, but our plans changed and we haven't rescheduled yet.
I have built my own build system and package manager, but have not yet properly documented and tested it, and I haven't figured out if all licenses allow me to distribute binaries and where I would host these.
Can you describe exactly what I could do to support your ideas?
Or if you like we could set op a call.

I was planning to meet the maker of Code::Blocks (who lives about a 1 hour drive from me) to discuss how winlibs and Code::Blocks can come together, but our plans changed and we haven't rescheduled yet.

That's nice idea. I really appreciate about the compiler suite and other tools you made.

I have built my own build system and package manager, but have not yet properly documented and tested it, and I haven't figured out if all licenses allow me to distribute binaries and where I would host these.

Currently, I use msys2/mingw64 gcc and other packages, they are very easy to install. One issue is that they are dropping the windows 7 OS support sooner or later, but some of my clients are still using windows 7.
If you are using a new build system, that's be great!

Can you describe exactly what I could do to support your ideas?
Or if you like we could set op a call.

I'm one of Code::Blocks developer, and what I want to do is to use pedeps tool to copy all the dependency dlls when I build a Code::Blocks from github actions. It is nearlly done, see this post: Re: build bot in the github, I see one nice project
I think many other project hosted in github can use your pedeps tool to ship their dependency dlls, and this make creating a release package very easily.
And currently, I think a call is not necessary, my English is not good enough. Thanks.

FYI:
In my project: asmwarrior/x86-codeblocks-builds: Automatically built codeblocks for both 32b and 64b Windows systems.
I can now create a full C::B win64 bit release package(include all the exe and dll) from github action. The copying of the dlls are mainly done by pedeps.

That's great news.
copypedeps should copy all files that it finds in the PATH environment variable that are not in the Windows folder.
So your script may need to add locations to the PATH environment variable if everything is not in the same folder.

That's great news.
copypedeps should copy all files that it finds in the PATH environment variable that are not in the Windows folder.
So your script may need to add locations to the PATH environment variable if everything is not in the same folder.

In the github action, I run your copypedepes.exe inside a msys2 ming64 shell, and I see the bin folder of the gcc is already added in the PATH, so dll files get copied correctly.
I see the result C::B works OK.


Suggestion: Asciinema style recording

It would be neat if stories could be recorded still but only by typing text into a file (the extension would monitor the key presses and see at what time they were pressed), and it is converted to a GIF and played back (or by updating the content of the current file).

That's what I've also thought. We could e.g. record every single cursor movement and every file addition and play it afterwards.

tried it, but I think I like gif better


VR V3 AlchemyAPI

Hi everybody . Is it still necessary in Visual Recognition V3 to use the Alchemy API and then set the environments variables in manual mode as fig. attached.

You don't need an alchemy key to use Visual recognition v3 and the environment variables you have are probably from an old version of the app.


Digest Access Authentication

Hi, I tried your application and I think it's great !
However, I found an issue with one camera: the app tries to connect for a very long time, but nothing happens and no error is shown (it seems the app waits forever with no time-out).
I discovered the problem is because the cam uses digest access authentication, and I guess the app does not support it.
Is it possible to add support for digest authentication?
Thank you.

Hi!
The app supports digest authentication. Can you try connection to you camera via VLC app to check it? (Possible the reason is unsupported TCP/UDP mode or something else.)

I tried with with VLC Android app and can't connect.
I also tried with the latests Windows version of VLC (Ver 3.0.18 win32) on a desktop PC, with the same result: can't connect.
It asks for user & password on a dialog box, but when I press OK, the connection can't be established and the dialog box appears again asking for the user credentials.
The dialog box appears whether you put the credentials embedded in the URI or not (it's indistinct).
I copy here some lines of the log:
main warning: Password in a URI is DEPRECATED
live555 error: Failed to connect with rtsp://<IP_ADDRESS>:80/live/0/h264.sdp
satip error: Failed to setup RTSP session
access_realrtsp warning: Cseq mismatch, got 1, assumed 0
access_realrtsp warning: only real/helix rtsp servers supported for now
I guess you're right with some unsupported mode.
I was checking wikipedia (https://en.wikipedia.org/wiki/Digest_access_authentication), and mentions that the original standard was RFC 2069, but later revised and upgraded in RFC 2617.
"Many of the security options in RFC 2617 are optional. If quality-of-protection (qop) is not specified by the server, the client will operate in a security-reduced legacy RFC 2069 mode"
I don't know wich version of the protocol (RFC) is supported by the cam and VLC, but maybe there is a mistmatch and they're not understanding.

Well, I hope next libVLC stable version 4.X will decide this issue.

I upgraded to Ver 2.3 and this problem persists.
Which version of libVLC are you using now?

The stable 4х version is still not released.
I'll update the libVLC as soon as possible after release.


"About YakYak" pops up whenever I click on any link/image/profile icon.

As demonstrated here....

Whenever I click on an image in a chat, or someone's profile icon, or any hyperlink in a chat message, the link opens as normal but that "About YakYak" popup appears every single time.
On YakYak v1.5.0, Windows 10 64bit version 1703 build 15063.0

Same here with 1.5.1 on Mac.

:D this bug is nice!

Fixed in https://github.com/yakyak/yakyak/commit/2e167d693812659092ba9da88693eb76af84edb2


[TOPIC-GPIO] drivers: sensor: hts221: update to new GPIO API

Correct DRDY active level to default active-high, switch to new interrupt configuration.
Tested on nrf52_pca20020 (Thingy:52) and (after much pain unrelated to this change) frdm_k64f with the x_nucleo_iks01a2 shield.
NOTE: This PR also contains #19862 and #19870 on which it depends.

Maybe we should add DNM label to make sure this PR will not get merged before  #19862 and #19870?


body styles

Is there a way to style the body with Radium?

You can use the <Style /> component. I usually use this to define global styles on generic HTML elements.

Yep :)
I like to include some global styles and normalize/reset with the Style component. You can also include a separate CSS file for that stuff, of course, but Style saves you an extra HTTP request.

sweet :+1:

Wait, I have to render the component for it to work on the body? That feels weird

Yes, unfortunately. I suggest rendering the style component with body styles in your main App component.


consume counts should change when changing items in/out of combat

Setting count= just updates countData; counts aren't actually updated until a BAG_UPDATE_DELAYED. We could just force call the relevant countLang function at buttonLang.count time, but there's gotta be a cleaner way.


fixing security audit

Pull Request Test Coverage Report for Build 417

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 83.631%

Totals

Change from base Build 415:
0.1%

Covered Lines:
2595

Relevant Lines:
2929

💛  - Coveralls


setup.sh script is broken on WSL2 Ubuntu 20.04

Description
Running the setup.sh script in the repo breaks using Ubuntu 20.04 on WSL2.
A quick google search shows that this has to do with the setup.sh script having Windows-style line endings \r\n. After running sed $'s/\r$//' on the setup.sh script, I was able to execute it as normal, however, the resulting /usr/local/share/chruby/auto.sh and /usr/local/share/chruby/chruby.sh had the same issue. After using the above sed command on both scripts did chruby finally behave as expected.
Steps To Reproduce
Steps to reproduce the bug:

Clone repository
Run sudo ./scripts/setup.sh as seen in the README

Expected Behavior
The setup script should run as normal
Actual Behavior
/usr/bin/env: ‘bash\r’: No such file or directory
Environment
$ bash --version
`GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)`

Closing this. This was an error on my part, found out when I had the same issue with another project.
Warning to anyone in the future: be careful with your gitconfig. In my case, I had autocrlf = true, which was converting LFs to CRLFs when I would clone repositories.


[Masternode] Offline signing endpoints

Add two new endpoints

Build the join message for signing
Broadcast the join transaction with a signed message

Add verification of the collateral signature

Hi Rowan, what was the requirement for this? (Looks good though)

Nothing formal AFAIK, just thought it might be more useful to split the method up this way to enable an offline collateral address to join.


pallet-drand-client application

Project Abstract
Current sources of verifiable randomness in Substrate are limited, and developers need to rely on deriving randomness themselves, or by taking it from BABE. Both of these sources contain randomness that is always derived from information within the network. Another approach is to take randomness from an external source, if that source can be proven to be secure. One such source could be a trusted drand (decentralised randomness beacon), which allows the participants "to produce collective, publicly verifiable, unbiased, unpredictable random values at fixed intervals". This is what is being used by the League of entropy.
This application shows our interest in implementing a drand client as a Substrate pallet. Details can be found in our Markdown application.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] A BTC, Ethereum (USDT/USDC/DAI) or Polkadot/Kusama (aUSD) address for the payment of the milestones is provided inside the application.
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Are you also in contact with protocol labs/filecoin? We discussed this previously with them directly and could theoretically involve them in the project.

We're not in contact with them, but we'd love to have them involved in the direction and scoping of this for a variety of reasons. That said, we would like be the primarily implementors of at least the first iteration of this, from a personal development and grant recipient perspective.

Would it be possible to share some additional information about your previous work either here or via email (david@web3.foundation)?

If we're in the right ballpark regarding the project scope and grant amount, and previous work experience proves to be satisfactory, @random-bacon suggested we start with adding you and/or the rest of the grants team to our private repos, as we've already started the implementations of the client and pallet/node. Thoughts?

hey! I work for protocol labs on the drand team and would love to hear more/help out where I can.

@random-ham sounds good, feel free to add me to the repo. The grant application looks pretty good and to be honest, I think we can get this approved in the coming days/weeks.

@random-ham I don't see any contact details on your github profile so i'll post here: drand has a slack workspace - if you have any questions or want to discuss any of the details, feel free to join: https://join.slack.com/t/drandworkspace/shared_invite/zt-1il4cggya-u~6xaE7aO~RzVMck6bQjjg

Hey @Noc2 , please check the master branch of drand-substrate-client for our submission for the Milestone 1 requirements.

@random-ham Thanks for pinging me here. But could you officially deliver the milestone here https://github.com/w3f/Grant-Milestone-Delivery Thanks!

Hi @random-ham how is milestone 2 coming along?

@random-ham friendly ping.

Hi @random-ham seeing as there has been no activity on the drand-client repo in 5 months, and neither of your personal accounts either, I think it's best that we close the grant and you can open in the future should you want to continue working on it.
If we don't hear back in a couple of weeks, the committee will file a PR to cancel the grant. Thanks for your work so far and let me know if you have any questions.


question about data retention period in cassandra

Hi team, quick question.
for the tracing data written in cassandra, is there a retention period ?
I do not see any purge job which deprecates the data after certain time.
just wondering if I miss any existing purge job (or anything similar) or that is on the to-do list or we have to implement the job by ourselves.
look forward to the reply!
thanks a lot !
Dan

If you used https://github.com/uber/jaeger/blob/master/plugin/storage/cassandra/schema/create.sh to initialize the keyspace, it defaults the TTL for everything to 2 days.

Hi Dan,
Cassandra supports TTLs, which we utilize in our schema. See https://github.com/uber/jaeger/blob/ec6378a7a5094ff3019c506163108da29f3f2f99/plugin/storage/cassandra/schema/create.sh#L11 for defaults.

thank you guys, this is super helpful !


Volumes Stuck in Detaching

We are having an issue with a freshly deployed cluster (that had previously been working stably with a different Node OS until a handful of mistakes were made) where none of our volumes are unmounting. They get stuck in a detaching status and the only way to fix it seems to be to delete the PVCs from Rancher. During node deployment, we made the recommended changes to /etc/iscsi/iscsid.conf on each node, as per this article:
https://github.com/longhorn/longhorn/blob/master/docs/csi-config.md
All nodes are talking to one another properly (network is solid) and all Longhorn pods are green. The only Longhorn pod with restarts is longhorn-csi-plugin with 4 restarts.
Node OS: Rancher OS v1.5.4
Engine Version: 18.9.8
Rancher Version: v2.3.2
Kubernetes Version: v1.16.2
Longhorn Version: 0.7.0
Node Type: Physical
Node Quantity:

1x ETC/Control Plane
4x Worker

RancherOS Config:
#cloud-config

# Set the hostname for this machine (takes precedence over hostname assigned by DHCP lease).
hostname: ehs-kn-**
# Authorize SSH keys for the `rancher` sudoer user
ssh_authorized_keys:
  - ssh-rsa AAAA***********

mounts:
  - ["/dev/sdc1", "/media/longhorn-00", "xfs", ""]
  - ["/dev/sdd1", "/media/longhorn-01", "xfs", ""]
  - ["/dev/sde1", "/media/longhorn-02", "xfs", ""]
  - ["/dev/sdf1", "/media/longhorn-03", "xfs", ""]

rancher:
  console: ubuntu
  services_include:
    kernel-headers: true
    kernel-extras: true
    open-iscsi: true
  network:
    # Configure nameserver (Default: <IP_ADDRESS>, <IP_ADDRESS> if not DHCP assigned)
    # Caveat: On first boot, System-Docker may use the default or DHCP assigned nameservers
    # before the cloud-init configuration has been applied (race).
    dns:
      search:
        - ********.**
        - ********.**
        - ********.**
        - ********.**
      nameservers:
        - **.**.**.**
        - **.**.**.**
      override: true # Override DHCP assigned nameservers
    # Assign static IP configuration to interfaces (Default: use DHCP)
    # Ref: https://rancher.com/docs/os/v1.x/en/installation/networking/interfaces/
    interfaces:
      "mac=**:**:**:**:**:**":
        address: **.**.**.**/23
        gateway: **.**.**.**
        mtu: 9000
        dhcp: false
  # Configure Sysctl parameters.
  # Example: Satisfy host requirement for running elasticsearch containers.
  sysctl:
    vm.max_map_count: 262144

write_files:
  - container: ntp
    path: /etc/ntp.conf
    permissions: "0644"
    owner: root
    content: |
      server 0.pool.ntp.org iburst
      server 1.pool.ntp.org iburst
      server 2.pool.ntp.org iburst
      server 3.pool.ntp.org iburst

      # Allow only time queries, at a limited rate, sending KoD when in excess.
      # Allow all local queries (IPv4, IPv6)
      restrict default nomodify nopeer noquery limited kod
      restrict <IP_ADDRESS>
      restrict [::1]

CSI-Attacher Log - Primary:
I1203 20:11:07.063055 1 controller.go:175] Started VA processing "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.063105 1 csi_handler.go:89] CSIHandler: processing VA "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.063118 1 csi_handler.go:140] Starting detach operation for "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.063191 1 csi_handler.go:147] Detaching "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.063216 1 csi_handler.go:542] Found NodeID ehs-kn-14 in CSINode ehs-kn-14
I1203 20:11:07.063234 1 connection.go:180] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1203 20:11:07.063245 1 connection.go:181] GRPC request: {"node_id":"ehs-kn-14","volume_id":"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74"}
I1203 20:11:07.067246 1 controller.go:175] Started VA processing "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.067277 1 csi_handler.go:89] CSIHandler: processing VA "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.067289 1 csi_handler.go:140] Starting detach operation for "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.067340 1 csi_handler.go:147] Detaching "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.067362 1 csi_handler.go:542] Found NodeID ehs-kn-14 in CSINode ehs-kn-14
I1203 20:11:07.067376 1 connection.go:180] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1203 20:11:07.067390 1 connection.go:181] GRPC request: {"node_id":"ehs-kn-14","volume_id":"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3"}
I1203 20:11:07.069397 1 connection.go:183] GRPC response: {}
I1203 20:11:07.070293 1 connection.go:184] GRPC error: rpc error: code = Aborted desc = The volume pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74 is detaching
I1203 20:11:07.070321 1 csi_handler.go:428] Saving detach error to "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.071113 1 connection.go:183] GRPC response: {}
I1203 20:11:07.071879 1 connection.go:184] GRPC error: rpc error: code = Aborted desc = The volume pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3 is detaching
I1203 20:11:07.071896 1 csi_handler.go:428] Saving detach error to "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.074134 1 controller.go:141] Ignoring VolumeAttachment "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3" change
I1203 20:11:07.074168 1 csi_handler.go:439] Saved detach error to "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3"
I1203 20:11:07.074201 1 csi_handler.go:99] Error processing "csi-f9b94a27d6ed6020e1f53b38671e93d5c88adbc559b129fde22e21d6b4e5b8c3": failed to detach: rpc error: code = Aborted desc = The volume pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74 is detaching
I1203 20:11:07.081725 1 controller.go:141] Ignoring VolumeAttachment "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290" change
I1203 20:11:07.081754 1 csi_handler.go:439] Saved detach error to "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290"
I1203 20:11:07.081837 1 csi_handler.go:99] Error processing "csi-87d2d8e688caa4bdb4190102867e76cb3427f7dd96c8cb703676544871b85290": failed to detach: rpc error: code = Aborted desc = The volume pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3 is detaching
I1203 20:11:09.097389 1 leaderelection.go:258] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io
I1203 20:11:14.103556 1 leaderelection.go:258] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io
I1203 20:11:18.244227 1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.Node total 49 items received

Engine-Manager Log:
[longhorn-instance-manager] time="2019-12-03T19:54:03Z" level=info msg="engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: waiting for state "
[longhorn-instance-manager] time="2019-12-03T19:54:03Z" level=info msg="wait for process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da to shutdown"
[longhorn-instance-manager] time="2019-12-03T19:54:03Z" level=debug msg="Process Manager: wait for process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da to shutdown before unregistering process"
[longhorn-instance-manager] time="2019-12-03T19:54:03Z" level=debug msg="Process Manager: wait for process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da to shutdown before unregistering process"
[longhorn-instance-manager] time="2019-12-03T19:54:03Z" level=info msg="engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: waiting for state "
[pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da] time="2019-12-03T19:54:03Z" level=error msg="Error when shutting down frontend:failed to shutdown frontend: failed to call gRPC FrontendShutdownCallback for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530: rpc error: code = Unknown desc = failed to callback for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530 frontend shutdown: device pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88: fail to stop SCSI device: Fail to logout target: Failed to logout target: Failed to execute: nsenter [--mount=/host/proc/6354/ns/mnt --net=/host/proc/6354/ns/net iscsiadm -m node -T iqn.2014-09.com.rancher:pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88 --logout -p <IP_ADDRESS>], output iscsiadm: Could not logout of [sid: 1, target: iqn.2014-09.com.rancher:pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88, portal: <IP_ADDRESS>,3260].\niscsiadm: initiator reported error (9 - internal error)\niscsiadm: Could not logout of all requested sessions\nLogging out of session [sid: 1, target: iqn.2014-09.com.rancher:pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88, portal: <IP_ADDRESS>,3260]\n, error exit status 9"
[pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da] time="2019-12-03T19:54:03Z" level=info msg="Closing: <IP_ADDRESS>:10000"
time="2019-12-03T19:54:03Z" level=info msg="Closing: <IP_ADDRESS>:10015"
time="2019-12-03T19:54:03Z" level=info msg="Closing: <IP_ADDRESS>:10015"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=info msg="Process Manager: process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da error out, error msg: exit status 1"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=debug msg="Process update: pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: state error: Error: exit status 1"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=info msg="Process Manager: successfully unregistered process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=debug msg="Process update: pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: state error: Error: exit status 1"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=warning msg="Engine Manager need to cleanup frontend before unregistering engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=debug msg="engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530: finishing frontend shutdown"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=info msg="Process Manager: successfully unregistered process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:04Z" level=debug msg="Process update: pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: state error: Error: exit status 1"
[longhorn-instance-manager] time="2019-12-03T19:54:05Z" level=info msg="Shutdown SCSI device for <IP_ADDRESS>:iqn.2014-09.com.rancher:pvc-83343c09-8945-49e1-b1fa-6f4e6e298333"
[longhorn-instance-manager] time="2019-12-03T19:54:15Z" level=info msg="Engine Manager: prepare to delete engine pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:54:15Z" level=debug msg="engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7: prepare to stop engine pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679"
[longhorn-instance-manager] time="2019-12-03T19:54:15Z" level=debug msg="Process Manager: prepare to delete process pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=warning msg="Engine Manager need to cleanup frontend before unregistering engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=debug msg="engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7: finishing frontend shutdown"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=warning msg="Engine Manager need to cleanup frontend before unregistering engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=debug msg="engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7: finishing frontend shutdown"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=warning msg="failed to get the related process info for engine pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679: rpc error: code = NotFound desc = cannot find process pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679"
[longhorn-instance-manager] time="2019-12-03T19:54:27Z" level=warning msg="failed to get the related process info for engine pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679: rpc error: code = NotFound desc = cannot find process pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679"
[longhorn-instance-manager] time="2019-12-03T19:54:28Z" level=info msg="Shutdown SCSI device for <IP_ADDRESS>:iqn.2014-09.com.rancher:pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=debug msg="engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530: succeed to stop engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da at <IP_ADDRESS>:10000"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=info msg="Engine Manager: deleted engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="failed to get the related process info for engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da: rpc error: code = NotFound desc = cannot find process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=debug msg="Engine Manager starts to unregistered engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=warning msg="Engine Manager need to cleanup frontend before unregistering engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=debug msg="engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530: finishing frontend shutdown"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=info msg="Engine Manager: prepare to delete engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=debug msg="engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530: prepare to stop engine pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:51Z" level=debug msg="Process Manager: prepare to delete process pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530-0f50c3da"
[longhorn-instance-manager] time="2019-12-03T19:54:52Z" level=info msg="Shutdown SCSI device for <IP_ADDRESS>:iqn.2014-09.com.rancher:pvc-83343c09-8945-49e1-b1fa-6f4e6e298333"
[longhorn-instance-manager] time="2019-12-03T19:55:15Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:55:16Z" level=info msg="Shutdown SCSI device for <IP_ADDRESS>:iqn.2014-09.com.rancher:pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88"
[longhorn-instance-manager] time="2019-12-03T19:55:39Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530"
[longhorn-instance-manager] time="2019-12-03T19:55:40Z" level=info msg="Shutdown SCSI device for <IP_ADDRESS>:iqn.2014-09.com.rancher:pvc-83343c09-8945-49e1-b1fa-6f4e6e298333"
[longhorn-instance-manager] time="2019-12-03T19:56:03Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7"
[longhorn-instance-manager] time="2019-12-03T19:56:03Z" level=warning msg="failed to get the related process info for engine pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679: rpc error: code = NotFound desc = cannot find process pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7-1871a679"

Longhorn-Manager Log - Node 1:
time="2019-12-03T19:54:13Z" level=info msg="Longhorn replica longhorn-system/pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-r-4454c8b7 has been deleted"
time="2019-12-03T19:54:14Z" level=info msg="Longhorn replica longhorn-system/pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-r-29ab6ce9 has been deleted"
time="2019-12-03T19:54:14Z" level=info msg="Longhorn replica longhorn-system/pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-r-91eebed9 has been deleted"
time="2019-12-03T19:54:15Z" level=info msg="Longhorn engine longhorn-system/pvc-5b5179ba-73f0-4797-8170-32eaa8065342-e-11b1847d has been deleted"
time="2019-12-03T19:54:15Z" level=info msg="Longhorn replica longhorn-system/pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-r-612f4662 has been deleted"
time="2019-12-03T19:54:15Z" level=info msg="Longhorn engine longhorn-system/pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-e-7689f9b7 has been deleted"
time="2019-12-03T19:54:15Z" level=info msg="Longhorn engine longhorn-system/pvc-f4ac28bb-3fc9-4ea1-8667-7b309501d1f7-e-1794384f has been deleted"
time="2019-12-03T19:54:16Z" level=info msg="Longhorn replica longhorn-system/pvc-f4ac28bb-3fc9-4ea1-8667-7b309501d1f7-r-1466cbd2 has been deleted"
time="2019-12-03T19:54:16Z" level=info msg="Longhorn replica longhorn-system/pvc-5b5179ba-73f0-4797-8170-32eaa8065342-r-730032cf has been deleted"
time="2019-12-03T19:54:16Z" level=info msg="Longhorn replica longhorn-system/pvc-5b5179ba-73f0-4797-8170-32eaa8065342-r-9d7c7f57 has been deleted"
time="2019-12-03T19:54:16Z" level=info msg="Longhorn replica longhorn-system/pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-r-45544865 has been deleted"
time="2019-12-03T19:54:17Z" level=info msg="Longhorn replica longhorn-system/pvc-83343c09-8945-49e1-b1fa-6f4e6e298333-r-fe6b4828 has been deleted"
time="2019-12-03T19:54:18Z" level=info msg="Longhorn replica longhorn-system/pvc-f4ac28bb-3fc9-4ea1-8667-7b309501d1f7-r-6b8d2ca0 has been deleted"
time="2019-12-03T19:54:18Z" level=info msg="Longhorn replica longhorn-system/pvc-5b5179ba-73f0-4797-8170-32eaa8065342-r-92b6d5bd has been deleted"
time="2019-12-03T19:54:18Z" level=info msg="Longhorn replica longhorn-system/pvc-f4ac28bb-3fc9-4ea1-8667-7b309501d1f7-r-d48416b1 has been deleted"
time="2019-12-03T19:54:18Z" level=info msg="Longhorn volume longhorn-system/pvc-83343c09-8945-49e1-b1fa-6f4e6e298333 has been deleted"
time="2019-12-03T19:54:19Z" level=info msg="Longhorn volume longhorn-system/pvc-5b5179ba-73f0-4797-8170-32eaa8065342 has been deleted"
time="2019-12-03T19:54:19Z" level=info msg="Longhorn volume longhorn-system/pvc-f4ac28bb-3fc9-4ea1-8667-7b309501d1f7 has been deleted"
time="2019-12-03T19:54:51Z" level=info msg="Longhorn engine longhorn-system/pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88-e-41b3c530 has been deleted"
time="2019-12-03T19:54:51Z" level=info msg="Longhorn volume longhorn-system/pvc-6581790b-c8e8-48cf-8e4d-05573ea44d88 has been deleted"
time="2019-12-03T19:58:34Z" level=debug msg="Replica controller ehs-kn-17 picked up pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-1ad8171d"
time="2019-12-03T19:58:34Z" level=debug msg="Instance handler updated instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-1ad8171d state, old state , new state stopped"
time="2019-12-03T19:58:34Z" level=debug msg="Replica controller ehs-kn-17 picked up pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-db0067a2"
time="2019-12-03T19:58:34Z" level=debug msg="Instance handler updated instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-db0067a2 state, old state , new state stopped"
time="2019-12-03T19:58:36Z" level=debug msg="Replica controller ehs-kn-17 picked up pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537"
time="2019-12-03T19:58:36Z" level=debug msg="Replica controller ehs-kn-17 picked up pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7"
time="2019-12-03T19:58:40Z" level=debug msg="Prepare to create instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537"
time="2019-12-03T19:58:40Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537\", UID:\"865b14e9-255d-4080-bf3d-1e5af844d493\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2578907\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537"
time="2019-12-03T19:58:40Z" level=debug msg="Prepare to create instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537"
time="2019-12-03T19:58:40Z" level=debug msg="Instance handler updated instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537 state, old state stopped, new state starting"
time="2019-12-03T19:58:41Z" level=debug msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537 starts running, IP <IP_ADDRESS>"
time="2019-12-03T19:58:41Z" level=debug msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537 starts running, Port 10000"
time="2019-12-03T19:58:41Z" level=debug msg="Instance handler updated instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-r-fe7d2537 state, old state starting, new state running"
time="2019-12-03T19:58:42Z" level=debug msg="Prepare to create instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7"
time="2019-12-03T19:58:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7\", UID:\"124a03c0-95dc-4efc-8465-0b94fc050b40\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2578940\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7"
time="2019-12-03T19:58:42Z" level=debug msg="Prepare to create instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7"
time="2019-12-03T19:58:42Z" level=debug msg="Instance handler updated instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7 state, old state stopped, new state starting"
time="2019-12-03T19:58:43Z" level=debug msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7 starts running, IP <IP_ADDRESS>"
time="2019-12-03T19:58:43Z" level=debug msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7 starts running, Port 10015"
time="2019-12-03T19:58:43Z" level=debug msg="Instance handler updated instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-r-9660e4f7 state, old state starting, new state running"

Longhorn-Manager Log - Node 2:
time="2019-12-03T20:27:21Z" level=debug msg="Prepare to delete instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:21Z" level=debug msg="Prepare to delete instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38\", UID:\"ab9bc699-8eab-461e-abc5-7f2c8a88af16\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579917\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852\", UID:\"9286e522-2c78-46b0-b76e-04af57aa7182\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579918\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:21Z" level=error msg="failed to receive log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:21Z" level=warning msg="cannot get crash log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:21Z" level=error msg="failed to receive log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:21Z" level=warning msg="cannot get crash log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=debug msg="Prepare to delete instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:51Z" level=debug msg="Prepare to delete instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38\", UID:\"ab9bc699-8eab-461e-abc5-7f2c8a88af16\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579917\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852\", UID:\"9286e522-2c78-46b0-b76e-04af57aa7182\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579918\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:51Z" level=error msg="failed to receive log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:51Z" level=warning msg="cannot get crash log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:51Z" level=error msg="failed to receive log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=warning msg="cannot get crash log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=debug msg="Prepare to delete instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:51Z" level=debug msg="Prepare to delete instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852\", UID:\"9286e522-2c78-46b0-b76e-04af57aa7182\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579918\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:51Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:27:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38\", UID:\"ab9bc699-8eab-461e-abc5-7f2c8a88af16\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579917\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:27:51Z" level=error msg="failed to receive log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=warning msg="cannot get crash log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:27:51Z" level=error msg="failed to receive log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:27:51Z" level=warning msg="cannot get crash log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=debug msg="Prepare to delete instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:28:21Z" level=debug msg="Prepare to delete instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:28:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852\", UID:\"9286e522-2c78-46b0-b76e-04af57aa7182\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579918\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:28:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38\", UID:\"ab9bc699-8eab-461e-abc5-7f2c8a88af16\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579917\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:28:21Z" level=error msg="failed to receive log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=warning msg="cannot get crash log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=error msg="failed to receive log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=warning msg="cannot get crash log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=debug msg="Prepare to delete instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:28:21Z" level=debug msg="Prepare to delete instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:28:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852\", UID:\"9286e522-2c78-46b0-b76e-04af57aa7182\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579918\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 is state , error message: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=warning msg="Instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 crashed on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, try to get log"
time="2019-12-03T20:28:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38\", UID:\"ab9bc699-8eab-461e-abc5-7f2c8a88af16\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2579917\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38"
time="2019-12-03T20:28:21Z" level=error msg="failed to receive log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852: rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=warning msg="cannot get crash log for instance pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-4f51cfb2-d7e6-4838-bc95-60bfc59b4ad3-e-4930b852-51a6872b"
time="2019-12-03T20:28:21Z" level=error msg="failed to receive log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38: rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"
time="2019-12-03T20:28:21Z" level=warning msg="cannot get crash log for instance pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38 on Instance Manager instance-manager-e-dab1acdb at ehs-kn-14, error rpc error: code = NotFound desc = cannot find process pvc-cb2be6c5-7071-4e25-9f9b-6d1afcde4d74-e-8fb8db38-ee00e89b"

This is a bug and I am suspecting the root cause is the same as #854 :
The engine manager deleted the process for the volume but cannot unregister the related launcher. Then the volume detachment gets stuck there.
The PR for this issue is here: https://github.com/longhorn/longhorn-engine/pull/436
One possible workaround:

Find the related engine manager pod and the node on which the volume gets stuck.
Detach the volumes on that node.
Restart the engine manager pod.

I see the PR failed two weeks ago, is there a rough ETA on when this issue might be fixed? It's blocking us from migrating to our new cluster. Has the entire project on hold because we can't afford to have pods stuck in detaching during off hours.

@DarkRider1768 It will be fixed in the next release, due in late January. In fact, it's a bit weird in your case, since the problem is a race condition, it should be rarely triggered. If all of your volumes hanging on detach, that might be something else.
You mentioned you're using another node OS before. Maybe give that a try?

We were running a minimal install of CentOS but the maintenance requirements of deploying that across a larger cluster is going to prevent us from going that route. The above numbers are a pre-rollout figure. Plus, the cluster as a whole wasn't terribly stable. We were having nodes do all sorts of weird things here and there, for no apparent reason. Longhorn worked great but other things would break.

@DarkRider1768 As a first step, can you send us a support bundle at<EMAIL_ADDRESS>? We can get more information for debugging in this way.

Alright, sent out the support bundle. You guys should be getting that any time.

@shuo-wu
would please provide reproduce steps for the issue?

Reproduce Step:

Launch Longhorn v0.7.0/master
Create and attach a volume. There should be only one volume in the Longhorn system.
Log into the corresponding engine manager pod, run

tgtadm --lld iscsi --mode logicalunit --op delete --tid 1 --lun 1

Detach the volume

For Longhorn v0.7.0, the volume will get stuck in the Detaching state. And the engine will keep printing the error logs like:

[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:05Z" level=info msg="Engine Manager: prepare to delete engine test1-e-5ee0c1f3"
[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:05Z" level=debug msg="engine launcher test1-e-5ee0c1f3: prepare to stop engine test1-e-5ee0c1f3-85b5a5f7"
[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:05Z" level=debug msg="Process Manager: prepare to delete process test1-e-5ee0c1f3-85b5a5f7"
[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:05Z" level=debug msg="Engine Manager: start getting logs for engine test1-e-5ee0c1f3"
[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:05Z" level=debug msg="Process Manager: start getting logs for process test1-e-5ee0c1f3-85b5a5f7"
[instance-manager-e-592460e3] [longhorn-instance-manager] time="2020-01-06T20:00:22Z" level=warning msg="failed to get the related process info for engine test1-e-5ee0c1f3-85b5a5f7: rpc error: code = NotFound desc = cannot find process test1-e-5ee0c1f3-85b5a5f7"

......

For Longhorn master, the volume can be detached after waiting for a while and the engine manager won't constantly print out error logs.

......

[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:07Z" level=warning msg="failed to get the related process info for engine test1-e-0ca6c17d-adabf8c2: rpc error: code = NotFound desc = cannot find process test1-e-0ca6c17d-adabf8c2"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:07Z" level=debug msg="engine launcher test1-e-0ca6c17d: finishing frontend shutdown"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:08Z" level=info msg="Shutdown SCSI target iqn.2014-09.com.rancher:test1"
[instance-manager-e-65072b26] tgtd: target_mgmt(147) Failed to unbind by address: ALL
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:08Z" level=error msg="Engine Manager fails to cleanup frontend before unregistering engine launcher test1-e-0ca6c17d, will retry it: failed to callback for engine test1-e-0ca6c17d frontend shutdown: device test1: fail to stop SCSI device: Fail to delete target: Failed to execute: tgtadm [--lld iscsi --op unbind --mode target --tid 1 -I ALL], output tgtadm: this access control rule does not exist\n, error exit status 22"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:08Z" level=error msg="Engine Manager fails to cleanup frontend, will give up it and continue unregistering engine launcher test1-e-0ca6c17d"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:08Z" level=info msg="Engine Manager had successfully unregistered engine launcher test1-e-0ca6c17d, deletion completed"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:08Z" level=warning msg="failed to get the related process info for engine test1-e-0ca6c17d-adabf8c2: rpc error: code = NotFound desc = cannot find process test1-e-0ca6c17d-adabf8c2"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:10Z" level=error msg="Engine Manager fails to cleanup frontend, will give up it and continue unregistering engine launcher test1-e-0ca6c17d"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:10Z" level=info msg="Engine Manager had successfully unregistered engine launcher test1-e-0ca6c17d, deletion completed"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:10Z" level=warning msg="failed to get the related process info for engine test1-e-0ca6c17d-adabf8c2: rpc error: code = NotFound desc = cannot find process test1-e-0ca6c17d-adabf8c2"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:11Z" level=error msg="Engine Manager fails to cleanup frontend, will give up it and continue unregistering engine launcher test1-e-0ca6c17d"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:11Z" level=info msg="Engine Manager had successfully unregistered engine launcher test1-e-0ca6c17d, deletion completed"
[instance-manager-e-65072b26] [longhorn-instance-manager] time="2020-01-06T20:19:11Z" level=warning msg="failed to get the related process info for engine test1-e-0ca6c17d-adabf8c2: rpc error: code = NotFound desc = cannot find process test1-e-0ca6c17d-adabf8c2"

Do cleanup by running

tgtadm --lld iscsi --mode target --op delete --tid 1

Validation: PASSED