Fragments shouldn't have variables in default constructor

Describe the bug
In both GooglePayFragment and  PaymentSheetFragment we create a fragment and inject variables in the constructor. This is considered a bad practice and can lead to undesired side effects.  Reason for having an 0 arg constructor is that Android can recreate the fragment itself when config changes.
See this link  for a good explanation of the issue
To Reproduce
We got issues in our library that the app crashes randomly when users put the app in the background. See: https://github.com/flutter-stripe/flutter_stripe/issues/996.
Reproduction is hard some could reproduce it like:

Enable "Don't keep activities" in Developer Options.
After calling Stripe.instance.initPaymentSheet(...) somewhere, put the app in background.
Re-enter the app.
Result: The app crashes after splash with the following stacktrace:

E/AndroidRuntime( 1575): java.lang.RuntimeException: Unable to start activity ...
androidx.fragment.app.Fragment$InstantiationException: Unable to instantiate fragment com.reactnativestripesdk.PaymentSheetFragment: could not find Fragment constructor
E/AndroidRuntime( 1575): 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4166)
E/AndroidRuntime( 1575): 	at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4312)
E/AndroidRuntime( 1575): 	at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:101)
E/AndroidRuntime( 1575): 	at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
E/AndroidRuntime( 1575): 	at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
E/AndroidRuntime( 1575): 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2571)
E/AndroidRuntime( 1575): 	at android.os.Handler.dispatchMessage(Handler.java:106)
E/AndroidRuntime( 1575): 	at android.os.Looper.loopOnce(Looper.java:226)
E/AndroidRuntime( 1575): 	at android.os.Looper.loop(Looper.java:313)
E/AndroidRuntime( 1575): 	at android.app.ActivityThread.main(ActivityThread.java:8741)
E/AndroidRuntime( 1575): 	at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime( 1575): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
E/AndroidRuntime( 1575): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)
E/AndroidRuntime( 1575): Caused by: androidx.fragment.app.Fragment$InstantiationException: Unable to instantiate fragment com.reactnativestripesdk.PaymentSheetFragment: could not find Fragment constructor
E/AndroidRuntime( 1575): 	at androidx.fragment.app.Fragment.instantiate(Fragment.java:678)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentContainer.instantiate(FragmentContainer.java:57)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentManager$3.instantiate(FragmentManager.java:507)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentState.instantiate(FragmentState.java:81)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentStateManager.<init>(FragmentStateManager.java:85)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentManager.restoreSaveStateInternal(FragmentManager.java:2505)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentManager.attachController(FragmentManager.java:2665)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentController.attachHost(FragmentController.java:117)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentActivity.lambda$init$3$androidx-fragment-app-FragmentActivity(FragmentActivity.java:140)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentActivity$$ExternalSyntheticLambda0.onContextAvailable(Unknown Source:2)
E/AndroidRuntime( 1575): 	at androidx.activity.contextaware.ContextAwareHelper.dispatchOnContextAvailable(ContextAwareHelper.java:99)
E/AndroidRuntime( 1575): 	at androidx.activity.ComponentActivity.onCreate(ComponentActivity.java:362)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.FragmentActivity.onCreate(FragmentActivity.java:217)
E/AndroidRuntime( 1575): 	at io.flutter.embedding.android.FlutterFragmentActivity.onCreate(FlutterFragmentActivity.java:279)
E/AndroidRuntime( 1575): 	... 15 more
E/AndroidRuntime( 1575): Caused by: java.lang.NoSuchMethodException: com.reactnativestripesdk.PaymentSheetFragment.<init> []
E/AndroidRuntime( 1575): 	at java.lang.Class.getConstructor0(Class.java:2363)
E/AndroidRuntime( 1575): 	at java.lang.Class.getConstructor(Class.java:1759)
E/AndroidRuntime( 1575): 	at androidx.fragment.app.Fragment.instantiate(Fragment.java:663)
E/AndroidRuntime( 1575): 	... 30 more

Expected behavior
The fragments should have an empty constructor so Android can recreate it when needed.
Desktop (please complete the following information):
Android

Getting lots of reports of this crash from our app in prod

Any reason this has not been fixed yet? Curious, as wee see crashes from this in our production app.

This is the same issue that's causing https://github.com/stripe/stripe-react-native/issues/1425 and https://github.com/flutter-stripe/flutter_stripe/issues/1282#issuecomment-2422406442 within flutter_stripe, as outlined in this comment.
Whatever payment method you choose, whether it's GooglePay or something else, and whether you use initPaymentSheet or confirmPayment, when you are directed back to your application it will crash 100% of the time if you follow the steps here.

Any chance of this issue being fixed any time soon? We're getting more and more users that are encountering crashes due to it. thanks!

We are having many crash rates related to this issue.

Any update on this one? Would also appreciate a solution very much!!

Bump! We are having crashes related to this.

Lots of crashes because of this. Pls fix!

Thanks for letting us know about this! We are looking into fixing this

We also have many many crashes because of this, we hope you can fix it short term!

@amk-stripe Is there an update on this?

A year has passed in silence.

@amk-stripe 290 crashes in the last 7 days:


About the dataset and the nnunet framework

Thank you for your valuable work. I noticed that the code you uploaded on GitHub is the core module. I would like to ask if I should download the code you provided and place it within the nnUNet framework for training and testing, correct? Additionally, is the skeleton information extracted from the original data and then incorporated into the nnUNet data format, or is the skeleton information automatically extracted during training?
Thanks again and I looking forward your reply.

Also, could you please let me know what version of the virtual environment I need to install? Is it the same as the one for nnUNet V2? Additionally, should I install the libraries mentioned in your README, such as:
pip install monai
For CUDA 12.x
pip install cucim-cu12
pip install cupy==12.3
For CUDA 11.x
pip install cucim-cu11
pip install cupy-cuda11x

Hello, thank you for your questions!
Regarding your question, there are two ways to implement the cbDice loss in your project.
Option 1: Using the nnUNet Framework
If you are using the nnUNet framework, you can integrate the cbDice loss directly by copying the loss code to the nnUNet-2.2/nnunetv2/training/loss/ directory. Additionally, move the nnUNetTrainer_variants code to the nnUNet-2.2/nnunetv2/training/nnUNetTrainer/ directory. After setting this up, you will need to install the nnUNet environment, along with monai, cucim, and cupy.
Option 2: Without the nnUNet Framework
Alternatively, if you aren’t using the nnUNet framework, you can integrate cbDice_loss.py and compound_cbdice_loss.py directly into your training code. Check out the example code on GitHub (nnUNetTrainer_CE_DC_CBDC.py#L25) for further details on parameter setup. For instance, you can configure the loss as follows:
loss = DC_and_CE_and_CBDC_loss(
    {'batch_dice': True, 'smooth': 1e-5, 'do_bg': False, 'ddp': False},
    {},
    {'iter_': 10, 'smooth': 1e-3},
    weight_ce=lambda_ce, weight_dice=lambda_dice, weight_cbdice=lambda_cbdice,
    ignore_label=None, dice_class=MemoryEfficientSoftDiceLoss
)

In this setup, you only need a few packages such as torch, monai, cucim, and cupy.
Skeleton Information
The skeleton information is automatically extracted during training when using cbDice loss. You can refer to the specific extraction steps in the cbDice loss code or the clDice loss code if you're using that variant.
Installation of Required Libraries
For the virtual environment, you should set up an environment compatible with nnUNet V2. The required libraries and installation steps are as follows:

Install monai:
pip install monai

For CUDA 12.x:
pip install cucim-cu12
pip install cupy==12.3

For CUDA 11.x:
pip install cucim-cu11
pip install cupy-cuda11x

These libraries support GPU-accelerated operations, like the distance transform, which cbDice loss uses for efficient training (link).
Thank you once again, and feel free to reach out with any additional questions.

@PengchengShi1220 Thank you for your very detailed answers; they are incredibly helpful. I appreciate it!

Hello, i integrated cbDice_loss.py and compound_cbdice_loss.py into my training code, but i got an error that i can't fix:
RuntimeError
CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
  File "/ai/code/project/imed_vision_main_mlla_skip/loss/cbdice_loss.py", line 127, in get_weights
    distances = distance_transform_edt(mask).float()
  File "/ai/code/project/imed_vision_main_mlla_skip/loss/cbdice_loss.py", line 47, in forward
    q_vl, q_slvl, q_sl = get_weights(y_true, skel_true, dim, prob_flag=False)
  File "/ai/code/project/imed_vision_main_mlla_skip/loss/compound_cbdice_loss.py", line 60, in forward
    cbdice_loss = self.cbdice(net_output, target, t_skeletonize_flage=t_skeletonize_flage) if self.weight_cbdice != 0 else 0
  File "/ai/code/project/imed_vision_main_mlla_skip/vessel_segmentation_train.py", line 553, in main_worker
    loss = seg_loss(pred, mask)
  File "/ai/code/project/imed_vision_main_mlla_skip/vessel_segmentation_train.py", line 127, in main
    main_worker(args.gpu, ngpus_per_node, args, config)
  File "/ai/code/project/imed_vision_main_mlla_skip/vessel_segmentation_train.py", line 722, in <module>
    main(config)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

torch: 2.1.1+cu118
monai：1.3.2 or 1.4.0 (both tried)
Is it because the version of PyTorch I am using is not compatible with the MONAI version?
Could you tell me which versions of PyTorch and MONAI are you using?

@PengchengShi1220

Hello, in the cbDice_cal_demo folder, there are three scenarios: translation, deformation, and diameter imbalance. However, I am uncertain about which specific scenario I should use the evaluation metrics from these files for, as I believe there is some overlap in functionality between these three files. Additionally, I would like to ask if I can directly place this code in the nnUNet/nnunet/evaluation/ path for seamless integration with nnU-Net. Furthermore, I noticed that the implementation code for Betti error and Betti matching is provided, but I would like to inquire where the implementation for the μ error, as mentioned in the paper, can be found. Is the μ error the same as the Betti matching error?
Thanks again and I looking forward your reply.

Hello,
Thank you for your questions! I’m happy to clarify:

About the cbDice_cal_demo folder: This folder provides examples of different evaluation metrics calculated in 2D for three scenarios: translation, deformation, and diameter imbalance. These examples are designed to demonstrate the functionality for 2D evaluation. If you want to calculate metrics like Betti error in 3D, I recommend using the official TopCoW evaluation method provided seg_metrics.

Integration with nnU-Net: Yes, you can integrate the code into nnunetv2/evaluation/evaluate_predictions.py.

Implementation of the μ error: The μ error is distinct from the Betti matching error. The Betti matching implementation provided is specific to 2D images. The original repository for Betti matching is available Betti-matching, and the relevant code for calculations can be found calculation.

If you have further questions, feel free to ask!

@PengchengShi1220
Thanks for you reply. I truly appreciate it.

@PengchengShi1220
Hi,
I have a question that I find a bit confusing. Why is it necessary to rewrite the nnUNetTrainer.py file to remove the deep supervision components (creating nnUNetTrainerNoDeepSupervision.py)?
Can I still use the deep supervision mechanism along with the loss function? What are the differences between using deep supervision and not using it?
I look forward to your response.
Thank you!

Hi,
Thank you for your question!
Yes, you can use deep supervision with the loss function. For example, check out the deep supervision versions of cbDice and clDice. Time comparisons are also available here: Time Comparisons.
In our paper, we used the version without deep supervision because Swin UNETR does not support it. To ensure fairness, all models excluded deep supervision. The primary role of deep supervision is to apply loss constraints to the decoder features at different scales (stages).

@PengchengShi1220 Hello! Does b_dou_loss.py only support 2D data? I would like to apply it to 3D data, but the results are not ideal. Could you please provide a version of the BoundaryDoULoss that supports 3D data? I would greatly appreciate it! Looking forward to your response. Thank you!

Hello,
The current b_dou_loss.py only supports 2D data, as noted in the original repository: BoundaryDoULoss.

@PengchengShi1220  Hello, I integrated your code into my nnU-Net network, but I encountered a very strange error. This error sometimes appears at the beginning of training, and other times it shows up after a few epochs. I'm not sure what the issue is. I checked that CUDA is working fine and set the number of threads to 1, but the error still occurs. Do you know what might be causing this error? Thank you.
2024-12-24 23:59:47.358380: Epoch 0
2024-12-24 23:59:47.358465: Current learning rate: 0.01
using pin_memory on device 0
terminate called after throwing an instance of 'c10::Error'
what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f486ce944d7 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f486ce5e36b in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f486cf30b58 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3:  + 0x1250cb7 (0x7f486e241cb7 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x4d5a16 (0x7f48d38afa16 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #5:  + 0x3ee77 (0x7f486ce79e77 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #6: c10::TensorImpl::~TensorImpl() + 0x1be (0x7f486ce7269e in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f486ce727b9 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8:  + 0x75afc8 (0x7f48d3b34fc8 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: THPVariable_subclass_dealloc(_object*) + 0x305 (0x7f48d3b35355 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x590cde]
frame #11: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x5b64e6]
frame #12: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x4e0973]
frame #13: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x58e9bc]
frame #14: PyFunction_NewWithQualName + 0x509 (0x4f4d69 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #15: _PyEval_EvalFrameDefault + 0x1d3d (0x4ef0ad in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #16: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #18: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #19: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #20: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #21: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #22: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #24: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #27: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #29: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #30: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x5417d3]
frame #31: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x57db1a]
frame #32: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x56ee2f]
frame #33: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x4fd124]
frame #34: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #35: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #36: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #37: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #38: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #39: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x50832e]
frame #40: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #41: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #42: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #43: _PyObject_FastCallDictTstate + 0xcd (0x4f5a4d in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #44: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x506068]
frame #45: _PyObject_MakeTpCall + 0x2ab (0x4f65cb in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #46: _PyEval_EvalFrameDefault + 0x4deb (0x4f215b in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #47: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #48: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #49: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #50: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #51: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #52: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #53: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #54: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #55: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #56: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #57: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x5417d3]
frame #58: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x57db1a]
frame #59: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x56ee2f]
frame #60: /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python() [0x4fd124]
frame #61: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #62: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
frame #63: _PyEval_EvalFrameDefault + 0x731 (0x4edaa1 in /mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/python)
Exception in thread Thread-5 (results_loop):
Traceback (most recent call last):
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
using pin_memory on device 0
2024-12-25 00:10:09.574858: train_loss -0.135
2024-12-25 00:10:09.575069: val_loss -0.4926
2024-12-25 00:10:09.575137: Pseudo dice [0.3565]
2024-12-25 00:10:09.575204: Epoch time: 622.22 s
2024-12-25 00:10:09.575253: Yayy! New best EMA pseudo Dice: 0.3565
2024-12-25 00:10:12.176431:
2024-12-25 00:10:12.176547: Epoch 1
2024-12-25 00:10:12.176640: Current learning rate: 0.00982
2024-12-25 00:21:20.733166: train_loss -0.6756
2024-12-25 00:21:20.733335: val_loss -0.8606
2024-12-25 00:21:20.733397: Pseudo dice [0.6318]
2024-12-25 00:21:20.733454: Epoch time: 668.56 s
2024-12-25 00:21:20.733501: Yayy! New best EMA pseudo Dice: 0.384
2024-12-25 00:21:23.700732:
2024-12-25 00:21:23.700881: Epoch 2
2024-12-25 00:21:23.700993: Current learning rate: 0.00964
2024-12-25 00:32:50.951932: train_loss -0.8531
2024-12-25 00:32:50.952155: val_loss -0.9592
2024-12-25 00:32:50.952236: Pseudo dice [0.6839]
2024-12-25 00:32:50.952487: Epoch time: 687.25 s
2024-12-25 00:32:50.952558: Yayy! New best EMA pseudo Dice: 0.414
2024-12-25 00:32:55.227058:
2024-12-25 00:32:55.227180: Epoch 3
2024-12-25 00:32:55.227287: Current learning rate: 0.00946
2024-12-25 00:43:14.290602: train_loss -0.9139
2024-12-25 00:43:14.290775: val_loss -0.9777
2024-12-25 00:43:14.290849: Pseudo dice [0.7089]
2024-12-25 00:43:14.290909: Epoch time: 619.06 s
2024-12-25 00:43:14.290957: Yayy! New best EMA pseudo Dice: 0.4435
2024-12-25 00:43:16.982140:
2024-12-25 00:43:16.982423: Epoch 4
2024-12-25 00:43:16.982525: Current learning rate: 0.00928
2024-12-25 00:53:49.466644: train_loss -0.9736
2024-12-25 00:53:49.473568: val_loss -1.0333
2024-12-25 00:53:49.473711: Pseudo dice [0.7404]
2024-12-25 00:53:49.473777: Epoch time: 632.49 s
2024-12-25 00:53:49.473825: Yayy! New best EMA pseudo Dice: 0.4732
2024-12-25 00:53:52.134172:
2024-12-25 00:53:52.134284: Epoch 5
2024-12-25 00:53:52.134372: Current learning rate: 0.0091
2024-12-25 01:04:44.714201: train_loss -0.9785
2024-12-25 01:04:44.719198: val_loss -0.8979
2024-12-25 01:04:44.719267: Pseudo dice [0.7017]
2024-12-25 01:04:44.719314: Epoch time: 652.58 s
2024-12-25 01:04:44.719350: Yayy! New best EMA pseudo Dice: 0.496
2024-12-25 01:04:47.126955:
2024-12-25 01:04:47.127060: Epoch 6
2024-12-25 01:04:47.127145: Current learning rate: 0.00891
Traceback (most recent call last):
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/bin/nnUNetv2_train", line 33, in
sys.exit(load_entry_point('nnunetv2', 'console_scripts', 'nnUNetv2_train')())
File "/mnt/hdd/sdb/changyw/U-Mamba/umamba/nnunetv2/run/run_training.py", line 268, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/mnt/hdd/sdb/changyw/U-Mamba/umamba/nnunetv2/run/run_training.py", line 204, in run_training
nnunet_trainer.run_training()
File "/mnt/hdd/sdb/changyw/U-Mamba/umamba/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1272, in run_training
val_outputs.append(self.validation_step(next(self.dataloader_val)))
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in next
item = self.__get_next_item()
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Exception in thread Thread-4 (results_loop):
Traceback (most recent call last):
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/mnt/hdd/sdb/changyw/anaconda3/envs/umamba/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message

Hello,
Based on your error description, it seems you might be encountering a potential memory leak, which is similar to the issue discussed in [this GitHub issue](https://github.com/PengchengShi1220/cbDice/issues/2). One way to verify this is by monitoring your GPU memory usage during training to see if it gradually increases. If this is indeed the case, it's likely that there may be an issue with the installation of cucim and cupy.
I noticed that you're using CUDA version 10. If possible, I recommend upgrading to CUDA 11 or 12, as they might resolve the issue. You can install the necessary versions as follows:

For CUDA 12.x:
pip install cucim-cu12
pip install cupy==12.3

For CUDA 11.x:
pip install cucim-cu11
pip install cupy-cuda11x

Let me know if this helps, or if the issue persists!


Code Ready Containers becomes 'Not Reachable' due to etcd crashing and unable to restart

General information

OS: Linux
Hypervisor: KVM
Did you run crc setup before starting it (Yes/No)? Yes

CRC version
# Put the output of `crc version`
CodeReady Containers version: 1.21.0+68a4cdd7
OpenShift version: 4.6.9 (embedded in executable)

CRC status
# Put the output of `crc status`
CRC VM:          Running
OpenShift:       Not Reachable (v4.6.9)
Disk Usage:      25.71GB of 74.6GB (Inside the CRC VM)
Cache Usage:     27.04GB
Cache Directory: /home/crcuser/.crc/cache

CRC config
# Put the output of `crc config view`
- consent-telemetry                     : no
- cpus                                  : 12
- disk-size                             : 70
- enable-cluster-monitoring             : true
- memory                                : 48000

Host Operating System
# Put the output of `cat /etc/os-release` in case of Linux
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

Steps to reproduce

Start CRC, leave it running
After 1-7 days (perhaps more... unclear)
CRC OpenShift API stops responding and status shows 'Not Reachable'
Some container workloads (e.g. other pods, services, routes for applications) stay operational
Stopping / restarting crc does not recover

Expected
CRC OpenShift API functions allowing oc login and other actions on the cluster against the API:
oc login -u developer -p developer
Login successful.

You have one project on this server: "victim"

Using project "victim".

Actual
Unable to login with cli or web console:
oc login -u developer -p developer
error: dial tcp <IP_ADDRESS>:6443: connect: connection refused - verify you have provided the correct host and port and that the server is currently running.

Logs
You can start crc with crc start --log-level debug to collect logs.
Link to gist with logs

pretty much a duplicate of https://github.com/code-ready/crc/issues/1786
copying my notes from that to here:
Possible to recover using crc stop; crc delete; crc start
When restarting crc w/o using delete the etcd pod is failing at the environment variable checking InitContainer with a log message of (after ssh to crc vm):
# cat etcd-crc-fdm75-master-0_openshift-etcd_etcd-ensure-env-vars-d5249245463ac1eadc44c03d0bd8217efd6a24f4f598c8e93c31831e5394be45.log
2020-12-22T00:35:21.651476376+00:00 stderr F Expected node IP to be <IP_ADDRESS> got <IP_ADDRESS>

If crc is restarted with stop then start it will come up ok as this is done before the time when the API becomes unresponsive. This message only appears after a crc status command shows that OpenShift is "Not Reachable".

pretty much a duplicate of https://github.com/code-ready/crc/issues/1786
copying my notes from that to here:
Possible to recover using crc stop; crc delete; crc start
When restarting crc w/o using delete the etcd pod is failing at the environment variable checking InitContainer with a log message of (after ssh to crc vm):
# cat etcd-crc-fdm75-master-0_openshift-etcd_etcd-ensure-env-vars-d5249245463ac1eadc44c03d0bd8217efd6a24f4f598c8e93c31831e5394be45.log
2020-12-22T00:35:21.651476376+00:00 stderr F Expected node IP to be <IP_ADDRESS> got <IP_ADDRESS>

If crc is restarted with stop then start it will come up ok as this is done before the time when the API becomes unresponsive. This message only appears after a crc status command shows that OpenShift is "Not Reachable".

Before restarting to gather debug logs, I also checked on the CRC vm and noted the same issue:
# tail -f /var/log/containers/etcd-crc-lf65c-master-0_openshift-etcd_etcd-ensure-env-vars-a9061e40b584245587e27a4efc00c8866c1b8ce500b4f8903e7678294e23e70e.log
2021-01-26T00:49:15.330307319+00:00 stderr F Expected node IP to be <IP_ADDRESS> got <IP_ADDRESS>

Before restarting to gather debug logs, I also checked on the CRC vm and noted the same issue:
# tail -f /var/log/containers/etcd-crc-lf65c-master-0_openshift-etcd_etcd-ensure-env-vars-a9061e40b584245587e27a4efc00c8866c1b8ce500b4f8903e7678294e23e70e.log
2021-01-26T00:49:15.330307319+00:00 stderr F Expected node IP to be <IP_ADDRESS> got <IP_ADDRESS>

@guillaumerose as you commented on the code in the etcd operator in the other issue, I looked a little more at the state of the failing vm configuration and compared that to a working crc instance.
For the failing VM - in the /etc/kubernetes/manifests/etcd-pod.yaml file all values that had the IP <IP_ADDRESS> are now showing <IP_ADDRESS>

@guillaumerose as you commented on the code in the etcd operator in the other issue, I looked a little more at the state of the failing vm configuration and compared that to a working crc instance.
For the failing VM - in the /etc/kubernetes/manifests/etcd-pod.yaml file all values that had the IP <IP_ADDRESS> are now showing <IP_ADDRESS>

In this case, the host machine is an Intel-based VM running on IBM Cloud. I have seen this problem as well on a  I've seen on a native AMD CPU.
One difference between working and failed vm's is the change of IP address in the /etc/kubernetes/manifests/etcd-pod.yaml file.

In this case, the host machine is an Intel-based VM running on IBM Cloud. I have seen this problem as well on a  I've seen on a native AMD CPU.
One difference between working and failed vm's is the change of IP address in the /etc/kubernetes/manifests/etcd-pod.yaml file.

Details from dmidecode and lscpu:  https://gist.github.com/timroster/02b185d97b51fd64c3a77eda1a438e59

Details from dmidecode and lscpu:  https://gist.github.com/timroster/02b185d97b51fd64c3a77eda1a438e59

@gbraad as to running a full OpenShift install to this particular VM type, it's complicated. It may be easier for me to round up an intel-based host to replicate.

@gbraad as to running a full OpenShift install to this particular VM type, it's complicated. It may be easier for me to round up an intel-based host to replicate.

Thanks @timroster for the issue. There is clearly something wrong with the dummy interface.

Thanks @timroster for the issue. There is clearly something wrong with the dummy interface.

The dummy interface still is present on the system - and it appears to still be the interface of record for the hostname:
/sbin/ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:fd:fc:07:21:82 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global dynamic noprefixroute enp1s0
       valid_lft 3051sec preferred_lft 3051sec
    inet6 fe80::50fd:fcff:fe07:2182/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
7: eth10: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 72:ce:aa:19:5d:f5 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global noprefixroute eth10
       valid_lft forever preferred_lft forever
    inet6 fe80::9bf2:9c50:7e4b:6bef/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
 ...

[root@crc-lf65c-master-0 ~]# hostname
crc-lf65c-master-0
[root@crc-lf65c-master-0 ~]# hostname -i
<IP_ADDRESS>
[root@crc-lf65c-master-0 ~]# hostname -I
<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>
[root@crc-lf65c-master-0 ~]# ping `hostname`
PING crc-lf65c-master-0.crc.testing (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=3 ttl=64 time=0.042 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=4 ttl=64 time=0.037 ms
^C

The dummy interface still is present on the system - and it appears to still be the interface of record for the hostname:
/sbin/ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:fd:fc:07:21:82 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global dynamic noprefixroute enp1s0
       valid_lft 3051sec preferred_lft 3051sec
    inet6 fe80::50fd:fcff:fe07:2182/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
7: eth10: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 72:ce:aa:19:5d:f5 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global noprefixroute eth10
       valid_lft forever preferred_lft forever
    inet6 fe80::9bf2:9c50:7e4b:6bef/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
 ...

[root@crc-lf65c-master-0 ~]# hostname
crc-lf65c-master-0
[root@crc-lf65c-master-0 ~]# hostname -i
<IP_ADDRESS>
[root@crc-lf65c-master-0 ~]# hostname -I
<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>
[root@crc-lf65c-master-0 ~]# ping `hostname`
PING crc-lf65c-master-0.crc.testing (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=3 ttl=64 time=0.042 ms
64 bytes from crc-lf65c-master-0 (<IP_ADDRESS>): icmp_seq=4 ttl=64 time=0.037 ms
^C

For the failing VM - in the /etc/kubernetes/manifests/etcd-pod.yaml file all values that had the IP <IP_ADDRESS> are now showing <IP_ADDRESS>

So something happened (I'd suspect some openshift component crashed?), which caused /etc/kubernetes/manifests/etcd-pod.yaml to be regenerated, which then breaks the cluster because the dummy IP is not used in this manifest.
Does this happen on 'pristine' clusters (crc delete && crc start and then wait until it breaks)? Or are you doing more steps before leaving the clusetr for a few days (installing more operators/applications, enabling some operators which are disabled, other modifications to the VM?)

For the failing VM - in the /etc/kubernetes/manifests/etcd-pod.yaml file all values that had the IP <IP_ADDRESS> are now showing <IP_ADDRESS>

So something happened (I'd suspect some openshift component crashed?), which caused /etc/kubernetes/manifests/etcd-pod.yaml to be regenerated, which then breaks the cluster because the dummy IP is not used in this manifest.
Does this happen on 'pristine' clusters (crc delete && crc start and then wait until it breaks)? Or are you doing more steps before leaving the clusetr for a few days (installing more operators/applications, enabling some operators which are disabled, other modifications to the VM?)

When your system is failing, can you also check systemctl status kubelet?
If I understand your setup correctly, this is a CentOS VM running on IBM cloud on Intel-based hardware,  and this CentOS VM is then used to run crc? Wondering in on the CentOS host, there are some errors in /var/log/libvirt/qemu/crc.log?

When your system is failing, can you also check systemctl status kubelet?
If I understand your setup correctly, this is a CentOS VM running on IBM cloud on Intel-based hardware,  and this CentOS VM is then used to run crc? Wondering in on the CentOS host, there are some errors in /var/log/libvirt/qemu/crc.log?

So something happened (I'd suspect some openshift component crashed?), which caused /etc/kubernetes/manifests/etcd-pod.yaml to be regenerated, which then breaks the cluster because the dummy IP is not used in this manifest.

Yes, that appears to be what has happened, simply patching the manifest does not appear to resolve and a short period after patching it is re-written to the value without the dummy IP

Does this happen on 'pristine' clusters (crc delete && crc start and then wait until it breaks)? Or are you doing more steps before leaving the clusetr for a few days (installing more operators/applications, enabling some operators which are disabled, other modifications to the VM?)

I've seen it happen at least one time where I have only done a crc delete && crc start and just let it sit.

Once you are ssh'ed into the VM, I wonder if journalctl or sudo crictl ps -a/ sudo crictl logs could be useful to understand what happened. There might be hints buried in there, but openshift logs are very verbose, and I don't know what to look for.

I might have missed a window on this cycle. Since I've restarted the vm to get the debug logs I don't have journal history from the potential event. Will need to look next time.

So something happened (I'd suspect some openshift component crashed?), which caused /etc/kubernetes/manifests/etcd-pod.yaml to be regenerated, which then breaks the cluster because the dummy IP is not used in this manifest.

Yes, that appears to be what has happened, simply patching the manifest does not appear to resolve and a short period after patching it is re-written to the value without the dummy IP

Does this happen on 'pristine' clusters (crc delete && crc start and then wait until it breaks)? Or are you doing more steps before leaving the clusetr for a few days (installing more operators/applications, enabling some operators which are disabled, other modifications to the VM?)

I've seen it happen at least one time where I have only done a crc delete && crc start and just let it sit.

Once you are ssh'ed into the VM, I wonder if journalctl or sudo crictl ps -a/ sudo crictl logs could be useful to understand what happened. There might be hints buried in there, but openshift logs are very verbose, and I don't know what to look for.

I might have missed a window on this cycle. Since I've restarted the vm to get the debug logs I don't have journal history from the potential event. Will need to look next time.

When your system is failing, can you also check systemctl status kubelet?

gist with systemctl status kubelet
The output in this capture shows kubelet trying to reach etcd at <IP_ADDRESS>... and localhost, both failing as etcd is not running. If I check a working crc instance with netstat, the only connections I see to tcp 2379 are to <IP_ADDRESS>.

If I understand your setup correctly, this is a CentOS VM running on IBM cloud on Intel-based hardware, and this CentOS VM is then used to run crc? Wondering in on the CentOS host, there are some errors in /var/log/libvirt/qemu/crc.log?

Does not appear to be anything untoward between last successful vm start and the point where I shut down the vm to capture the debug logs:
2021-01-21 21:03:36.344+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
2021-01-21T21:03:36.762936Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2021-01-26T01:10:55.151492Z qemu-kvm: terminating on signal 15 from pid 154321 (/usr/sbin/libvirtd)
2021-01-26 01:10:55.796+0000: shutting down, reason=shutdown
2021-01-26 01:12:45.031+0000: starting up libvirt version: 6.0.0, package: 28.module_el8.3.0+555+a55c8938 (CentOS Buildsys<EMAIL_ADDRESS>2020-11-04-01:04:00, ), qemu version: 4.2.0qemu-kvm-4.2.0-34.module_el8.3.0+613+9ec9f184.1, kernel: 4.18.0-193.14.2.el8_2.x86_64, hostname: tor-crc-schematics-instance

gist with full file contents

When your system is failing, can you also check systemctl status kubelet?

gist with systemctl status kubelet
The output in this capture shows kubelet trying to reach etcd at <IP_ADDRESS>... and localhost, both failing as etcd is not running. If I check a working crc instance with netstat, the only connections I see to tcp 2379 are to <IP_ADDRESS>.

If I understand your setup correctly, this is a CentOS VM running on IBM cloud on Intel-based hardware, and this CentOS VM is then used to run crc? Wondering in on the CentOS host, there are some errors in /var/log/libvirt/qemu/crc.log?

Does not appear to be anything untoward between last successful vm start and the point where I shut down the vm to capture the debug logs:
2021-01-21 21:03:36.344+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
2021-01-21T21:03:36.762936Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2021-01-26T01:10:55.151492Z qemu-kvm: terminating on signal 15 from pid 154321 (/usr/sbin/libvirtd)
2021-01-26 01:10:55.796+0000: shutting down, reason=shutdown
2021-01-26 01:12:45.031+0000: starting up libvirt version: 6.0.0, package: 28.module_el8.3.0+555+a55c8938 (CentOS Buildsys<EMAIL_ADDRESS>2020-11-04-01:04:00, ), qemu version: 4.2.0qemu-kvm-4.2.0-34.module_el8.3.0+613+9ec9f184.1, kernel: 4.18.0-193.14.2.el8_2.x86_64, hostname: tor-crc-schematics-instance

gist with full file contents

The kubelet is still started with kubelet --node-ip=<IP_ADDRESS> even though its logs show it's trying to access <IP_ADDRESS>

The kubelet is still started with kubelet --node-ip=<IP_ADDRESS> even though its logs show it's trying to access <IP_ADDRESS>

One correction related to kubelet. I was looking further and realized that in fact the kubelet logs shown by systemctl in the gist are for a sub-process, specifically kube-apiserver . When this problem is occurring it looks like the configuration resources in /etc/kubernetes/manifests and /etc/kubernetes/static-pod-resources for both etcd and kube-apiserver have changed the target IP from <IP_ADDRESS> to <IP_ADDRESS> for the endpoint for etcd

One correction related to kubelet. I was looking further and realized that in fact the kubelet logs shown by systemctl in the gist are for a sub-process, specifically kube-apiserver . When this problem is occurring it looks like the configuration resources in /etc/kubernetes/manifests and /etc/kubernetes/static-pod-resources for both etcd and kube-apiserver have changed the target IP from <IP_ADDRESS> to <IP_ADDRESS> for the endpoint for etcd

Very interesting! I started a cluster and look for <IP_ADDRESS>. I don't see it in /etc/kubernetes nor in any etcd keys/values. It must be something updated by an operator.

Very interesting! I started a cluster and look for <IP_ADDRESS>. I don't see it in /etc/kubernetes nor in any etcd keys/values. It must be something updated by an operator.

Can you check the content of etcd operator and apiserver operator ? Also look at stopped containers for these containers, there is maybe a clue inside.

Can you check the content of etcd operator and apiserver operator ? Also look at stopped containers for these containers, there is maybe a clue inside.

I appear to have (at least temporarily) recovered the cluster. Several manual things were attempted, as I was experimenting with the pod configmaps in /etc/kubernetes/static-pod-resources . This directory appears to hold a history of static pod definitions with multiple versions. I had removed the latest versions, and then updated the etcd-pod.yaml values in /etc/kubernetes/manifests and then used crc stop; crc start without any success - in fact the etcd-pod.yaml contents had reverted.
Most recently, I ssh'd to the vm and then updated in place the /etc/kubernetes/static-pod-resources/etcd-pod-3/configmaps/etcd-pod/pod.yaml file switching <IP_ADDRESS> back to <IP_ADDRESS>, followed by fixing IP in /etc/kubernetes/manifests/etcd-pod.yaml. Immediately after this I performed systemctl restart kubelet .  This time - the kubelet appears to have read the new static pod definitions and the cluster began to recover.
$ crc status
CRC VM:          Running
OpenShift:       Degraded (v4.6.9)
Disk Usage:      27.79GB of 74.6GB (Inside the CRC VM)
Cache Usage:     27.04GB
Cache Directory: /home/crcuser/.crc/cache

I will monitor, and if it goes out again, I'll see if only the edits to the two resource files for the etcd static pod followed by a systemctl restart kubelet are sufficient to recover.

I appear to have (at least temporarily) recovered the cluster. Several manual things were attempted, as I was experimenting with the pod configmaps in /etc/kubernetes/static-pod-resources . This directory appears to hold a history of static pod definitions with multiple versions. I had removed the latest versions, and then updated the etcd-pod.yaml values in /etc/kubernetes/manifests and then used crc stop; crc start without any success - in fact the etcd-pod.yaml contents had reverted.
Most recently, I ssh'd to the vm and then updated in place the /etc/kubernetes/static-pod-resources/etcd-pod-3/configmaps/etcd-pod/pod.yaml file switching <IP_ADDRESS> back to <IP_ADDRESS>, followed by fixing IP in /etc/kubernetes/manifests/etcd-pod.yaml. Immediately after this I performed systemctl restart kubelet .  This time - the kubelet appears to have read the new static pod definitions and the cluster began to recover.
$ crc status
CRC VM:          Running
OpenShift:       Degraded (v4.6.9)
Disk Usage:      27.79GB of 74.6GB (Inside the CRC VM)
Cache Usage:     27.04GB
Cache Directory: /home/crcuser/.crc/cache

I will monitor, and if it goes out again, I'll see if only the edits to the two resource files for the etcd static pod followed by a systemctl restart kubelet are sufficient to recover.

I think I possibly found the issue. We define 2 times the same parameter node-ip in /etc/systemd/system/kubelet.service
ExecStart=/usr/bin/hyperkube \
    kubelet \
      --node-ip=<IP_ADDRESS> \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \

I don't know where this KUBELET_NODE_IP env variable is coming but it is definitely suspect!

I think I possibly found the issue. We define 2 times the same parameter node-ip in /etc/systemd/system/kubelet.service
ExecStart=/usr/bin/hyperkube \
    kubelet \
      --node-ip=<IP_ADDRESS> \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \

I don't know where this KUBELET_NODE_IP env variable is coming but it is definitely suspect!

Not sure if that is directly related. But something at some point while the cluster is running seems to be causing the static pod resource for etcd to be switched from <IP_ADDRESS> to <IP_ADDRESS>. And perhaps at the same time, or a later time, when kubelet restarts this pod, the conflict seems to keep it from starting (env-vars initContainer), causing the api server to start failing.

Not sure if that is directly related. But something at some point while the cluster is running seems to be causing the static pod resource for etcd to be switched from <IP_ADDRESS> to <IP_ADDRESS>. And perhaps at the same time, or a later time, when kubelet restarts this pod, the conflict seems to keep it from starting (env-vars initContainer), causing the api server to start failing.

I experimented a bit and found out that with cobra, only the the last parameter is accepted. It makes node-ip is always empty in crc.
kubelet code is then doing a lookup to get the IP. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L225
Then, it is compared to the list of all interfaces to see if it's valid (net.InterfaceAddrs()). Can you give me the output of ifconfig and route -n ?
If it doesn't match, it picks the interface of the default gateway.
I guess we fall in the last case.

I experimented a bit and found out that with cobra, only the the last parameter is accepted. It makes node-ip is always empty in crc.
kubelet code is then doing a lookup to get the IP. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L225
Then, it is compared to the list of all interfaces to see if it's valid (net.InterfaceAddrs()). Can you give me the output of ifconfig and route -n ?
If it doesn't match, it picks the interface of the default gateway.
I guess we fall in the last case.

From the system (now recovered) output is rather long so captured in ifconfig and route -n gist
These don't look too different from any other crc system.

From the system (now recovered) output is rather long so captured in ifconfig and route -n gist
These don't look too different from any other crc system.

Can you try with this bundle and see if it's better? https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/code-ready_snc/315/pull-ci-code-ready-snc-master-e2e-snc/1354454524017774592/artifacts/e2e-snc/gather-snc/

Can you try with this bundle and see if it's better? https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/code-ready_snc/315/pull-ci-code-ready-snc-master-e2e-snc/1354454524017774592/artifacts/e2e-snc/gather-snc/

Can you try with this bundle and see if it's better?

Just to confirm steps, use crc setup as usual, then replace the qcow2 file with the one from the bundle before crc start?

Can you try with this bundle and see if it's better?

Just to confirm steps, use crc setup as usual, then replace the qcow2 file with the one from the bundle before crc start?

figured it out:
crc start -p pull-secret
...
INFO Loading bundle: crc_libvirt_4.6.9.crcbundle ...
INFO Verifying bundle crc_libvirt_4.6.9.crcbundle ...
INFO Creating CodeReady Containers VM for OpenShift 4.7.0-0.nightly-2021-01-27-110023...

figured it out:
crc start -p pull-secret
...
INFO Loading bundle: crc_libvirt_4.6.9.crcbundle ...
INFO Verifying bundle crc_libvirt_4.6.9.crcbundle ...
INFO Creating CodeReady Containers VM for OpenShift 4.7.0-0.nightly-2021-01-27-110023...

Thanks @timroster for the issue. There is clearly something wrong with the dummy interface.

https://github.com/code-ready/crc/issues/1786#issuecomment-753557751
Reported that a while back, but wasn't sure about this. Glad this new issue got rid of some noise and moved forward; often several opinions/comments in a single issue make it harder to follow.

Thanks @timroster for the issue. There is clearly something wrong with the dummy interface.

https://github.com/code-ready/crc/issues/1786#issuecomment-753557751
Reported that a while back, but wasn't sure about this. Glad this new issue got rid of some noise and moved forward; often several opinions/comments in a single issue make it harder to follow.

figured it out:
INFO Creating CodeReady Containers VM for OpenShift 4.7.0-0.nightly-2021-01-27-110023...

I will keep this instance running and monitor status. May take a little while to see any clear signs one way or another.

OpenShift 4.7 nightly had not been marked as stable last time I checked. Reported this as yellow on the status report for CRC. @guillaumerose ?
You use a different bundle with the -b option on start.

figured it out:
INFO Creating CodeReady Containers VM for OpenShift 4.7.0-0.nightly-2021-01-27-110023...

I will keep this instance running and monitor status. May take a little while to see any clear signs one way or another.

OpenShift 4.7 nightly had not been marked as stable last time I checked. Reported this as yellow on the status report for CRC. @guillaumerose ?
You use a different bundle with the -b option on start.

Yes not the best bundle to test sorry. I need to prepare the snc 4.6 branch for the next release, there will be a better bundle available then.

Yes not the best bundle to test sorry. I need to prepare the snc 4.6 branch for the next release, there will be a better bundle available then.

@guillaumerose fwiw - the crc instance that I "recovered" and subject of this issue has failed again, whereas the other instance that I started with the bundle you recommended is still running ok.
@cfergeau looking at relatively recent output from sudo crictl ps -a seems to look like there was some processes kicking off in the cluster 8 and 12 hours before the command was run, and there are a broad range of containers that are cycling due to etcd no longer available. Based on journalctl timestamps, the failure appeared to be about 8 hours ago.
output from journalctl was quite voluminous, but I was able to find the minute where things appeared to go from ok to bad by searching on Readiness probe for "etcd-crc. Critical event appears to be happening here (times UTC):
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722255  657366 secret.go:207] Received secret openshift-cluster-version/default-token-b6lgm containing (4) pieces of data, 16604 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722087  657366 configmap.go:188] Setting up volume config for pod f867fc1c-6ece-4518-ae47-366636ee052d at /var/lib/kubelet/pods/f867fc1c-6ece-4518-ae47-366636ee052d/volumes/kubernetes.io~configmap/config
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722965  657366 configmap.go:212] Received configMap openshift-cluster-machine-approver/machine-approver-config containing (0) pieces of data, 0 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.721078  657366 secret.go:183] Setting up volume serving-cert for pod 77e7a34e-2384-4fe6-bf94-4f708f705b9b at /var/lib/kubelet/pods/77e7a34e-2384-4fe6-bf94-4f708f705b9b/volumes/kubernetes.io~secret/serving-cert
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.723326  657366 secret.go:207] Received secret openshift-cluster-version/cluster-version-operator-serving-cert containing (2) pieces of data, 4408 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 systemd[1]: var-lib-containers-storage-overlay-f103536cdc9d497779f99fc4b38fc945cad4b6a914b9ff47c623d77ff967dd79-merged.mount: Consumed 0 CPU time
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.801030  657366 prober.go:126] Liveness probe for "dns-default-j5mbf_openshift-dns(2713c6c8-bf0b-4b70-86ba-b7f9ee86cb92):dns" succeeded
Jan 30 06:39:08 crc-lf65c-master-0 crio[2947]: time="2021-01-30 06:39:08.810244239Z" level=info msg="Removed container f62a82007e3cbd6d8a9a90e9e6eed0728ca4e8364f071abc0109099d5e62b8c6: openshift-apiserver/apiserver-664d95b7b9-8pqh6/openshift-apiserver-check-endpoints" id=92b30c3d-5c20-4a3e-922b-f40a6a5450de name=/runtime.v1alpha2.RuntimeService/RemoveContainer
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.026067  657366 file.go:201] Reading config file "/etc/kubernetes/manifests/etcd-pod.yaml"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: E0130 06:39:09.028045  657366 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/etcd-pod.yaml": /etc/kubernetes/manifests/etcd-pod.yaml: couldn't parse as pod(Object 'Kind' is missing in 'null'), please check config file
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.028282  657366 file.go:201] Reading config file "/etc/kubernetes/manifests/etcd-pod.yaml"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.036731  657366 kubelet.go:1844] SyncLoop (REMOVE, "file"): "etcd-crc-lf65c-master-0_openshift-etcd(5e4e08ff-7916-4fe5-8388-3f19cd1c0b9f)"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.036895  657366 kubelet_pods.go:1212] marking pod pending termination "5e4e08ff-7916-4fe5-8388-3f19cd1c0b9f"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037160  657366 kubelet.go:1834] SyncLoop (ADD, "file"): "etcd-crc-lf65c-master-0_openshift-etcd(66d597e3-33b3-44d8-9d5e-296d86fbe8c3)"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037717  657366 topology_manager.go:233] [topologymanager] Topology Admit Handler
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037927  657366 kubelet_pods.go:1247] Killing unwanted pod "etcd-crc-lf65c-master-0"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038167  657366 kuberuntime_container.go:635] Killing container "cri-o://aeb9c6fff966f1cf16dabb76c1455ae29c7fef396eac8ece754af5861c45dc69" with a 30 second grace period
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038168  657366 kuberuntime_container.go:635] Killing container "cri-o://9948985c0e41c648e3866d1e0c3f7fd8dbf78f8f1185bbf344d7e3369ef7324c" with a 30 second grace period
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038221  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcdctl"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.078116  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcd"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.078152  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcd-metrics"

After this etcd never fully restarts to pass readiness checks. It looks like there is some corruption of the /etc/kubernetes/manifests/etcd-pod.yaml to me from the error, or at least an issue in reading the file. Maybe this causes it to be re-synced, and in the process the IP address gets changed. Sort of speculation...
After a few timeouts in health checks, about 3 minutes later the kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver goes offline and never recovers:
Jan 30 06:41:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:25.268581  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:35 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:35.259032  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:45 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:45.259149  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:55 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:55.275737  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:42:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:05.267048  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:42:15 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:15.264939  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:25.265320  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:35 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:35.261177  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:50 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:50.409196  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": dial tcp <IP_ADDRESS>:6443: connect: connection refused
Jan 30 06:43:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:05.245746  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:10 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:10.408849  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:15 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:15.246660  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:20 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:20.409739  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:25.247370  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: TLS handshake timeout
Jan 30 06:43:45 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:45.245856  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": dial tcp <IP_ADDRESS>:17697: connect: connection refused
Jan 30 06:44:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:44:05.245706  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@guillaumerose fwiw - the crc instance that I "recovered" and subject of this issue has failed again, whereas the other instance that I started with the bundle you recommended is still running ok.
@cfergeau looking at relatively recent output from sudo crictl ps -a seems to look like there was some processes kicking off in the cluster 8 and 12 hours before the command was run, and there are a broad range of containers that are cycling due to etcd no longer available. Based on journalctl timestamps, the failure appeared to be about 8 hours ago.
output from journalctl was quite voluminous, but I was able to find the minute where things appeared to go from ok to bad by searching on Readiness probe for "etcd-crc. Critical event appears to be happening here (times UTC):
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722255  657366 secret.go:207] Received secret openshift-cluster-version/default-token-b6lgm containing (4) pieces of data, 16604 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722087  657366 configmap.go:188] Setting up volume config for pod f867fc1c-6ece-4518-ae47-366636ee052d at /var/lib/kubelet/pods/f867fc1c-6ece-4518-ae47-366636ee052d/volumes/kubernetes.io~configmap/config
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.722965  657366 configmap.go:212] Received configMap openshift-cluster-machine-approver/machine-approver-config containing (0) pieces of data, 0 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.721078  657366 secret.go:183] Setting up volume serving-cert for pod 77e7a34e-2384-4fe6-bf94-4f708f705b9b at /var/lib/kubelet/pods/77e7a34e-2384-4fe6-bf94-4f708f705b9b/volumes/kubernetes.io~secret/serving-cert
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.723326  657366 secret.go:207] Received secret openshift-cluster-version/cluster-version-operator-serving-cert containing (2) pieces of data, 4408 total bytes
Jan 30 06:39:08 crc-lf65c-master-0 systemd[1]: var-lib-containers-storage-overlay-f103536cdc9d497779f99fc4b38fc945cad4b6a914b9ff47c623d77ff967dd79-merged.mount: Consumed 0 CPU time
Jan 30 06:39:08 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:08.801030  657366 prober.go:126] Liveness probe for "dns-default-j5mbf_openshift-dns(2713c6c8-bf0b-4b70-86ba-b7f9ee86cb92):dns" succeeded
Jan 30 06:39:08 crc-lf65c-master-0 crio[2947]: time="2021-01-30 06:39:08.810244239Z" level=info msg="Removed container f62a82007e3cbd6d8a9a90e9e6eed0728ca4e8364f071abc0109099d5e62b8c6: openshift-apiserver/apiserver-664d95b7b9-8pqh6/openshift-apiserver-check-endpoints" id=92b30c3d-5c20-4a3e-922b-f40a6a5450de name=/runtime.v1alpha2.RuntimeService/RemoveContainer
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.026067  657366 file.go:201] Reading config file "/etc/kubernetes/manifests/etcd-pod.yaml"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: E0130 06:39:09.028045  657366 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/etcd-pod.yaml": /etc/kubernetes/manifests/etcd-pod.yaml: couldn't parse as pod(Object 'Kind' is missing in 'null'), please check config file
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.028282  657366 file.go:201] Reading config file "/etc/kubernetes/manifests/etcd-pod.yaml"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.036731  657366 kubelet.go:1844] SyncLoop (REMOVE, "file"): "etcd-crc-lf65c-master-0_openshift-etcd(5e4e08ff-7916-4fe5-8388-3f19cd1c0b9f)"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.036895  657366 kubelet_pods.go:1212] marking pod pending termination "5e4e08ff-7916-4fe5-8388-3f19cd1c0b9f"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037160  657366 kubelet.go:1834] SyncLoop (ADD, "file"): "etcd-crc-lf65c-master-0_openshift-etcd(66d597e3-33b3-44d8-9d5e-296d86fbe8c3)"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037717  657366 topology_manager.go:233] [topologymanager] Topology Admit Handler
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.037927  657366 kubelet_pods.go:1247] Killing unwanted pod "etcd-crc-lf65c-master-0"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038167  657366 kuberuntime_container.go:635] Killing container "cri-o://aeb9c6fff966f1cf16dabb76c1455ae29c7fef396eac8ece754af5861c45dc69" with a 30 second grace period
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038168  657366 kuberuntime_container.go:635] Killing container "cri-o://9948985c0e41c648e3866d1e0c3f7fd8dbf78f8f1185bbf344d7e3369ef7324c" with a 30 second grace period
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.038221  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcdctl"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.078116  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcd"
Jan 30 06:39:09 crc-lf65c-master-0 hyperkube[657366]: I0130 06:39:09.078152  657366 event.go:291] "Event occurred" object="openshift-etcd/etcd-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container etcd-metrics"

After this etcd never fully restarts to pass readiness checks. It looks like there is some corruption of the /etc/kubernetes/manifests/etcd-pod.yaml to me from the error, or at least an issue in reading the file. Maybe this causes it to be re-synced, and in the process the IP address gets changed. Sort of speculation...
After a few timeouts in health checks, about 3 minutes later the kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver goes offline and never recovers:
Jan 30 06:41:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:25.268581  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:35 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:35.259032  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:45 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:45.259149  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:41:55 crc-lf65c-master-0 hyperkube[657366]: I0130 06:41:55.275737  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:42:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:05.267048  657366 prober.go:126] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" succeeded
Jan 30 06:42:15 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:15.264939  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:25.265320  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:35 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:35.261177  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): HTTP probe failed with statuscode: 500
Jan 30 06:42:50 crc-lf65c-master-0 hyperkube[657366]: I0130 06:42:50.409196  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": dial tcp <IP_ADDRESS>:6443: connect: connection refused
Jan 30 06:43:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:05.245746  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:10 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:10.408849  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:15 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:15.246660  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:20 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:20.409739  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver" failed (failure): Get "https://<IP_ADDRESS>:6443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 30 06:43:25 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:25.247370  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: TLS handshake timeout
Jan 30 06:43:45 crc-lf65c-master-0 hyperkube[657366]: I0130 06:43:45.245856  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": dial tcp <IP_ADDRESS>:17697: connect: connection refused
Jan 30 06:44:05 crc-lf65c-master-0 hyperkube[657366]: I0130 06:44:05.245706  657366 prober.go:117] Liveness probe for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(9c9a7be9-44dd-46f8-96d9-064e3bdaa0de):kube-apiserver-check-endpoints" failed (failure): Get "https://<IP_ADDRESS>:17697/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Also - it was possible to recover the cluster by using the following:
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/manifests/etcd-pod.yaml
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/static-pod-resources/etcd-pod-6/configmaps/etcd-pod/pod.yaml
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/static-pod-resources/etcd-pod-6/etcd-pod.yaml
# systemctl restart kubelet

Note that the latest generation /etc/kubernetes/static-pod-resources/etcd-pod-## should be targeted, the timestamp of this directory matched when the failure appeared - apparently a new static generation is being created.

Also - it was possible to recover the cluster by using the following:
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/manifests/etcd-pod.yaml
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/static-pod-resources/etcd-pod-6/configmaps/etcd-pod/pod.yaml
# sed -i 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/kubernetes/static-pod-resources/etcd-pod-6/etcd-pod.yaml
# systemctl restart kubelet

Note that the latest generation /etc/kubernetes/static-pod-resources/etcd-pod-## should be targeted, the timestamp of this directory matched when the failure appeared - apparently a new static generation is being created.

Every 24 hours kubelet node certs are rotated. The cluster machine approver approves it. This is what we see here.
Maybe the rotation triggers a reload of kubelet and it re-reads all options (and network interfaces?).

Every 24 hours kubelet node certs are rotated. The cluster machine approver approves it. This is what we see here.
Maybe the rotation triggers a reload of kubelet and it re-reads all options (and network interfaces?).

It doesn't seem to happen every 24 hours however. I've seen crc run for a few days before this has happened on some systems. I've never seen it run for an indefinite time without it happening, but maybe I'm unlucky.
I wrote a short script that can be run from the VM that seems to repair the failures I've seen on KVM with Intel (nested) and AMD (host) virtualization.

It doesn't seem to happen every 24 hours however. I've seen crc run for a few days before this has happened on some systems. I've never seen it run for an indefinite time without it happening, but maybe I'm unlucky.
I wrote a short script that can be run from the VM that seems to repair the failures I've seen on KVM with Intel (nested) and AMD (host) virtualization.

Hi @timroster ,
The release is almost out. Can you try with http://mirror.openshift.com/pub/openshift-v4/clients/crc/1.22.0/ ?
Thanks.

@guillaumerose - thanks for heads up. Spun up new vm and installed this version. I see a single command line argument for --node-ip=<IP_ADDRESS> for kubelet in the process table. Will let it run and also subject it to some use.

6 days running w/o a crash or other problem with the CRC api server availability

Great!

@guillaumerose I've not seen 1.22 fail with this type of problem on two different systems and with >7 days running time. closing and thanks for the investigation and fixes.


Add support for fetching wbGeometry objects💡

💡 Is your feature request related to a problem? Please describe.
We need to be able to retrieve wbGeometry objects for a given wellbore
Describe the solution you'd like
The Witsml library should be extended with type mappings for fetching wbGeometry objects from a Witsml server
Describe alternatives you've considered
N/A
Additional context
This will be implemented within our team, this issue is mostly for informational purposes

Fixed by #1320 and #1638


Use voluptuous for Alarm.com

Description:
Migration of the configuration check to voluptuous.
Related issue (if applicable): fixes 127528299
Example entry for configuration.yaml (if applicable):
alarm_control_panel:
  platform: alarmdotcom
  name: "HA Alarm"
  code: PASSCODE
  username: YOUR_USERNAME
  password: YOUR_PASSWORD

@Xorso, would be nice if you could take a look at the changes and run a quick test. Thanks.

LGTM - though you have swapped your vol ordering. You said it was Required -> Optional -> Alpha
Now you're doing Optional -> Required -> Alpha
Not a big deal.

:dolphin:


Request: Add gradio auth (potential security issue)

First, great work on this!
However, I recently discovered some images that I definitely did NOT generate in my google drive. I can only assume that this is due to someone accidentally (or worse, purposefully) using the same gradio server/url. It's possible that people are just trying random hashes/urls to see if they're live. Or it could just be a bug on gradio's end.
I know I can tweak the code myself to enable auth, but it'd be nice if the notebook supported it via the UI. (IMO, auth should be enabled by default, as a security precaution).
It would also be nice to add a config option that disables output capture. I think it's useful to see what's going on behind the scenes.
Cheers

Uncheck the box "user_gradio_server" to use a more secure tunnel


Allow to emphasize certain completion items

Currently, Visual Studio Code JSON IntelliSense does not highlight (or in any other way emphasize) default values:

(es3 in this example)
I'd like to suggest to highlight default values, so the programmer is aided in what values will be used when (s)he lints (probably large) configuration files for redundant entries.

We put the label 'default' to the detail attribute of completion item. Unfortunately these are only seen when the entry is selected.

@jrieken Any tips/tricks on how to better emphasize certain items?

@aeschli Sorting? Make it so that es3 goes on top?

Or adding a (default) text to the corresponding value?

The default value is not the most preferred proposal, so sorting isn't helpful here.
Default is really just additional information here. I guess we can put it in the regular label.


readme translate

i translate readme to english.

I confirmed this pull request.
This is very useful.
Thank you very much ! !


envdir setup is not easily greppable / envprint utility?

I am considering using envdir for a Django project, but get the impression that it may become more difficult to grep / search for settings and their values, given that they are split into filename and file content.
For example, you cannot just grep your settings / project folder anymore for DEBUG, EMAIL_ etc, without having to also grep (and then look into) the output of find.
Given that I am also looking into django-configurations, because it allows to have local, production and staging settings in a single file, this feels uncomfortable.
OTOH, envdir makes it easier already to have environment settings in a central location (envs/{local,prod,...}), which are scattered around otherwise (virtualenv's postactivate, wsgi server setting files, ...).
Just wanted to leave this as feedback - maybe there's an approach to make it easier to work with it?
Maybe a script / option could be provided, which only displays settings from a specific env dir in the form of NAME=VALUE.
Or when invoked in a directory with subdirs, would display it like grep as in:
$ envprint ./envs
prod: DEBUG=False
prod: FOO=1
local: DEBUG=True

$ envprint ./envs/local
DEBUG=True

@blueyed this works too:
➜   myproj [master] ⚡ tail envs/local/*
==> envs/local/DATABASE_URL <==
postgres://postgres@localhost:5432/election2016

==> envs/local/DEBUG <==
true

==> envs/local/SECRET_KEY <==
poop

==> envs/local/STAGE <==
dev


Not working on my NEX-5T

I successfully installed the app and checked the "disable recording limit," but my camera still stops recording before 30 minutes.

That becouse Sony 5T will over heating on 30 minute...


add name in contributors list

Pull Request Template

Please provide all of the details below. Pull requests missing any of the information below will be closed!
Providing the issue number/URL helps us identify the problem being solved and keep the issues tab clean and tidy

Sumarise what this PR fixes
<Detail what your PR changes/fixes>
I add my name in the contributors list.
What ISSUE NUMBER does this fix?
Fixes #
Checklist:
(Add an x between the brackets to check the items)

[ ] I have summarised what this PR fixes above
[ ] I have provided the issue ID/URL above
[ ] This PR fixes only the issue mentioned above (create separate PRs for separate issues)
[ ] I was the first one to claim the issue or I created the issue

Hey,
Thanks for creating the PR, its important that you are accredited for your work, however in accordance with Hacktoberfest's rule I must mark this with the invalid tag otherwise Hacktoberfest can blacklist the repo:

Thanks for your understanding and please do not let this discourage you.


upgrade package.json to latest and add html template for project

@sandangel Special thanks to your contribution.


Activity > Modularise various step-pages

[ ] Step 1
[ ] Step 2
[ ] Step 3
[ ] Step 4
[ ] Step Success! (5)

Not sure it's relevant. Most steps are simple forms, better to stay there. See /src/components/Activity.vue


Drawer under status bar

Is it possible to display the drawer under the status bar?

What's your meaning?
Show the status bar, right?

@Pluto-Y, ah yes. That's what I meant.
When displaying the drawer, the status bar is also displayed.

Please try this, SlideMenuOptions.hideStatusBar = false.

@mggeray  Is it ok?

@Pluto-Y, sorry for the late reply. Yes this is OK.
Thank you!

How can i show navigation controller bar?


[Docs] How to write automated tests

Should include how to write tests for a service whether or not it has a plugin.  With a plugin, the user can validate their custom functionality.

Every generated projects now ships with tests.


Ability to allow Maps to recieve proper touch inside scrollview - Android

Issue relating too: https://github.com/amay077/Xamarin.Forms.GoogleMaps/issues/336
Using this library with the map nested inside a ScrollView, the scrolling of the ScrollView is badly interfering with the map.
Using the below code resolves the issue:
public override bool DispatchTouchEvent(MotionEvent e)
       {
           base.Parent.RequestDisallowInterceptTouchEvent(true);
           // Handle MapView's touch events.
           return base.DispatchTouchEvent(e);
       }

I am proposing that we set some flag or something that allows Devs to disallow touch on the ScrollView when the touch is being made on the map. It might simple override or not override the above method
iOS didnt seem to have this issue.

This looks like a simple solution.  Any reason why this issue is still not fixed?

Any news on this issue?

Still awaiting an elegant fix for this

@amay077 bump. Can this be implemented in the repo?

ping 👋

bump :(


Missing note for running command or invaild example for updating policy link

For the section that instructing on how to associate the WAF policy with the Azure Front Door resource it's not mentioned that you need to escape the quotation marks. Seem like it might be a similar error found here https://github.com/Azure/azure-cli/issues/10011 for gateway.
copy of command given
az network front-door update --name <> --resource-group <> --set frontendEndpoints[0].webApplicationFirewallPolicyLink='{"id":"<>"}' // the quotes by id and <> need to be escaped for this to work
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7528220b-b6b5-1465-f33f-4f282af5e6de
Version Independent ID: f8151b07-016c-4f85-d465-8650fdc7be83
Content: Tutorial: Scale and protect a web app by using Azure Front Door and Azure Web Application Firewall (WAF)
Content Source: articles/frontdoor/front-door-waf.md
Service: frontdoor
GitHub Login: @duongau
Microsoft Alias: duau

Thank you for reaching out. At this time we are reviewing the ask and will provide an update as appropriate

@AceOfSpades29 This command is working alright and is not throwing any errors. Please reach out to Azure Support if you are running into any issues with the same.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Check for jQuery / Underscore before including

Noticed an issue when trying to embed notes on our article pages – they work, but cause a cascade of errors to the rest of the page with anything else that requires jQuery.
We assign jQuery.noConflict to a different global var ($j) when we call it, but since jQuery is also delivered via note_embed.js it reassigns, breaking other functions called by the footer (or anything that runs after the embed script does).
Could the vendor scripts be checked and called as part of the asset loader rather than minified into note_embed.js to avoid reassigning to the default global vars? While less common, I imagine this would also be an issue for sites that already call Underscore, but reassign to a different global var.

Hey @seanokey, we're also calling noConflict with the intent to side step precisely this issue: https://github.com/documentcloud/documentcloud-notes/blob/master/src/js/note_embed.js#L13 so i'm hoping we haven't messed anything up there.
I dropped into check on a document posted on the CNN site but i'm not sure if what i'm seeing is what you're talking about: http://www.cnn.com/2016/06/28/us/orlando-pulse-police-logs/index.html
Do you have a sample page we could take a look at?
Best,
-Ted

Hi @knowtheory – right, the full document embeds work great. It's the note-specific embeds that are presenting an issue. Because of that, we don't have any live examples, but I'll work something up and message you a test link privately shortly.
Thanks!
-Sean

Oh! duh, sorry about that :) yes of course.
If you have a test page handy i'd love to take a look, otherwise i can put something together to simulate it.

This ended up being a really small but significant oversight on my part.
jQuery.noConflict() needs to be passed true so as not to step on other noConflict'd versions of jQuery.
Fixed in 6152599a3f237c41468c877ab6ba59bc19a30e2c


optimization | vs code | test launch profiles

[ ] setup launch profiles to make it easy to launch critical test cases

[x] basic launch.json template
[x] customize for 1 test case
[x] add default configuration
[x] added configuration for orcatest.py
[x] validate config then close
[x] add additional use cases based on @vamseeachanta input

hi @vamseeachanta closing since we have completed this.


Low SQuADv2 F1 & EM Score

Hi,
I used the default settings and run_squad.py script (with the exception of a batch size of 4 since my GPU has low memory) to train for 3 epochs. Turns out I got an EM & F1 score of 43% and 48% respectively. AvNA looks decent at 65%. Is this due to a small number of epochs or the small batch size? Note that training for 3 epochs on my single 1070-TI took 12 hours.
Thanks.

@thomwolf
Hi, it seems there is something wrong with the training code in this repo.
I used Google's official BERT training code and I could get decent results even with a small batch size:
{'EM': 73.80717341230668, 'F1': 77.11048422305339, 'AvNA': 80.78315235274762}
Here's the settings I used for Google's and this repo's:
--vocab_file=uncased_L-12_H-768_A-12/vocab.txt
--bert_config_file=uncased_L-12_H-768_A-12/bert_config.json
--init_checkpoint=uncased_L-12_H-768_A-12/bert_model.ckpt
--do_train=True
--train_file=../squad/data/train-v2.0.json
--do_predict=True
--predict_file=../squad/data/dev-v2.0.json
--train_batch_size=5
--learning_rate=3e-5
--num_train_epochs=2.0
--max_seq_length=384
--doc_stride=128
--output_dir=gugel_bert
--version_2_with_negative=True
--do_lower_case=True

--bert_model=bert-base-uncased
--output_dir=try_bert_2
--train_file=data/train-v2.0.json
--predict_file=data/dev-v2.0.json
--do_train
--do_predict
--do_lower_case
--train_batch_size=5
--predict_batch_size=5
--num_train_epochs=2.0
--learning_rate=3e-5
--version_2_with_negative

Forgot to say that I also can get the expected results using SQuAD v1.1 with this repo's code.