Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain

SYSTEM Infomation
ubuntu16.04
cuda7.5
cudnn5
gtx1060
tensorflow1.0.1
python3.5
memory 15.6G ,used 5.3GB
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0  On |                  N/A |
| 33%   38C    P2    27W / 120W |   6012MiB /  6071MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1536      G   /usr/lib/xorg/Xorg                           273MiB |
|    0      3210      G   compiz                                       115MiB |
|    0      3811      G   ...-token=C6D7A354DD6B35830E1B2860115A47BF    82MiB |
+-----------------------------------------------------------------------------+

run train script get this error:
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Called with args:
Namespace(cfg_file='./lstm/lstm.yml', gpu_id=0, max_iters=700000, network_name='LSTM_train', pre_train=None, randomize=False, restore=0, set_cfgs=None)
Using config:
{'CHARSET': '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ',
 'EXP_DIR': 'lstm_ctc',
 'FONT': 'fonts/Ubuntu-M.ttf',
 'GPU_ID': 0,
 'IMG_SHAPE': [180, 60],
 'LOG_DIR': 'lstm_ctc',
 'MAX_CHAR_LEN': 6,
 'MAX_LEN': 6,
 'MIN_LEN': 4,
 'NCHANNELS': 1,
 'NCLASSES': 64,
 'NET_NAME': 'LSTM',
 'NUM_FEATURES': 60,
 'POOL_SCALE': 2,
 'RNG_SEED': 3,
 'ROOT_DIR': '/srv/python/lstm_ctc_ocr_with_tf_1.0.1',
 'SPACE_INDEX': 0,
 'SPACE_TOKEN': '',
 'TEST': {},
 'TIME_STEP': 90,
 'TRAIN': {'BATCH_SIZE': 32,
           'DISPLAY': 100,
           'GAMMA': 1.0,
           'LEARNING_RATE': 0.001,
           'LOG_IMAGE_ITERS': 100,
           'MOMENTUM': 0.9,
           'NUM_EPOCHS': 2000,
           'NUM_HID': 128,
           'NUM_LAYERS': 2,
           'SNAPSHOT_INFIX': '',
           'SNAPSHOT_ITERS': 2000,
           'SNAPSHOT_PREFIX': 'lstm',
           'SOLVER': 'RMS',
           'STEPSIZE': 2000,
           'WEIGHT_DECAY': 1e-05},
 'VAL': {'BATCH_SIZE': 128,
         'NUM_EPOCHS': 1000,
         'PRINT_NUM': 5,
         'VAL_STEP': 500}}
Output will be saved to `/srv/python/lstm_ctc_ocr_with_tf_1.0.1/output/lstm_ctc`
Logs will be saved to `/srv/python/lstm_ctc_ocr_with_tf_1.0.1/logs/lstm_ctc/lstm_train/2017-11-11-11-13-49`
/gpu:0
Tensor("data:0", shape=(?, ?, 60), dtype=float32)
Tensor("conv4/BiasAdd:0", shape=(?, ?, 30, 1), dtype=float32)
Tensor("time_step_len:0", shape=(?,), dtype=int32)
Use network `LSTM_train` in training
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
E tensorflow/core/common_runtime/direct_session.cc:137]
Internal: failed initializing StreamExecutor
for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain:
CUDA_ERROR_OUT_OF_MEMORY; total memory reported:<PHONE_NUMBER>3709551615
Traceback (most recent call last):
  File "./lstm/train_net.py", line 89, in <module>
    restore=bool(int(args.restore)))
  File "./lstm/../lib/lstm/train.py", line 187, in train_net
    with tf.Session(config=config) as sess:
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1176, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 552, in __init__
    self._session = tf_session.TF_NewDeprecatedSession(opts, status)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

I will be grateful to anyone for helping me
Thanks everyone

It looks like there is a problem with this:
for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain:
CUDA_ERROR_OUT_OF_MEMORY; total memory reported:<PHONE_NUMBER>3709551615

Can you try with:
gpu_fraction = 0.1
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=gpu_fraction)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue? Please update the label and/or status accordingly.

It seems that there is no space in GPU, you would like to kill one process and re-run you code, or if you have  more than one gpu, set the gup number you want to use .

I had an issue similar to this...
I was trying to execute the tensorflow in a jupyter lab notebook.
The issue stopped after I ran jupyter with sudo jupyter lab --allow-root

It looks like there is a problem with this:
for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain:
CUDA_ERROR_OUT_OF_MEMORY; total memory reported:<PHONE_NUMBER>3709551615

Can you try with:
gpu_fraction = 0.1
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=gpu_fraction)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

I tried, but it still doesn' t work.

jupyter lab --allow-root worked for me.

Hi brothers
i have 4 gpus on my PC. But i could utilize only one gpu. Others can not be used. Can you help me in this case. Actually i tried to use all gpus for dirrerent git repos.
error:
Attempting to fetch value instead of handling error Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported:<PHONE_NUMBER>2

@Harry-KIT ,
please set the gpu numbers to the process... you can do this in multiple ways as per the context:

dlib:

  from dlib import cuda
  cuda.set_device(<gpu number>

OPENCV
  cv2.cuda.setDevice(<gpu number>- More generic method

install numactl (sudo apt install numactl)
export CUDA_VISIBLE_DEVICES=<gpu_number>;numactl --cpunodebind= ./executable

Hope this helps :+1:

Thank you.


Hierarchical structure paths do not create filled indexes

Documentation recommends hierarchical structure for the files in Writing Pages | Bloggrify, such as 2024/02/24/post01.md.
The index between pages does not create directory listings. For example 2024/02/24/ does not list posts within the directory.
Suggested Fix
Create logic using the Nuxt Content queryContent() on the requested path to list entries within the directory.
Alternative implementation could use the Nuxt Content <ContentList> component.
Feedback and suggestions are welcome!

Hi,
Thanks for your report.
I'm not sure to understand the expectation here.
You want to have one page per directory that list all pages in this directory ?
For instance, on my own blog, you imagine a page
https://eventuallycoding.com/2024/11 listing all pages for november ?
This is indeed not the expected behaviour. There is one "index page", the archives pages, for instance : https://mistral.bloggrify.com/archives
Could you elaborate a bit the use case ?

Hi, @hlassiege
Thanks for your quick reply.

I'm not sure to understand the expectation here.

It is a feature request.

You want to have one page per directory that list all pages in this directory ?

Yes an index page that lists all posts within that directory, similar to WordPress for example with the path /news/2024/11/ November 2024 | WordPress.

For instance, on my own blog, you imagine a page
https://eventuallycoding.com/2024/11 listing all pages for november ?

Yes, like this and be customisable.

This is indeed not the expected behaviour. There is one "index page", the archives pages, for instance :  https://mistral.bloggrify.com/archives

Agreed, the archives path should still remain, the new indexes would be a feature request.

Could you elaborate a bit on the use case?

To create directory structures that extend beyond the overarching tags and category navigation.
For example, end users create directories for the organisation of posts with a structure.

User story: As a travel blogger, I want to organise my blog posts by country and region.

I will label this issue as a feature request to be more clear.

Extended User Story
Details

As a non-technical Travel Blogger, I want to organise my blog posts into directories and subdirectories to better manage and navigate my content. For example:

Country Level: Create a directory for each country.

Japan:

Subdirectory for regions like Kansai, Kanto, etc.

Spain:

Subdirectory for regions like Catalonia, Andalusia, etc.

Canada:

Subdirectory for provinces like Quebec, Ontario, etc.

This structure will help me and my readers easily find posts related to specific regions within a country, enhancing the overall user experience and content organisation.

Additional Example
Details

As a Train Enthusiast, I want the flexibility to organise my data in various ways that suit my needs. For example:

Train Enthusiast: Create directories for different rail lines.

Rail Line Level:

Subdirectory for specific trains or routes.
Blog posts about meeting specific trains or experiences on particular routes.

Customising Indexes
Details

To further enhance organisation, an index.md file could be created within each directory to title the folder. For example:

November: An index.md file in the 2024/11 directory to title it "November 2024" in the front-matter block.
Kansai: An index.md file in the Japan/Kansai directory to title it "Kansai" in the front-matter block.

This approach allows for clear labelling and better navigation within the directory structure, making it easier for users to understand and manage their content.
Implementation suggestion
2024/11/index.md
---
title: November 2024
archive: true
---

Ok I understand
Indeed, having an index.md page in some folders is a flexible solution that could work for people who need to have intermediate index pages.
Then, there is two solutions :

having a special type in the frontmatter saying it's a listing page

I'm not a big fan because it's not something really easy to customize for the end user.

having a custom component

This solution would allows the user to create a specific content for each index page
For instance
The kansai is a wonderful region and you can find here all the different blog posts I've made here :
::listing
::

The syntax is maybe not exactly this one but that would be the idea

Ok I understand
Indeed, having an index.md page in some folders is a flexible solution that could work for people who need to have intermediate index pages.

Agreed

Then, there is two solutions :

having a special type in the frontmatter saying it's a listing page

We may need a front matter property for the following reasons:

The layout may be being different compared to a regular post
If the page metadata is being set (using existing front matter)

e.g. Page title, description etc. whereas a component controlling page metadata may confuse users

I'm not a big fan because it's not something really easy to customize for the end user.

I understand, the front matter is tightly embed in the core bloggrify project, rather than extensible through themes.
Some customisations could come in the form of a listings layout,  with the rest in a content component. Creating a layout similar to archive with a space for markdown would allow for content and page layout customisation in a theme. The markdown space would allow for user made components including listing.

having a custom component

This solution would allows the user to create a specific content for each index page
For instance
The kansai is a wonderful region and you can find here all the different blog posts I've made here :
::listing
::

That sounds like a great idea, it may be a good addition to a page just like you highlighted. It is similar to the WordPress Query Loop block which allows for matching rulesets.
It could be further expanded to provide list of posts for internal linking.

Example: A post about a dining experience review could use a listing component to offer other dining experience reviews to readers.Well that's all of my thoughts on my time at Restaurant A, check out some of my other German restaurants reviews:

::listing{title="Suggested German Restaurant Reviews" tag="restaurant" dir="germany" max=3}

See something you like? Don't forget to share the reviews on social media!

#footerSlot
Find the full list of reviews in the [Germany Section](/countries/germany/).
::

Idea: Users could customise the components/content/listing.vue component within a theme to show cards inside of a list, similar to related content under an article e.g. PWC Insight Post: Retail Outlook 2025

The syntax is maybe not exactly this one but that would be the idea

I think it clearly illustrates how it could work.

Example:
2024/11/index.md
---
title: November 2024
description: All Posts for November 2024
listing: true
---

November has been a busy month this year with the new recipe suggestions sent in by all of you wonderful people.

See all of this months recipes below:

::listing{title="November 2024 Recipes"}
::

In conclusion, a mixed approach may be best. A front matter property to set the listings layout, along with the customisation offered in the theme's layout and listing content component, provides flexibility and control being used, providing flexibility and control.

yes indeed, a mixed approach could be good.
In a way, it's already possible for a theme creator to use a different layout.
It's done here on the bento theme : https://bento.bloggrify.com/portfolio
the corresponding markdown is :
---
title: "Portfolio"
description: "Portfolio"
listed: false
layout: "portfolio"
---

A equivalent system for index page could looks like your example, except that I would use the layout property instead of a new one
---
title: November 2024
description: All Posts for November 2024
layout: listing
---

November has been a busy month this year with the new recipe suggestions sent in by all of you wonderful people.

See all of this months recipes below:

::listing{title="November 2024 Recipes"}
::

That would give a good flexibility for theme creation.
What do you think ?

@hlassiege, the layout option is great.
I am very happy with how this feature's approach is looking.

Why automated listing pages might not be the right solution:
Users may expect the app to automatically create index listing pages for dates.
Hopefully the flexibility in creating index listings overcomes this expectation, provided it is clearly documented.
Additionally, the automated approach may force strict / standard date structures, cause false positives etc. that are undesirable.


doc: fix schedule manifest missing config file

:tada: This PR is included in version 0.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


kubernetes: fix interpolation error and move services to own target

Motivation for this change
The first commit is a straight up fix for a bug where enabling the cni network
plugin will make evaluation fail.
The second part shifts the kubernetes services into their own slice and makes
the launch happen via a dedicated kubernetes target.
The reason for moving into its own slice and target is that it makes starting
and stopping far easier plus it allows for resource allocations.
Cc: @offlinehacker
Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Thanks!

Why did you need toString? imap nethod should always return numbers.

Why did you need toString? imap nethod should always return numbers.

Because 10+i isn't being interpolated automatically inside the string.

ah yes, of course, tnx


Submitted By (Name) field on Add event form

Allow people to provide their name (optionally) to help identify who has submitted the event

Assigning to self as would require DB change


chore: merge tmp_develop to develop

merge tmp_develop to develop

:tada: This PR is included in version 5.3.0-beta.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 5.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Regenerate IDL temperature response data

The IDL temperature response curves should be recalculated using the correct right gain value for the CCD. See https://github.com/HinodeXRT/xrtpy/pull/284#issue-2504338811

Maybe a duplicate of #221?


Remove one level of indirection

make it CPU cache friendly

@redbaron Can you please add benchmarks for this change?

there is an empirical evidence - why waste 32 cache lines for each shard struct, when they all can be compact together in 32*sizeof(shard)/64 lines.
I'll add benchmarks soon, but just to show that there is no performance degradation. I expect improvement not to be seen  in synthetic runs as there is not cache pressure when single library is tested. In real apps & virtualised environments CPU  cache is a very scarce resource, that is where I expect this change to have some effect.

Great, I am expecting your benchmarks.
Thanks a lot for your support and input to this project!
On Sun, Aug 14, 2016 at 4:07 PM, Maxim Ivanov<EMAIL_ADDRESS>wrote:

there is an empirical evidence - why waste 32 cache lines for each shard
struct, when they all can be compact together in 32*sizeof(shard)/64
lines.
I'll add benchmarks soon, but just to show that there is no performance
degradation. I expect improvement not to be seen in synthetic runs as there
is not cache pressure when single library is tested. In real apps &
virtualised environments CPU cache is a very scarce resource, that is where
I expect this change to have some effect.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/streamrail/concurrent-map/pull/34#issuecomment-239672617,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVy5gRnDNmi73c8mYHVOJQgxs7G37RVks5qfxMSgaJpZM4Jj2nE
.


Support @pattern

https://typespec.io/docs/standard-library/built-in-decorators/#@pattern

I added initial support, but currently using both @pattern, and @minLength and/or @maxLength doesn't work. It will use the @pattern without constraining the length.
Tracking issue on the fast-check side: https://github.com/dubzzz/fast-check/issues/5562


[FLINK-5481] Add type extraction from collection

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[ ] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[ ] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[ ] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

@zentol I rewrite as you pointed

I will shepherd this PR.

I am a bit skeptical about the special case handling or Row in the type exatractor there.
Why specifically support row and not other types? To me, Row does not seem like a most common type to be put into the collection inputs that it needs such special case handling.
Also, this does not detect if the different rows have conflicting types in the columns (such as `row1[x] = int, row2[x] = String).
What do you think about simply improving how to generate a row type info (similar as we did with TypeHint for other types)? Then StreamExecutionEnvironment.fromCollection(rows, type) should work quite well.

I would also be fine with StreamExecutionEnvironment.fromCollection(rows, type). What do you think @tonycox?

@StephanEwen I agree with types conflicting and that it's not common type for collection.
So how should generating a row type info look like?

StreamExecutionEnvironment.fromCollection(
  rows,
  new RowTypeInfo(BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO))

You could think of adding a utility method on the RowTypeInfo:
@SafeVarAgrs
RowTypeInfo.from(Class<?>... types)

and use it like
StreamExecutionEnvironment.fromCollection(
  rows,
  RowTypeInfo.from(String.class, int.class));

I think all it's just a syntax sugar
      val data = List(
        Row.of(Row.of("data_1", "dob"), Row.of("info_4", "dub")),
        Row.of(Row.of("data_1", "dob"), Row.of("info_4", "dub")),
        Row.of(Row.of("data_1", "dob"), Row.of("info_4", "dub")))

      val info= RowTypeInfo.from(
        RowTypeInfo.from(classOf[String], classOf[String]),
        RowTypeInfo.from(classOf[String], classOf[String])
      )

      execEnv.fromCollection(data, info)

instead of
      val info = new RowTypeInfo(
        Array[TypeInformation[_]](
          new RowTypeInfo(
            Array[TypeInformation[_]](
              BasicTypeInfo.STRING_TYPE_INFO,
              BasicTypeInfo.STRING_TYPE_INFO),
            Array("name", "age")),
          new RowTypeInfo(
            Array[TypeInformation[_]](
              BasicTypeInfo.STRING_TYPE_INFO,
              BasicTypeInfo.STRING_TYPE_INFO),
            Array("more_info", "and_so_on"))),
        Array("person", "additional")
      )

      execEnv.fromCollection(data, info)

Hmmm... what would you suggest to do instead?

I suggest add a little bit more sugar
TypeInformation<?> infoNew = RowTypeInfo.from(
    new String[]{"1c", "2c"},
    RowTypeInfo.from(
        new String[]{"1.1", "1.2c"},
        String.class, String.class),
    RowTypeInfo.from(
        new String[]{"2.1c", "2.2c"},
        String.class, String.class));

instead of

TypeInformation<?> infoOld = new RowTypeInfo(
    new TypeInformation<?>[]{
        new RowTypeInfo(
            new TypeInformation<?>[]{
                BasicTypeInfo.STRING_TYPE_INFO,
                BasicTypeInfo.STRING_TYPE_INFO},
            new String[]{"1.1", "1.2c"}),
        new RowTypeInfo(
            new TypeInformation<?>[]{
                BasicTypeInfo.STRING_TYPE_INFO,
                BasicTypeInfo.STRING_TYPE_INFO},
            new String[]{"2.1c", "2.2c"}
            )},
    new String[]{"1c", "2c"});

val infoNew = RowTypeInfo.from(
      Array("person", "additional"),
      RowTypeInfo.from(
        Array("name", "age"),
        classOf[String], classOf[String]),
      RowTypeInfo.from(
        Array("name", "age"),
        classOf[String], classOf[String])
    )

instead of

val infoOld = new RowTypeInfo(
        Array[TypeInformation[_]](
          new RowTypeInfo(
            Array[TypeInformation[_]](
              BasicTypeInfo.STRING_TYPE_INFO,
              BasicTypeInfo.STRING_TYPE_INFO),
            Array("name", "age")),
          new RowTypeInfo(
            Array[TypeInformation[_]](
              BasicTypeInfo.STRING_TYPE_INFO,
              BasicTypeInfo.STRING_TYPE_INFO),
            Array("more_info", "and_so_on"))),
        Array("person", "additional")
      )

In the Table API we implemented org.apache.flink.table.api.Types to improve the user experience. We could also think about moving this to core. I would be helpful to have a class that enumerates all supported types. It could also support methods like Types.ROW(Types.INT(), Types.DOUBLE()). If you then import all members statically, you end up with ROW(INT(), DOUBLE()).

@twalthr I think it's good way.
Should I move TimeIntervalTypeInfo to core as well or not?

import Types._
      ROW(Array("person", "additional"),
        ROW(Array("name", "age"), STRING, STRING),
        ROW(Array("more_info", "and_so_on"), STRING, STRING))

I think I should rename this issuea to "Simplify Row creation".
What do you think?

I think this goes into a good direction.
I am wondering, though, if we need types like "time interval" in flink-core. I would prefer to keep then in the Table API scope.
There can be a class Types in flink-core and one in flink-table - the table one extends the core one. That way you can have common types in core, extended types in table, and still get access to all by importing the flink-table:Types class.

@StephanEwen @twalthr What do you think about this PR ?

Hi @StephanEwen
I added javadoc

Looks good to me, +1
@twalthr @fhueske Any concerns about merging this?

Found one more small concern (inline comment)

@StephanEwen I fixed

Hi @StephanEwen could you continue review this PR?

Looks good, thank you!
Merging this...


[Suggestion] Use alias for module imports

Rollup-plugin-alias could be useful while importing from "far away" folders and during recode / restructuring operations.
If you are interested I can create a PR 🙂.

I find your ideas intriguing and wish to subscribe to your newsletter 😁
There are probably a several spots that would benefit tremendously from that. Most of the really chewy ones are just off root, so is it possible to create an alias to the root of the src?

Yes, sure!
My initial idea was to create an alias for each src folder.
I've created a little POC

I'm curious of the need for that plugin since Rollup accepts a paths config (similar to RequireJS) where you just use a global-ish name, and Rollup does resolution for you.
Also not sure if Ractive should have this. I think it's best to leave it to the bundler in use since it's much more flexible.

The biggest benefit for me would be during refactoring, when splitting a file up the relative paths are inevitably off and I have to actually think about how many ../s there should be. And then fix it when the build blows up 😁... Twice. I'm not good at picking the correct relative path.

I'm curious of the need for that Rollup plugin since Rollup accepts a paths

I've tried to use path but I've found this issue: it seems that path should be used for include  external dependencies.


Assist: reorder fields

This assist should be applicable to record literals and patterns, and should reorder fields in the same order as in the definition. Ie, it should be able to cleanup this:

I wonder if there is some caveat in the order of evaluation of record literal fields initializers

Assists are not guaranteed to preserve semantics.

I think this could get tricky:
struct Foo {
    a: String,
    b: String,
}

impl Foo {
    fn new() -> Self {
        let s = String::new();
        Self {
            b: s.clone(),
            a: s,
        }
    }
}

I'd think the output of an assist should always compile at least?

I'd think the output of an assist should always compile at least?

Not necessary, assist is just a tool that applies some syntactic transformation which is helpful, it doesn't need to be complete or "correct". For example, in fill match arms, we deliberately use {} and not todo!() as a placeholder, because {} is easier to delete.
The goal of assist is to be useful, and in your example I would be totally willing to apply the assist and move the clone call manually afterwards.

Hi,
First-time contributor here. I'd love to give this a try !

I believe this issue can be closed, as it was implemented in #3925


Conjugate heat transfer

Description of the problem

Lethe had no capability for solid regions and conjugate heat transfer problems

Description of the solution
This ended up being a major undertaking

Added the capability to have solid regions in the parser and the physical property manager
Added an automatic identification of solid regions in the Navier-Stokes equations
Added a mechanism in the assembly to skip assembling equation in these regions
Added mechanism in the heat transfer assembler to correctly identify the region

How Has This Been Tested?

One unit test is added
An entire example has been added

Documentation

Physical property.rst
Full example has been added

Future changes

Will need to be extended to other physics (tracer and chemical reactions when we have them)

Good work. I only have a couple of comments related to the example in the documentation. Also, I would suggest to remove concentric-heat-exchanger/images/schematic.svg since it is not being used.

I understand what you mean for the SVG, but this enables us to remake the schematic if it is modified (the svg is modifiable by inkscape directly, it's what I use, so I'd rather keep it :) )


config.json is not provided in the cet_1_9_2.zip

as per:   https://wiki.cybermods.net/cyber-engine-tweaks/getting-started/configuration
it mentions:
The config file is found in: /bin/x64/plugins/cyber_engine_tweaks/config.json
however when open the ZIP there is nothing....

config.json is deprecated in new versions. you can change all settings in-game now using the console overlay and clicking on the settings tab along the top.

config.json is deprecated in new versions. you can change all settings in-game now using the console overlay and clicking on the settings tab along the top.

@alefnull , ah ok  thanks for the clarification.
then the wiki  needs updating then.

@alefnull , ah ok  thanks for the clarification.
then the wiki  needs updating then.

This is not an issue :P
We do not package it cause of the reasons alefnull said.
Also, we were effectively reseting settings by each update, this should in general be much better solution I believe.
@yamashi can be closed

This is not an issue :P
We do not package it cause of the reasons alefnull said.
Also, we were effectively reseting settings by each update, this should in general be much better solution I believe.
@yamashi can be closed


Adding exercise05

Very well done. You have some very nice graphs and I like how you built functions to create then so they're easily reproducible. You did put two of the functions in module_affordable_housing.py, but you didn't import and call them from the module. I'm happy to work on this with you at office hours so you get this last step.


improved UI

Pull Request Description
Improved Ui for the individual pages

Hi @CobbyElsonfx ,
Status: Approved ✔
Excellent job for creating the basic schema and models. Keep it up 👏
Your project is complete! There is nothing else to say other than... it's time to merge it 💪
Congratulations! 🎉

[Optional] suggestions:

Kindly please Add the link of the frontend app in all the PR you will create next time.


Suddenly started returning 500 on metrics endpoint

Running kminion 2.1.0. It was working fine and then suddenly it started returning a 500 error on /metrics.
Turned on debug logs and tried upgrading to 2.2.0 but it's still returning 500 and according to the logs everything is returning successful. Any ideas?
Edit: Seemed to fix itself after a rolling restart of the Kafka cluster, still odd there was 0 errors or warnings.

Hi @rmb938 ,
are you sure this is returned by KMinion? How long does it take until this is served? What happens if you try to call it's /ready endpoint?
As you can see here: https://github.com/redpanda-data/kminion/blob/master/main.go#L101 , the Prometheus lib itself is fully in charge of serving the response and I have never seen a 500 error returned there either. I would expect some error log messages at least.

I'm pretty sure, I went into the container and curl'ed it on localhost. In the debug logs I could see it send all the requests to Kafka and right when it stopped logging the 500 was returned, this happened fairly quickly, a second or less.
Unfortunately the issue fixed itself after a Kafka cluster rolling restart... I'll keep an eye out of it happening again and run it against the /ready endpoint.


Windows installer creates incorrect folder in %USERS% folder

v1.0.7
During installation it creates c:\Users\RUNNER~1\ folder

I could replicate it. We will debug more to find the reason and try to fix it as soon as possible.


adding redirects for s.dvc.org

Per https://github.com/iterative/dvc.org/issues/2967#issuecomment-956455055
"for now to just use redirects-list entries with some convention that the new solution will follow?"

adds the following redirects for s.dvc.org
s.dvc.org/g/exp/run -> /doc/user-guide/experiment-management/running-experiments
s.dvc.org/g/exp -> /doc/user-guide/experiment-management
s.dvc.org/g/ -> /doc/user-guide/
s.dvc.org/s/ -> /doc/start/
s.dvc.org/c/ -> /doc/use-cases/
s.dvc.org/r/ -> /doc/command-reference
Closes #2968
Closes #2967

Closes #2967

I don't think this closes https://github.com/iterative/dvc.org/issues/2967#issuecomment-956455055

That would probably take another month to implement in the engine.

front-matter yes, removing prefixes (and use some custom link) - it can be done even today. frontmatter is just an implementation detail, I don't thing @skshetry is specifically concerned about the mechanism

That would probably take another month to implement in the engine.

I could have waited if this was going to be worked on in the next month. 😄  I had a PR that only added 2 links: #2968, but I did not get any response (that adding those redirections is okay for now).

@skshetry please feel free to add any redirects under the s.dvc.org namespace with the redirects.json file. I think that should resolve the problem for now (if let's say you still don't like the verbosity of the Emre's redirects)?


Suggestion: Replace Result<Async, E> with Async<T, E>

I have just read this article: https://www.coredump.ch/2017/07/05/understanding-the-tokio-reactor-core/ and the double wrapping of future return values (e.g. return Ok(Async::Ready(self.answer))) looks pretty cumbersome to me.
My suggestion is to add an Err variant to the futures::Async enum (and rename Ready to Ok) so that it becomes:
pub enum Async<T, E> {
    Ok(T),
    Err(E),
    NotReady,
}

Then Ok(Async::Ready(self.answer)) can simply become Async::Ok(self.answer).

Thanks for the report! I believe this is a duplicate of https://github.com/alexcrichton/futures-rs/issues/141 which has been closed, so I'm going to close this as well.


MNT: compatibility with cython 3

This change is required to compile with cython 3 pre-releases.  I think that this may be revealing a bug upstream (xref https://github.com/cython/cython/issues/3613) but opening this PR as well out of an abundance of pragmatism.
This is the result of an intentional change upstream: https://github.com/cython/cython/issues/2564
TODO:

[x] Unit tests and/or doctests in docstrings
[x] tox -e py38 passes locally (there is a flake8 failure but unrelated to these changes)
[x] Docstrings and API docs for any new/modified user-facing classes and functions
[x] Changes documented in docs/release.rst
[x] tox -e docs passes locally
[ ] AppVeyor and Travis CI passes
[ ] Test coverage to 100% (Coveralls passes)

py38 run-test: commands[3] | flake8 numcodecs
numcodecs/categorize.py:46:43: E741 ambiguous variable name 'l'
ERROR: InvocationError for command /home/tcaswell/source/bnl/zarr-developers/numcodecs/.tox/py38/bin/flake8 numcodecs (exited with code 1)

https://github.com/cython/cython/pull/3615 was merged to cython which eliminates the need for this change, closing.


We need CI

The package template needs CI coverage — the combination of features, flags, moving Zeek versions etc is too fragile to assume it just works. @rsmmr got bitten by this recently, and I once nearly pushed a breaking change to master because I didn't try out an offending combination of settings.
This relates to zeek/action-zkg-install#5: we should verify that the CI feature actually runs successfully.

Yeah, in my branch for more Spicy features I'm using a shell-script as a manual, poor-man's-CI substiute for now: https://github.com/zeek/package-template/blob/31668ba6af3af6b63c360d4a5a5b83e9530d33c7/test-spicy-features


Added CARO import

added CARO import to CL and fixed unsat axioms see #1306 for more details
Fixes #1306

error:
2022-01-13 16:34:05,375 ERROR org.obolibrary.robot.ReasonOperation - Only equivalent classes that have been asserted are allowed. Inferred equivalencies are forbidden.
2022-01-13 16:34:05,376 ERROR org.obolibrary.robot.ReasonOperation - Equivalence: <http://purl.obolibrary.org/obo/CL_0000003> == <http://purl.obolibrary.org/obo/GO_0005623>
make: *** [cl-full.owl] Error 1
Makefile:616: recipe for target 'cl-full.owl' failed

Seems like CARO is making a equiv class for cell to an obsolete term

Related to: https://github.com/obophenotype/caro/issues/23

closing this as imports systems have changed


lightbox: Add "share" button in bottom app bar

The share_plus plugin has an API for sharing a file: https://pub.dev/documentation/share_plus/latest/share_plus/Share/shareXFiles.html
We should try it out, for a button to share the image.
As with #42, I didn't include this in the prototype because I think it'll involve an API request; see there for details.

I would like to work on it

We got a request for this in beta feedback, emphasizing that the legacy app's "share" button in the lightbox is pretty unhelpful because it shares a string of text instead of the image file itself.
From the issue description, it seems like there's a straightforward way to share the file itself; we should do that.


[HOLD #16364][$1000] The emoji recommendation list closes on switching to next tab

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to staging dot on chrome
Go to a chat > Type in any emoji within colon. Example :dis and the recommendation list appears.
Switch to the next tab and come back to the application.
Result : See that the emoji recommendation list closes
Follow the above steps in slack and see that the emoji recommendation doesn’t close even when we switch to the next tab

Expected Result:
The emoji recommendation list shouldn’t close on switching to the next tab
Actual Result:
The emoji recommendation list closes on switching to the next tab
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.2.90-4
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/43996225/228036455-823e9754-ea41-44d6-b36d-5f1e6d31a0b0.mp4
https://user-images.githubusercontent.com/43996225/228036470-f93e8a4d-5bef-4cb0-9025-d7a7ee0d102c.mp4
Expensify/Expensify Issue URL:
Issue reported by: @avi-shek-jha
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1679907956819009
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~010419d96d34cc6f01
Upwork Job ID:<PHONE_NUMBER>138916864
Last Price Increase: 2023-05-15

Triggered auto assignment to @michaelhaxhiu (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)

If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead

[ ] This bug is reproducible using the reproduction steps in the OP. S/O

If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
If the reproduction steps aren't clear and you determine the correct steps, please update the OP.

[ ] This issue is filled out as thoroughly and clearly as possible

Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.

[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

Maybe dupe of https://github.com/Expensify/App/issues/16364

@michaelhaxhiu Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

@michaelhaxhiu 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

Thanks @Pujan92.
I'd like to cc @0xmiroslav - curious what you think about this bug? If we agree with your latest comment on https://github.com/Expensify/App/issues/16364#issuecomment-1493702177 then I think we would proceed with calling this a bug?

So can we confirm that consistent behavior should be to show recommendation on both focus and selection change always on all platforms?

I can't decide if this is a feature or bug, basically?

Discussed on slack and it's decided as not a priority for the moment. Closing it out.

We had a change of heart after discussing further, let's re-open and prioritize because this is a valid inconsistency in the product (and thus - we should fix it).

Job added to Upwork: https://www.upwork.com/jobs/~010419d96d34cc6f01

Current assignee @michaelhaxhiu is eligible for the External assigner, not assigning anyone new.

Triggered auto assignment to @marcochavezf (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The emoji recommendation list automatically closes if a user switches to a different browser tab and returns back to the Expensify Chat tab.
What is the root cause of that problem?
The emoji recommendation list is set to hide with a call to this.resetSuggestedEmojis  in the onBlur event handler for the composer on ReportActionCompose.
What changes do you think we should make in order to solve the problem?
We can remove this reset method call, and check if the user manually changes focus away from the composer input.
Alternatively, we can add code to check the content of the composer when it regains focus and re-display the emoji suggestion based on the current input and the cursor position.
What alternative solutions did you explore? (Optional)
None.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The emoji recommendation list closes on switching to the next tab
What is the root cause of that problem?
In https://github.com/Expensify/App/blob/a5d171b6ccd6c2cbef00285c4d984796bf4b845d/src/pages/home/report/ReportActionCompose/index.js#L859 we're resetting the suggested emojis when the Composer is blurred, thus close the emoji recommendation list.
If we mirror the correct behavior of Slack, in the following 2 cases we should not close the emoji recommendation list when the composer is blurred:

When we switch to another browser tab
When we open another native app (aside from the browser)

What changes do you think we should make in order to solve the problem?

In here https://github.com/Expensify/App/blob/a5d171b6ccd6c2cbef00285c4d984796bf4b845d/src/pages/home/report/ReportActionCompose/index.js#L859, we need to check that the window is currently visible and focused, if yes then we'll reset the emoji list since that means the user proactively blurs the input. If no, we should not reset the emoji list since the user only switches to another tab/app and will come back and focus on the input again.
We can use Visibility.isVisible here https://github.com/Expensify/App/blob/4607dcb69b6d5bb7607f652f8a78327c7e002183/src/libs/Visibility/index.js#L8 for this. However there's currently a bug in  Visibility.isVisible in web that causes it to return true even when we open another native app, that's being addressed here https://github.com/Expensify/App/issues/17242. That issue will be fixed by updating the Visibility.isVisible to return document.visibilityState === 'visible' && document.hasFocus() as per the accepted proposal there.
That's exactly what we need so after that change is merged, we can just update https://github.com/Expensify/App/blob/a5d171b6ccd6c2cbef00285c4d984796bf4b845d/src/pages/home/report/ReportActionCompose/index.js#L859 to
if (Visibility.isVisible()) {
    this.resetSuggestedEmojis();
}

What alternative solutions did you explore? (Optional)
NA

@marcochavezf @michaelhaxhiu @0xmiroslav this issue is now 3 weeks old. There is one more week left before this issue breaks WAQ and will need to go internal. What needs to happen to get a PR in review this week? Please create a thread in #expensify-open-source to discuss. Thanks!

@marcochavezf, @michaelhaxhiu, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!

Because composer is blurred, blue outline is removed but emoji suggestions still remaining.

@0xmiroslav Slack also works like that so I think that's fine. I think the user in that case is ready to return to the app to continue to select emojis so keeping the emoji suggestions is ok.

@marcochavezf @michaelhaxhiu @0xmiroslav this issue is now 4 weeks old and preventing us from maintaining WAQ, can you:

Decide whether any proposals currently meet our guidelines and can be approved as-is today
If no proposals meet that standard, please take this issue internal and treat it as one of your highest priorities
If you have any questions, don't hesitate to start a discussion in #expensify-open-source

Thanks!

Current assignee @0xmiroslav is eligible for the Internal assigner, not assigning anyone new.

Current assignee @michaelhaxhiu is eligible for the External assigner, not assigning anyone new.

Current assignee @0xmiroslav is eligible for the External assigner, not assigning anyone new.

Current assignee @marcochavezf is eligible for the External assigner, not assigning anyone new.

@0xmiroslav Have you had a chance to review my proposal?

@0xmiroslav Have you had a chance to review my proposal?

@akinwale yes I checked. Your solution doesn't work for me.

Are you testing in Chrome or Safari? I can take another look and try to come up with a better solution.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The emoji suggestions are hidden when we open a new tab
What is the root cause of that problem?
In the onBlur of the composer we are reseting the emojis suggestions
What changes do you think we should make in order to solve the problem?

We can subscribe to AppState change to know if the app is in the background or not:
AppState.addEventListener('change', this._handleAppStateChange);
Unfortunely the appState change happens after the onBlur so in the onBlur instead of calling resetSuggestedEmojis  we can:

put this.state.shouldShowSuggestionMenu to false
in 20ms put this.state.suggestedEmojis to [] if the app State is "active" because that means we are still in the app

In the new method _handleAppStateChange, if the app is now in the background and this.state.suggestedEmojis is not [] that means the suggested emojis were opened and we can put shouldShowSuggestionMenu to true so that when the user comes back to the app he will see the suggested emojis again.
That's the result I have:
https://user-images.githubusercontent.com/19537677/234995922-1df94149-171a-4a3e-8032-12da14e2316d.mp4
It's not clear in the video but when I click another app (for example NotPad) the suggested emojis are hidden, I didn't understand if it is what we want or not.
What alternative solutions did you explore? (Optional)

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

Checking in - how is this coming along so far @0xmiroslav? Do we have a proposal that's viable or need to go internal?

Hi @0xmiroslav, could you take a look at the current proposals?

@0xmiroslav Hello, thanks for reviewing my proposal. Can you give me more feedback about my proposal so that I can improve and maybe make a better proposal ? You don't like it because it uses AppState or because of something else ?

Also, considering the posted proposals, what do you think would be the best approach to address this issue? Since the problem occurs when switching to another tab, I'm wondering if the solution should be limited to the web only, thoughts?

@0xmiroslav should this be on hold for https://github.com/Expensify/App/pull/18065? Let's update the GH title if so.

@0xmiroslav sure thing
Proposal updated

Not overdue, waiting for @0xmiroslav to review the updated proposal

On hold for https://github.com/Expensify/App/issues/16364

still held 🕳

On hold

On hold

same hold situation

Still on hold

going weekly because melvin is getting on my bad side 😂  (also - "hold" in title is the req for weekly label)

Still on hold

Still on hold

Still on hold

Still on hold

Still on hold https://github.com/Expensify/App/issues/16364

Note: I'm preparing to go OOO for ~2 weeks, and need a buddy to oversee this while I'm gone.
@sonialiap in terms of next steps:

This is on "hold" until https://github.com/Expensify/App/issues/16364 is finalized, since there's a dependency of code on the proposal here and what's being implemented in #16364.
There's a good chance this doesn't see any movement while I'm gone. In which case I'll just take it over when I'm back!

Since this is on hold for a weekly issue that's also on hold, I think it would make sense to change it to weekly

Moving to weekly

Back from OOO, unassigning Sonia.

two GHs are holding this one, and so I'm using monthly for now.

No progress on this still as it's blocked on. I'm going to manually re-assign to @sonialiap since she had an eye on it before.
Next steps (same as before):

This is on "hold" until https://github.com/Expensify/App/issues/16364 is finalized, since there's a dependency of code on the proposal here and what's being implemented there.
If this isn't fixed by the time I return, I'm glad to take it back! fingers crossed 🤞 it gets unblocked fixed though

still on hold for https://github.com/Expensify/App/issues/16078

still on hold for https://github.com/Expensify/App/issues/16078

@DylanDylann reassigning, please take over as C+. If you don't have bandwidth, unassign yourself. Thanks

@sonialiap Could you help to update the title: Hold on 16078 and update this issue to weekly because 16078 also be weekly

Still on hold

still on hold for https://github.com/Expensify/App/issues/16078

Still on hold for https://github.com/Expensify/App/issues/16078

Still on hold for #16078

Still on hold for https://github.com/Expensify/App/issues/16078

still on hold

Still on hold

Still on hold for https://github.com/Expensify/App/issues/16078

Still on hold for #16078

Still on hold for https://github.com/Expensify/App/issues/16078

Still on hold. @sonialiap Should we update label to Monthly

Still on hold. I think weekly is fine, the other issue was moved to weekly too

Still on hold

The PR for the issue that we're holding for was recently merged, but let's wait a bit more in case of regressions

https://github.com/Expensify/App/issues/16078 has merged and the reported issue is reproducible

Asking team input on value of issue - slack

After discussing we've decided to close this issue as low priority


Improve URLProvider

Right now if i want to make this works, i have to do this.
const App = () => (

    <Url.URLProvider urls={urls}>
      {[

Content

      ]}
    </Url.URLProvider>

);

Instead of this.
const App = () => (

    <Url.URLProvider urls={urls}>

Content

    </Url.URLProvider>

);

Thanks for the fix! I already published it in npm with version 1.0.1


Init NextJS with typescript

Fixed in https://github.com/easy-deep-learning/nextjs-project-template/commit/33308171be6ce7eb4898374adcf95d7b2b028349


minimize the proteus.linux2.yaml stack

By that I mean try commenting out as much of the stack as we can.  The main dependencies we all know: petsc, hdf5, superlu, triangle, tetgen, but in some of the newer modules there are others like chrono, scipy, etc. I believe as long as you just comment out packages the stack rebuilds should go quickly and if you just remove proteus/linux2 the proteus modules won't rebuild either, so testing the stack by hand shouldn't take all that long and will shave some time off the downloads of the binaries.  On the other hand the really big packages are probably petsc and hdf5, which we can't do anything about.. By that I mean try commenting out as much of the stack as we can.  The main dependencies we all know: petsc, hdf5, superlu, triangle, tetgen, but in some of the newer modules there are others like chrono, scipy, etc. I believe as long as you just comment out packages the stack rebuilds should go quickly and if you just remove proteus/linux2 the proteus modules won't rebuild either, so testing the stack by hand shouldn't take all that long and will shave some time off the downloads of the binaries.  On the other hand the really big packages are probably petsc and hdf5, which we can't do anything about.

From a quick look at the listed packages, I don't see any exposure in the code to julia or cgal. The other packages that might not generally be used are packages for plotting like openjpeg, matplotlib, matlab and for manipulating certain data formats like GRIB ( grib_api).

Some of the packages are needed in the C++ code (e.g. scorec), which would require more than just deleting proteus/linux2 to reset the build. So unless it's too expensive, I'd rather just let Travis handle testing the stack profile.

It might be useful to add a note next to each package indicating how they are used. That way, we can have a better idea of when packages become obsolete.

On 1) yes, and we can use pip install for matplotlib if it's not being used for any tests. On 2) hashdist is going to ensure that scorec dependencies are met, so unless there are undeclared dependencies in the scorec.yaml file deleting proteus/linux2 will regenerate the stack properly. hashdist uses a sort of build jail to  try to prevent linking to external dependencies accidentally. It's fine to let travis do the checking if you want to, but I was trying to save you time. Good idea on 3) about adding a note. You could just put a generic "optional package" in the ones you comment out for now.

It turns out matplotlib is used by the testing suite.
My concern was really that if a C++ extension/module was already compiled, the .so and .a files are not stored in proteus/linux2 so even if I commented out some packages and deleted proteus/linux2, it wouldn't prompt a rebuild of the C++ libraries.

I've commented out a good number of packages that can be seen in the following erdc/hashstack to hashdist/hashstack PR: hashdist/hashstack#997
If anyone else has any other packages that they think can be removed, let me know.
The following is the Travis test associated with the test stack:
https://travis-ci.org/erdc/proteus/jobs/337657934


[21.02] luci-app-https-dns-proxy: massive providers update

introduce 31 new providers
uniform names (FQDNs) in provider files
other minor fixes

Signed-off-by: Stan Grishin<EMAIL_ADDRESS>(cherry picked from commit fae45215aa3a42b944270e0e75bfa1ba23e5702f)

Hi Stan. Given that this list is surely likely to grow as time goes on (and manually providing these might become untenable), how about an extra 'custom' field where users can provide their own at config time?
Possibly an idea for future iteration.

Hi Stan. Given that this list is surely likely to grow as time goes on (and manually providing these might become untenable), how about an extra 'custom' field where users can provide their own at config time?
Possibly an idea for future iteration.

Paul, this is a frequently requested feature and it's totally planned for after transition to javascript. I really don't want to implement it in Lua now since the transition is inevitable (although not immenent).
I'd welcome a PR for either (transition or a custom provider implementation in lua).

I hear you on the lua thing...


Navigation via arrow keys does not work if focus is lost

Expected behavior
Currently it is somehow possible to use the arrow keys left and right to navigate through the slides.
Actual behavior
However, this is only possible if the navigation buttons at the top have been clicked at first and keep the focus, if they loose it, because the user clicks somewhere onto the slide, then these click events are not handled anymore.
Need help
These arrow-key events are handled in src/elm/Lia/View.elm by function onNavigationKey:
https://github.com/LiaScript/LiaScript/blob/ab8013838f6b7f3f5ea198a4a872e2bcba7f1332/src/elm/Lia/View.elm#L35-L64
However, I have no idea why this happens if the focus is lost. The event-handling happens on the main div and thus work for all child elements, if they do not disable event-bubbling... at least in theory.
Hints and suggestions are welcome ;-)

Commit: 7051380
does not fix this issue, it circumvents it. The event handling for key press events for navigation has been moved to JS (TypeScript), similar to swipe events ...

Commit: 7051380
does not fix this issue, it circumvents it. The event handling for key press events for navigation has been moved to JS (TypeScript), similar to swipe events ...


KSML 1.0.0 - Joins not working. Optimisation setting not used

When running a simple Stream-Table join KSML only picks up table entries made after the application started.
A restart will wipe this state, causing missed joins.
The cause of the issue is a regression in the KafkaStreamsRunner and TopologyGenerator classes.
The Optimisation property is not forwarded and used in the TopologyGenerator build method.

Fixed by PR #136


Logging Operator Not Delivering Logs to Elastic

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):

Follow the documentation to install Rancher 2.5 Logging, (See: https://rancher.com/docs/rancher/v2.x/en/logging/v2.5/#enabling-logging-for-rancher-managed-clusters).
Configure logging to send to Elastic:

The contents of the flow.yml file:
kind: ClusterFlow
metadata:
  creationTimestamp: "2020-10-28T16:24:35Z"
  generation: 2
  name: all-logs
  namespace: cattle-logging-system
  resourceVersion: "69296331"
  selfLink: /apis/logging.banzaicloud.io/v1beta1/namespaces/cattle-logging-system/clusterflows/all-logs
  uid: 8a3ddcfd-55bc-49a6-bfe2-719c3873faaf
spec:
  filters:
  - record_transformer:
      enable_ruby: true
      records:
      - log_id: ${(0...32).map { (65 + rand(26)).chr }.join}
  - throttle:
      group_bucket_limit: 1000
      group_bucket_period_s: 10
      group_key: $.kubernetes.container_name
  globalOutputRefs:
  - elastic

The contents of the output.yml file:
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  creationTimestamp: "2020-10-28T16:24:14Z"
  generation: 3
  name: elastic
  namespace: cattle-logging-system
  resourceVersion: "69296576"
  selfLink: /apis/logging.banzaicloud.io/v1beta1/namespaces/cattle-logging-system/clusteroutputs/elastic
  uid: 5569b503-0c4e-4930-a511-68bc7f99cc4a
spec:
  elasticsearch:
    default_elasticsearch_version: "7"
    host: instance_id.elastic.cloud
    id_key: log_id
    include_timestamp: true
    index_name: rancher-logging
    password:
      valueFrom:
        secretKeyRef:
          key: [ . . . REDACTED . . . ]
          name: [ . . . REDACTED . . . ]
    port: 9243
    reconnect_on_error: true
    reload_connections: false
    reload_on_failure: true
    request_timeout: 15s
    rollover_index: true
    scheme: https
    suppress_type_name: true
    user: k8s-logging

Result:
Pod are not being forwarded to elastic. The following errors were found in the fluentbit pod log:
2020-10-27T13:55:00.479478954Z [2020/10/27 13:55:00] [error] [src/flb_io.c:171 errno=111] Connection refused
2020-10-27T13:55:00.479505189Z [2020/10/27 13:55:00] [error] [io] connection               #132              failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240

And also this:
2020-10-27T13:54:20.480609189Z [2020/10/27 13:54:20] [error] [io] connection               #292              failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240
2020-10-27T13:54:20.480614232Z [2020/10/27 13:54:20] [error] [output:forward:forward.0] no upstream connections available

Other details that may be helpful:
Some logs from k8s events are being sent, but nothing pertaining to any of the deployed workloads.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.1
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): VMware, 8 CPU/32GB RAM
Kubernetes version (use kubectl version): 1.16.15
Docker version (use docker version): 19.3.8

Client:
 Debug Mode: false

Server:
 Containers: 315
  Running: 160
  Paused: 0
  Stopped: 155
 Images: 73
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local pxd
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-121-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.41GiB
 Name: k8s-node-1
 ID: [ . . . REDACTED . . . ]
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

WARNING: No swap limit support

gz#13058

i've met the same problem, hope these messages can help you
Environment information
Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.1
Installation option (single install/HA): single
Cluster information
Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): cloud 4 CPU/8GB RAM
Kubernetes version (use kubectl version): v1.19.3
Docker version (use docker version): 19.03.13
here is the error logs from fluentbit pod log:
[2020/11/08 13:40:39] [error] [src/flb_io.c:171 errno=111] Connection refused
[2020/11/08 13:40:39] [error] [io] connection #58 failed to: rancher-logging-rke-fluentd.cattle-logging-system.svc:24240
[2020/11/08 13:40:39] [error] [output:forward:forward.0] no upstream connections available
[2020/11/08 13:40:39] [error] [src/flb_io.c:171 errno=111] Connection refused
[2020/11/08 13:40:39] [error] [io] connection #41 failed to: rancher-logging-rke-fluentd.cattle-logging-system.svc:24240
[2020/11/08 13:40:39] [error] [output:forward:forward.0] no upstream connections available
[2020/11/08 13:40:39] [ warn] [engine] failed to flush chunk '1-1604841983.492702482.flb', retry in 54 seconds: task_id=1485, input=tail.0 > output=forward.0
[2020/11/08 13:40:39] [ warn] [engine] failed to flush chunk '1-1604842785.17854207.flb', retry in 685 seconds: task_id=1696, input=tail.0 > output=forward.0

is there any way to fix the problem before the v2.5.3 release, such as change the development yaml files?

Rancher version: v2.5.1
I created a Custom cluster of 4 nodes with all roles (etcd/control plane/worker).
I installed rancher-logging to the System project and created the ClusterOutput and ClusterFlow.
ClusterOutput
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  creationTimestamp: "2020-11-09T18:36:54Z"
  generation: 2
  name: clusterout1
  namespace: cattle-logging-system
  resourceVersion: "16635"
  selfLink: /apis/logging.banzaicloud.io/v1beta1/namespaces/cattle-logging-system/clusteroutputs/clusterout1
  uid: 8ccf8a28-9fab-4f04-8fab-92a46cd44db9
spec:
  elasticsearch:
    default_elasticsearch_version: "7"
    host: <REDACTED>
    include_timestamp: true
    index_name: rancher-logging
    password:
      valueFrom:
        secretKeyRef:
          key: password
          name: elasticpass
    port: 9200
    reconnect_on_error: true
    reload_connections: false
    reload_on_failure: true
    request_timeout: 15s
    rollover_index: true
    scheme: http
    suppress_type_name: true
    user: elastic

ClusterFlow
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  creationTimestamp: "2020-11-09T18:39:37Z"
  generation: 1
  name: clusterflow1
  namespace: cattle-logging-system
  resourceVersion: "16913"
  selfLink: /apis/logging.banzaicloud.io/v1beta1/namespaces/cattle-logging-system/clusterflows/clusterflow1
  uid: ef59baed-25ab-4427-b3cd-47df7d34f8bf
spec:
  filters:
  - record_transformer:
      enable_ruby: true
      records:
      - log_id: ${(0...32).map { (65 + rand(26)).chr }.join}
  - throttle:
      group_bucket_limit: 1000
      group_bucket_period_s: 10
      group_key: $.kubernetes.container_name
  globalOutputRefs:
  - clusterout1

I create a project: testproj1
I create a namespace: testns1
And created a workload in testns1 using paynejacob/loggenerator:latest
I see the logs flowing to the ElasticSearch backend with this configuration but I observed that if I create an Index Pattern with timestamp I don't see all the logs, most of the observed logs when using the @timestamp pattern setting are from container/kubernetes events.

But If I select to not use  @timestamp for the index pattern settings I see all the logs from the kubernetes cluster (from above test workload for example).
I also noticed the fluent-bit errors reported in this issue but these happened each time a configcheck was done and applied. Like in the case of a new created ClusterFlow, but the logs were flowing to the elasticsearch backend after a successful flush.
Sample fluentbit pod log:
[2020/11/09 18:23:32] [ info] [engine] started (pid=1)
[2020/11/09 18:23:32] [ info] [storage] version=1.0.5, initializing...
[2020/11/09 18:23:32] [ info] [storage] root path '/buffers'
[2020/11/09 18:23:32] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/09 18:23:32] [ info] [storage] backlog input plugin: storage_backlog.1
[2020/11/09 18:23:32] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 4.8M
[2020/11/09 18:23:32] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2020/11/09 18:23:32] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2020/11/09 18:23:32] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2020/11/09 18:23:32] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK
[2020/11/09 18:23:32] [ info] [sp] stream processor started
[2020/11/09 18:23:32] [ info] inotify_fs_add(): inode=262321 watch_fd=1 name=/var/log/containers/canal-qqtxx_kube-system_flexvol-driver-8507bc9c29936dc100d7de9d4f554e651266651374b731f328cadcad0914604e.log
[2020/11/09 18:23:32] [ info] inotify_fs_add(): inode=391287 watch_fd=2 name=/var/log/containers/rancher-logging-fluentd-0_cattle-logging-system_fluentd-cadeef9b0bc3ae5b2c3fa2493aaefbccfb35ef4e4688ddd915ceb0558e4dcf5a.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=261723 watch_fd=3 name=/var/log/containers/canal-qqtxx_kube-system_calico-node-9433a10375f2f37bfb39a9220323d25a3f95d27abaa39b8b99826f8a1f9c74af.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=262235 watch_fd=4 name=/var/log/containers/canal-qqtxx_kube-system_install-cni-65afe28054aede75bef1706ce0236626a35ddfae4d6b610a5774fabcbcbb27e0.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=262467 watch_fd=5 name=/var/log/containers/canal-qqtxx_kube-system_kube-flannel-8ef43d39cee544f9daa78719358621f0e6a57e27034e01f9d2cf1f4d976f1d70.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=262146 watch_fd=6 name=/var/log/containers/cattle-node-agent-twbwg_cattle-system_agent-8b380b6d2877c868fdb0fd043f95c4da9addd4d778b6a24e9ba00eac99a13d63.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=262562 watch_fd=7 name=/var/log/containers/kube-api-auth-dptfd_cattle-system_kube-api-auth-a513f06650f0d5b1cdc9c8d95ab8eaf51d8cbde381827585929dfc3d2edeae5d.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=390889 watch_fd=8 name=/var/log/containers/nginx-ingress-controller-cj488_ingress-nginx_nginx-ingress-controller-a4fd8863279ea83684fdf33604eebf8b06775ae9e6c8527439ef494031ba12ea.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=391481 watch_fd=9 name=/var/log/containers/rancher-logging-rke-containers-fluentbit-2nxhc_cattle-logging-system_fluent-bit-2a5d5297d37a67e3b42b836b15efe9a82990a208af79d803137400b825855cbf.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=391505 watch_fd=10 name=/var/log/containers/rancher-logging-rke-fluentbit-r59dn_cattle-logging-system_fluent-bit-60424432c2c0158e2e8a8583211915e4bad47aad346b51eb5db0ab674a0b5950.log
[2020/11/09 18:23:33] [ info] inotify_fs_add(): inode=391091 watch_fd=11 name=/var/log/containers/rancher-logging-rke-fluentd-configcheck-c3054702_cattle-logging-system_fluentd-f9cc987b7fb5f4e35522f6e3f70a8365242b91ec5d838c0d4c70830535b14097.log
[2020/11/09 18:25:44] [error] [src/flb_io.c:171 errno=110] Connection timed out
[2020/11/09 18:25:44] [error] [io] connection #42 failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240
[2020/11/09 18:25:44] [error] [output:forward:forward.0] no upstream connections available
[2020/11/09 18:25:44] [ warn] [engine] failed to flush chunk '1-1604946212.229106576.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=forward.0
[2020/11/09 18:25:44] [ info] inotify_fs_add(): inode=391001 watch_fd=12 name=/var/log/containers/rancher-logging-5d9f8c9f9d-mxw64_cattle-logging-system_rancher-logging-eb32ed5469ad4f8b8091038073d4e417324eae186fc8284e0c0314caa8589404.log
[2020/11/09 18:25:44] [ info] inotify_fs_add(): inode=391529 watch_fd=13 name=/var/log/containers/rancher-logging-fluentbit-6x24q_cattle-logging-system_fluent-bit-59febe00f99f554ab77e540aa71a1ad9d19dc2ee29ea906ed62578cb76d6ef4a.log
[2020/11/09 18:25:44] [ info] inotify_fs_add(): inode=391558 watch_fd=14 name=/var/log/containers/rancher-logging-fluentd-0_cattle-logging-system_config-reloader-2d0e1edc34f3611d55ad452df14f83598146fdfacce8979708d1507c8a57d95d.log
[2020/11/09 18:25:49] [ info] [engine] flush chunk '1-1604946212.229106576.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
[2020/11/09 18:40:09] [ info] inotify_fs_remove(): inode=391091 watch_fd=11
[2020/11/09 18:40:27] [error] [src/flb_io.c:171 errno=111] Connection refused
[2020/11/09 18:40:27] [error] [io] connection #33 failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240
[2020/11/09 18:40:27] [error] [output:forward:forward.0] no upstream connections available
[2020/11/09 18:40:27] [ warn] [engine] failed to flush chunk '1-1604947226.470797876.flb', retry in 7 seconds: task_id=0, input=tail.0 > output=forward.0
[2020/11/09 18:40:28] [error] [src/flb_io.c:171 errno=111] Connection refused
[2020/11/09 18:40:28] [error] [io] connection #42 failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240
[2020/11/09 18:40:28] [error] [output:forward:forward.0] no upstream connections available
[2020/11/09 18:40:28] [ warn] [engine] failed to flush chunk '1-1604947227.305666549.flb', retry in 11 seconds: task_id=1, input=tail.0 > output=forward.0
[2020/11/09 18:40:29] [error] [src/flb_io.c:171 errno=111] Connection refused
[2020/11/09 18:40:29] [error] [io] connection #42 failed to: rancher-logging-fluentd.cattle-logging-system.svc:24240
[2020/11/09 18:40:29] [error] [output:forward:forward.0] no upstream connections available
[2020/11/09 18:40:29] [ warn] [engine] failed to flush chunk '1-1604947228.305801382.flb', retry in 6 seconds: task_id=2, input=tail.0 > output=forward.0
[2020/11/09 18:40:34] [ info] [engine] flush chunk '1-1604947226.470797876.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
[2020/11/09 18:40:35] [ info] [engine] flush chunk '1-1604947228.305801382.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
[2020/11/09 18:40:39] [ info] [engine] flush chunk '1-1604947227.305666549.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0

yes, my ES works well now. after waiting for a long time, I can get the data in ES. And in this scene, different logs use multipul indexes which is hard for analysis, is there any method to optimize this?

Loki works, but not ES.
connection failed to rancher-logging-fluentd.cattle-logging-system.svc

Please fix this issue, lokki data isn't full for me. I need details in my logs

@JoeyGaojingxing How did you solve this issue? we are also getting the same error in a fluent-bit pod.
can you please help me related this issue?

@JoeyGaojingxing How did you solve this issue? we are also getting the same error in a fluent-bit pod.
can you please help me related this issue?

just waiting ,,, the next day you will see data in your ES.
and remember, when you are creating index pattern, DO NOT choose any timestamp or you will not able to see all the data. That's because logs from each container have same index when using the ClusterOutputs.

@paynejacob Can you file a ticket in bonzai about the timestamp issue for ES?

Hello
The following flow doesn't work:
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: "all-logs-loki"
  namespace: "cattle-logging-system"
spec:
  globalOutputRefs:
    - loki
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
 name: loki
 namespace: "cattle-logging-system"
spec:
 loki:
   url: http://loki-loki-distributed-distributor.logging:3100
   configure_kubernetes_labels: true
   buffer:
     timekey: 1m
     timekey_wait: 30s
     timekey_use_utc: true
---

# Rancher v2.5.5
 REDACTED  MCW0CDP3YY  ~  Desktop  Git  home   master  1✎  5+  $   k version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:01:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Fluent Bit v1.6.4
--
Mon, Feb 22 2021 2:35:20 pm | * Copyright (C) 2019-2020 The Fluent Bit Authors
Mon, Feb 22 2021 2:35:20 pm | * Copyright (C) 2015-2018 Treasure Data
Mon, Feb 22 2021 2:35:20 pm | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
Mon, Feb 22 2021 2:35:20 pm | * https://fluentbit.io
Mon, Feb 22 2021 2:35:20 pm |
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [engine] started (pid=1)
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [storage] version=1.0.6, initializing...
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [storage] root path '/buffers'
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [storage] backlog input plugin: storage_backlog.1
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 4.8M
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
Mon, Feb 22 2021 2:35:20 pm | [2021/02/22 14:35:20] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [sp] stream processor started
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3105469 watch_fd=1 name=/var/log/containers/alertmanager-prometheus-kube-prometheus-alertmanager-0_monitoring_alertmanager-58d8e8dbe5d95e9e03a9a4d0f8337129bfa5a83af306cab318975554eea191fd.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842425 watch_fd=2 name=/var/log/containers/alertmanager-prometheus-kube-prometheus-alertmanager-0_monitoring_alertmanager-c96c9b5ee0e0b95a8a0ff3204aa2349ee40b7db0fc034d5c0be459e21a27146f.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3105993 watch_fd=3 name=/var/log/containers/alertmanager-prometheus-kube-prometheus-alertmanager-0_monitoring_config-reloader-068e5fa5225c50f74fdd556c6a4e24a8506778ab35b6804375c6ec5e2e9e0c86.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842479 watch_fd=4 name=/var/log/containers/alertmanager-prometheus-kube-prometheus-alertmanager-0_monitoring_config-reloader-a95bc039d25cd88860860dab8ec368d3156283f77535cd090baf696e42316bd2.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3363928 watch_fd=5 name=/var/log/containers/alertmanager-sns-forwarder-64f6c457cc-qwrwq_monitoring_alertmanager-sns-forwarder-460f5ef49979171324b1af6581f0128a98730685501d2297876f57762ab70d25.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2843221 watch_fd=6 name=/var/log/containers/alertmanager-sns-forwarder-64f6c457cc-qwrwq_monitoring_alertmanager-sns-forwarder-ef3432dbc9236c0c72badce8e30c9218c5fc36d98ba4474a1f3a367718386a84.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3367065 watch_fd=7 name=/var/log/containers/cattle-node-agent-5gnj7_cattle-system_agent-3eebe9e55b7c4a8e9516ba5dc3c6cdd715d40446bdf3e0bca01595f2fc7b9f43.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3105259 watch_fd=8 name=/var/log/containers/cert-manager-webhook-5f57f59fbc-cgmbp_cert-manager_cert-manager-46963e23b63059a54499fe103e998fe92e777d2db6bb5ae470555d7e6292ab1d.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2588438 watch_fd=9 name=/var/log/containers/cert-manager-webhook-5f57f59fbc-cgmbp_cert-manager_cert-manager-7caf66d739ab4423e574082178859794c9a51bb7f6aa76d06607a91004ac706f.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842323 watch_fd=10 name=/var/log/containers/coredns-6f85d5fb88-hfn8c_kube-system_coredns-510a85c9ebe607741cc4b1939bc01ba9daa202d8128db8a6b2ff2a48886b4430.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3105263 watch_fd=11 name=/var/log/containers/coredns-6f85d5fb88-hfn8c_kube-system_coredns-f26cc9891bf115c07de170ac6d0f536c43d0b204d7d56e36b6c84082fc2c22e5.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842048 watch_fd=12 name=/var/log/containers/coredns-autoscaler-79599b9dc6-wpgtp_kube-system_autoscaler-85b2b3821b79abbde84957c931ece855d60deab8022007f09c1321c55dbb5485.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2848284 watch_fd=13 name=/var/log/containers/coredns-autoscaler-79599b9dc6-wpgtp_kube-system_autoscaler-ab7e7152b2bfe193ddc1d0287f6980f2aef686b11022a48d9f70d5b8f7d8bba7.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2843062 watch_fd=14 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_cleanup-334919c4d5e081a1fba9d6914bb6b8f4d2c02ea5019fa73e28629b884e7e772b.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1821575 watch_fd=15 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_cleanup-ab8a4175e5f625597f5dd28186f5949128ae56bed61955727f8da1ba6905d156.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841834 watch_fd=16 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_csi-driver-4725284ba0e4c27c4509ab64010ad095a29cce9f11555fe0691536e02f450e7d.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3369918 watch_fd=17 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_csi-driver-5f5a98fa40a4c02a97266be7a42669f38cda76676f28dd45d4155b1321599dc0.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842217 watch_fd=18 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_driver-registrar-852d9abb45459ffb44bca73372765eeba149656cdd1da48aede28e6ae37e0926.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1821527 watch_fd=19 name=/var/log/containers/freenas-nfs-democratic-csi-node-qg2m9_democratic-csi_driver-registrar-d5c612bb9f262e0637ef720f1708fd98591d14f5d4ad5adc86d6d1950eca538c.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3360151 watch_fd=20 name=/var/log/containers/heimdall-84978bf7f7-cxhmr_default_heimdall-f3c566d3c50c7854f5e8ae760ccdb1e36a1c23245659e687e10f26e8f781b7f7.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3363649 watch_fd=21 name=/var/log/containers/loki-loki-distributed-distributor-7fb65f6cdf-kbxhw_logging_loki-48f0bed848be0d9a67e21098fec3517b2212240b8a0a72e721c6574c4ced89af.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3105721 watch_fd=22 name=/var/log/containers/loki-loki-distributed-gateway-56ffdd7d4c-sfdmr_logging_nginx-c218aa0119f4182cf2005810f1c19975b7d90a792eba5d81463151e2b251ebe6.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841623 watch_fd=23 name=/var/log/containers/loki-loki-distributed-gateway-56ffdd7d4c-sfdmr_logging_nginx-e9135ef534c57d35e844dbf3cfc7f80b55abd2dc67ec9d8de5bf3ab0093b5e78.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3363722 watch_fd=24 name=/var/log/containers/loki-loki-distributed-ingester-0_logging_loki-6965e825f50562321f9ebcc07d9d12f6ef784fd124cccd2efa6a8204a3399788.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3106219 watch_fd=25 name=/var/log/containers/metrics-server-8449844bf-tljsb_kube-system_metrics-server-21141d46d30398a98a31a9e695311873523cf897dd29706531d787fe8abef2fe.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842303 watch_fd=26 name=/var/log/containers/metrics-server-8449844bf-tljsb_kube-system_metrics-server-5e79951aa1af00494e7f7dda0ec63a8093ba2015f522742f8e500d4e96245985.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842628 watch_fd=27 name=/var/log/containers/minio-operator-6f5b8cdcff-gc5xh_minio_minio-operator-4366ea66ab2cbe7aa2214f175e39eb63d4b69bb6aba2f9482e35f7b5cef0e443.log
Mon, Feb 22 2021 2:35:25 pm | [2021/02/22 14:35:25] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3106121 watch_fd=28 name=/var/log/containers/minio-operator-6f5b8cdcff-gc5xh_minio_minio-operator-8e3154e6957ddf5477a9afab7463a82725bb0127f39a825e7917362104d406c4.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3369830 watch_fd=29 name=/var/log/containers/postgres-operator-844b7d4f84-kbmwz_default_postgres-operator-ac7fe8d0e93db238c15d43eccaf3feeb48c3d5336f9cebfbb62902a3f06ed648.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2346579 watch_fd=30 name=/var/log/containers/postgres-operator-844b7d4f84-kbmwz_default_postgres-operator-bd83a7810d1bf9298ece61dbf4037b22b592d2346783eb9611448c50c4e031b3.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3364747 watch_fd=31 name=/var/log/containers/prometheus-grafana-6bb7d6d96d-mttn6_monitoring_download-dashboards-56ee551b2df6d6c9424cf0dcae981f5ab96574418f5dc8d571ebe5bf8ae08005.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3366951 watch_fd=32 name=/var/log/containers/prometheus-grafana-6bb7d6d96d-mttn6_monitoring_grafana-79b6752ba66ee63a5e20f0076061c2f113a8381e8487ce903f5a9ef271884809.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3364792 watch_fd=33 name=/var/log/containers/prometheus-grafana-6bb7d6d96d-mttn6_monitoring_grafana-sc-dashboard-083dc6118f151a8ed3152c20f2faebb9237366ad78fa12b3dbb85b93bd488c75.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3364757 watch_fd=34 name=/var/log/containers/prometheus-grafana-6bb7d6d96d-mttn6_monitoring_grafana-sc-datasources-eea30bbd8419a590751c870e9e33ac8938b20bb360a086b7ed8e24f64091d0a0.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3364316 watch_fd=35 name=/var/log/containers/prometheus-grafana-6bb7d6d96d-mttn6_monitoring_init-chown-data-75b3fb3fd6c0a6def0a8d608751bf32a86336f266a24474889abcb2184e72f5c.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3363987 watch_fd=36 name=/var/log/containers/prometheus-prometheus-node-exporter-zsdt6_monitoring_node-exporter-b1c96fe0b72630ba5a8f9cffb5e0d7d63feb16f02db6c8bee72a30ab6fed6e12.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2588771 watch_fd=37 name=/var/log/containers/prometheus-prometheus-node-exporter-zsdt6_monitoring_node-exporter-f50e3d1121d790eb34ab2a6aaed7686f9edd83a4180578c978b3843e6dc05a40.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3367828 watch_fd=38 name=/var/log/containers/rancher-logging-rke-aggregator-hbcxs_cattle-logging-system_fluentbit-1908901497d2c12c530f68f0484ae5d69f53508d22ab2d35238f5365829d98b0.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1562785 watch_fd=39 name=/var/log/containers/speaker-l5ks4_metallb-system_speaker-fc97546dd44b3ffbdeb5abb270139f0a365ce64b3946e4cf45853e645d4f1363.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3354719 watch_fd=40 name=/var/log/containers/thanos-bucket-78994c7579-5gdlh_monitoring_thanos-bucket-5ab898ecb2787af909573f9f6b4a02e14c93fef67a69e8b68cbc9bb73100573c.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842039 watch_fd=41 name=/var/log/containers/thanos-bucket-78994c7579-5gdlh_monitoring_thanos-bucket-fa235616ce9dcaae36953a7b570df03a2fd6d9a86e11006e3fa70dc7329e4325.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3354650 watch_fd=42 name=/var/log/containers/thanos-compact-86ddc89ccf-pbvc6_monitoring_thanos-compact-8ab78031778c45e50bdfaa32b20c1324a0f587201b1cc46c7af18ca61ed20a73.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841361 watch_fd=43 name=/var/log/containers/thanos-compact-86ddc89ccf-pbvc6_monitoring_thanos-compact-957159d437c2e147b72e7e0d0d8e4d95deebcd712d99315c42488161265309cf.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841583 watch_fd=44 name=/var/log/containers/thanos-store-0-7d6bf588b5-rq6nf_monitoring_thanos-store-b8eb651c4cb3fa0037d44de430caa0bda5586a15768cd9604d3c9730ab73decb.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3363721 watch_fd=45 name=/var/log/containers/thanos-store-0-7d6bf588b5-rq6nf_monitoring_thanos-store-f184584fb314ecadaa1c811b30f2b4ac01874d574189d2f47bec2555dd1d07b3.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2842994 watch_fd=46 name=/var/log/containers/vernemq-0_default_vernemq-a4790b1c51ee2ab869f4d9c5f97270897753c4fb131dcf1b8fed62f4d4a288d0.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3368133 watch_fd=47 name=/var/log/containers/vernemq-0_default_vernemq-ff206b0a1fa49fddd9b2930394153f11ca2cbb2f119f7ce37c4d21a308120fc2.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2586890 watch_fd=48 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-accbed7a1c69a878e882d499fbef7a757bb32bcded4f448f651af8da1cfd47dd.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1562968 watch_fd=49 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-npc-080b61c6215883001179d95b976b1633a9aee104c60f7c5be0499197cc2bb9df.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841350 watch_fd=50 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-plugins-87db8d5c503ce1cc44702039482ea66c9edeaeb68cc9f3d93585d07889579d15.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1563025 watch_fd=51 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-plugins-a2e4b96f4364d6ccc521331ccb4702ae5804e402f0930e665789171d4680c985.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3378438 watch_fd=52 name=/var/log/containers/nginx-ingress-nginx-controller-5ccc45b566-4kvkb_default_controller-714223aaff6c9af9a98a9c38299ea6a6be4c9b488d60f44ce4befc83e692de42.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3368065 watch_fd=53 name=/var/log/containers/rancher-logging-rke-fluentbit-5htcd_cattle-logging-system_fluent-bit-1198a5331f77e3067443da99333a0b5530dd9e5b45a4906e75764083c36f17f0.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2066644 watch_fd=54 name=/var/log/containers/speaker-l5ks4_metallb-system_speaker-d31f456a369881453ac2368053cf2a2e64034dc365c1c9c71dd8fe91c49a2237.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2843264 watch_fd=55 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-f64bef2a1f03a2b3cac9f21c31fefac059af571990da4039f0b1e43902e7e8f3.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3368111 watch_fd=56 name=/var/log/containers/rancher-logging-fluentbit-ltdpn_cattle-logging-system_fluent-bit-ee0a887471a6159d664cfc939edf2da70a09fedf10bceb68c81a80fe611ee090.log
Mon, Feb 22 2021 2:35:26 pm | [2021/02/22 14:35:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2841258 watch_fd=57 name=/var/log/containers/weave-net-p2bjj_kube-system_weave-npc-b97d95924c90523cf5aad962d192ee198437d5be704f2910c02d668e44870bb5.log
Mon, Feb 22 2021 2:37:56 pm | [2021/02/22 14:37:56] [error] [io] TCP connection failed: rancher-logging-fluentd.cattle-logging-system.svc:24240 (Connection refused)
Mon, Feb 22 2021 2:37:56 pm | [2021/02/22 14:37:56] [error] [output:forward:forward.0] no upstream connections available
Mon, Feb 22 2021 2:37:56 pm | [2021/02/22 14:37:56] [ warn] [engine] failed to flush chunk '1-1614004675.726803953.flb', retry in 9 seconds: task_id=0, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [error] [io] TCP connection failed: rancher-logging-fluentd.cattle-logging-system.svc:24240 (Connection refused)
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [error] [output:forward:forward.0] no upstream connections available
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [ warn] [engine] failed to flush chunk '1-1614004676.247612751.flb', retry in 11 seconds: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [error] [io] TCP connection failed: rancher-logging-fluentd.cattle-logging-system.svc:24240 (Connection refused)
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [error] [output:forward:forward.0] no upstream connections available
Mon, Feb 22 2021 2:37:57 pm | [2021/02/22 14:37:57] [ warn] [engine] failed to flush chunk '1-1614004676.905288708.flb', retry in 11 seconds: task_id=2, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:38:05 pm | [2021/02/22 14:38:05] [ info] [engine] flush chunk '1-1614004675.726803953.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:38:08 pm | [2021/02/22 14:38:08] [ info] [engine] flush chunk '1-1614004676.247612751.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:38:08 pm | [2021/02/22 14:38:08] [ info] [engine] flush chunk '1-1614004676.905288708.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:41:44 pm | [2021/02/22 14:41:44] [error] [io] TCP connection failed: rancher-logging-fluentd.cattle-logging-system.svc:24240 (Connection refused)
Mon, Feb 22 2021 2:41:44 pm | [2021/02/22 14:41:44] [error] [output:forward:forward.0] no upstream connections available
Mon, Feb 22 2021 2:41:44 pm | [2021/02/22 14:41:44] [ warn] [engine] failed to flush chunk '1-1614004904.190854514.flb', retry in 7 seconds: task_id=0, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:41:45 pm | [2021/02/22 14:41:45] [error] [io] TCP connection failed: rancher-logging-fluentd.cattle-logging-system.svc:24240 (Connection refused)
Mon, Feb 22 2021 2:41:45 pm | [2021/02/22 14:41:45] [error] [output:forward:forward.0] no upstream connections available
Mon, Feb 22 2021 2:41:45 pm | [2021/02/22 14:41:45] [ warn] [engine] failed to flush chunk '1-1614004904.245904447.flb', retry in 11 seconds: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:41:51 pm | [2021/02/22 14:41:51] [ info] [engine] flush chunk '1-1614004904.190854514.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0
Mon, Feb 22 2021 2:41:56 pm | [2021/02/22 14:41:56] [ info] [engine] flush chunk '1-1614004904.245904447.flb' succeeded at retry 1: task_id=1, input=tail.0 > output=forward.0

I had the same issue with elastic. Please execute the following to shed some light into your problems:
kubectl exec -n cattle-logging-system rancher-logging-fluentd-0 -- cat /fluentd/log/out
This shows your if everything is ok with your ES or whatever output target  connection:
[warn]: #0 [clusterflow:cattle-logging-system:xxx:clusteroutput:cattle-logging-system:xx] Could not communicate to Elasticsearch, resetting connection and trying again. connect_read timeout reached
Hope this helps.

Interesting. Those logs should be written to stdout like everything else. Also fluentd doesn't handle OOM nicely.

Can anybody consistently reproduce this with elastic where it does not work after 1 hour?  The messages about fluentd not being available are typical on a fresh installation because fluentbit starts faster.  Clusters with little to no log activity can also see delays in delivery because they do not hit the threshold to flush a buffer and logs are only flushed after a timeout.

@paynejacob - I can consistently reproduce this issue across multiple clusters and over several days.  I see a very small subset of the expected logs with no apparent pattern on what does and does not get forwarded.  We abandoned the baked-in operator and switched to using filebeat for the majority of our production workloads, which has no issues forwarding logs to the same elastic cluster.  We're actively working the issue with Rancher support and have provided extensive debug information.

Assigning this to QA to reproduce in currently supported Rancher versions.

Closing as stale. If it's seen in a supported Rancher version, this can be reopened.