Ruff server not closing with neovim instances - vol 2

I am still seeing this problem originally reported (and fixed) in #11207
I am on ruff 0.4.5 and neovim 0.9.5 on Linux Fedora 39.
Here's the output from Lsp logs
[START][2024-05-26 16:20:07] LSP logging initiated
  16021 [ERROR][2024-05-26 16:20:07] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "wa        rning: The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the follow        ing options in `.venv/lib64/python3.12/site-packages/pandas/pyproject.toml`:\n  - 'ignore' -> 'lint.ignore'\n  - 'select' -> 'lint.        select'\n  - 'typing-modules' -> 'lint.typing-modules'\n  - 'unfixable' -> 'lint.unfixable'\n  - 'per-file-ignores' -> 'lint.per-fi        le-ignores'\n"
  16022 [ERROR][2024-05-26 16:20:07] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "wa        rning: `PGH001` has been remapped to `S307`.\n"
  16023 [ERROR][2024-05-26 16:20:07] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "           0.700874s WARN ruff_server::server LSP client does not support dynamic capability registration - automatic configuration reloading         will not be available.\n"
  16024 [ERROR][2024-05-26 16:20:16] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "           9.789324s INFO ruff_server::server::connection Shutdown request received. Waiting for an exit notification...\n"

Originally posted by @danielhollas in https://github.com/astral-sh/ruff/issues/11207#issuecomment-2128365016
cc @dhruvmanila

Hey, thanks for providing the logs. I've a couple of follow-up questions:

Are there no other logs apart from this?
Do you get any diagnostics in the editor regardless of what the logs are saying?

I think it'll also be useful to provide INFO level logs. You'll need to set the log level for the Neovim LSP client to INFO with the following snippet where you setup your LSP configs.
vim.lsp.set_log_level(vim.lsp.log_levels.INFO)
require('vim.lsp.log').set_format_func(vim.inspect)

Are there no other logs apart from this?

Nope.

Do you get any diagnostics in the editor regardless of what the logs are saying?

The ruff server functionality (diagnostics / code actions) work as expected AFAIK.
Here are the logs including INFO level.
[START][2024-05-27 13:30:49] LSP logging initiated
  [INFO][2024-05-27 13:30:49] .../vim/lsp/rpc.lua:662»    "Starting RPC client"»  {
    args = { "server", "--preview" },
    cmd = "/home/hollas/.local/share/nvim/mason/bin/ruff",
    extra = {
      cwd = "/home/hollas/atmospec/aiida-core"
    }
  }
  [INFO][2024-05-27 13:30:49] .../lua/vim/lsp.lua:1344»   "LSP[ruff]"»    "server_capabilities"»  {
    server_capabilities = {
      codeActionProvider = {
        codeActionKinds = { "quickfix", "source.fixAll.ruff", "source.organizeImports.ruff", "notebook.source.fixAll.ruff", "notebook.sourc        e.organizeImports.ruff" },
        resolveProvider = true,
        workDoneProgress = true
      },
      diagnosticProvider = {
        identifier = "Ruff",
        interFileDependencies = false,
        workDoneProgress = true,
        workspaceDiagnostics = false
      },
      documentFormattingProvider = true,
      documentRangeFormattingProvider = true,
      hoverProvider = true,
      notebookDocumentSync = {
        notebookSelector = { {
            cells = { {
                language = "python"
              } }
          } },
        save = false
      },
      positionEncoding = "utf-16",
      textDocumentSync = {
        change = 2,
        openClose = true,
        willSave = false,
        willSaveWaitUntil = false
      },
      workspace = {
        workspaceFolders = {
          changeNotifications = true,
          supported = true
        }
      }
    }
  }
  [ERROR][2024-05-27 13:30:49] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "warning:   The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the following options   in `.venv/lib64/python3.12/site-packages/pandas/pyproject.toml`:\n  - 'ignore' -> 'lint.ignore'\n  - 'select' -> 'lint.select'\n  - 'typi  ng-modules' -> 'lint.typing-modules'\n  - 'unfixable' -> 'lint.unfixable'\n  - 'per-file-ignores' -> 'lint.per-file-ignores'\n"
  [ERROR][2024-05-27 13:30:49] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "warning:   `PGH001` has been remapped to `S307`.\n"
  [ERROR][2024-05-27 13:30:50] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "   0.270  272s WARN ruff_server::server LSP client does not support dynamic capability registration - automatic configuration reloading will not be   available.\n"
  [INFO][2024-05-27 13:31:24] .../lua/vim/lsp.lua:1875»   "exit_handler"» { {
      _on_attach = <function 1>,
      attached_buffers = { true },
      cancel_request = <function 2>,
      commands = {},
      config = {
        autostart = true,
        capabilities = {
          textDocument = {
            callHierarchy = {
              dynamicRegistration = false
            },
            codeAction = {
              codeActionLiteralSupport = {
                codeActionKind = {
                  valueSet = { "", "quickfix", "refactor", "refactor.extract", "refactor.inline", "refactor.rewrite", "source", "source.org                  anizeImports" }
                }
              },
              dataSupport = true,
              dynamicRegistration = false,
              isPreferredSupport = true,
              resolveSupport = {
                properties = { "edit" }
              }
            },
             completion = {
              completionItem = {
                commitCharactersSupport = true,
                deprecatedSupport = true,
                documentationFormat = { "markdown", "plaintext" },
                insertReplaceSupport = true,
                insertTextModeSupport = {
                  valueSet = { 1, 2 }
                },
                labelDetailsSupport = true,
                preselectSupport = true,
                resolveSupport = {
                  properties = { "documentation", "detail", "additionalTextEdits", "sortText", "filterText", "insertText", "textEdit", "ins                  ertTextFormat", "insertTextMode" }
                },
                snippetSupport = true,
                tagSupport = {
                  valueSet = { 1 }
                }
              },
              completionItemKind = {
                valueSet = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 }
              },
              completionList = {
                itemDefaults = { "commitCharacters", "editRange", "insertTextFormat", "insertTextMode", "data" }
              },
              contextSupport = true,
              dynamicRegistration = false,
              insertTextMode = 1
            },
            declaration = {
              linkSupport = true
            },
            definition = {
              linkSupport = true
            },
            documentHighlight = {
              dynamicRegistration = false
            },
            documentSymbol = {
              dynamicRegistration = false,
              hierarchicalDocumentSymbolSupport = true,
              symbolKind = {
                valueSet = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26 }
              }
            },
            hover = {
              contentFormat = { "markdown", "plaintext" },
              dynamicRegistration = false
            },
            implementation = {
              linkSupport = true
            },
            publishDiagnostics = {
              relatedInformation = true,
              tagSupport = {
                valueSet = { 1, 2 }
              }
            },
            references = {
              dynamicRegistration = false
            },
            rename = {
              dynamicRegistration = false,
              prepareSupport = true
            },
            semanticTokens = {
              augmentsSyntaxTokens = true,
              dynamicRegistration = false,
              formats = { "relative" },
              multilineTokenSupport = false,
              overlappingTokenSupport = true,
              requests = {
                full = {
                  delta = true
                },
              serverCancelSupport = false,
              tokenModifiers = { "declaration", "definition", "readonly", "static", "deprecated", "abstract", "async", "modification", "doc              umentation", "defaultLibrary" },
              tokenTypes = { "namespace", "type", "class", "enum", "interface", "struct", "typeParameter", "parameter", "variable", "proper              ty", "enumMember", "event", "function", "method", "macro", "keyword", "modifier", "comment", "string", "number", "regexp", "o              perator", "decorator" }
            },
            signatureHelp = {
              dynamicRegistration = false,
              signatureInformation = {
                activeParameterSupport = true,
                documentationFormat = { "markdown", "plaintext" },
                parameterInformation = {
                  labelOffsetSupport = true
                }
              }
            },
            synchronization = {
              didSave = true,
              dynamicRegistration = false,
              willSave = true,
              willSaveWaitUntil = true
            },
            typeDefinition = {
              linkSupport = true
            }
          },
          window = {
            showDocument = {
              support = true
            },
            showMessage = {
              messageActionItem = {
                additionalPropertiesSupport = false
              }
            },
            workDoneProgress = true
          },
          workspace = {
            applyEdit = true,
            configuration = true,
            didChangeWatchedFiles = {
              dynamicRegistration = false,
              relativePatternSupport = true
            },
            semanticTokens = {
              refreshSupport = true
            },
            symbol = {
              dynamicRegistration = false,
              hierarchicalWorkspaceSymbolSupport = true,
              symbolKind = {
                valueSet = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26 }
              }
            },
            workspaceEdit = {
              resourceOperations = { "rename", "create", "delete" }
            },
            workspaceFolders = true
          }
        },
        cmd = { "/home/hollas/.local/share/nvim/mason/bin/ruff", "server", "--preview" },
        cmd_cwd = "/home/hollas/atmospec/aiida-core",
        filetypes = { "python" },
        flags = {},
        get_language_id = <function 3>,
        handlers = <1>{},
        init_options = vim.empty_dict(),
        log_level = 2,
        message_level = 2,
        name = "ruff",
        on_attach = <function 4>,
        on_exit = <function 5>,
        on_init = <function 6>,
        root_dir = "/home/hollas/atmospec/aiida-core",
        settings = {},
        single_file_support = true,
        workspace_folders = <2>{ {
            name = "/home/hollas/atmospec/aiida-core",
            uri = "file:///home/hollas/atmospec/aiida-core"
          } },
        <metatable> = <3>{
          __tostring = <function 7>
        }
      },
      handlers = <table 1>,
      id = 1,
      initialized = true,
      is_stopped = <function 8>,
      messages = {
        messages = {},
        name = "ruff",
        progress = {},
        status = {}
      },
      name = "ruff",
      notify = <function 9>,
      offset_encoding = "utf-16",
      request = <function 10>,
      request_sync = <function 11>,
      requests = {},
      rpc = {
        is_closing = <function 12>,
        notify = <function 13>,
        request = <function 14>,
        terminate = <function 15>
      },
      server_capabilities = {
        codeActionProvider = {
          codeActionKinds = { "quickfix", "source.fixAll.ruff", "source.organizeImports.ruff", "notebook.source.fixAll.ruff", "notebook.sou          rce.organizeImports.ruff" },
          resolveProvider = true,
          workDoneProgress = true
        },
        diagnosticProvider = {
          identifier = "Ruff",
          interFileDependencies = false,
          workDoneProgress = true,
          workspaceDiagnostics = false
        },
        documentFormattingProvider = true,
        documentRangeFormattingProvider = true,
        hoverProvider = true,
        notebookDocumentSync = {
          notebookSelector = { {
              cells = { {
                  language = "python"
                } }
            } },
          save = false
        },
        positionEncoding = "utf-16",
        textDocumentSync = {
          change = 2,
          openClose = true,
          willSave = false,
          willSaveWaitUntil = false
        },
        workspace = {
          workspaceFolders = {
            changeNotifications = true,
            supported = true
          }
        }
      },
      stop = <function 16>,
      supports_method = <function 17>,
      workspace_did_change_configuration = <function 18>,
      workspace_folders = <table 2>
    } }
  [ERROR][2024-05-27 13:31:24] .../vim/lsp/rpc.lua:734»   "rpc"»  "/home/hollas/.local/share/nvim/mason/bin/ruff"»"stderr"»       "  35.157  125s INFO ruff_server::server::connection Shutdown request received. Waiting for an exit notification...\n"

I'll need some time to get Fedora running in a VM setup but before that can you try running it with the following minimal config: https://gist.github.com/dhruvmanila/1c7afb9825c56c28c8d98a1afd96d05b
In the above gist, there are two files:

minimal.lua - Neovim setup containing only LSP config for Ruff
test.py - Test file which will highlight two diagnostic if Ruff is running correctly

Run the above setup with:
/path/to/nvim --clean -u /path/to/minimal.lua /path/to/test.py

Or, using the debug log level:
NVIM_LSP_LOG_LEVEL=debug /path/to/nvim --clean -u /path/to/minimal.lua /path/to/test.py

[!IMPORTANT]
Replace the "/path/to" to the actual path.

Can you provide the debug logs using this setup?

Will do, thanks! I'll also try to get Neovim 0.10 going to see if it makes any difference. (Fedora didn't yet updated the neovim package, but I'll try to install it from Flathub)

If it's available, I could also look at your Neovim config to get the exact setup which might help as well. I see that you're using mason.nvim but apart from that I don't know what your setup looks like.

Here's my config (using kickstart.nvim)
https://github.com/danielhollas/kickstart.nvim/tree/dh

Hey @danielhollas, so I've got a VM running with Fedora 38 and I'm using the same setup that you've provided with Neovim 0.9.5 and Ruff 0.4.5 and unfortunately it's working fine on my machine. I'm sorry to ask yet another request but can you provide DEBUG logs?
vim.lsp.set_log_level(vim.lsp.log_levels.DEBUG)

Thanks so much for taking time to investigate this. That's frustrating that you cannot reproduce. How exactly did you install Neovim?
Here's the DEBUG log.
nvim.log

How exactly did you install Neovim?

I just installed it via dnf and it gave me 0.9.5
sudo dnf install neovim

Thanks for the debug logs, I'm looking at it now.

I don't really have anything else. I'm sorry that you're facing that. Is this making Ruff unusable i.e., is every instance of Ruff doesn't stop at all?

Yeah, it's every time I open and close with, 100% reproducible AFAICT. I see them with pgrep ruff and have to kill them all from time to time with killall ruff. I'll investigate some more and report back.

Ok, I can reproduce it now. I'm not sure why it was not showing up earlier.

Although, I'm not reliably able to reproduce this. Most of the times the process remains alive but sometimes the server does shutdown.

I noticed that I cannot reproduce with Neovim 0.10 installed via Flatpack.

Interesting, let me try that.

@danielhollas How are you running it via flatpak? I'm still seeing that the server doesn't quit.
I followed the instruction as mentioned at https://flathub.org/apps/io.neovim.nvim:
flatpak install neovim
flatpak run io.neovim.nvim

Yes, that's how I run it. (nvim config is located in Flatpack specific dir elsewhere as you probably know)
Heh, seems like we have the opposite problem wrt reproduction 😅

I should note just in case it is relevant that I've upgraded to ruff 0.4.7 in the meantime.

Maybe https://github.com/davidbarsky/tracing-tree/pull/82 ?

https://github.com/astral-sh/ruff/pull/11747 will remove tracing-tree, just FYI.

This should be resolved by https://github.com/astral-sh/ruff/commit/ded010cf9c9eb91f6c981cbafeb051fdd6ce6774 which was released v0.4.8. Feel free to comment or open a new issue if anyone faces this issue again.


adding wtforms

All forms should be using WTForms.
Shouldn't be too hard to add this in now, as only login form needs to be converted to use WTForms.
I have experience with WTForms 1.0 but not anything beyond that, though I can't imagine it has changed all that much so it's probably going to be fine.

Before you start creating forms.py I think we need to think ahead where we might have many forms, so having a single forms.py is not going to scale very well.
If you created a module called forms with an init.py and a forms/auth.py then you can put all auth forms (login form) in there.

I have updated the forms to use wtforms. However I am not sure if the view is correct. The form itself is minimal at the moment with little markup around it. Please review.

The view code looks fine to me, will merge soon.

The presentation (template) doesn't need to be that good at this point anyway as we have no design or theme yet, so anything will do for now.


fix: remove image-cache-it jcenter dependency

2.0.3 is hosted only in jcenter

fixed here https://github.com/triniwiz/nativescript-plugins/commit/d1dfb6334cdcbf4f42c6368d401aa7842034a522


Need to display error when Docker is not detected

As long as we're leaning on Docker, we need to communicate to the user if it's not available.

see also #93


[10.x] Ensure array cache considers milliseconds

@timacdonald Hello. I brought the ArrayCache fix. True, I did not return the condition greater than equal, because RateLimiter breaks. Because of this, I have .001ms in my test, not .000ms.
Text from PR #48573:
With the recent fix to the array cache driver, https://github.com/laravel/framework/pull/48497, it has surfaced another problem with it that was previously hidden by the initial issue.
The array cache driver, unlike the redis cache driver, does not take milliseconds into account.
This PR fixes that.
This is technically a breaking change - but one I think we should make as it is more of a bugfix. I think the use of the array driver in prod is somewhat of an edgecase.
fixes https://github.com/laravel/framework/issues/48569

@timacdonald I checked the code not only on laravel tests, but also on the tests of my project - they pass. Your pipe has failed due to docker.

Retriggered the build

We'll explore any changes to the array cache internally. cc @timacdonald


[LOG4J2-928] Mostly wait-free MemoryMappedFileManager

A pull request for https://issues.apache.org/jira/browse/LOG4J2-928.
This PR is not complete - it lacks tests and benchmarks (I'm going to add them in the next few weeks), but the core implementation is reviewable already.

Roman, I started to review your patch. Initial feedback below (still reading):

MemoryMappedFileManager::switchRegion should not throw Error: generally a logging problem should not bring down the application. See also the {{ignoreExceptions}} [configuration|https://logging.apache.org/log4j/2.x/manual/appenders.html#MemoryMappedFileAppender] attribute that allows applications to choose to receive logging exceptions.

What do you think of the idea of moving some of the newly added classes to the core.async package? ThreadHints, ThreadHintsMH, UninterruptibleCountDownLatch and UnlockedAwaitableReentrantLock. Perhaps also some of MemoryMappedfileManager's inner classes (need to look more at these)?

in MemoryMappedfileManager::getFileManager, please call AbstractManager::narrow on the result of super.getManager(). This will show users a better error message when log4j2 is misconfigured (LOG4J2-1908). This is a recent change I made, may have introduced a (small) merge conflict, sorry about that.

Question: The javadoc for MemoryMappedFileManager::SPIN_WAIT_MINIMIZING_LOCKED_WRITE_THRESHOLD mentions a 32 KB threshold for stack traces. What happens if a user logs a stack trace exceeding this length?

... still reading, also looking forward to tests & perf results.
On the topic of perf testing:  I'd be interested to see a throughput comparison of the new MemoryMappedFileAppender vs. RandomAccessFileAppender and FileAppender with a varying number of logging threads (1, 2, 4, 8, 16, 32, 64 if possible).
A throughput comparison with Async Loggers would also be interesting but be aware that JMH quickly fills up the disruptor ringbuffer. To solve this, configure with a non-existing appender (so the background thread essentially logs to /dev/null). See /log4j-perf/src/main/java/org/apache/logging/log4j/perf/jmh/AsyncLoggersBenchmark.java
Some indication of latency behaviour would also be interesting.

Roman, one more thing: since this is a non-trivial patch, would you mind submitting an ICLA?

@remkop

MemoryMappedFileManager::switchRegion and ::closeAndSwitchRegionWhileLocked should not throw Error: generally a logging problem should not bring down the application. See also the {{ignoreExceptions}} [configuration|https://logging.apache.org/log4j/2.x/manual/appenders.html#MemoryMappedFileAppender] attribute that allows applications to choose to receive logging exceptions.

I follow the approach of java.util.concurrent internals here, where throw new Error(...) is used in the face of unrecoverable and totally unexpected errors. the alternative is AssertionError, or assert statements.

@remkop note that this PR is primarily about latency, not throughput at all. I will do some testing, but if the throughput will degrade (that I expect in a single-thread case), and latency with improve, I consider this a success.

Latency comparison
java -jar log4j-perf/target/benchmarks.jar ".*Log4j2AppenderLatencyBenchmark.end2end" -f 1 -wi 1 -i 30 -t 2 -p loggerType=MMap

On my MBP with 4 cores and SSD.
https://cdn.rawgit.com/leventov/288f6ff4ea960998e6479001976b0755/raw/aeb2f65560241deaa9394ff1ef8044bb961ea37e/plotFiles.html (red line is before this PR, blue line is this PR)
Tail latency is much better than it used to be, but worse than I expected. Calling MappedByteBuffer.load() significantly improves latency around 99.99-99.999 percentiles, but dramatically drops the extreme tail latency, from 4.5 milliseconds to ~300 milliseconds. I don't know the reason for that.
Also quite "fun" than hand-rolled version of MappedByteBuffer.load(): https://github.com/leventov/logging-log4j2/tree/wait-free-MemoryMappedFileManager-crash crashes the JVM when I try to run the same command
java -jar log4j-perf/target/benchmarks.jar ".*Log4j2AppenderLatencyBenchmark.end2end" -f 1 -wi 1 -i 30 -t 2 -p loggerType=MMap

Could somebody else please try it on other machines?

Throughput comparison
First number is the number of threads, "150" is the message size.
Before this PR
1 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  988106.677 ± 44561.265  ops/s
2 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  1378707.634 ± 45234.050  ops/s
3 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  1507720.952 ± 42527.798  ops/s
4 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  1615561.064 ± 10197.025  ops/s

1 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  1406710.542 ± 53654.749  ops/s
2 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  2032867.062 ± 33273.059  ops/s
3 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  2182597.197 ± 19446.011  ops/s
4 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  2128503.272 ± 40132.820  ops/s

With this PR
1 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  848596.439 ± 29081.144  ops/s
2 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  1302047.611 ± 21627.635  ops/s
3 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       10  1702653.587 ± 36385.194  ops/s
4 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.end2endMMap                   150  thrpt       15  1880038.309 ± 35208.682  ops/s

1 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  1283001.528 ± 40095.635  ops/s
2 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  1803333.701 ± 27412.900  ops/s
3 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  2506996.207 ± 40209.437  ops/s
4 - o.a.l.l.p.j.Log4j2AppenderComparisonBenchmark.appenderMMap                   150  thrpt       10  2723771.920 ± 103479.124  ops/s

On the same MacBook with 4 cores.
You can see that with 1 or 2 threads older version is faster, but it's the limit of it's scalability, and with 3 or 4 threads the wait-free version takes over.

I'll try to run those benchmarks on an AWS instance next week, though that's not an ideal place to benchmark due to noisy neighbours and that sort of thing.

I tried to run:
java -jar log4j-perf/target/benchmarks.jar ".*Log4j2AppenderLatencyBenchmark.end2end" -f 1 -wi 1 -i 30 -t 2 -p loggerType=MMap

but got this result:
# VM invoker: /opt/jvm/jdk1.8.0_131/jre/bin/java
# VM options: <none>
# Warmup: 1 iterations, 1 s each
# Measurement: 30 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 2 threads, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.logging.log4j.perf.jmh.Log4j2AppenderLatencyBenchmark.end2end
# Parameters: (loggerType = MMap)

# Run progress: 0.00% complete, ETA 00:00:31
# Fork: 1 of 1
# Warmup Iteration   1: 818183.651 ops/s
Iteration   1: 677355.347 ops/s
Iteration   2: <failure>

java.lang.ArrayIndexOutOfBoundsException: value outside of histogram covered range. Caused by: java.lang.IndexOutOfBoundsException: index 21813
        at org.HdrHistogram.AbstractHistogram.handleRecordException(AbstractHistogram.java:492)
        at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:484)
        at org.HdrHistogram.AbstractHistogram.recordValue(AbstractHistogram.java:397)
        at org.apache.logging.log4j.perf.jmh.Log4j2AppenderLatencyBenchmark.end2end(Log4j2AppenderLatencyBenchmark.java:106)
        at org.apache.logging.log4j.perf.jmh.generated.Log4j2AppenderLatencyBenchmark_end2end.end2end_thrpt_jmhStub(Log4j2AppenderLatencyBenchmark_end2end.java:125)
        at org.apache.logging.log4j.perf.jmh.generated.Log4j2AppenderLatencyBenchmark_end2end.end2end_Throughput(Log4j2AppenderLatencyBenchmark_end2end.java:70)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:198)
        at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:180)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

# Run complete. Total time: 00:00:04

Benchmark    Mode  Samples  Score   Error  Units

@mikaelstaldal it is configured for at most 1 second latency. Maybe on your machine some pause takes more.
BTW jvm crash is not a mystery anymore, it was because of reads after buffer unmapping.

Then I suggest that you raise the maximum latency, since it can obviously happen.
And maybe this long latency on my machine is a flaw in the implementation?

BTW, the test fails on Java 9:
2017-07-02 14:19:04,650 org.apache.logging.log4j.perf.jmh.Log4j2AppenderLatencyBenchmark.end2end-jmh-worker-1 ERROR Memo
ryMappedFileManager target/MemoryMappedFileAppenderTest.log Unable to remap: java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio" to unnamed module @1417a6e9 java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio" to unnamed module @1417a6e9

Roman, thanks for the feedback and the benchmarks.

Understood about keeping the new package-private classes in the same package as MemoryMappedFilemanager. No problem.

Thanks for updating to use the new narrow method and for clarifying how extremely long stack traces are handled.

I still think Log4j components should never throw an Error. I understand the rationale behind throwing Errors in java.util.concurrent: if these low level components detect an anomaly it means that the JVM cannot guarantee sane behaviour for the application. Our case is different because Log4j is an auxiliary library: logging may fail, but the application should be able to continue.
It is up to the application to decide whether they want to receive logging exceptions or not, and users can configure their choice with the ignoreExceptions attribute on MemoryMappedFileAppender. New implementations of MemoryMappedFileAppender should respect this contract.

About testing latency:
Please take a look at  /log4j-core/src/test/java/org/apache/logging/log4j/core/async/perftest/ResponseTimeTest.java
ResponseTimeTest incorporates various lessons I learned from Gil Tene's presentations on measuring latency as well as discussions on the Mechanical Sympathy mailing list. (The inner Pacer class is Gil's.) It distinguishes between response time and service time (see the sidebar on the Log4j perf page), and it allows us to control the load. This is important because latency behaviour is often different under various workloads.
I am not a fan of using JMH for measuring latency because it only measures service time, not response time, and there is no way to control the workload, it just runs at max speed. (Gil explicitly mentioned we probably don't want to measure latency under maximum load, when the system is under pressure and latency becomes unpredictable).
Also, it looks like Log4j2AppenderLatencyBenchmark also measures how long it takes to generate a random number, which is probably not the intention.
The throughput numbers look very promising, I will try to run these tests on enterprise hardware this week when I have a chance, and report back the results.
All in all, this is great stuff! Keep 'em coming!

P.S. Roman, are you okay with submitting an ICLA? I am still reviewing the patch but we need the ICLA before it can be merged. If you put "Apache Logging" in the form field "notify project", the secretary will let us know when your ICLA is on file.

As promised, here are the throughput results on enterprise hardware.
Hardware: total 64 cores (4x 16-core) Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
Java Hotspot 64-bit server 1.8.0_112
Results look good: as soon as we introduce concurrency, the lock-free MMap appender starts outperforming the other file appenders.
1 thread
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples          Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10    1344494.961 ± 221110.545  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10    1563396.748 ±  12497.136  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10    1223215.440 ±  14661.487  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    1008537.505 ±   9213.181  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10     998564.075 ±  12207.661  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10     785114.881 ±   8827.768  ops/s
{code}

4 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10 -t 4

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score          Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1678485.793 ±   260632.925  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1412574.559 ±   447410.040  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1386414.915 ±   333117.992  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    959662.994 ±   333541.548  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    929405.390 ±    66673.436  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1013681.438 ±   155916.949  ops/s
{code}

8 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10 -t 8

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score         Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1164538.492 ±   62482.965  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1194865.200 ±   72251.118  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1338960.644 ±   71104.991  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    944670.900 ±   69190.255  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    930248.735 ±   28523.086  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10    998676.286 ±   36327.805  ops/s
{code}

16 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10 -t 16

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1214786.965 ±   7217.940  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1192036.036 ±  13696.830  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1548961.297 ±  27141.206  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10   1003235.059 ±  10365.921  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    994834.287 ±   9057.470  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1323043.186 ±  19960.127  ops/s
{code}

32 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10 -t 32

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1372729.015 ±  40762.524  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1185351.707 ±  16992.542  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1784377.615 ± 179359.793  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    918029.971 ±  38434.750  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10   1122310.566 ± 152840.084  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1898762.678 ±  81312.274  ops/s
{code}

64 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".*Log4j2AppenderThroughput.*" -wi 5 -f 1 -i 10 -t 64

Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score         Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1193371.846 ±   32893.485  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1231136.529 ±   26895.411  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1409277.639 ±   91900.204  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    881261.726 ±   35023.552  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    929691.738 ±   22616.873  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1216108.561 ±   41321.530  ops/s
{code}

Thank you for running the benchmarks. Is this worth doing over on the
current Java 8 runtime (_131)?
Gary
On Wed, Jul 5, 2017 at 6:59 AM, Remko Popma<EMAIL_ADDRESS>wrote:

As promised, here are the throughput results on enterprise hardware.
Hardware: total 64 cores (4x 16-core) Intel(R) Xeon(R) CPU E5-2697A v4 @
2.60GHz
Java Hotspot 64-bit server 1.8.0_112
Results look good: as soon as we introduce concurrency, the lock-free MMap
appender starts outperforming the other file appenders.
1 thread
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples          Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10    1344494.961 ± 221110.545  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10    1563396.748 ±  12497.136  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10    1223215.440 ±  14661.487  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    1008537.505 ±   9213.181  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10     998564.075 ±  12207.661  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10     785114.881 ±   8827.768  ops/s
{code}
4 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10 -t 4
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score          Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1678485.793 ±   260632.925  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1412574.559 ±   447410.040  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1386414.915 ±   333117.992  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    959662.994 ±   333541.548  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    929405.390 ±    66673.436  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1013681.438 ±   155916.949  ops/s
{code}
8 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10 -t 8
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score         Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1164538.492 ±   62482.965  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1194865.200 ±   72251.118  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1338960.644 ±   71104.991  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    944670.900 ±   69190.255  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    930248.735 ±   28523.086  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10    998676.286 ±   36327.805  ops/s
{code}
16 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10 -t 16
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1214786.965 ±   7217.940  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1192036.036 ±  13696.830  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1548961.297 ±  27141.206  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10   1003235.059 ±  10365.921  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    994834.287 ±   9057.470  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1323043.186 ±  19960.127  ops/s
{code}
32 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10 -t 32
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score        Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1372729.015 ±  40762.524  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1185351.707 ±  16992.542  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1784377.615 ± 179359.793  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    918029.971 ±  38434.750  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10   1122310.566 ± 152840.084  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1898762.678 ±  81312.274  ops/s
{code}
64 threads
{code}
-sh-4.2$ java -jar benchmarks.jar ".Log4j2AppenderThroughput." -wi 5 -f 1 -i 10 -t 64
Benchmark                                                  (loggerType)  (messageSizeBytes)   Mode  Samples         Score         Error  Units
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             File                 150  thrpt       10   1193371.846 ±   32893.485  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender              RAF                 150  thrpt       10   1231136.529 ±   26895.411  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.appender             MMap                 150  thrpt       10   1409277.639 ±   91900.204  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              File                 150  thrpt       10    881261.726 ±   35023.552  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end               RAF                 150  thrpt       10    929691.738 ±   22616.873  ops/s
o.a.l.l.p.j.Log4j2AppenderThroughputBenchmark.end2end              MMap                 150  thrpt       10   1216108.561 ±   41321.530  ops/s
{code}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/logging-log4j2/pull/87#issuecomment-313111330,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABIfN9oh2p20v8fnX4yBVzi-pzelEgr1ks5sK5bPgaJpZM4OEhNj
.

I don't have access to hardware with that version of Java. I am fairly sure that 131 doesn't have performance improvements (we checked whether we should upgrade but it was too close to our go-live and 131 didn't have anything particular we needed).

@leventov The log4j community is discussing releasing a 2.9 version soon (in July). I would like to merge your contribution but have you had a chance to look at the ICLA?

@remkop I have not competed this PR yet, there are a few more things need to be done. I will try to complete PR in the next weekend. Also I've just sent ICLA to<EMAIL_ADDRESS>

Thanks, we got notification of the ICLA.

New latency benchmarking results "done right" (with Pacer), On 4-core MBP with SSD, benchmarking with 3 threads:

200 messages/sec/thread
2k messages/sec/thread
20k messages/sec/thread
100k messages/sec/thread

I still use JMH to piggy-back threading, warmup, setup/teardown, parameter setting, etc.

Things still TODO / TBD in this PR:

Why on extreme loads the new version has significantly worse tail latency?
StringBuilderEncoder.DEFAULT_BYTE_BUFFER_SIZE (8k) effectively overrides MemoryMappedFileManager.DEFAULT_REGION_LENGTH (32k), is that reasonable?
Java 9:

Buffer unmapping, as pointed here: https://github.com/apache/logging-log4j2/pull/87#issuecomment-312488652
WriteTouch should have an opportunistic impl based on VarHandles.

Not doing stuff "in background" on 1-core machines; probably not doing write touch on "small" machines (< 4 cores?)
Cleanup and comments to the recently added write touch and background stuff

Also recent addition of "write touch" seems to improve throughput considerably, so the numbers posted above are irrelevant now.

The Java 9 stuff sounds interesting. We may need to add a multi-version JAR build for log4j-core as well (we have one for log4j-api so far) if it can't reasonably done via reflection.

Following new developments with interest.
Looking forward to javadoc on the WriteTouch stuff. I just very briefly scanned the commit on my phone while commuting to work, but didn't see anything Java 9-ish, perhaps I misunderstood. What does this do?
Good point about StringBuilderEncoder and MemoryMappedFileManager buffer sizes. We have several options here: we can match up the buffer sizes so they are always the same, or we can document in the MemoryMappedFile appender manual that it is advisable for users to increase the StringBuilderEncoder  buffer size (I believe there is a system property). Any preference anyone?

Most folks are unlikely to look to closely at the docs. IMO we should make the docs as good as we can AND come up with default values that cover 80% of the cases.

@leventov We may have a Log4j 2.9 release starting as soon as this weekend. Let me know if you want to make the remaining changes to complete this PR for merging into master before that release.

@leventov Are you still interested in contributing this?

@rgoers in principle, I would like, but I won't have time to do this in foreseeable future.

@leventov @remkop @jvz Is there value in accepting this as-is. It has merge conflicts that have to be resolved. I have not checked to see how many there are or how difficult they would be to resolve.  It is targeted at Master and I don't believe I would try to backport it to release-2.x due to its complexity.  If no action is going to be taken then we should close it as the last activity was 3 years ago.

Closing since no one responded to my last inquiry.


Builder#parse_env does not override manually set filter level

Consider the following example (runnable version):
std::env::set_var("RUST_LOG", "debug");

let mut builder = env_logger::Builder::new();
builder.filter_level(log::LevelFilter::Trace);

// The following call should override the log level to Debug, as read from the environment
builder.parse_default_env();

builder.init();
assert_eq!(log::max_level(), log::LevelFilter::Trace);

According to the documentation, the parse_env function "allows a builder to be configured with default parameters, to be then overridden by the environment". For this example, that would mean that log::max_level() should be set to Debug at the end of this example. In reality, it's set to Trace instead.
Removing the call to builder.filter_level(log::LevelFilter::Trace) correctly sets the log level to Debug instead.

Yeah, the docs are pretty clear here. A PR fixing this would be welcome.

I provided both #199 and #200 as alternative potential fixes to this problem, depending on whether we want directive names to be unique in general (#200), or not (#199).

Hey @jplatte, friendly ping on this. Would #200 be an acceptable way to fix this, or is there a better approach? Thanks for taking a look :-)
Kind regards,
Bram

I'll have time on the weekend to take a look at the PR.

Thank you! But no rush, I was just checking it was still being considered 🙂

@jplatte I've taken a look at both of the PRs. Both look good.
I don't have strong preference for either PR, but if I'd have to choose I'd say we should merge #200, as it means more cosistent behaviour across the library in general by always overwriting previous directives. I can't think of an example off the top of my head where this might not be wanted.

I've had a look, I agree that #200 seems like the better solution.

Hey,
Just checking if anything is expected/needed from my end here.
Kind regards,
Bram

No. I merged #200. Sorry about the long delay.


Work on two routers

it pulled my ewelink info into the sonoff.Jason file but when I go to
Current entities in Developer Tools no sonoffs are there.
any other place to look or other settings ?
I even tried putting one in with the ID and switch in the config but nothing.
could this be because all 12 of my ewelink sonoffs are on different wifi router and SSID .
my HA HassIO new fresh install but its connected with Ethernet to my main DLS modem?router. and all my ewelink devises are on a separate router that is connected to the DSL moden/router.
not sure why this would matter because my shellys using factory setting MQTT show up and they are on the same SSID as the ewelink sonoffs.
HassOS 3.10 Home Assistant 0.105.3
raspberry pi 4

Devices are detected by multicast. Multicast really may not pass between different routers.

So you saying It can't find because on a separate router on same network?

That was the problem
I connected the pi to the same router and all sonoffs were found

Yes, you can read how multicast traffic works. Many devices use it.


[MIGRATE] Model.yaml -> System.yaml

migrate the model.yaml into the system.yaml
Now that we have ui, and arent really expecting the user to edit this manually we should merge these two files together

update manipulator calibration
https://github.com/DeMarcoLab/fibsem/blob/fd20bed3d0754d2d6433a87681a50ab06d88e764/fibsem/ui/FibsemManipulatorWidget.py#L154

manipulator calibration fixed

@LucileNaegele  @Rohit-Kannachel has this been resolved?


Only catch action exceptions in production mode

Rasa Core version:
Master (commit 241979ca4fbe22083aec7b1b6c37810cc0ce2e6c)
Python version:
3.5.2
Operating system (windows, osx, ...):
linux
Issue:
Hi, I noticed that coding/execution errors in the actions are not reported anywhere, when running the bot.
Even macroscopic errors like using an undefined variable are just silenced. The execution of the actions stops and the conversation keeps on.
It would seem like somewhere under the hood you are running the action code with a very wide exception catching policy. I don't think this should be happening. Please just catch expections that are generated by rasa, and pass the other ones up so we can deal with them ourselves.
Thanks for your help,
Andrea.

If you run the bot with debug mode enabled (so pass --debug to the rasa_core.run script) all the exceptions are shown

Yes we are catching all exceptions there. But you are right, we should probably only do that in production mode or something like that, because otherwise the bot might be unreachable because of a small bug.

I must admit I am not sure what the best thing here is. I see how you do not want the bot to crash on a conversation just because of an exception.
I think the choice to keep the bot running is the best one, but some uttering should warn the user that something went wrong. in my case the exception was preventing some data to be retrieved from a db, which made the rest of the conversation completely meaningless.
You could maybe introduce a RasaCoreError and a RasaCoreCritical exception that users can raise in their code whenever some crucial part fails.


String.truncate assumes the string is null-terminated

When a string is truncated to its own .size(), there should be no change in the string data, but a non-null-terminated string loses its last character.
let s = String.from_iso_array(recover ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'] end)

env.out.print(s.string())
env.out.print("size: " + s.size().string())

s.truncate(s.size())

env.out.print("-------truncated to size-------")

env.out.print(s.string())
env.out.print("size: " + s.size().string())

abcdefgh
size: 8
-------truncated to size-------
abcdefg
size: 7

I don't exactly know how to fix this since the current .truncate() behavior is surprising in that you can truncate to a value larger than the _size as long as it's less than the _alloc. This means that you can truncate a string down and then back up, and you'll have a 0 byte where the null terminator was inserted. It would also seem to pose potential security issues.
actor Main
    new create(env: Env) =>
        let s = String.from_iso_array(recover ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'] end)

        env.out.print("size: " + s.size().string())

        s.truncate(3) // truncate down to 3
        s.truncate(8) // truncate back up to 8

        env.out.print("size: " + s.size().string())

        for c in s.string().values() do
            env.out.print(recover String.from_utf32(c.u32()) end)
        end

Output:
size: 8
size: 7
a
b
c

e
f
g

So it would seem like the len to which you're truncating should have an upper limit of the _size instead of the _alloc, and that there shouldn't be any way to "expand" to the _alloc without writing data to that expanded space.

Since Praetonus pointed out to me that preallocated space in arrays does not get zeroed in a release build, it seems like it would be a potential heartbleed-esque security issue to allow String.truncate() to continue to accept a value greater than _size unless it zeros that memory.
So do we:

Change the current behavior to cap the truncated value to the _size
Keep the behavior, and zero the bytes above the _size

The former is a breaking change, but alters the behavior to match Array.truncate.
The latter isn't breaking, but we do still have the issue of the 0 being left in the middle of a String that was truncated down then back up.

There's another case in builtin that allows access to uninitialised memory:
let memory = Array[U8].undefined(1 << 20)

Here, memory has access to a megabyte of uninitialised memory.
Both "truncate to expand" (which follows the same model as truncate for files on Linux) and "undefined to expand" result in exposing uninitialised memory, and for the same reason: performance. For example, creating a large array of numbers which will be filled in as the function proceeds, so the programmer doesn't want to write to the array twice.
It's quite different from Heartbleed (which allowed buffer over-reads to arbitrary memory locations), but it does allow "searching" for previously deallocated memory. Or, rather, it doesn't inherently, but it's possible to imagine an application that could be exploited that way by user input causing repeated undefined array allocations of various sizes.
This is used in network code, reading blocks from files, reading from spawned processes, etc.
I didn't present any solutions here :) It's an interesting and important trade-off. Anyone else have thoughts on this?

I agree with @sylvanc.
I think the main issue here is the null terminator. We don't inherently need it, maybe it would be good to only care about it in cstring by lazily adding it when the function is called.

Thanks everyone.
I think I may have been confused on one thing. Does the allocator ensure that the previous data in that new memory location has been in some way overwritten? I understood that it does not, but I see the undefined docs say something about populating elements with random memory.

No, that's not guaranteed. The undefined docs mean that the contents of the Array isn't predictable since it will use the memory as is. Maybe "random" isn't the best choice of word here.

Thanks for the clarification @Praetonus.
I have several questions, but I think I'll take most of them to IRC.
If we lazily null-terminate, then is is_null_terminated would seem to just become _alloc > _size (and perhaps be renamed), since cstring would simply add the 0 in that case, otherwise create a terminated copy. Other things will be impacted too, like space should always return _alloc.

I think we could even remove is_null_terminated. Yes, that would be a pretty big change, a RFC on that would have to consider all of those cases. Also, to be efficient, I think it would need some specialisation on ref vs val receiver for cstring, to lazily write for ref and to copy and write for val. I've been thinking about a RFC on that for some time, I'll have to write it up.

Then for now I'm going to simply fix the case that chops off the last byte, and leave the case that inserts a null byte in the middle. That way it's partially fixed, and this can be closed and discussion moved to your forthcoming RFC.
I'll post the questions I have surrounding "expand to greater" on IRC at some point in the future.


Promote mqm version and add off switch to octane integration

Add support for new mqm-client-lib
Add off switch for octane jenkins plugin
Remove unused code
fix unittest

approved to test build

approved for test build

:white_check_mark: Build hp-application-automation-tools-plugin octane_latest.5-uvsxgvyo completed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/1a4829657f by @m-seldin)

approved for build

:white_check_mark: Build hp-application-automation-tools-plugin octane_latest.5-rpwmgasv completed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/044ae3de36 by @m-seldin)

approved for test build

:x: Build hp-application-automation-tools-plugin octane_latest.5-insruisp failed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/044ae3de36 by @m-seldin)

:x: Build hp-application-automation-tools-plugin octane_latest.5-lxkhkqem failed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/0227448ee9 by @m-seldin)

approved for test build

approved for test build

:x: Build hp-application-automation-tools-plugin octane_latest.5-tagbjexu failed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/e63e64ffc6 by @m-seldin)

:x: Build hp-application-automation-tools-plugin octane_latest.5-ahhbstpl failed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/0b89f5a726 by @m-seldin)

:white_check_mark: Build hp-application-automation-tools-plugin octane_latest.5-fjfoqthx completed (commit https://github.com/hpsa/hp-application-automation-tools-plugin/commit/230bcfda68 by @m-seldin)

approved for test build

approved to merge


error: Metrics not available for pod

well I have install promethus and grafana according instructions and all pods is running and ready. And I ran cmd:
kubectl top nodes
and works fine. but when i ran
kubectl top pods -n monitoring
I got:
error: Metrics not available for pod <podname>
and I ran:
kubectl logs prometheus-adapter-79c588b474-nclg9 -n monitoring
and I got error logs like below:
1012 02:52:14.165401       1 provider.go:191] unable to fetch CPU metrics for pod &PartialObjectMetadata{ObjectMeta:ObjectMeta{Name:prometheus-adapter-79c588b474-l6v4v,GenerateName:prometheus-adapter-79c588b474-,Namespace:monitoring,SelfLink:,UID:5daa7433-1fb2-440e-94e9-b67a67551ef8,ResourceVersion:212510,Generation:0,CreationTimestamp:2023-10-12 02:12:06 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app.kubernetes.io/component: metrics-adapter,app.kubernetes.io/name: prometheus-adapter,app.kubernetes.io/part-of: kube-prometheus,app.kubernetes.io/version: 0.11.1,pod-template-hash: 79c588b474,},Annotations:map[string]string{checksum.config/md5: 3b1ebf7df0232d1675896f67b66373db,cni.projectcalico.org/containerID: 809e5b40519426479a23cf5271446f5789687d3073d216a45d12e7c3fdfa7346,cni.projectcalico.org/podIP: <IP_ADDRESS>/32,cni.projectcalico.org/podIPs: <IP_ADDRESS>/32,},OwnerReferences:[]OwnerReference{OwnerReference{Kind:ReplicaSet,Name:prometheus-adapter-79c588b474,UID:6b59d352-4d4d-48db-a1ca-2bb0a60abdde,APIVersion:apps/v1,Controller:*true,BlockOwnerDeletion:*true,},},Finalizers:[],ManagedFields:[]ManagedFieldsEntry{ManagedFieldsEntry{Manager:kube-controller-manager,Operation:Update,APIVersion:v1,Time:2023-10-12 02:12:06 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:metadata":{"f:annotations":{".":{},"f:checksum.config/md5":{}},"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6b59d352-4d4d-48db-a1ca-2bb0a60abdde\"}":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"prometheus-adapter\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":6443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:readOnlyRootFilesystem":{}},"f:startupProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/adapter\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/tmp\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/var/run/serving-cert\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:nodeSelector":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"config\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"tmpfs\"}":{".":{},"f:emptyDir":{},"f:name":{}},"k:{\"name\":\"volume-serving-cert\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}},Subresource:,},ManagedFieldsEntry{Manager:calico,Operation:Update,APIVersion:v1,Time:2023-10-12 02:12:09 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}},Subresource:status,},ManagedFieldsEntry{Manager:kubelet,Operation:Update,APIVersion:v1,Time:2023-10-12 02:12:17 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"<IP_ADDRESS>\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},Subresource:status,},},},}, skipping
I'm not familiar with promethus and I'm confused why it happened. Please give me some help.

You can confirm config of prometheus-adpater,
kubectl -n monitoring get cm adapter-config -oyaml  | grep nodeQuery
check it has resources.overrides.pod property,like this:
resources:
overrides:
namespace:
resource: namespace
pod:
resource: pod

I'm running into the same error. Looking at the configuration for the adapter, there are rules configured for cpu and memory, and both sections have the resources section:
        "resources":
          "overrides":
            "namespace":
              "resource": "namespace"
            "node":,
              "resource": "node"
            "pod":
              "resource": "pod"

I'm running the latest commit of the release-0.13 branch (11447edd8f3644cde5287d0adc7c2a554ba5728d), via jsonnet.

What happened?:
We tried to upgrade our Prometheus and used e7fa2f0 by copying kubernetes manifests and adding patches on top of it as per our requirement.
We are only seeing this error at pod startup. After a while, we can kubectl top nodes' and kubectl top pods -A' and we were able to see cpu and memory. We ran it on specific pods that we saw the logs and we ran it on all pods and didn't see an issue.
Please provide the prometheus-adapter config:
This is the output of k describe configmaps adapter-config-749fgk78h9 -n monitoring
Name:         adapter-config-749fgk78h9
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>

Data
====
config.yaml:
----
rules:
- seriesQuery: '{ __name__="nginx_ingress_controller_nginx_process_connections",state="active"}'
  name:
    as: nginx_ingress_active_connections_1m_average
  resources:
    overrides:
      controller_namespace:
        resource: namespace
      controller_pod:
        resource: pod
  metricsQuery: 'avg_over_time(<<.Series>>{<<.LabelMatchers>>,state="active"}[1m])'
- seriesQuery: '{__name__="rabbitmq_celery_queue_size"}'
  name:
    as: rabbitmq_celery_queue_size
  resources:
    overrides:
      namespace:
        resource: namespace
      service:
        resource: service
  metricsQuery: '<<.Series>>{<<.LabelMatchers>>}'
resourceRules:
  cpu:
    containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>,container!="POD",container!="",pod!=""}[5m])) by (<<.GroupBy>>)
    nodeQuery: sum(1 - rate(node_cpu_seconds_total{mode="idle"}[5m]) * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:{<<.LabelMatchers>>}) by (<<.GroupBy>>)
    resources:
      overrides:
        node:
          resource: node
        namespace:
          resource: namespace
        pod:
          resource: pod
    containerLabel: container
  memory:
    containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,container!="POD",container!="",pod!=""}) by (<<.GroupBy>>)
    nodeQuery: sum(node_memory_MemTotal_bytes{job="node-exporter",<<.LabelMatchers>>} - node_memory_MemAvailable_bytes{job="node-exporter",<<.LabelMatchers>>}) by (<<.GroupBy>>)
    resources:
      overrides:
        instance:
          resource: node
        namespace:
          resource: namespace
        pod:
          resource: pod
    containerLabel: container
  window: 1m

Please provide the HPA resource used for autoscaling:
This is the output of  k describe replicaset prometheus-adapter-5c85857884 -n monitoring
Name:           prometheus-adapter-5c85857884
Namespace:      monitoring
Selector:       app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=5c85857884
Labels:         app.kubernetes.io/component=metrics-adapter
                app.kubernetes.io/name=prometheus-adapter
                app.kubernetes.io/part-of=kube-prometheus
                app.kubernetes.io/version=0.11.2
                pod-template-hash=5c85857884
Annotations:    deployment.kubernetes.io/desired-replicas: 2
                deployment.kubernetes.io/max-replicas: 3
                deployment.kubernetes.io/revision: 9
Controlled By:  Deployment/prometheus-adapter
Replicas:       2 current / 2 desired
Pods Status:    2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=metrics-adapter
                    app.kubernetes.io/name=prometheus-adapter
                    app.kubernetes.io/part-of=kube-prometheus
                    app.kubernetes.io/version=0.11.2
                    pod-template-hash=5c85857884
  Annotations:      checksum.config/md5: 3b1ebf7df0232d1675896f67b66373db
  Service Account:  prometheus-adapter
  Containers:
   prometheus-adapter:
    Image:      registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2
    Port:       6443/TCP
    Host Port:  0/TCP
    Args:
      --cert-dir=/var/run/serving-cert
      --config=/etc/adapter/config.yaml
      --metrics-relist-interval=1m
      --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/
      --secure-port=6443
      --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
    Limits:
      cpu:     250m
      memory:  200Mi
    Requests:
      cpu:        102m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=5s #success=1 #failure=5
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=5s #success=1 #failure=5
    Startup:      http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=18
    Environment:  <none>
    Mounts:
      /etc/adapter from config (rw)
      /tmp from tmpfs (rw)
      /var/run/serving-cert from volume-serving-cert (rw)
  Volumes:
   tmpfs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   volume-serving-cert:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      adapter-config-ggb4hk8mh7
    Optional:  false
Events:        <none>

Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:
We are seeing following in the logs of prometheus-adapter-### pod.
I0203 15:09:07.348750       1 adapter.go:114] successfully using in-cluster auth
I0203 15:09:10.448711       1 serving.go:342] Generated self-signed cert (/var/run/serving-cert/apiserver.crt, /var/run/serving-cert/apiserver.key)
I0203 15:09:13.546833       1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
W0203 15:09:13.551282       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
I0203 15:09:13.649827       1 secure_serving.go:210] Serving securely on [::]:6443
I0203 15:09:13.649928       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0203 15:09:13.649942       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0203 15:09:13.650001       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0203 15:09:13.650020       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0203 15:09:13.650042       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0203 15:09:13.650048       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0203 15:09:13.650109       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/run/serving-cert/apiserver.crt::/var/run/serving-cert/apiserver.key"
I0203 15:09:13.650241       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0203 15:09:14.046466       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0203 15:09:14.050882       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0203 15:09:14.150604       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0203 15:10:02.150494       1 provider.go:191] unable to fetch CPU metrics for pod &PartialObjectMetadata{ObjectMeta:ObjectMeta{Name:prometheus-adapter-5c85857884-f7jqp,GenerateName:prometheus-adapter-5c85857884-,Namespace:monitoring,SelfLink:,UID:ff4ee260-d173-4341-9b25-25f3e92d5c13,ResourceVersion:871410981,Generation:0,CreationTimestamp:2024-02-03 15:09:40 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app.kubernetes.io/component: metrics-adapter,app.kubernetes.io/name: prometheus-adapter,app.kubernetes.io/part-of: kube-prometheus,app.kubernetes.io/version: 0.11.2,pod-template-hash: 5c85857884,},Annotations:map[string]string{checksum.config/md5: 3b1ebf7df0232d1675896f67b66373db,},OwnerReferences:[]OwnerReference{OwnerReference{Kind:ReplicaSet,Name:prometheus-adapter-5c85857884,UID:828a4229-d099-404b-a749-9366e18d70aa,APIVersion:apps/v1,Controller:*true,BlockOwnerDeletion:*true,},},Finalizers:[],ManagedFields:[]ManagedFieldsEntry{ManagedFieldsEntry{Manager:kube-controller-manager,Operation:Update,APIVersion:v1,Time:2024-02-03 15:09:40 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:metadata":{"f:annotations":{".":{},"f:checksum.config/md5":{}},"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"828a4229-d099-404b-a749-9366e18d70aa\"}":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"prometheus-adapter\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":6443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:readOnlyRootFilesystem":{},"f:runAsNonRoot":{},"f:seccompProfile":{".":{},"f:type":{}}},"f:startupProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/adapter\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/tmp\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/var/run/serving-cert\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:nodeSelector":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"config\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"tmpfs\"}":{".":{},"f:emptyDir":{},"f:name":{}},"k:{\"name\":\"volume-serving-cert\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}},Subresource:,},ManagedFieldsEntry{Manager:kubelet,Operation:Update,APIVersion:v1,Time:2024-02-03 15:09:50 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"<IP_ADDRESS>\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},Subresource:status,},},},}, skipping
E0203 15:10:02.150605       1 provider.go:191] unable to fetch CPU metrics for pod &PartialObjectMetadata{ObjectMeta:ObjectMeta{Name:node-exporter-zmr6b,GenerateName:node-exporter-,Namespace:monitoring,SelfLink:,UID:0d8399b6-b2a4-4ff5-90cf-a2137ab47820,ResourceVersion:871410347,Generation:0,CreationTimestamp:2024-02-03 15:08:49 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app.kubernetes.io/component: exporter,app.kubernetes.io/name: node-exporter,app.kubernetes.io/part-of: kube-prometheus,app.kubernetes.io/version: 1.7.0,controller-revision-hash: 5964cf9676,pod-template-generation: 3,},Annotations:map[string]string{kubectl.kubernetes.io/default-container: node-exporter,},OwnerReferences:[]OwnerReference{OwnerReference{Kind:DaemonSet,Name:node-exporter,UID:4f102835-b129-427b-80ac-4a25e8ff2f9d,APIVersion:apps/v1,Controller:*true,BlockOwnerDeletion:*true,},},Finalizers:[],ManagedFields:[]ManagedFieldsEntry{ManagedFieldsEntry{Manager:kube-controller-manager,Operation:Update,APIVersion:v1,Time:2024-02-03 15:08:49 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/default-container":{}},"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:controller-revision-hash":{},"f:pod-template-generation":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"4f102835-b129-427b-80ac-4a25e8ff2f9d\"}":{}}},"f:spec":{"f:affinity":{".":{},"f:nodeAffinity":{".":{},"f:requiredDuringSchedulingIgnoredDuringExecution":{}}},"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"kube-rbac-proxy\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"IP\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":9100,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:hostPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:readOnlyRootFilesystem":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{},"f:seccompProfile":{".":{},"f:type":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}},"k:{\"name\":\"node-exporter\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:add":{},"f:drop":{}},"f:readOnlyRootFilesystem":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/host/root\"}":{".":{},"f:mountPath":{},"f:mountPropagation":{},"f:name":{},"f:readOnly":{}},"k:{\"mountPath\":\"/host/sys\"}":{".":{},"f:mountPath":{},"f:mountPropagation":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:hostNetwork":{},"f:hostPID":{},"f:nodeSelector":{},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:tolerations":{},"f:volumes":{".":{},"k:{\"name\":\"root\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}},"k:{\"name\":\"sys\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}}}}},Subresource:,},ManagedFieldsEntry{Manager:kubelet,Operation:Update,APIVersion:v1,Time:2024-02-03 15:08:58 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.248.XXX.XX\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},Subresource:status,},},},}, skipping

Environment:

prometheus-adapter version: 0.11.2
prometheus version: 2.48.1 (quay.io/prometheus/prometheus:v2.48.1)
Kubernetes version (use kubectl version): 1.28
Cloud provider or hardware configuration: AWS


Allow first argument of coda fetch function to be a file path

For cases where you only want to call coda.fetch (or coda_fetch) once, it would be very useful if the coda.fetch does the open/close of a product itself.
pf = coda.open('filename')
data = coda.fetch(pf)
coda.close(pf)

would then become a single
data = coda.fetch('filename')

This means that instead of the first parameter be a product handle or cursor, it should also be possible to let it be a string providing a file path.
This change applies to the Python, IDL, and MATLAB interfaces only.

We can't do this for the IDL interface, since that interface might return datahandles. And we can't close the file and return datahandles.
Since the MATLAB interface is more similar to IDL than to Python, we won't implement this convenience function there either.
So, we only implement this for Python.

Implemented


window.focus() offsets the window position

In #2241, I've added a new focus() command via these lines. Then in this commit, 3b1b decided to focus() the window every time on initialization, i.e. also when the new reload() command is used.
The problem is that I had to use a workaround in that focus() command since the pyglet activate() method didn't have any effects for me. That's why I did a simple
self._window.set_visible(False)
self._window.set_visible(True)

This consistently restores focus on the window, however it also offsets its position for me, which is quite annoying as the window might even jump around when reaching the corners of the screen.

[!tip]
I've opened a new issue on the pyglet repo here to track this issue.

Thanks for the partial fix via #2274, at least the window doesn't jump around now when I use reload() (but of course doesn't get focus either, but this is a better state than beforehand). Hopefully, the team at pyglet will find a solution such that focus() works properly.


[SPARTA-360] [SPARTA-510] Integration with GoSec

Description
With this feature you can configure a SSO server. For this purpose you need to follow these steps:

Add in your /etc/hosts:

<IP_ADDRESS> gosec01.dev.stratio.com
<IP_ADDRESS> anistal

Download the certification from: https://gosec01.dev.stratio.com:9005/cas/login (export Gosec's SSL certification with chrome/firefox/etc.)
Create cacerts file:

keytool -import -alias ca -file Stratio\ Inc\ CA -keystore cacerts -storepass stratio

Run sparta with this option -Djavax.net.ssl.trustStore=/path/to/cacerts
Configure your application.conf with oauth configuration:

oauth2 {
  enable = "true"

  url {
    authorize = "https://gosec01.dev.stratio.com:9005/cas/oauth2.0/authorize"
    accessToken = "https://gosec01.dev.stratio.com:9005/cas/oauth2.0/accessToken"
    profile = "https://gosec01.dev.stratio.com:9005/cas/oauth2.0/profile"
    logout = "https://gosec01.dev.stratio.com:9005/cas/logout"
    callBack = "http://anistal:9090/login"
    onLoginGoTo = "/"
  }
  client{
    id = "anistal-client"
    secret = "fg345vfdefgh4gweb424rtvwg24"
  }
  cookieName="user"
}

Starts sparta and go to the main page. When it appears you should be redirected to the login page
Enter username and password: jmgomez/1234 (test user to check).
The user is logged. If you need to do a logout: http://localhost:9090/logout. The cookie will be removed.

Definition of Done
GIVEN a running Sparta instance
WHEN you try to access to the main page
THEN you should be redirected to the SSO page
Documentation
https://stratio.atlassian.net/wiki/display/SPARKTA/6.+Configuration#id-6.Configuration-5.8.Oauth

trying to test but redirect bug has been detected again. Pending @anistal and @EvaMGC review

+1

:shipit:

Comment accepted. Will treat it ASAP

Squashing and checks in progress.


Frozen LM for object-level grounding

Hi, firstly thanks for your nice work! I was curious to know which frozen Language Model did you use for object-level grounding, as mentioned in Sec 4.1 of the paper?
Please let me know. Thanks!

We used a frozen bert-model for pre-training the alignment between pcd object encoder and language model.


bug(@vtmn/css): VtmnRating stars out of the box with many reviews.

Describe the bug

Stars of VtmnRating component don't fit in the box when they have a lot of ratings and the mobile is 360px.

Steps to reproduce
On any page with a card carousel, staying in Viewport > 359px
Notice that each product carousel review star component is overflowing the box when there are many reviews.
Expected behavior

For example, one solution might be to add a wrapper to the flex container and center it. It could look like this:
Browsers affected
- Chrome
- Firefox
- Edge
- Safari

Version affected

@vtmn/assets": "^1.1.1",
@vtmn/css-design-tokens": "^1.1.0",
@vtmn/css-tailwind-preset": "^1.1.2",
@vtmn/css-utilities": "^1.1.1",
@vtmn/icons": "^1.2.1",
@vtmn/svelte": "^2.0.1"

Hi, for your context you should switch to the compact version of this component which is typically designed for this type of usecase.

Thanks @MARIEDELATTRE for your answer to @tony3fk.
I close this issue accordingly.
Feel free to reopen if you feel that could not be possible or if you have another request.
Thank you again.


[#760] Support JSON output in -compileonly

Added the support for -json flag and transform the errors into array JSON format.

Thank you! :D


🐞 Haptic Feedback on Launch

Description
Haptic feedback is triggered on launch if project windows are open. This is most likely in response to the @yapryntsev's PR we merged for snapping the sidebar. When the sidebar snaps to the default width, haptic feedback is given.
To Reproduce

Open CodeEdit after Quitting with a few project windows open.
Notice haptic feedback is given.

Expected behavior
No haptic feedback is given.
Version information
CodeEdit: 0.0.1
macOS: 13.0.1
Additional context
No response

@yapryntsev would you be able to look at this one?

@yapryntsev would you be able to look at this one?

Hi, I'm working on a fix for this bug.

@austincondiff Can we close this issue as it looks like the pull request by @Cubik65536 solved it?

I think we can close this. I did mention this problem and give a suggestion in PR #984 so it won't introduce this problem again. Or we can close this after 984 is merged.

I think this issue was reintroduced by PR #984 :( /cc @austincondiff


using sandbox skeleton

Not ready for review

@rudigiesler | @smn Please review

other than the typo, :+1:


Enhancement: Optimize generated classes with character data

Is there an existing issue for this?

[X] I have searched the existing issues

Enhancement description
Currently generated classes with character data check if the codepoint belongs to this class by using some kind of binary search between the ranges this class has. This causes O(log(n)) complexity to find if the class contains the codepoint in the worst case.
I think this can be optimized in the following way:
We can divide ranges into sub-groups by applying a shift-right bit operation (codepoint >> 16). This will group ranges based on their higher bits. As a result, we will have smaller groups of ranges to apply our binary search and we will be able to filter out codepoints that cannot belong to the class by comparing the result of a shift-right operation with known values for this class.
fun contains(codepoint: Int): Boolean {
  if (codepoint < minCodepoint || codepoint > maxCodepoint) {
    return false
  }
  return when (codepoint shr 8) {
    in 0x0..0x2 -> // binary search in this group
    0x10 -> // binary search in this group
    else -> false
  }
}

Any attempt to group ranges and keep the correctness did not give any performance benefits. Benchmarks show that the performance only gets worse (2-3 times worse).
I will close this issue for now. If new ideas on how to optimize the generated code come to my mind (or somebody else's) I will reopen this issue.


Is there a way not to generate paralanguage tags

I get a text: hi everybody hi everyone this is  the.
Is there a way not to generate paralanguage tags like  ?

Hi, can you give a bit more information ?  What do you mean by "paralanguage tags" in your text above?  Do you mean punctuation ?  Or do you mean tags like <laugh>  ?

oh sorry, the fully text is: hi everybody hi everyone this is  the.
Is there a way to remove ? and how can I get text with punctuation marks?
thank you for you answer!

oh sorry, the fully text is: hi everybody hi everyone this is  the.
Is there a way to remove ? and how can I get text with punctuation marks?
thank you for you answer!

sorry , the fully text is :hi everybody hi everyone this is <laugh> the.
is there any way to remove <laugh>? and how can I get text with punctuation marks?
Thank you for your reply！


Fix for wrong City Data

Proposed change

the Zamg data for the city Klagenfurt and Innsbruck do not match between the list of measurement points and the realtime data.
This fix adds the numbers used in the realtime data to the list of measurement points.
Realtime data: http://www.zamg.ac.at/ogd/
Measurement point list: https://www.zamg.ac.at/cms/en/documents/climate/doc_metnetwork/zamg-observation-points
Like this the auto search for the nearest location works and also all citys can be used.
The way this works may not be the best, maybe somebody should even highlight this mismatch to Zamg?
(im from Innsbruck and was wondering why i could not use the City and the auto search found something quite far away, where i knew that Zamg has a location just next to me)
Type of change

[ ] Dependency upgrade
[x] Bugfix (non-breaking change which fixes an issue)
[ ] New integration (thank you!)
[ ] New feature (which adds functionality to an existing integration)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml
sensor:
  - platform: zamg
  - latitude: 47.26
  - longitude: 11.4

This is located in Innsbruck, it should NOT give "Patscherkofel" as nearest station, with this Bugfix it returns "Innsbruck"
Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:

Checklist

[x] The code change is tested and works locally.
[x] Local tests pass. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist
[x] The code has been formatted using Black (black --fast homeassistant tests)
[ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
[ ] New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

[x] No score or internal
[ ] 🥈 Silver
[ ] 🥇 Gold
[ ] 🏆 Platinum

Hi @designer2k2,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

Hey,
Thanks for the contribution but we won't be able to merge it at this moment.
This integration contains protocol specific information that should have been put into a PyPI package. This means that we cannot accept any changes/fixes to this integration until it has been fixed.
I also think that you should report these issues to ZAMG directly, as it's their API that is reporting wrong values.

Ok, i wrote a Email to ZAMG.
Meanwhile i keep the bugfixed version here locally as its useless in my region of Austria otherwise.
Could you give me a hint or example about the "protocol specific information move to PyPI package" ?
As i would then like to try to tackle that fix.

The class ZamgData should be published as a standalone library to PyPI. It should work independently of Home Assistant. Then you can update manifest.json to rely on the new PyPI package and it can be used.


Support high-resource testing on RDev [CI/CD]

For RDev to more fully support all modes of testing (ad hoc, automated, QA testing), it must be provide compute resources that are capable of working with prod-equivalent data and with performance that is closer to our deployment environments. Currently, API calls and selected data can causes OOM errors or cause systems to operate too slowly, making RDev insufficient for some testing scenarios.

ECS backend instances need to be up-scaled appropriately.
Batch instances need to be up-scaled appropriately

Implementation notes:

Determine the appropriate levels of scaling needed
Assess cost of upscaling to the desired levels. Once we have estimates, let's get approval from Arathi before increasing AWS costs.
As needed, investigate the use of ECS soft and hard memory limits to prevent over-provisioning of RDev compute resources.
As needed, work with the Happy Team to understand if RDev auto-scaling is possible.
Document how RDev cost can be monitored so cost impact can be effectively managed by engineering team.

Tasks:

Once all the tests are in place, run the tests in current RDEV config to see if tests pass
If not, figure out how much to scale up given estimated parallel runs (includes ad-hocs and CI/CD provisioned stacks), and assess costs and get Arathi's approval budget

Hey team! Please add your planning poker estimate with Zenhub @atolopko-czi @Bento007 @danieljhegeman @ebezzi

team agreed with 3

Here is the current configuration for rdev
https://github.com/chanzuckerberg/single-cell-infra/blob/d5783679da0611e020eef727e1fa7730a737c783/terraform/envs/dev/dp-rdev/main.tf#L71-L73
It uses a larger machine than dev
https://github.com/chanzuckerberg/single-cell-infra/blob/d5783679da0611e020eef727e1fa7730a737c783/terraform/envs/dev/dp-dev/main.tf#L53-L55

Increasing the size of the ECS cluster will not significantly increase the cost of development. We could double it and it would only be an additional $200 a month. We are also configured to use 1-2 servers for happy and are only using 1. We have plenty of room to grow. There is no need to change the current configuration for running the data portal server.

No changes need to be made at the moment to support more rdev environments.