TSM client

Motivation for this change
Tivoli Storage Manager is IBM's backup solution. This pull request brings the client software and a NixOS module to use the client for regular system backups. Note that the client also provides an API to be linked against.
This solftware was renamed to IBM Spectrum Protect recently, but the previous name prevails in file names of the software as well as in package names of other distros. A I would expect possible users to look for the old name, I stick to it.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS: x86_64-linux
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests) (no such tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip" (none)
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date: there is none
[ ] Fits CONTRIBUTING.md.

Updated the branch following the suggestions of https://github.com/NixOS/nixpkgs/pull/59891 . Also added some examples to module options.

Service is inlined (assertion also, but remaining stuff is so involved, so it's probably better comprehendable as separate declaration).
I also realized that the service should be created even if no autoTime is set as it would be useful to permit starting the service manually.
I hardcoded tsm-client as directory in /etc and /var/lib everywhere now.  I have to add that I made up this name; there is no "official" name for these directories provided by IBM (at least none that I heard of).
Regarding DSM_LOG, this variable tells the software where to write logfiles to.  It must point to a directory where the invoking user has write access.  The way it is declared in the wrapper allows the user to override it at will.  However, if the user doesn't touch it, the wrapper lets it point to the user's HOME directory, as that is the only directory where it is very likely the invoking user has write access to.  I wanted to use --set-default to implement the override effect, but makeWrapper uses single quotes to enclose the argument in the resulting wrapper script which prevents $HOME from being resolved.  Therefore I implemented a command that's similar to --set-default but actually uses the value of $HOME here.
The construct ''${ is needed to escape the string ${ as nix would try to resolve it as an antiquotation.  Prefixing with two single quotes is actually the recommended way (given in the Nix manual) to escape the ${ construct.
Regarding the directories /var/lib/tsm-client/*: The client needs a directory where it stores the password it uses to connect to the server; that directory is declared in the configuration file (in /etc/tsm-client/cli.dsm.sys).  The client also needs a directory where it stores certificates received from the server on the first connection (and more files that I cannot explain); it stores those files somewhere in $HOME.  As all those files are created at some point, then must be kept around for future invocations, I considered them to be state and I put them in /var/lib/tsm-client.  I have to add that my statements above derive in parts from the documentation of IBM, in parts from personal experience of tsm's behaviour I couldn't find documentation for.
Thanks, @flokli, for all the efforts you invest in my pull requests.  I really appreciate that!

I did some research on the syntax of dsm.sys and some tests.  The youngest update of the branch is based on the results.
First off, some notes on server options:

dsm.sys contains sections per server, so-called "stanzas", similar to "Host" sections in ssh_config.  Each stanza has a name, followed by that stanza's configuration.  In contrast to ssh, users may only connect to servers declared in the system-wide dsm.sys file.
To represent the stanza structure, this pull request's tsm module provides the option tsmClient.servers, containing an attrset that maps stanza names to their respective configuration attrsets (similar to users.users with user configuration).  The stanza's options include some of the most important or commonly used options (server, port, ...), furthermore the option extraConfig containing another attrset that is directly mapped into key-value-pairs in the respective stanza.
The declaration of stanzas with tsmClient.servers happens independently of an optional systemd service that might invoke dsmc.  Depending on the usecase, these stanzas and the TSM client in general may be used for things other than pure backup operations.  Therefore, the global configuration option is now called tsmClient instead of tsmBackup.
To use a stanza for regular systemd-controlled system backup, the user may provide the name of the stanza to the option tsmClient.service.servername.

I also tried to feed logging into syslog/journal, ultimately to no avail:

There are several ways to control where the client writes its log output to:

The dsm.sys option errorlogname takes a filename where logs are to be written to.
The environment variable $DSM_LOG takes a filename where logs are to be written to.  If $DSM_LOG ends with a slash, it is treated as a directory name and the base name dsmerror.log will be appended automatically.
If none is given, TSM acts as if $DSM_LOG points to the current working directory.  Consequently, $DSM_LOG/dsmerror.log will be used.

TSM tries to create the logging directory if it does not exist.
If TSM cannot create its log file/dir, it aborts.
To summarize, I failed to convince TSM to use syslog.  However, the systemd service writes detailed output to stdout/stderr in addition to the logfile.
As TSM uses a (sensible) default for $DSM_LOG, I removed the default from the wrapper scripts.
TSM creates several logfiles.  I observed dsminstr.log, citScanOutput.xml being created in the same directory that also contains dsmerror.log.  As not all filenames document their ancestry clearly, I decided to use /var/lib/tsm-client/ as logging directory of the system backup service.  Putting these files directly into /var/log (as did the outdated versions of this pull request) would litter that directory with confusingly-named log files.

Further notable changes:

It turned out that dsmc works well with $DSM_CONFIG unset, so I removed this variable from the wrapper scripts as well.
The backup systemd service uses /var/lib/tsm-client as $HOME directly.
Some options changed their names to be more consistent.
The docs on the commmethod stanza key explain that TSM uses IPv4 or IPv6 (depending on dns lookup result) if v6tcpip is selected.  So that's now the default as I guess it's what most users would like.
The default value of inclucdeExclude is removed as the mkForce-comment seemed to be too confusing.  Compression is off by default, anyway.
Several assertions check for inconsistencies (e.g. default servername not in stanza attrset).  In particular, as TSM ignores upper and lower case in server/stanza names and option names, several checks are in place to prevent duplicate definition of server/stanza names and option names via differing capitalization.
I learned that stanza names cannot have more than 64 characters and network ports (used by TSM) cannot be higher than 32767.  The option declarations reflect these restrictions.

@spacefrogg could you take a look at this? You seem to be familiar with this ecosystem ;-)

Thanks for your suggestions, @spacefrogg. It is really good to see that I will not be the sole user of that package! I applied most of your requested changes.
Considering your suggestion to create a wrapper script in a NixOS module, I beg to differ. Since creating that wrapper concerns several of your per-line requests, I will concentrate my response to that concept here.
I am convinced that the symlink to dsm.sys in /etc from within the package is a better approach than reassembling the DSM_DIR (and adding dsm.sys) outside of the core package, for the following reasons:

It is certainly right that a reference to /etc constitutes a dependency on state outside of the package. However, this is standard practice in most packages containing system software in nixpkgs. For instance, most server software that handles logins reference /etc/pam.d somewhere inside their binaries to find out how to authenticate users and alike. While such software references /etc via a hardcoded path inside some binary in most cases (or is patched to do so, e.g. fuse), using a symlink is no different. In particular, a symlink causes no further problems.
According to IBM's documentation (https://www.ibm.com/support/knowledgecenter/en/SSEQVQ_8.1.7/client/c_cfg_spevunix.html), the directory pointed to by DSM_DIR must contain most of the executables, dsm.sys, and "resource files". Those, in turn, encompass the language directories (EN_US and similar), but probably more. All of those files are simply static data, with dsm.sys being the lone exception. Therefore, it's no problem to let the environment variable point to the directory inside the package. In contrast, reassembling the DSM_DIR outside of the package is complicated, occupies additional space, and is prone to missing files. For instance, the example wrapper for the NixOS module is only symlinking EN_US thereby losing support for other languages.
Removing the wrappers from the tsm-client package would render that package useless without the NixOS module, e.g., if used on other distributions where Nix is installed, but NixOS modules can't be used. Users would likely have to write a similar wrapper by hand.
I expect this to be even more of a problem for API users, as software linking against the tsm-client certainly brings no script for rebuilding the DSMI_DIR, leaving that task again for the user.
Adding other packages to the PATH via a wrapper in the derivation is a common method in nixpkgs -- grepping in nixpkgs for makeWrapper yields many examples for this, git and vim amongst them. Removing the wrapper from the tsm-client package cuts the dependency on procps and jdk for non-NixOS users. The unwrapped binaries actually do depend on outside state as they will invoke whatever happens to be in PATH. I can't imagine a use case where it is necessary to call the command line utilities without procps or (optionally) java in the PATH.
Adding DSM_CONFIG and DSM_LOG to the wrapper does make it behave the same whether called by systemd or by hand, but there is no need for that. In fact, the environment variables are used to modify the behaviour of dsmc. If the user wants to connect to a different server, s/he will likely provide her/his own dsm.opt file.
In contrast, providing config files to systemd services by adding environment variables like DSM_CONFIG is a standard method. Most software uses command line options for this (like sshd -C /path/to/config-file) but it is the same principle. (Note that I replaced the environment variable with the -se command line option now.)

On a sidenote: I used to have a similar setup on my machines -- I reassembled the DSM_DIR (the directory where my package's symlink dsm_dir points to) in /etc/tsm-client and -- from time to time -- had to adjust my copy of the directory as it turned out I missed some important files in the package's directory. I think that now I understand much better how IBM wants that directory to be used. Based on my understanding, the symlink to /etc appears to me to be a solution that is much more robust and less complicated than reassembling the DSM_DIR directory by script or wrapper.

@Yarny0 Thank you for your effort! I understand your reasoning and will address each of your remarks in turn. Please note, that I am in no authority over merging this PR or demanding any changes of you. My code reviews are based on my experience and current understanding of how nixpkgs PRs are supposed to be designed to acceptable to people that actually do have the authority. So, feel free to ignore me.

No file-based references to outside a nixpkg. This is not negotiable. We try very hard to stick to this rule. Even if you find a package breaking this rule. This one is a normal use-case and does not demand such exception. I am quite sure the upstream hydra server does reject this package as it is right now.
TSM is of exceptionally poor design (as many closed-source applications) as it does not properly differentiate between static and runtime data. You don't need to copy anything around, and my suggested wrapper script was not doing this at all. You create a tree of links with the proper files in it, which costs virtually nothing. If you look closely, the structures of my suggested approach and your solution slightly differ: In my solution, all configuration files are first generated into the /nix/store and then referenced by the wrapper. So, still nothing points to outside the store. With a heavyweight config-abstraction you could even make sure that the tsm-client never breaks, as it never sees a broken config file. In your approach, you link to modifiable data outside the store and hope nothing goes wrong. Why is that important even if anything works today? Well, in general, you could not vouch for TSM to not deprecate a config option or note choke on a perfectly reasonable config file in the future. Abstraction through modules gives you a way to properly migrate or warn/inform users. Depending on usable external state does not.
This is true but is a price we are willing to pay. The reasoning goes as follows: This application is broken by design, as it is only useful with a proper configuration file, which is not taken from a reasonable location by default. The approach to get there in nixos is via wrappers in modules. People outside nixos have to write their own wrappers to get to there. No shortcuts that break nixos. This is actually true for most server software on nixos. The apache package, for instance, is completely useless outside nixos for this very reason.
Why should software linking against tsm-client need that? Anyways, linking software against a library is a package maintainer's task not an ordinary user task. We expect package maintainers to take the burden. (Which you could lighten by putting up a comment explaining how to properly link against TSM including wrappers.)
True. Inside nixos the package is never used directly, see apache for another example, but only through a nixos module. For outside nixos you can use the runtimeDependencies attribute to unconditionally add the packages to the closure. You might actually want to keep the wrapper for dsmj (making it double wrapped in the end) and only add procps to runtimeDependencies.
True. If you look closely to my suggested wrapper, it allows the respective environment variables to be overridden from the outside at call time. Still, you should at least pin DSM_LOG, as I found it annoying that it writes the log to your current directory. So again, TSM comes without sane defaults.
True. I have nothing against providing the data via environment variables in systemd directly. It was merely a shortcut, once you have the proper values in the wrapped binaries anyway.

Dear @spacefrogg, I will concentrate on the question of the (im)possibility of the symlink into /etc.  That question is probably paramount to the design of this pull request.

No file-based references to outside a nixpkg. This is not negotiable. We try very hard to stick to this rule. Even if you find a package breaking this rule. This one is a normal use-case and does not demand such exception. I am quite sure the upstream hydra server does reject this package as it is right now.

I'm not sure if your argument goes against a symlinks in particular or against any kind of reference that points outside of the Nix store.  Please forgive if I'm bringing forward unnecessary arguments.  Anyway, here's my stance on references that point out of the Nix store:

It is common for packages to reference files outside of /nix/store.  vim, bash, tmux, ssh{,d}, git, and many more consult configuration files in /etc when invoked.  Those references are no problem as their targets are properly documented or at least common knowledge.
A symlink -- while being more prominent -- has the same effect as a simple file containing a reference.  Binaries can use and follow a symlink like they follow a path that is denoted elsewhere in the package.
Symlinks to targets beyond the Nix store are also common practice in nixpkgs, see e.g. linux-pam, spacefm, roundcube, bluez, openfire, and more.  Most of those point to out-of-package configuration files in /etc, thereby accomplishing exactly what I want to accomplish with tsm-client.  Hydra seems to have no problems with these packages as long as the license is acceptable.
Last but not least, I simply can't spot a problem with NixOS that the symlink would provoke, as long as it points to a sane and fixed location.  It is common and apparently legitimate for packages in nixpkgs to reference files with configuration, state or runtime data outside of the Nix store.

Is there any documentation or otherwise documented consensus that explicitely forbids -- or frowns upon -- references pointing out of the Nix store?  What is the rational to avoid such references in general?  Although I still consider myself a novice w.r.t. Nix packaging, the ban on outside references stands in stark contrast to everything in nixpkgs I've seen so far.  On the other hand, not using the symlink would hamstring many of my use cases.  So please understand my skepticism and excuse my insistence.
While looking for statements about the (in)validity of references pointing out of /nix/store, I found a script that checks for references to the build directory (<nixpkgs/pkgs/build-support/setup-hooks/audit-tmpdir.sh>), as such references could pose serious problems.  However, neither this script nor any other one I looked at checks for references pointing outside of the store in general.

[...] In your approach, you link to modifiable data outside the store and hope nothing goes wrong. Why is that important even if anything works today? Well, in general, you could not vouch for TSM to not deprecate a config option or note choke on a perfectly reasonable config file in the future. Abstraction through modules gives you a way to properly migrate or warn/inform users. Depending on usable external state does not.

The safety and comfort of a NixOS module is independent of the usage of /etc.  This pull request uses a module to place the config file in /etc (similar to other software, e.g. sshd), providing the abstraction of the module.  Using a wrapper and sidestepping /etc does not give any benefit over the symlink to a config file in /etc if that file is generated by the NixOS module.  On non-NixOS systems -- and even on NixOS if root explicitely chooses to do so -- root may also provide a handmade config file.  This would be a conscious decision by root, and I consider this possibility an improvement as it facilitates usage of tsm-client on non-NixOS systems.  Of course, root has to ensure that the file is correct, but this is no different from other software using config files in /etc (irrespective of the distribution).

This is true but is a price we are willing to pay. The reasoning goes as follows: This application is broken by design, as it is only useful with a proper configuration file, which is not taken from a reasonable location by default. The approach to get there in nixos is via wrappers in modules. People outside nixos have to write their own wrappers to get to there. No shortcuts that break nixos. This is actually true for most server software on nixos. The apache package, for instance, is completely useless outside nixos for this very reason.

Do you refer to Apache's httpd?  I'm not sure it fully compares to the problem at hand: httpd accepts a config file on its command line, and it is a server process not ment to be invoked by ordinary users.  However, I tried it, and httpd from nixpkgs runs fine without a wrapper script -- even when called by non-root (on a port beyond 1024) and even on a non-NixOS machine.  I'd like to reach a similar solution with tsm-client: The binaries should work "out of the box", without any wrapper scripts but just the configuration file being present, irrespective of the underlying distribution.

Why should software linking against tsm-client need that? Anyways, linking software against a library is a package maintainer's task not an ordinary user task. We expect package maintainers to take the burden. (Which you could lighten by putting up a comment explaining how to properly link against TSM including wrappers.)

The software would need it as TSM's API needs the DSMI_DIR environment variable containing a properly prepared dsm.sys (like the command-line client that needs a DSM_DIR).  A package maintainer, not knowing the content of dsm.sys in advance, would face the same dilemma and would likely end up with the same symlink solution that I prefer, since this solution doesn't require the package to be rebuild whenever the configuration file dsm.sys needs modification (assuming that the maintainer uses another Nix recipe to prepare the package linking against tsm-client).

True. Inside nixos the package is never used directly, see apache for another example, but only through a nixos module. For outside nixos you can use the runtimeDependencies attribute to unconditionally add the packages to the closure. You might actually want to keep the wrapper for dsmj (making it double wrapped in the end) and only add procps to runtimeDependencies.

Why would I add procps to runtimeDependencies?  TSM doesn't link against procps (not sure if that would be possible at all), but TSM actually calls the executable ps, so ps must be in the PATH.  The wrappers in this pull request accomplish exactly that.
Let me add and emphasize that I'm very greatful of this discussion, @spacefrogg.  Irrespective of the outcome of this particular question, I'm aiming at a solution that covers as much use cases as possible, in a way that is as user/admin friendly as possible.  So I would be very inclined if the version that is finally merged covers your use cases as well as mine.  I'm still convinced that the symlink to /etc is a dependable solution, nevertheless (or maybe for exactly that reason), I'd like to know if the pull request in its current version could also cover your use cases.

No file-based references to outside a nixpkg.

I'm not sure if your argument goes against a symlinks in particular or against any kind of reference that points outside of the Nix store.

My argument goes against what I wrote just above. File-based references, i.e., softlinks and hardlinks.

It is common for packages to reference files outside of /nix/store.  vim, bash, tmux, ssh{,d}, git, and many more consult configuration files in /etc when invoked.  Those references are no problem as their targets are properly documented or at least common knowledge.

These are not references of the same kind. The main difference between the two is in expectations. Resolving a configuration file from inside a running program is in the responsibility of the program itself. Providing a filesystem without broken links is not. You may not like this distinction, but I, myself, regard this as common consensus (by mere observation) among Linux distribution maintaining folks.

A symlink -- while being more prominent -- has the same effect as a simple file containing a reference.  Binaries can use and follow a symlink like they follow a path that is denoted elsewhere in the package.

Maybe, but this similarity is not relevant. See my previous remark.

Symlinks to targets beyond the Nix store are also common practice in nixpkgs, see e.g. linux-pam, spacefm, roundcube, bluez, openfire, and more.  Most of those point to out-of-package configuration files in /etc, thereby accomplishing exactly what I want to accomplish with tsm-client.  Hydra seems to have no problems with these packages as long as the license is acceptable.

True, and the term you are looking for is purity, impure package and descendants. We try to keep packages pure as much and as often as possible. A package should not reference anything outside the known scope of the nix package manager with very few exceptions, e.g., runtime data (usually databases of any kind). And even that is a concession to practicality. I give you the very idea of NixOS:

The system is (ideally) stateless.
Consequently, a /nix/store path to a system configuration uniquely identifies the complete closure of the system, its applications, their configuration.
Consequently, calling an application from the closure of a particular system configuration always yields the same result (when executed on the same machine).
This even holds when a different system configuration was active in the meantime.

The consequence from 3. is, that applications cannot have state, which is impractical, which is why local application state is treated separately. But even then, look at the postgres NixOS module and you will find that it uses versioned directories for its application state. This way, you at least have the possibility to downgrade again, should the database upgrade mess up your end-user application. On a 'normal' Linux distribution you would have literally messed up when you upgraded without making backups first.
As a sidenote: With pam you have picked a special case of package impurity, that is really hard to solve due to its basic service it provides to a system. You need it everywhere under insanely varied circumstances. The others look like bad quality packages (without having looked into it). roundcube is definitely a case for bad quality software design, as it expects to be allowed to write into its own code base. This is the case for a lot of so-called "web applications".

Last but not least, I simply can't spot a problem with NixOS that the symlink would provoke, as long as it points to a sane and fixed location.  It is common and apparently legitimate for packages in nixpkgs to reference files with configuration, state or runtime data outside of the Nix store.

Okay, now to the other side of the story... As mentioned, NixOS allows application state, although it strives for purity through statelessness. If it would have been practically achievable and sustainable (i.e., maintainable), there would be no /etc directory. You can see that we strive very hard to get rid of the statefulness of /etc by the way we literally generate a complete shadow tree and copy everything to the real directory on each system startup and each system configuration change. Allowing references to /etc is a concession to broken software (well, no longer valid design assumptions) and practicality. We avoid statefulness in /etc (configuration files in general) as much as possible by:

Directly referencing the constructed configuration file in the /nix/store via command-line switches etc.,
Wrapping binaries, so that they point to the right incarnation of the configuration file or, at least, by
Making /etc/<configfile> a symlink to a /nix/store path.

We even support copying the file from the /nix/store to /etc and setting its owner/mode bits on each start if an application really does not accept a symlink (for god's sakes!). Look at the openafs module for an instance of this. Also look at the cups module for the fireworks we burn to provide a writable state directory, yet a stateless configuration, all in the same directory. And if that still is not enough to please your application, you can use an activation script to execute code on every boot or system configuration change to do something useful.
So... stepping back, looking at the infrastructure NixOS actually provides you with to avoid statefulness. No, breaking this rule lightly and uncalled for is not acceptable.

Is there any documentation or otherwise documented consensus that explicitely forbids -- or frowns upon -- references pointing out of the Nix store? What is the rational to avoid such references in general? Although I still consider myself a novice w.r.t. Nix packaging, the ban on outside references stands in stark contrast to everything in nixpkgs I've seen so far. On the other hand, not using the symlink would hamstring many of my use cases. So please understand my skepticism and excuse my insistence.

I can only refer you to IRC or the discourse forum for further guidance. But anyway, could you please concisely list which of your use cases it would hinder to not set up the symlink but to follow the approach I showed you? I think, I haven't understood your problem domain, yet.

The safety and comfort of a NixOS module is independent of the usage of /etc. This pull request uses a module to place the config file in /etc (similar to other software, e.g. sshd), providing the abstraction of the module. Using a wrapper and sidestepping /etc does not give any benefit over the symlink to a config file in /etc if that file is generated by the NixOS module. On non-NixOS systems -- and even on NixOS if root explicitely chooses to do so -- root may also provide a handmade config file. This would be a conscious decision by root, and I consider this possibility an improvement as it facilitates usage of tsm-client on non-NixOS systems. Of course, root has to ensure that the file is correct, but this is no different from other software using config files in /etc (irrespective of the distribution).

I hope, I could prove to you, that your conclusions are wrong, here. Sidestepping /etc is exactly what makes tsm-client in incarnation X differ from its incarnation Y. X references the configuration from X, and Y from Y. In your case, both would refer to the configuration of whatever system configuration is active at the time, e.g., Z. This brakes the consistency guarantees you may expect (and we strive to provide) from a NixOS system configuration.

The software would need it as TSM's API needs the DSMI_DIR environment variable containing a properly prepared dsm.sys (like the command-line client that needs a DSM_DIR). A package maintainer, not knowing the content of dsm.sys in advance, would face the same dilemma and would likely end up with the same symlink solution that I prefer, since this solution doesn't require the package to be rebuild whenever the configuration file dsm.sys needs modification (assuming that the maintainer uses another Nix recipe to prepare the package linking against tsm-client).

But it is the very responsibility of a package maintainer to correctly instate the configuration of a package that depends on another one. If we could automate that task, we would do so. If it is as easy as providing every descendent with the right DSMI_DIR environment variable, it's even a no-brainer (maybe not trivial to write, just the architectural idea is). You could write a package setup hook in tsm-client that does all the magic to the application binaries whenever you add tsm-client to the descendant's buildInputs. That is exactly what happens when using cmake or autobuild or... In this sense, linking to /etc from inside the /nix/store breaks a second rule. A nix package should be described declaratively. You describe the result you want to achieve and let nix care for how to get there. You reference content, not locations. The locations that provide you with the content are resolved by nix.

Why would I add procps to runtimeDependencies? TSM doesn't link against procps (not sure if that would be possible at all), but TSM actually calls the executable ps, so ps must be in the PATH. The wrappers in this pull request accomplish exactly that.

Because it adds procps to the runtime closure of tsm-client. So, whenever somebody installs tsm-client, procps also get's installed. Thus, procps is always in PATH for tsm-client. And yes, the wrappers do that. It is an optimization to avoid double-wrapping but not a necessary thing to do. Actually, calling tsm-client is in no way performance critical (in the sense of "is usually called 1000 times per second"). Just go ahead, leave your wrappers (except DSM_DIR) and wrap again in the nixos module.

I'm still convinced that the symlink to /etc is a dependable solution, nevertheless (or maybe for exactly that reason), I'd like to know if the pull request in its current version could also cover your use cases.

I am not, but I think, it would cover my use cases.
I am always willing to share my insights.

Thanks for your response that helped me understand your objections more clearly.  I updated the request in order to tweak some minor details (explained below).  However, irrespective of other packaging details, I am still confused by the interplay of symlinks and purity:

My argument goes against what I wrote just above.  File-based references, i.e., softlinks and hardlinks.

vim, bash, tmux, ssh{,d}, git, and many more consult configuration files in /etc when invoked.

These are not references of the same kind.  The main difference between the two is in expectations.  Resolving a configuration file from inside a running program is in the responsibility of the program itself.  Providing a filesystem without broken links is not.

This leaves me with the impression that it is considered alright when a program looks into /etc to find configuration as long as there is no symlink (or hardlink) involved.  However, this one contests the impression:

A package should not reference anything outside the known scope of the nix package manager with very few exceptions, e.g., runtime data (usually databases of any kind).  And even that is a concession to practicality.  I give you the very idea of NixOS:

The system is (ideally) stateless.
Consequently, a /nix/store path to a system configuration uniquely identifies the complete closure of the system, its applications, their configuration.
Consequently, calling an application from the closure of a particular system configuration always yields the same result (when executed on the same machine).
This even holds when a different system configuration was active in the meantime.

[...]
I hope, I could prove to you, that your conclusions are wrong, here. Sidestepping /etc is exactly what makes tsm-client in incarnation X differ from its incarnation Y.  X references the configuration from X, and Y from Y.  In your case, both would refer to the configuration of whatever system configuration is active at the time, e.g., Z.  This brakes the consistency guarantees you may expect (and we strive to provide) from a NixOS system configuration.

These definitions are independent of the usage of symlinks/hardlinks.  In particular, by this definition, the programs listed above (and many more) suffer impurity.  E.g. the command ssh hostname still depends on whatever the administrator has written in  /etc/ssh/ssh_config.  By that standard, even Nix itself is impure as it consults /etc/nix/nix.conf.
I still can't grasp why dependence on outside state is acceptable unless a symlink is involved.  While a filesystem with a "broken symlink" (as criticised above) isn't nice, it won't do any additional harm when compared to a missing config file that is referenced without passing through a symlink. For instance:

sshd works alright if a proper configuration file is placed in /etc/ssh/sshd_config as documented by the man page.  If it is missing, sshd says so and fails.
dsmc on classical Linux distributions works alright if a proper configuration file is placed in TSM's directory structure as ordered by the documentation.  If it is missing, dsmc says so and fails.
dsmc as declared in this pull request works alright if a proper configuration file is placed in /etc/tsm-client/ as denoted in the package's description.  If it is missing, dsmc says so and fails.

All those behaviours are certainly impure to the degree that local configuration is not included in the package itself.  Yet, it matches most users' expectations, and it works flawlessly on NixOS and on non-NixOS systems.  How the config file is discovered in the end is an implementation detail that is of no interest to the admin as long as where the file is to be placed is documented.
To aid the admin, I added a hint about the configuration file's placement in the longDescription of the package.  Also, I expanded the module to create the API's configuration file as well.

We avoid statefulness in /etc (configuration files in general) as much as possible by:

Directly referencing the constructed configuration file in the /nix/store via command-line switches etc.,
Wrapping binaries, so that they point to the right incarnation of the configuration file or, at least, by
Making /etc/ a symlink to a /nix/store path.

We even support copying the file from the /nix/store to /etc and setting its owner/mode bits on each start if an application really does not accept a symlink (for god's sakes!).  Look at the openafs module for an instance of this.  Also look at the cups module for the fireworks we burn to provide a writable state directory, yet a stateless configuration, all in the same directory.  And if that still is not enough to please your application, you can use an activation script to execute code on every boot or system configuration change to do something useful.
So... stepping back, looking at the infrastructure NixOS actually provides you with to avoid statefulness.  No, breaking this rule lightly and uncalled for is not acceptable.

This leaves me confused.  This pull request uses the method advertised in the third bullet point of yours to spare the admin the hassle of having to prepare the config file by hand.  So, if you want to ensure that TSM's config file is in the Nix store, this pull request satisfies your needs.  By using the NixOS module in this pull request, you will have a config file that is generated in the store and then referenced by a symlink in /etc.  Note that only on non-NixOS systems, where the NixOS module is not applicable, the actual place of the configuration file is in /etc; but even there the administrator may still decide to symlink back into the Nix store from /etc.

I can only refer you to IRC or the discourse forum for further guidance.  But anyway, could you please concisely list which of your use cases it would hinder to not set up the symlink but to follow the approach I showed you?  I think, I haven't understood your problem domain, yet.

As I'm still not sure how to balance practicability and purity here, I will consult Discourse.
Regarding the use cases I'd like to cover, they represent most combinations of

(a) NixOS and (b) Nix on non-NixOS
(a) command line client and (b) linking aginst the API
(a) called by root or systemd and (b) called by non-root user

in addition to

the config file contains several server stanzas
the config file sees regular changes

The combination 2b+3a is of no real interest to me, the combination 2a+3b is used occasionally only.  All other combinations are very important.
Talking about expectations, non-NixOS systems are a concern for me in particular, as admins are used to just modifying the config file and expecting the change to be active immediately (like adding a certificate to ssh_known_hosts is for ssh).  It would likely be a source of constant mishaps if the admin and/or users would have to remember to rebuild wrapper packages whenever the config changes.  Moreover, it would deprive users of the possibility to simply use nix-env to install a usable tsm-client (again, as is normal with ssh and most other tools, including also Apache's httpd, as discussed earlier).

But it is the very responsibility of a package maintainer to correctly instate the configuration of a package that depends on another one.
[...]
You could write a package setup hook in tsm-client that does all the magic to the application binaries whenever you add tsm-client to the descendant's buildInputs.

dsm.sys is a file the administrator of a system has to provide, not the package maintainer.  The package maintainer can only design the package such that providing a config file is as easy as possible for the administrator.  So while using a setup hook to supply descendent packages with DSMI_DIR-creating wrappers is a beautiful concept, we would ultimately wind up again with the question of the source/placement of the configuration file: Either baked into a wrapper package in the store, or discovered in a documented place outside of the store via a symlink.

Why would I add procps to runtimeDependencies? TSM doesn't link against procps (not sure if that would be possible at all), but TSM actually calls the executable ps, so ps must be in the PATH.

Because it adds procps to the runtime closure of tsm-client.  So, whenever somebody installs tsm-client, procps also get's installed. Thus, procps is always in PATH for tsm-client.

When trying what you suggested, dsmc responded sh: ps: command not found.  And this is consistent with the nixpkgs manual: runtimeDependencies are added to the RPATH of executables.  This is completely unrelated to the PATH environment variable.  So I will stick to adding procps to the PATH with makeWrapper (together with the optional packages jdk8 and acl).
Yet I want to thank you for reminding me of the runtimeDependencies argument.  As mentioned earlier in this discussion thread, TSM requires "libdevmapper.so v1.01 or higher" for "image backup and restore functions for LVM2 volumes".  autoPatchElf doesn't detect any dependency on libdevmapper.so by itself, so adding lvm2 (which contains libdevmapper) to buildInputs wouldn't make a difference.  I suspect TSM accesses libdevmapper only when needed, then likely through dlopen and dlsym.  Like adding acl to the PATH to enable ACL functions, adding lvm2 to runtimeDependencies should therefore enable LVM2 volume functions.  I updated the pull request to do so (and added null as default for optional packages), although I can't test it.  As with acl, I expect the increase in closure size to be too small to warrant a separate tsm-client-with-lvm package.

I am not, but I think, it would cover my use cases.
I am always willing to share my insights.

Thanks for you offer, and moreover thanks for the assessment of your use cases.  This feeds me with optimism regarding this pull request.

This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/symlink-to-etc-from-within-a-package-vs-purity/3257/1

Executive summary, see below for details:

Make the dsmc nixpkg provide wrapped binaries that point to the configuration in /etc and that set DSMC_DIR to a location in the nix store. Do not put a link to outside the nix store in there.
Make a nixos module that consistently creates the configuration file pointed to by the wrapper. On NixOS, this is the way, dsmc is installed, not via nix-env.

This should suite both, NixOS and non-NixOS cases without relying on this broken symlink.

These are not references of the same kind.  The main difference between the two is in expectations.  Resolving a configuration file from inside a running program is in the responsibility of the program itself.  Providing a filesystem without broken links is not.

This leaves me with the impression that it is considered alright when a program looks into /etc to find configuration as long as there is no symlink (or hardlink) involved. However, this one contests the impression:

It is considered acceptable, because it is not sustainable to make all applications work without hardcoded configuration files. Maybe it helps to look at it from a different angle. You actually don't care where a configuration file is stored on the harddrive. What you do care about is, making the application access the right configuration when it is run. Having a hardcoded location for a configuration file, is a broken, old-school way of providing this functionality. Whenever you can tell your application where to find the right configuration at calltime, you don't need a hardcoded location. This is what we do on NixOS, if possible. There are usage scenarios, where it is important for a user to know what's inside a configuration file. But this is an abuse of hardcoded paths for providing a stable interface altogether. Ideally you'd have a way to ask your application or operating system for providing this information.
Again, the question is, why are hardcoded paths a broken interface? Answer, they lack the notion of multiple (and different) consistent application states. This cannot be done using hardcoded paths. This is the reason why there is a separate /nix/store with weird paths that are not under user control.

[...]
These definitions are independent of the usage of symlinks/hardlinks. In particular, by this definition, the programs listed above (and many more) suffer impurity. E.g. the command ssh hostname still depends on whatever the administrator has written in /etc/ssh/ssh_config. By that standard, even Nix itself is impure as it consults /etc/nix/nix.conf.

Yes, they are independent of symlinks/hardlinks. On NixOS, programs generally do not suffer from impurity, because, to stay with you example, the administrator cannot write into /etc/ssh/ssh_config. It is a symlink into the nix store and, thus, (re-)generated by activating a particular system configuration. The same is true for Nix. If you read my statements closely, you will see, that I never stated that the way NixOS implements /etc was flawed or stateful or violates above definitions.

I still can't grasp why dependence on outside state is acceptable unless a symlink is involved. While a filesystem with a "broken symlink" (as criticised above) isn't nice, it won't do any additional harm when compared to a missing config file that is referenced without passing through a symlink. For instance:

What I wanted to convey with my previous remarks, was the following: Nothing you say is plain wrong, but on a NixOS system, we want more guarantees. You are arguing towards not providing these guarantees in the way that I seem acceptable (aka. what I think the NixOS community finds acceptable). Nothing more, nothing less.

sshd works alright if a proper configuration file is placed in /etc/ssh/sshd_config as documented by the man page.  If it is missing, sshd says so and fails.

You're comparing apples and oranges, here. On a NixOS system this is exactly not how you use sshd. On a NixOS system you activate its module and it provides two things for you:

The running service, whatever is necessary for it; stuff in PATH, systemd service etc.
A configuration file that is consistent with the administrators configuration and the necessities of this particular sshd version. For instance, for sometime, the module added an option to the configuration file that prevented a known security issue with sshd, thus, providing safe defaults.

This said, if you want a PR to be merged into NixOS, you should at least accept that this is the way software is used. No ad hoc hacking in /etc, no stateful installation of software. And, this is how you wrote your PR. So, everything is fine on that side of the story. :)

dsmc as declared in this pull request works alright if a proper configuration file is placed in /etc/tsm-client/ as denoted in the package's description.  If it is missing, dsmc says so and fails.

This is not good enough. If possible and sustainable, dsmc, particularly because it is more likely to be used as a regular service than a time-to-time user application, should be used via a proper nixos module. It never happens that dsmc on NixOS does not have a proper configuration file. This has nothing to do whether it is residing in /etc or not. This also makes providing nix-env support a low priority. Providing the wrapper from inside the nixos module does not do any harm. So, I fail to see why you argument so hard against it... I hope you see, that it at least does provide the benefits I described earlier.

All those behaviours are certainly impure to the degree that local configuration is not included in the package itself. Yet, it matches most users' expectations, and it works flawlessly on NixOS and on non-NixOS systems. How the config file is discovered in the end is an implementation detail that is of no interest to the admin as long as where the file is to be placed is documented.

True, not good enough. NixOS provides functionality, not inconsistent stuff that must be made consistent as an afterthought. This is where almost all other operating systems fail today.

This leaves me confused. This pull request uses the method advertised in the third bullet point of yours to spare the admin the hassle of having to prepare the config file by hand....

True, I never argued against this part of your implementation. I merely shared my insights into the architecture of NixOS with you. I argued against putting a broken symlink into the nix store.

Regarding the use cases I'd like to cover, they represent most combinations of

(a) NixOS and (b) Nix on non-NixOS
(a) command line client and (b) linking aginst the API
(a) called by root or systemd and (b) called by non-root user

in addition to

the config file contains several server stanzas
the config file sees regular changes

The combination 2b+3a is of no real interest to me, the combination 2a+3b is used occasionally only. All other combinations are very important.

Thanks for sharing.

Talking about expectations, non-NixOS systems are a concern for me in particular, as admins are used to just modifying the config file and expecting the change to be active immediately (like adding a certificate to ssh_known_hosts is for ssh). It would likely be a source of constant mishaps if the admin and/or users would have to remember to rebuild wrapper packages whenever the config changes. Moreover, it would deprive users of the possibility to simply use nix-env to install a usable tsm-client (again, as is normal with ssh and most other tools, including also Apache's httpd, as discussed earlier).

The expectation of ad hoc hacking configuration is not the focus of NixOS. If possible without violating purity, you may write the package in a way to also provide useful non-NixOS behaviour. If not, non-NixOS is not the focus, and folks there have to find a way around the issues. For your particular case, there is no such problem if you follow my suggestions. Non-NixOS systems can happily use dsmc with a wrapper that points to the config file in /etc.

dsm.sys is a file the administrator of a system has to provide, not the package maintainer. The package maintainer can only design the package such that providing a config file is as easy as possible for the administrator.

This is true only for non-NixOS systems. On NixOS, both, the application and the configuration are provided consistently at the same time. It is, thus, in the responsibility of the package maintainer to provide the necessary nixos module mechanics. Other distributions like Debian also expect from you to provide sane default configuration for a service you want to integrate in their system.

[...] So while using a setup hook to supply descendent packages with DSMI_DIR-creating wrappers is a beautiful concept, we would ultimately wind up again with the question of the source/placement of the configuration file: Either baked into a wrapper package in the store, or discovered in a documented place outside of the store via a symlink.

True. The decision has been made, wrap the binaries. No symlinks pointing outside the store. This decision hinges on filesystems not providing multiple consistent states on hardcoded paths.

Why would I add procps to runtimeDependencies? TSM doesn't link against procps (not sure if that would be possible at all), but TSM actually calls the executable ps, so ps must be in the PATH.

Because it adds procps to the runtime closure of tsm-client.  So, whenever somebody installs tsm-client, procps also get's installed. Thus, procps is always in PATH for tsm-client.

When trying what you suggested, dsmc responded sh: ps: command not found. And this is consistent with the nixpkgs manual: runtimeDependencies are added to the RPATH of executables. This is completely unrelated to the PATH environment variable. So I will stick to adding procps to the PATH with makeWrapper (together with the optional packages jdk8 and acl).

Good, good! Do this! I was not sure myself. :)

Yet I want to thank you for reminding me of the runtimeDependencies argument. As mentioned earlier in this discussion thread, TSM requires "libdevmapper.so v1.01 or higher" for "image backup and restore functions for LVM2 volumes". autoPatchElf doesn't detect any dependency on libdevmapper.so by itself, so adding lvm2 (which contains libdevmapper) to buildInputs wouldn't make a difference. I suspect TSM accesses libdevmapper only when needed, then likely through dlopen and dlsym. Like adding acl to the PATH to enable ACL functions, adding lvm2 to runtimeDependencies should therefore enable LVM2 volume functions. I updated the pull request to do so (and added null as default for optional packages), although I can't test it. As with acl, I expect the increase in closure size to be too small to warrant a separate tsm-client-with-lvm package.

Sounds good!

The pull request received a major overhaul:

The package now accepts paths to the config files as arguments.
The module uses these arguments to have the package point directly to the config file created by the module.

Details of changes in the package:

The package -- internally -- consists of an "unwrapped" derivation and a "wrapper" derivation.  The "wrapper" derivation is ultimatelly returned/used.
The "unwrapped" derivation is similar to the old package.  However, no dsm.sys is added to the installation directories (where DSM_DIR and DSMI_DIR have to point to) at all.  Binaries are left untouched (not enclosed in wrappers).  Hence, the "unwrapped" derivation can't be used directly.
The "wrapper" derivation is a collection of symlinks into the "unwrapped" one.  To each installation directory, the builder also adds symlinks that point to the config file paths that have been provided as build arguments.  Wrapper scripts are created to help the executables find the DSM_DIR installation directory.  Whenever the path to the configuration file changes, only the "wrapper" derivation will be rebuild.
If installed with nix-env -i on other distros, no paths can be set.  In that case, the paths default to /etc/..., as before.

Details of changes in the module:

pkgs.writeText is used to create the config file.  Its path is passed to pkgs.tsm-client as config file path.  The resulting package is used in the system environment and for the backup service.  /etc is no longer touched or referenced.
The text of the config file is also made available via an additional option.  API users can use the module, possibly define additional servers, and feed the configuration into their own API-linking packages, in order to sidestep /etc as well.
(Not related to the /etc issue): To enable users to install the tsm-client-withGui package with the module, I added a package option.

Dear @spacefrogg,
first of all, thanks again for your hints towards a better implementation.  I revised the pull request based on ideas I received on discourse and on what I distilled from your latest response.  Could you have a look at the current revision to judge whether the design satisfies purity requirements?
I tried to understand and follow your suggestions, but I'm still not sure if I graps your goals correctly.  In case the current design is not acceptable, this is, essentially, where I'm struggling to follow your response:

On NixOS, programs generally do not suffer from impurity, because, to stay with you example, the administrator cannot write into /etc/ssh/ssh_config.  It is a symlink into the nix store and, thus, (re-)generated by activating a particular system configuration.

Just to be sure so we're not talking cross purposes: In my initial design the files /etc/tsm-client/{cli,api}.dsm.sys were, like /etc/ssh/ssh_config (and /etc/ssh/sshd_config for the server) generated by the module in my PR and hence yielded symlinks pointing from /etc back into the store as well.  So, assuming ssh/sshd are based on well-designed modules, I can't see where my approach deviates.

Dear @spacefrogg, I updated the pull request and applied your requests (and responded to non-request comments).  Two further remarks are in order:

IBM published a new TSM client about a week ago (<IP_ADDRESS>).  The pull request has been updated to use the latest version.
The module has been split into a "program" module and a "service" module, as requested.  The "program" module creates the final, wrapped, tsm client package based on the module's configuration.  Since the "service" module needs the wrapped package to call the executable dsmc, the "program" module makes the wrapped package available via the new wrappedPackage option.

Tiny update: By looking at xfig in https://nixos.org/nixos/packages.html#xfig I realized that XML tags are no good in longDescription.

looks really nice so far, thanks for the hard work!

I saw Docbook tags being used in NixOS module descriptions - I assumed that was also possible for package descriptions - cc @grahamc

It seems thaht a subset of the XML is supported for options: https://github.com/NixOS/nixos-homepage/blob/master/nixos/options.tt#L279 but is not applied to packages. It'd be nice to port this over and support it in both!

It would be lovely if this could be integrated into https://github.com/NixOS/nixos-homepage/pull/209 (or a followup PR after that)


Rename "Doctrine" to "Doctrine ORM"

"Doctrine" is the name of the organization, "Doctrine ORM" is the tool in question.

Good point, thank you :+1:


adding some autoated test to the action

I created few tests for the action so that the macOS error that I'm currently facing don't reappear by accident.

create GitHub actions triggered by push/PR to master branch
install the checkout action
test installation process
verify version number
use a dummy test text and verify the output
special case for macOS to check a version < 3.1.1 (when the .pkg file name changed)

Challenges:

the macOS build don't work. I already created a patch that may solved it in #5 I think it's safe to ignore it in this PR
the tested text in windows don't return the expected result. I'm no pandoc expert, is it normal ?
the latest version is currently hard coded I don't want to spend more time on this PR at the moment but maybe open an issue to dynamically wire the latest one.

Demo:
I checked it on my side: https://github.com/12rambau/setup-pandoc/actions/runs/6417462490

This is very cool! Thanks!


EKS operator's logs flooded with error message when cluster is in error state

Rancher Server Setup

Rancher version: 2.6-head commit id: 628b1b9
Installation option (Docker install/Helm Chart): HA install - RKE1 local cluster

Information about the Cluster

Kubernetes version: 1.21
Cluster Type (Local/Downstream): EKS v2 cluster

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom) Admin

Describe the bug
EKS operator's logs flooded with error message when cluster is in error state
To Reproduce

Deploy an EKS v2 cluster
In the node group section, specific an AMI.
The cluster is deployed, but fails to come up Active in Rancher

Result

Error on Rancher UI : Waiting for API to be available:ResourceInUseException: Nodegroup cannot be updated as it is currently not in Active State { RespMetadata: { StatusCode: 409, RequestID: "x-x-x-x" }, ClusterName: "israeleksubuntucluster", Message_: "Nodegroup cannot be updated as it is currently not in Active State", NodegroupName: "israelubuntunodegroup" }
EKS operator logs are spammed every second with logs:

time="2022-05-09T20:59:37Z" level=error msg="error syncing 'cattle-global-data/c-sgdtw': handler eks-controller: ResourceInUseException: Nodegroup cannot be updated as it is currently not in Active State\n{\n RespMetadata: {\n StatusCode: 409,\n RequestID: \"0bad6cf5-9ed5-471f-97af-47a44fe08257\"\n },\n ClusterName: \"israeleksubuntucluster\",\n Message_: \"Nodegroup cannot be updated as it is currently not in Active State\",\n NodegroupName: \"israelubuntunodegroup\"\n}, requeuing"
Mon, May 9 2022 1:59:38 pm | time="2022-05-09T20:59:38Z" level=error msg="error syncing 'cattle-global-data/c-sgdtw': handler eks-controller: ResourceInUseException: Nodegroup cannot be updated as it is currently not in Active State\n{\n RespMetadata: {\n StatusCode: 409,\n RequestID: \"f38d5f2f-111a-4687-942e-86d125390beb\"\n },\n ClusterName: \"israeleksubuntucluster\",\n Message_: \"Nodegroup cannot be updated as it is currently not in Active State\",\n NodegroupName: \"israelubuntunodegroup\"\n}, requeuing"
Mon, May 9 2022 1:59:38 pm | time="2022-05-09T20:59:38Z" level=error msg="error syncing 'cattle-global-data/c-sgdtw': handler eks-controller: ResourceInUseException: Nodegroup cannot be updated as it is currently not in Active State\n{\n RespMetadata: {\n StatusCode: 409,\n RequestID: \"93989b6f-40a5-4133-aefb-75c14dda479f\"\n },\n ClusterName: \"israeleksubuntucluster\",\n Message_: \"Nodegroup cannot be updated as it is currently not in Active State\",\n NodegroupName: \"israelubuntunodegroup\"\n}, requeuing"
Mon, May 9 2022 1:59:39 pm | time="2022-05-09T20:59:39Z" level=error msg="error syncing 'cattle-global-data/c-sgdtw': handler eks-controller: ResourceInUseException: Nodegroup cannot be updated as it is currently not in Active State\n{\n RespMetadata: {\n StatusCode: 409,\n RequestID: \"f87d89c3-0ad6-4fd8-9c81-cffd269f3076\"\n },\n ClusterName: \"israeleksubuntucluster\",\n Message_: \"Nodegroup cannot be updated as it is currently not in Active State\",\n NodegroupName: \"israelubuntunodegroup\"\n}, requeuing"
Mon, May 9 2022 1:59:40 pm

Expected Result
EKS operator should not retry every second. There should be a backoff interval

EKS operator utilizes wrangler, which has a default backoff builtin. If this backoff is not being utilized, then there may be something causing the object to get updated, which skirts the backoff.

Both @sowmyav27 and I reproduced this on rancher 2.6.5.
When provisioning an EKS cluster via the rancher UI using a custom ubuntu AMI ID and a node volume size of < 30GB, this err appears because rancher expects a node disk size of >= 30GB for snapshots.

I updated the disk size to 30GB. Then the expected err appeared.

EKS operator logs were spammed with that error.

So wrangler has a controller timeout of 15 mins but since the eks-config-handler in the eks-operator is spamming the logs every second, that backoff interval is definitely not being used. When the operator spams the logs with that err, this line is hit, and via the comments.. what I think is happening is: the EKS operator assumes that a failure to update a node group is a currently pending update, so it logs the error message, then immediately sends the object back into the controller where the controller waits for the update 'to finish'.

Turns out, the behavior seen in this case results from the fact that the error message that is returned from EKS has metadata with a unique requestID in the body. Since the error message is different every time, the Status continues to get updated every second. Will modify PR.

Testing template
Root cause
When a call to UpdateNodeGroupVersion fails due to a custom AMI version causing node provisioning to fail in EKS, the failure is expected but lasso spams the EKS logs every second. This happens because the error message that is returned from EKS has metadata with a unique requestID in the body. Since the error message is different every time, this code is skipped and the Status gets updated every time which reenqueues the handler immediately instead of doing a backoff reenqueue and is spamming the logs.
What was fixed, or what changes have occurred
My solution is to remove the metadata from the EKS logs
What happens now when an EKS error is logged?

Raw err message is logged
Remove metadata from the error. If removing metadata fails because the EKS error is not in the expected format, continue with the original logic to prevent a regression.  The new error string will contain the cluster name, error message, and node group.
If the Status doesn't already contain the error (without metadata), add the error to the Status.

Areas or cases that should be tested

Go to rainbow cluster
Scale eks-operator deployment replicas to 0
Run an instance of https://github.com/rancher/eks-operator in golang with theKUBECONFIG env var to the kubeconfig for the rainbow cluster. This runs a local version of eks-operator on rainbow
Provision an EKS cluster via the rancher UI with a custom AMI and node gb of 30 instead of the default 20. You should see the following formatted error appear

What areas could experience regressions ?
Are the repro steps accurate/minimal ?

Verified in v2.6-head https://github.com/rancher/rancher/commit/069433410f08903ae5967e5178867c2622bce419.
ENVIRONMENT DETAILS

Rancher Install: Helm
Rancher version: v2.6-head https://github.com/rancher/rancher/commit/069433410f08903ae5967e5178867c2622bce419
Browser: Chrome

TEST RESULT
PASS
VERIFICATION STEPS

Setup Rancher and navigate to the UI in a browser.
Created a standard user and logged into Rancher as that user.
Provisioned a hosted Amazon EKS cluster. Ensured that the cluster met the following criteria:

Custom Ubuntu AMI
Node volume size = 20GB

Monitored the Provisioning Log in the cluster and ensured no unwarranted messages appear.
Noted the following error message:

This is expected as per the repro steps highlighted earlier in this issue. What's important is that there was NOT a flood of EKS operator messages present in the logging.

Repeat steps 3-5, but ensured that the Node volume size > 30GB:

The above is from colleague's testing effort to this issue as for some reason, my AMIs were not yielding successful result.
Verified that no flooding of EKS operator messages.

If the error returned from EKS is not in that format, the metadata removal will fail and get logged here so this bug may appear for future error formats we are unaware of.

Are there any workarounds to this? I'm seeing similar behaviour on Rancher 2.7.6 for a missing LaunchInstanceTemplateVersion on EKS:

time="2024-04-16T14:26:51Z" level=error msg="error syncing 'cattle-global-data/c-94tkb': handler eks-controller: error creating nodegroup: InvalidParameterException: Launch template could not be found : Could not find the specified version 3 for the launch template with ID lt-xxxxxxxx.\n{\n  RespMetadata: {\n    StatusCode: 400,\n    RequestID: "cfed5af4-343f-4c2d-80bb-60d273af2125"\n  },\n  Message_: "Launch template could not be found : Could not find the specified version 3 for the launch template with ID lt-xxxxxxxxx."\n}, requeuing"


Added item to the release checklist.

Once a stable branch is created, it needs to be added
to readthedocs so that PDF and ePub files can be generated.
Refs #24568

@ericholscher, currently we have to contact you to manually add the version since our branch names ("stable-1.8.x"" do not match our desired URLs "1.8.x"). Should this be the documented solution? If so, what's your preferred contact method?

Is there a reason you can't just use the stable branch name? It is what gets slugified automatically, and since users are never directly hitting the URLs, merely being served a download, it seems like the slug shouldn't matter?

Isn't it also used for URLs like http://django.readthedocs.org/en/1.8.x/ and other places the version is displayed (like the version switcher)? We could certainly switch if it's a hassle for you, but we'll have to make an update on our end to vary the URL based on the version.

Sure. If there was a standard way of truncating the slug that would work as
well. I don't know if we can really start doing something like real_slug =
slug.split('/')[-1] or something. We could special case django, but that
doesn't seem like a great solution either. The other option is to allow for
user-editable slugs on versions, but that is hard because we are sync'ing
with VCS, and so we could need to track state there. I think the best
solution is probably some kind of alias built on top of the real, existing
slug.
I can fix the version for now, but hopefully we'll have a proper solution
to this before the next release :)
On Thu, Apr 2, 2015 at 10:41 AM, Tim Graham<EMAIL_ADDRESS>wrote:

Isn't it also used for URLs like http://django.readthedocs.org/en/1.8.x/
and other places the version is displayed (like the version switcher)? We
could certainly switch if it's a hassle for you, but we'll have to make an
update on our end to vary the URL based on the version.
—
Reply to this email directly or view it on GitHub
https://github.com/django/django/pull/4436#issuecomment-88985907.

--
Eric Holscher
Maker of the internet residing in Portland, Or
http://ericholscher.com

Thanks Eric. HTML/ePub is working but not the PDF. We had the same issue for 1.7. Not sure if the root cause is the same or different.

I also noticed that the other PDFs haven't updated since October 2014. I guess that's probably related to the root cause:

https://media.readthedocs.org/pdf/django/1.7.x/django.pdf
https://media.readthedocs.org/pdf/django/latest/django.pdf

Merged in 418f75d55f2612066b5bd27171026c20af317716 and created ticket in readthedocs tracker for the PDF issue: https://github.com/rtfd/readthedocs.org/issues/1219


Fix build

Deploying doesn't work because of build issues:
https://github.com/aceade/sydfjords/actions/runs/11103175895

Poking around online found this issue where somebody had the same problem. Updating vue-tsc to V2 fixes it.


chore: update next with master

Description
This PR updates the next branch with the latest master changes.

This PR shall not be merged with squash and merge otherwise it will solve the commit issues going forward.


Store in every Event the content in JSON format that the Producer emit.

Store in every Event the content in JSON format that the Producer emit.

https://github.com/broker-game/broker-client/pull/31


[WIP] [#3] Skip CI if PR title starts with 'WIP'

Notes for Reviewer

Pre-Review Checklist for the PR Author

[ ] Add sensible notes for the reviewer
[ ] PR title is short, expressive and meaningful
[ ] Relevant issues are linked in the References section
[ ] Every source code file has a copyright header with SPDX-License-Identifier: Apache-2.0 OR MIT
[ ] Branch follows the naming format (iox2-123-introduce-posix-ipc-example)
[ ] Commits messages are according to this guideline

[ ] Commit messages have the issue ID ([#123] Add posix ipc example)
[ ] Commit author matches Eclipse Contributor Agreement (and ECA is signed)

[ ] Tests follow the best practice for testing
[ ] Changelog updated in the unreleased section including API breaking changes
[ ] Assign PR to reviewer
[ ] All checks have passed (except task-list-completed)

Checklist for the PR Reviewer

[ ] Commits are properly organized and messages are according to the guideline
[ ] Unit tests have been written for new behavior
[ ] Public API is documented
[ ] PR title describes the changes

Post-review Checklist for the PR Author

[ ] All open points are addressed and tracked via issues

References

Relates to #3

@elBoberido I would like to merge this. Can we do this?

@elfenpiff this does not work as expected. I'd like to be able to manually trigger the CI after the PT title changed but unfortunately, when the initially failed PR re-runs, it checks the old PR title and it is not possible to trigger the CI without having a dummy push after removing the [WIP]


CloudKidStudios PixiParticles wrappers

New pull request to develop. Is that good?

All Good :)

Great :) Thanks!

@halfabench Do you want me to release this to haxelib now or wait until we have a sample in place.

Well, I'll try to add some samples this week. So It's up to you :)

I know you haven't tested AnimatedParticle and I just found compiler issues with that and other classes as well.
pixi/plugins/particles/cloudkid/AnimatedParticle.hx:27: characters 51-52 : Unexpected )
pixi/plugins/particles/cloudkid/ParticleUtils.hx:27: characters 43-44 : Unexpected <
pixi/plugins/particles/cloudkid/ParticleUtils.hx:57: characters 15-61 : Class not found : pixi.core.Point
pixi/plugins/particles/cloudkid/ParticleUtils.hx:65: characters 15-57 : Type required for extern classes and interfaces
pixi/plugins/particles/cloudkid/PathParticle.hx:54: characters 1-28 : Type required for extern classes and interfaces

Don't worry I just fixed them, so please make sure you update your branch accordingly.

I know it's very hard to test each and every bit of functionality in the externs but please try to a little bit careful :)
I feel that creating samples and making them part of the build will at least ensure the core functionality is always working.
Just released v2.2.3 to haxelib. http://lib.haxe.org/p/pixijs

Sorry! You're absolutely right. I should have tested again that everything was ok. So I will definitely add some samples and try them before submitting updates. I'll be much more careful next time.
Anyway thanks for the release and thanks for the good work. I really enjoy working with haxe+pixi.js. It's such a breathe :+1:

Question: where do you want me to put the samples? In plugins/particles or in samples?

in samples

ok.


Create Haskell DSL for CDDL

We need a DSL that we can specify our CDDL spec in and use it to:

Generate CDDL files
Generate random binary data and test our decoders

The new DSL has been implemented as cuddle library and a huddle tool in the https://github.com/input-output-hk/cuddle repo
Here are the tickets that track using the new tooling:

#4126
#4125

This ticket can be considered done


Create class EntitySet

Create the class EntitySet, a Set implementation that compares added elements by their id.

Maybe add EntityMap, a Map implementation that its key is always its id.


Add VAM scene files to this repository to keep in sync

Good be important in case of changing/adding communication features, so app and scene keep compatibel. Must solve this: How to move scene into VAM folder automatically?

I understand what you want with this, like the app generating the scene file maybe so people don't have to use the var, (or maybe I misunderstand), but I think having the .var separate is no problem. I make sure the jsons in the var have the same version number as the app so I always know which version people are using.

That is fine, but I just want to keep the scene files inside the repository to guarantee compatibility.
Distribution is another thing. For releases distribute the scene in VAM hub and the app seperately as cx_frozen "compiled" app. But because the users always have to download the app outside of VAM itself, I think having the scene inside the downloadable app zip would also be okay.
I notice that my English is much worse than I thought -- sorry! :D


Clarify repositories structure in install kubeadm document

We received feedback that it's not clear enough that there's a dedicated package repository for each Kubernetes minor version (see #42810). This is a regression from #43407 and this PR is supposed to address this concern.
/assign @sftim @saschagrunert @cpanato @jeremyrickard
cc @kubernetes/release-engineering

I found a formatting issue in instructions for Red Had-based distros:

The issue is also present on k8s.io: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
I fixed this issue in the latest commit.

@Affan-7 I addressed your comments in the latest commit
/hold cancel

@sftim Addressed your comments, PTAL again

/lgtm
Ideally: squash to fewer commits

@sftim Squashed the PR to one commit, PTAL again

Yep, still looks good to me (Prow has kept the lgtm label).

Thanks.
/approve


CalendarDay - onKeyDown is not a function

Preconditions:

Using v16
Using SingleDatePicker Component
Entering the CalendarDay by pressing the down key

Pressing a key results in CalendarDay onKeyDown is not a function

Thanks, this is definitely a bug

oh no! will fix asap!


Update alizer dependency

What type of PR is this:
What does this PR do / why we need it:
Update Alizer to the latest commit https://github.com/redhat-developer/alizer/commit/ad2d3056c08d511e132c67ed8e353a8fc67d5927
Which issue(s) this PR fixes:
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

How to test changes / Special notes to the reviewer:

Added a comment on the related issue: #6210 (comment)

oh, you are right.
I've removed "fix" from the issue description. This PR doesn't fix that issue, but I think that updating alizer still makes sense.

License validation seems to fail. Not sure why wwhrd could not find the Apache 2.0 license for that indirect dependency..
Do we want to add an exception for this dependency?

I'll look into that


network: kata generates netlink sanity check warnings in dmesg

Description of problem
Whilst trying to debug [], I found a bunch of these in my dmesg:
[ 6382.972629] netlink: 'kata-runtime': attribute type 4 has an invalid length.

That is coming from what looks like a kernel netlink sanity check function
We should probably track this down - it might be a symptom of something?
Expected result
No kata warn or errors in dmesg would be nice :-)

kata-env info below. Note, I believe vsock is on here (but is not listed as such in the output... odd):
[Meta]
  Version = "1.0.12"

[Runtime]
  Debug = false
  [Runtime.Version]
    Semver = "1.1.0"
    Commit = "bf1cf68"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0 (v2.11.0-rc0-311-g36c0224931-dirty)\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  Msize9p = 8192
  Debug = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-2018-10-26-10:33:49.086322915+0100-osbuilder-b9b9410-agent-e395ac6"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67-16"
  Parameters = "agent.log=debug"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-6ddb006ad3f709cab018af9dc0bf9e756c3ce2cd"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-5fbf1f0919ce0bb1f2b7e85692cdf3058023926f"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.17.3-100.fc27.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  [Host.Distro]
    Name = "Fedora"
    Version = "27"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz"

I don't reproduce this warning.

Ah, OK. Maybe it is setup or host distro/kernel specific. I'll have to try to reproduce and confirm when I get back to my desk.  I think I saw this whilst trying to reproduce the vsock leak issue. So, I likely had vsock on, and could have been using either rootfs or initrd. Host was likely on a single ipv4 wired connection.


ISPN-7300 Operator '-' doesn't work in fulltext querying

https://issues.jboss.org/browse/ISPN-7300

LGTM, have a couple of questions though

@anistor ok, thanks for the clarification

Integrated, thanks @anistor !


Integrate ServiceWorker

Fix #3
Please confirm this PR when you are not busy!!
Done

[x] Add ServiceWorker Setting by using workbox-cli
[x] Fix manifest.json

I'm so sorry that the difference is very large 🙇
So, I'm OK 🙆♂  if this PR will not be merged.
The reason is ejection (43a2b8b)
Maybe, mainly confirm the following commits.

daf37e4
5fba19d
ad790e5
5d5fec8

Carefully!!
If you confirm the effect of Service Worker, you should  prepare for the staging environment (apply for https). You can't confirm the effect on the development environment.
Maybe, it is a little hard to prepare for staging environment,
so I think this PR should be merged when the access is less.
In my case, I debug by adding the almost same workbox configuration in the development webpack configuration.  (Configuration)
The following table shows the problem is solved and the score is improving by caching.

Before
After

But, it was difficult to make both src/img/*.png, gif or png cache and public/logo.png, favicon.io chache. (This is because webpack-dev-server doesn't bundle a real file)
So, in the case of  production, I used workbox-cli in order to make sw.js.
This sw.js create these caches at the same time. (you can confirm build/sw.js)

Hi, @nd-02110114 にっしーさん,
Thank you very much for this PR and the detailed comments + commit messages!
I will take a good look at it after I'm done with my exams, or perhaps after I fly back to Taiwan (5/13).
However, personally, I don't want to eject create-react-app so soon (at least within 3-4 months), so I'm wondering if there is another way to develop PWA without ejecting?
I think PWA is a fantastic idea and would definitely like to ask for your advice/contribution. It's just that currently there are some back-end modifications I want to make, and I would prefer not ejecting first since it is irreversible. I will probably merge this PR after I get most things settled.
Thank you for contributing again and I'll share my detailed thoughts after testing your PR on my local machine later this month!

Thank you for your prompt comments 😄

I will take a good look at it after I'm done with my exams, or perhaps after I fly back to Taiwan (5/13)

I got it ! First of all, you concentrate on your exam.  Good Luck!!!
And, I hope you could have a great time in Taiwan 👍
My PR is not bugfix, so I don't care  whether this PR is merged or not 🙆♂
It is not necessary to hurry!

However, personally, I don't want to eject create-react-app so soon (at least within 3-4 months), so I'm wondering if there is another way to develop PWA without ejecting?

It is OK! I submitted the another PR(#5) without ejecting.
If you confirm the effect of ServiceWorker on the production environment,
it is better to merge #5 PR.

It's just that currently there are some back-end modifications I want to make, and I would prefer not ejecting first since i is irreversible. I will probably merge this PR after I get most things settled.

Go for it ! I think some api responses are slow.
I expect that these api response is improved 😄 😄

Thank you for contributing again and I'll share my detailed thoughts after testing your PR on my local machine later this month!

Please ask me if there is something that you do not know !

Please refer to PR #5


Failed to obtain images from the video.

Failed to obtain images from the video.
SaveExporter('/data/dstdataset'): 0it [00:00, ?it/s]  ERROR:root:Video '/data/videos/18.mp4' is invalid, skipped
VideoSource('/data/videos/18.mp4'): 0it [00:03, ?it/s]
SaveExporter('/data/dstdataset'): 0it [00:03, ?it/s]
The video can be played, and there is no problem with the path.
从视频里面获取图像失败，
视频可以播放，路径没有问题。

closed due to a misusage.


docs: prevent mDNS browsing when unicast records are available

This PR adjusts the discovery procedure to ensure that when unicast records are available, these are used in preference to any mDNS records which may be available on the same network. This holds true unless the Node supports a configuration parameter for the user to choose between unicast and multicast DNS.

LGTM


Patch for Issue 1203

@dwightguth : there seem to be some issues with this change; using krewrite as such triggers a timeout in the unparsing tests. any suggestions on how to better address this?


One color per multiple labels results in gradients

🐛 (Probably a) Bug
Related to #716
Problem
I have a segmentation image, which I want to color by some metadata. Let's say, each segment id has a property A or B assigned. To color segments by properties, I supply a dictionary to the labels layer that looks like:
color_dict =  {0: 'black', 1: 'red', 2: 'red', 3: 'blue', 4: 'blue', etc}
viewer.add_labels(segmentation, color=color_dict)
Note that in comparison to #716, I have one color per multiple labels.
That is a sample resulting image I get:

As you can notice, there are gradients of red and blue present (the purple cells). I've tried the very same with Image layer and vispy Colormap as described here and I still get the gradients. Where do they come from? It is not such a big issue for two colors, but once I have more it gets worse.
Expected behavior
Only blue, red and black colors on the image. No gradients
Minimal reproducible example
def get_weird_gradients():
    sample_array = np.arange(100)
    np.random.shuffle(sample_array)
    labels_image = sample_array.reshape(10,10)
    colors = ['black', 'red', 'blue']

    color_map = {i: colors[np.random.randint(3)]
                 for i in range(100)}

    with napari.gui_qt():
        viewer = napari.Viewer()
        viewer.add_labels(labels_image, color=color_map)

Environment
napari: 0.3.6
Platform: Linux-4.17.9-041709-generic-x86_64-with-debian-buster-sid
Python: 3.7.6 | packaged by conda-forge | (default, Jun 1 2020, 18:57:50) [GCC 7.5.0]
Qt: 5.12.5
PyQt5: 5.12.3
NumPy: 1.18.4
SciPy: 1.4.1
Dask: 2.17.2
VisPy: 0.6.4
GL version: 4.6.0 NVIDIA 440.100
MAX_TEXTURE_SIZE: 32768
Plugins:

napari-plugin-engine: 0.1.5
svg: 0.1.3

Additional context
Alternatively, is there any better way to color by metadata?
cc @constantinpape

Alternatively, is there any better way to color by metadata?

For some context here, @vzinche and me have been looking into similar problems of coloring labels by data associated with the labels.
I think a good solution for this would be to color via the properties, which are also available in the labels layer now. As far as I can see there's no option to explicitly color by the properties right now.
See also #1516.

Thanks so much for flagging this, it does look like a bug from #1339 - maybe @ziyangczi can take a look.
It could either be in the colormap calculation here, https://github.com/napari/napari/blob/master/napari/utils/colormaps/colormaps.py#L119-L140 or in the mapping of integer labels to floats here https://github.com/napari/napari/blob/master/napari/layers/labels/labels.py#L556-L565. Somewhere things are getting out of sync.
It will be good to figure out where and add a test.

I think a good solution for this would be to color via the properties, which are also available in the labels layer now. As far as I can see there's no option to explicitly color by the properties right now.

I agree. I don't think we've supported that yet, but that was definitely one of the goals and probably the long term better approach. There was some discussion of that here by @kevinyamauchi https://github.com/napari/napari/pull/1339#issuecomment-639650482

should be fixed with https://github.com/napari/napari/pull/1537, feel free to try it out!

Closed by #1537 @constantinpape @vzinche please re-open if this isn't fixed as expected.


fix(dashboard): correct prometheus rule for OTHER Power Consumption by Pods

This commit fixes the issue where the wrong Prometheus rule was referenced for displaying OTHER Power Consumption(W) by Pods in the dashboard

Verified the same on OpenShift


require('highland') returns nothing

For whatever reason in the Node REPL, require('highland') fails:
var _ = require('highland');
console.log(_); // undefined

Additionally highland seems to be writing to the _ variable on the global scope by default:
require('highland');
console.log(_); // stuff!

That's a quirk of the Node REPL. _ is a special variable that refers to the result of the previous expression. Using a different variable name will work.

Ahhhh. Thanks.


Update placeholder images for non-existant images

Current placeholder looks bad. New placeholders should be silhouettes of a pet

Resolved with d9b62ca96af2d48efea0d1e6a073481ed8d9b590


i have some problems in downloading model.zip.

In experiment 4, the second downloading link has something wrong.
When i running this commend: wget -cO - https://cloudstor.aarnet.edu.au/plus/s/kya3ZjF5N3WqLVL/download > ./data/models.zip,
the commend telling me 403：Forbidden.
When i opening that link, the webpage telling me "Access denied. Download of a folder is not allowed."

Hi, thanks for reporting this!
@oravus do you know by any chance if this URL is still active?

Hi @Raywoo1, can you please try manually downloading through that link but without the download at the end, i.e., https://cloudstor.aarnet.edu.au/plus/s/kya3ZjF5N3WqLVL/
[Cloudstor, where the files are stored, is going to be decommissioned; thus they might have disabled folder downloads, we will host data somewhere else, but hope that the reduced link works for you for now.]

Dear @Attila94 and @oravus, I appreciate your efforts in providing a solution. However, when attempting to access the cloudstor, a prompt requesting me to select a university to log in appeared. Therefore, I kindly request that you provide me with a new link. Thank you very much for your assistance!

@Raywoo1, can you please try the following:
wget "$(curl -s 'https://universityofadelaide.app.box.com/index.php?folder_id=216398644945&q%5Bshared_item%5D%5Bshared_name%5D=3gu4beadjj3t2iulz63w1jzj6tmc0bve&rm=box_v2_zip_shared_folder' | grep -o '"download_url":"[^"]*' | cut -d':' -f2- | tr -d '"' | cut -c 11- | sed 's/\\\//\//g')" -O models.zip

@Raywoo1, can you please try the following: wget "$(curl -s 'https://universityofadelaide.app.box.com/index.php?folder_id=216398644945&q%5Bshared_item%5D%5Bshared_name%5D=3gu4beadjj3t2iulz63w1jzj6tmc0bve&rm=box_v2_zip_shared_folder' | grep -o '"download_url":"[^"]*' | cut -d':' -f2- | tr -d '"' | cut -c 11- | sed 's/\\\//\//g')" -O models.zip

Thanks, bro. I have tried this link, however, there is an error like http://: Invalid host name.

I have tried this link, however, there is an error like http://: Invalid host name.

that seems to be due to missing curl, can you install it: sudo apt install curl and then run the full download command I had shared.

I have tried this link, however, there is an error like http://: Invalid host name.

that seems to be due to missing curl, can you install it: sudo apt install curl and then run the full download command I had shared.

I have checked the curl. That is ok. The question is like this

That is strange, I successfully tested the command through two different networks.
Can you please check if you are able to manually download through this link (no need to sign up)?

That is strange, I successfully tested the command through two different networks. Can you please check if you are able to manually download through this link (no need to sign up)?

hhh, this link is working now, thank you so much again!!!

Great, but that (long) command is still giving error?

yes, and the error is still 'http://: Invalid host name'

@Attila94, can you please try that at your end?

@Attila94, can you please try that at your end?

Works on my end!


Group.LazyRun

LazyRun will allow to be called in TestMain, and Run container when Group.Run called first time in any Test* func.
After run lazily, the container will be reused in any other tests.

If the container is used by multi-tests over multi-packages concurrently, we need shared controller for exclusive control (another container, or lock file?)

Group.LazyRun and Group.Use is implemented in 0954f3ce21a26bec0a7d15fcea0540f6f9907230 .


What is ParagraphExit?

Hi, there!
This is not bug report, and this is a question about @textlint/ast-node-types.
That package several node types related to markdown syntax.
However, that package also contains ParagraphExit used in nowhere.
Why such types are provided?
Since it makes the developer misslead, I suggest to remove them if you have no reason.
Thank you.

*:exit is not actual AST node's type.
The rule creator use Syntax.DocumentExit instead of ${Syntax.Paragrap}:exit.
https://github.com/textlint/textlint/issues/267
It is just constant value for *:Exit
Exit constant is for traversing of AST.
https://textlint.github.io/docs/rule.html#docsNav

I see. Thank you for explanation.
Could you add some note in JSDOC about this topic?

I should notice you that eslint will report warning when we write switch-case clause for ASTNodeTypes even if we did not use.  Thus, we should write the rule for each node types or default: now. In my feeling, this is  unintuitive.
https://github.com/typescript-eslint/typescript-eslint/pull/972/files


ClassNotFoundException While ShuttingDown Apache Felix Using H2

Environment
HikariCP version: 2.6.2
JDK version     : 1.8.0_121
Database        : H2
Driver version  : 1.4.195

I got the same error as #298, even the DataSource is closed explicty. I use apache karaf 4.0.9, HikariCP 2.6.2 and H2 database 1.4.196. The maven project to reproduce the bug is attached.
The steps to reproduce the bug are as follows (hikaricp-h2-bug*.jar will be generated contained in the attached file):

Unzip the attached zip-file, and then
mvn clean install
Download the apache-karaf from https://repository.apache.org/content/groups/public/org/apache/karaf/apache-karaf/4.0.9/apache-karaf-4.0.9.tar.gz
Unzip the tgz-file
Start the karaf
bin/karaf
In the karaf console, execute the following commands:

karaf@root()> karaf@root()> install mvn:com.h2database/h2/1.4.196
Bundle ID: 52
karaf@root()> install mvn:com.zaxxer/HikariCP/2.6.2
Bundle ID: 53
karaf@root()> install mvn:foo/hikaricp-h2-bug/0.1.0-SNAPSHOT
Bundle ID: 54
karaf@root()> start 54
Created H2 DataSoruce
karaf@root()> list
START LEVEL 100 , List Threshold: 50
ID | State    | Lvl | Version        | Name
---------------------------------------------------------
52 | Resolved |  80 | 1.4.196        | H2 Database Engine
53 | Resolved |  80 | 2.6.2          | HikariCP
54 | Active   |  80 | 0.1.0.SNAPSHOT | hikaricp-h2-bug
karaf@root()> uninstall 54
Closing H2 DataSoruce
Closed H2 DataSoruce
karaf@root()> shutdown -f
karaf@root()>
Exception in thread "Thread-18" java.lang.NoClassDefFoundError: org/h2/mvstore/MVMap$2
	at org.h2.mvstore.MVMap.entrySet(MVMap.java:870)
	at org.h2.store.LobStorageMap.removeAllForTable(LobStorageMap.java:314)
	at org.h2.engine.Database.removeOrphanedLobs(Database.java:1340)
	at org.h2.engine.Database.close(Database.java:1261)
	at org.h2.engine.DatabaseCloser.run(DatabaseCloser.java:63)
Caused by: java.lang.ClassNotFoundException: Unable to load class 'org.h2.mvstore.MVMap$2' because the bundle wiring for org.h2 is no longer valid.
	at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1515)
	at org.apache.felix.framework.BundleWiringImpl.access$200(BundleWiringImpl.java:79)
	at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1958)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 5 more

hikaricp-h2-bug.zip

@xipki Thanks for the zip, but I don't have a Karaf installation, so it doesn't help that much.

@brettwooldridge Thanks for excellent project. I updated the issue description.


build(elixir): move default config to mix.exs file

Config values from config/config.exs are only loaded when building
from the app directory.
If the app is built as a dependency, configuration should be specified
in the mix.exs file.

To test these changes impact, use the branch name hairyhum/ockam-app-default-config in examples/elixir/get_started/setup.exs and run an example


✨ feat: anonymous ok (#1010)

Checklist

[ ] The title of my PR starts with a Conventional Commit prefix (fix:, feat:, docs: etc).
[ ] I have read the Contributor Guide and followed the process outlined for submitting PRs.
[ ] I have updated/added tests for new/updated/fixed functionality.
[ ] I have updated/added dartdoc comments with ///.

Breaking Change

[ ] Yes, this is a breaking change.
[ ] No, this is not a breaking change.

Related Issues

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (48d5b85) 80.18% compared to head (1acc13b) 81.21%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1011      +/-   ##
==========================================
+ Coverage   80.18%   81.21%   +1.03%
==========================================
  Files         302       47     -255
  Lines        3674      724    -2950
==========================================
- Hits         2946      588    -2358
+ Misses        728      136     -592

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Allow data collection script output to be hidden

The output of the data collection script is designed to be pasted directly into a github issue.
That results in a lot of output so using a cute trick which @dylanzr introduced us to, allow the output of the script to be hidden / unhidden by clicking on a "twistie". See the example at the top of https://github.com/kata-containers/runtime/issues/1347.

argh, embedded HTML ?? :-/ - but, yes, given it hides the huge volume of data... that will help with the PR readability. And, as we cannot (en)force the data being added as an attachment rather than a paste... this is probably a step in a good direction...


Geom collection trait attempt 3

disallow nested geometry collections (i.e. Geometry cannot hold GeometryCollections)
try to add in multiple lifetimes as suggested in https://github.com/kylebarron/geoarrow-rs/issues/172#issuecomment-1692441676

Still getting
error[E0597]: `inner_geom` does not live long enough
  --> src/io/wkb/writer/geometrycollection.rs:22:34
   |
16 | pub fn geometry_collection_wkb_size<'a: 'iter, 'iter>(
   |                                     -- lifetime `'a` defined here
...
21 |     for inner_geom in geom.geometries() {
   |         ---------- binding `inner_geom` declared here
22 |         sum += geometry_wkb_size(&inner_geom);
   |                ------------------^^^^^^^^^^^-
   |                |                 |
   |                |                 borrowed value does not live long enough
   |                argument requires that `inner_geom` is borrowed for `'a`
23 |     }
   |     - `inner_geom` dropped here while still borrowed

This was magically resolved!!! https://github.com/geoarrow/geoarrow-rs/issues/172#issuecomment-1857186075


[TACHYON-1249] Remove unnecessary isLeaderMode() from MasterBase.java

https://tachyon.atlassian.net/browse/TACHYON-1249

LGTM

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/6755/
Test PASSed.

+1 LGTM