The test_gmtime test produces a lot of output.

test-runs/test_gmdiff Is producing a lot of output.
The output of testsuite "HARNESS_VERBOSE=yes" is 49MiB in my configuration. The output of
 90-test_gmdiff.t ...................
 # The results of this test will end up in test-runs/test_gmdiff
 1..1
     # Subtest: ../../test/gmdifftest
     1..1000000
     ok 1 - test_gmtime - iteration 1
     ok 2 - test_gmtime - iteration 2
…
     ok 1000000 - test_gmtime - iteration 1000000
 ../../util/wrap.pl ../../test/gmdifftest => 0
 ok 1 - running gmdifftest
 ok

is 45MiB. Is it possible to shink this somehow, please? I have HARNESS_VERBOSE enabled so I see on a build box once something goes wrong without the need to re-run the whole thing. It helps a lot especially if the test does not fail on each run.
Sebastian

You could run them normally, scrape the output for failures and re-run those tests verbosely.

It runs automatically on a build box without any interaction. And if this a "one time" thing I lose everything.
I could this logic to the part where make test is handled but this would increase the build time in error case. The build on armhf takes ~1h, mip64el ~3h.
Now that I found VERBOSE_FAILURE would it do what I want? Be quiet and print verbose output of the test that failed?
Sebastian

Now that I found VERBOSE_FAILURE would it do what I want? Be quiet and print verbose output of the test that failed?

Probably.  See test/README.md

Yes. thank you.
For some reason I was using HARNESS_VERBOSE and a grep for it did not lead to test/README.md.
Thanks again.
Sebastian


Render Blok Components within RichText

Description
Right now, there's no built-in way to render Blok components within rich text.  Sure, there are several "solutions" thrown around, but they all have issues:

https://gist.github.com/SebbeJohansson/31b64c5c4f914abfec2660210ec7795a This only renders bloks at the top level. If one is embedded deeper in the hierarchy, it fails.
https://www.storyblok.com/faq/how-to-render-richtext-nuxt I shouldn't be required to have the runtime template compiler to make this work.
https://www.storyblok.com/tp/nuxt-js-multilanguage-website-tutorial This is recommending using a package that hasn't been updated in years. It's missing some features now and requires you to register the components that can be used, rather than using auto-imports.

The Astro Storyblok SDK has implemented this directly. I'm honestly quite surprised this hasn't been done for a while now.
Suggested solution or improvement
Implement a RichText component that will render blok component where bloks are used. Also, it would be nice to allow the rich text resolver to return VNodes so we can render custom components there, but if the first thing is implemented, I'm not sure what use I'd have for this.
Additional context
No response
Validations

[X] Follow our Code of Conduct

Hi, @joezimjs, you are completely right in your statement; thanks for sharing your concerns with us. Actually, the Astro implementation is our first try at introducing this in our SDKs (I see you fast there 😏). As Astro is kind of new as an SDK in our list, it is easy to test new initiatives (it is less used by our clients) to then introduce this in the rest of the SDKs, so sooner than later, this will be implemented to the Nuxt SDK too (@alexjoverm do you want to add anything here?).
For now, about the links you mention, the 3rd one you should ignore it since we are using Nuxt 3 here, and that article is for the old Nuxt 2. For the FAQ and the runtime template compiler, for now, this is a working approach. I know it is not the ideal case, but for now, it is the simplest way.
As we always say, this is a community effort, we are trying to provide the most features we can, but we don't have a dedicated team for the SDKs. We help when we have less workload, so please, if any day you want to do a pair programming session with me or one of my colleagues or open a PR with the features you need we will be so happy to see it happening!

I got something working in my own repository by creating a custom RichText component: https://github.com/joezimjs/nuxt-storyblok-poc/tree/main/components/rich-text
Basically converted everything into Vue components rather than using functions that return HTML. And the RTText component (for text nodes) handles all the different mark types. I'm not sure how to integrate something like this into storyblok-nuxt, primary because I'm not sure which of the auto-import features are available inside a plugin.

Hmm... now that I'm thinking about it, there may be a better way that relies on the marks/nodes functionality that is defined for renderRichText from @storyblok/js, which would allow this to automatically stay up-to-date if new node/mark types are added.

There! I changed it so it's all packed in one component and it takes advantage of the provided RichTextSchema and even allows a user to pass in a different schema. https://github.com/joezimjs/nuxt-storyblok-poc/blob/main/components/RichText.ts

I've done a very simple fix for this:
<script setup lang="ts">
type Props = { text: any };
defineProps<Props>();

const renderText = (text: any) => {
	const markup = { type: 'doc', content: [text] };
	return renderRichText(markup);
};
</script>

<template>
	<template v-if="text?.content" v-for="element in text?.content">
		<template v-if="element.type === 'blok'" v-for="blok in element?.attrs?.body" :key="blok.id">
			<component :is="blok.component" :blok />
		</template>
		<div v-else v-html="renderText(element)"></div>
	</template>
</template>

I haven't thoroughly texted it yet but seems to be working.

@PP-Tom Looks pretty good but I don't think this will catch bloks that are embedded in a lower level of structure. For example, if you put a blok into a bullet point, the bulleted list will hit the renderRichText call and the blok inside it will never be seen by this component. It's the same problem as the first example I linked in my original post.

Good point. I'm really surprised that StoryBlok's UI is built in Vue but it's been almost a year for a 1st party solution for this. I know it's a DX thing, but just surprised.

Good point. I'm really surprised that StoryBlok's UI is built in Vue but it's been almost a year for a 1st party solution for this. I know it's a DX thing, but just surprised.

Right?!?

Hi @joezimjs @PP-Tom @Youhan jumping here to give some good news on Richtext topic.
I'm currently working on a framework agnostic solution for the Richtext Renderer that is potentially extendable to each major framework (vue, react etc). Between all the features the use-case or rendering Blok components is covered and was successfully tested on the PoC. And yes, it includes as well overwriting resolves like adding the NuxtLink 😉
The community has been very patient about this topic so I want to be transparent that there a work in progress solution for this, we just need to wait a little bit more so it's rolled out since we are discussing internally the DX.
Hope this brings a little bit of light to this topic, will keep you posted.

Love to hear it!

Hai melompat ke sini untuk memberikan kabar baik tentang topik.Richtext
Saat ini saya sedang mengerjakan solusi agnostik kerangka kerja untuk Richtext Renderer yang berpotensi dapat diperluas ke setiap kerangka kerja utama (vue, react, dll). Di antara semua fitur, kasus penggunaan untuk merender komponen Blok tercakup dan berhasil diuji pada PoC. Dan ya, itu termasuk juga menimpa resolusi seperti menambahkan 😉NuxtLink
Komunitas telah sangat sabar tentang topik ini jadi saya ingin transparan bahwa ada solusi yang sedang dalam proses untuk ini, kita hanya perlu menunggu sedikit lagi sehingga diluncurkan karena kita sedang mendiskusikan DX secara internal.
Semoga ini membawa sedikit cahaya ke topik ini, akan membuat Anda tetap diposting.

So glad to hear that 😇

Sorry to bother this community but I wasted a lot of time trying to know why StoryblokRichText component wasn't getting imported in Nuxt when I check this issue.
It was so confusing to have Rich Text component example available on the documentation to see there is not implementation yet.
<StoryblokRichText :doc="blok.description" />

Please don't do that before implementing something because developers trust documentation.
I was getting:
Failed to resolve component: StoryblokRichText
If this is a native custom element, make sure to exclude it from component resolution via compilerOptions.isCustomElement.

Hi @moracabanas, the module should be exporting the StoryblokRichText component from version 6.2.0 on. I prepared a small reproduction to show it is in fact the case https://stackblitz.com/~/edit/nuxt-starter-cwrbwd?file=app.vue
Could you please check if you are using the latest version of the module? If that's the case could you provide me a minimal reproduction link so I can help you?
Thanks

@joezimjs Did you have a chance to test the new component? https://stackblitz.com/~/edit/nuxt-starter-cwrbwd?file=app.vue

Hi @moracabanas, the module should be exporting the StoryblokRichText component from version 6.2.0 on. I prepared a small reproduction to show it is in fact the case https://stackblitz.com/~/edit/nuxt-starter-cwrbwd?file=app.vue
Could you please check if you are using the latest version of the module? If that's the case could you provide me a minimal reproduction link so I can help you?
Thanks

That was my fault..
At first I considered to upgrade "@storyblok/nuxt" and it ended up in "^5.6.12"
I will try to upgrade to 6.
Thanks to point me to the right direction I will try that and see.

@moracabanas no worries, we are here to help 😊, let me know if you find any issue in the future.

Hi, @alvarosabu! I also tested StoryblokRichText in my project and it handles blok components just fine, but I get the Invalid watch source: undefined warning. It also appears in your reproduction as can be seen on a screenshot.
Also the real-time changes in Storyblok's visual editor don't seem to work properly when using this component, so I think it's worth to note that.

@alvarosabu I'm at v6.0.4 and I'm getting this issue. The StoryblokRichText component doesn't render. There are no warnings in the console. The component itself renders and is visible in vue devtools, however none of its content is being rendered. I noticed that resolvers are undefined, however. The example shown in this module's documentation doesn't pass any resolvers to the component.

I'm running v6 and I can't get custom resolvers to work no matter what. I've tried this every way to Sunday - it should be as straightforward as passing the richtext field's contents to a wrapper component but so far my resolvers are all being either completely ignored, or throwing a MarkTypes is undefined error. Hope I'm just missing something obvious here. https://stackblitz.com/edit/github-ugbuh1-mtyebacy?file=app.vue,package.json


make Camelia a hot link

make Camelia a hot link to perl6.org

Thanks. Fixed in https://github.com/perl6/modules.perl6.org/commit/f10acaadf5023eef1444f966e3a7216ca4271c73


use internal for constant variables, use uint

I would prefer to keep using uint256 as it is more explicit.

will be merging as discussed offline


regeneratorRuntime is not defined

Hi, I'm getting this error everytime I try to use redux-oidc. I've added it to the entry in the webpack.config.js and also imported it in the entry to my app. Any idea?

Yes. You need to include teh [babel-polyfill(https://babeljs.io/docs/usage/polyfill/) in your build configuration.

Yes, I've got that right I think. Don't worry, noone else has had this issue. It will be something silly on my end

The babel-polyfill is a strange beast. You either need to have it in your webpack entry like in the example-app or import it once using an ES6 import statement in your index.js. I know that I had my issues with this. Best of luck :-).

Were you able to solve this issue?

Yes, sorry. I wasn't sure of the protocol for closing the issue

No problem :)

How did you resolve this? I have added an import statement to my index.js but am still getting the error.

You have to do one of the following:
Add an import statement to index.js like this:
import 'babel-polyfill';
OR
add it to your webpack config entry file like in the example app.
Hope this helps,

I couldn't get it to work until I made
import 'babel-polyfill';
the first import statement in the file. I had to use the import statement because I cannot easily modify my webpack.config file as I am using a seed project from Facebook that essentially abstracts all configuration from the developer.
To be clear I will provide the exact code I use in my application's entry point...
import 'babel-polyfill';
import React from 'react';
import ReactDOM from 'react-dom';
import { Provider } from 'react-redux';
import { OidcProvider } from 'redux-oidc';
import configureStore from './store/configureStore';
import userManager from './config/UserManager';
import AppRouter from './routes';
import './index.css';

const store = configureStore();

ReactDOM.render(
  <Provider store={store}>
      <OidcProvider store={store} userManager={userManager}>
        <AppRouter />
      </OidcProvider>
    </Provider>,
  document.getElementById('root')
);


upgrade protobuf

I need this for upgrading bazel
Motivation

Test plan

See included automated tests.

Tests and linters are failing here; I did a quick check and found that the linters are failing because we're failing in bazel build //tools:compdb, if that gives you an idea of where to start looking.

I was able to sidestep this for now. Some day we should upgrade bazel but it isn't a blocker for me yet. Sorry for the noise!


Flymake in-buffer faces

Discussed in https://github.com/catppuccin/emacs/discussions/203

Originally posted by gs-101 December 10, 2024

Why are the Flymake faces set for descriptions and notes customized, but not the ones used in regions (that is, the ones found on the buffer)?
I'm asking this as I tried to configure them with Catppuccin's colors with set-face-attribute, but had no success.

flymake-error uses the default value of "Red1", for example.

@gs-101 I'm not sure, I think merging #205 is okay


Invalid resource type sslcertificate

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppe
t::Parser::AST::Resource failed with error ArgumentError: Invalid resource type
sslcertificate at /etc/puppet/modules/...pp

resolved by
https://github.com/voxpupuli/puppet-sslcertificate/pull/29

Should this issue be closed?

Hi,
based on a discussion in https://groups.io/g/voxpupuli/message/449 we decided to archive this repository. I'm going to close all issues and PRs. If you're interested in maintaining the module, please respond to our mailinglist.


handle case with append tag in config.js

would you like this to be released in the 0.15 series, or the next release
is ok?
Il giorno gio 9 apr 2015 alle 17:37 Peter Elger<EMAIL_ADDRESS>ha scritto:

You can view, comment on, or merge this pull request online at:
https://github.com/nearform/nscale-docker-analyzer/pull/13
Commit Summary

handle case with append tag in config.js

File Changes

M lib/images.js
https://github.com/nearform/nscale-docker-analyzer/pull/13/files#diff-0
(10)

Patch Links:

https://github.com/nearform/nscale-docker-analyzer/pull/13.patch
https://github.com/nearform/nscale-docker-analyzer/pull/13.diff

—
Reply to this email directly or view it on GitHub
https://github.com/nearform/nscale-docker-analyzer/pull/13.


Get-DbaBackupHistory -Force doesn't work

the doc says it will work but when I try it, it fails with Parameter set cannot be resolved using the specified named parameters

hey steven,
it will work, but not with -Last.
PowerShell parametersets arent very friendly because of this exact error message. I'll ask stuart to make it pretty in another issue


feat: upgrade to ESLint v7, standard-with-typescript v18

This pulls in some new rules, upgraded peer-dependencies, and drops support for Node.js 8.x, so it's a breaking change

##[error]error trying to get list of pull request's commits
9
Bad credentials
10
HttpError: Bad credentials

Hmmmm 🤔

Closed by https://github.com/toddbluhm/ts-standard/pull/82


Autoload tests only in the dev environment

Tests don't need to be registered in the production autoloader.

Good catch, thanks @stof.


Crash

the game crashes with an injection, when connected, the game crashes again. what to do?

Assuming you're on 10.25, have the injector opened (optional with admin privilege for some regions' client) right from the beginning, the game will crash even if you do nothing. The workaround is wait until the game fully loaded, you are in the arena, only then run the injector. See PR #29, quote:

only works, when injecting whilst already being ingame btw ^^

Which server, champion?

Russian. Ahri, Fiddl, Kata.  These are the heroes I played when I tried to use the skin changer.


apacheHttpd: 2.4.46 -> 2.4.47

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/apache-httpd/versions.
meta.description for apacheHttpd is: "Apache HTTPD, the world's most popular web server"
meta.homepage for apacheHttpd is: "http://httpd.apache.org/"
meta.changelog for apacheHttpd is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/envvars passed the binary check.

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/htpasswd had a zero exit code or showed the expected version

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/htdigest had a zero exit code or showed the expected version

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/htdbm had a zero exit code or showed the expected version

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/ab passed the binary check.

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/logresolve passed the binary check.

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/httxt2dbm had a zero exit code or showed the expected version

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/htcacheclean had a zero exit code or showed the expected version

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/rotatelogs passed the binary check.

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/checkgid passed the binary check.

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/fcgistarter had a zero exit code or showed the expected version

Warning: no invocation of /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/dbmmanage had a zero exit code or showed the expected version

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/apachectl passed the binary check.

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/envvars-std passed the binary check.

/nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin/httpd passed the binary check.

5 of 15 passed binary check by having a zero exit code.

7 of 15 passed binary check by having the new version present in output.

found 2.4.47 with grep in /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47

found 2.4.47 in filename of file in /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47

directory tree listing: https://gist.github.com/35a49e2024046e0ae568abb718ce0319

du listing: https://gist.github.com/06fe0fbedec0c8f9f354a66abced6445

Rebuild report (if merged into master) (click to expand)

667 total rebuild path(s)

406 package rebuild(s)

406 x86_64-linux rebuild(s)
103 i686-linux rebuild(s)
55 x86_64-darwin rebuild(s)
103 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
adminer
apacheHttpd
apacheHttpdPackages.apacheHttpd
apacheHttpdPackages.mod_auth_mellon
apacheHttpdPackages.mod_ca
apacheHttpdPackages.mod_crl
apacheHttpdPackages.mod_csr
apacheHttpdPackages.mod_dnssd
apacheHttpdPackages.mod_fastcgi
apacheHttpdPackages.mod_ocsp
apacheHttpdPackages.mod_perl
apacheHttpdPackages.mod_pkcs12
apacheHttpdPackages.mod_python
apacheHttpdPackages.mod_scep
apacheHttpdPackages.mod_spkac
apacheHttpdPackages.mod_tile
apacheHttpdPackages.mod_timestamp
apacheHttpdPackages.mod_wsgi
apacheHttpdPackages.mod_wsgi2
apacheHttpdPackages.mod_wsgi3
apacheHttpdPackages.php
apacheHttpdPackages.subversion
apacheHttpdPackages_2_4.apacheHttpd
apacheHttpdPackages_2_4.mod_auth_mellon
apacheHttpdPackages_2_4.mod_ca
apacheHttpdPackages_2_4.mod_crl
apacheHttpdPackages_2_4.mod_csr
apacheHttpdPackages_2_4.mod_dnssd
apacheHttpdPackages_2_4.mod_fastcgi
apacheHttpdPackages_2_4.mod_ocsp
apacheHttpdPackages_2_4.mod_perl
apacheHttpdPackages_2_4.mod_pkcs12
apacheHttpdPackages_2_4.mod_python
apacheHttpdPackages_2_4.mod_scep
apacheHttpdPackages_2_4.mod_spkac
apacheHttpdPackages_2_4.mod_tile
apacheHttpdPackages_2_4.mod_timestamp
apacheHttpdPackages_2_4.mod_wsgi
apacheHttpdPackages_2_4.mod_wsgi2
apacheHttpdPackages_2_4.mod_wsgi3
apacheHttpdPackages_2_4.php
apacheHttpdPackages_2_4.subversion
apacheHttpd_2_4
arcanist
bookstack
drush
engelsystem
gnome3.gnome-control-center
gnome3.gnome-user-share
gnome3.gnome_control_center

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A apacheHttpd https://github.com/r-ryantm/nixpkgs/archive/fe563d71a37c2d05ebfc3d73b67e4758d781a61c.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47
ls -la /nix/store/vik96vn1rjsa9wh50zwcv18hx1kii9g0-apache-httpd-2.4.47/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings
cc @lovek323 @peti for testing.

Result of nixpkgs-review pr 121233 at fe563d71 run on x86_64-linux 1

17 packages marked as broken and skipped:

hhvm
libsForQt512.kcachegrind
php73Extensions.pcs
php73Extensions.v8
php73Extensions.v8js
php73Extensions.zmq
php73Packages.phpmd
php74Extensions.pcs
php74Extensions.pthreads
php74Extensions.v8
...

345 packages skipped due to time constraints:

adminer
apacheHttpdPackages.mod_auth_mellon (apacheHttpdPackages_2_4.mod_auth_mellon)
mod_ca (apacheHttpdPackages.mod_ca)
mod_crl (apacheHttpdPackages.mod_crl)
mod_csr (apacheHttpdPackages.mod_csr)
mod_dnssd (apacheHttpdPackages.mod_dnssd)
mod_fastcgi (apacheHttpdPackages.mod_fastcgi)
mod_ocsp (apacheHttpdPackages.mod_ocsp)
mod_pkcs12 (apacheHttpdPackages.mod_pkcs12)
mod_python (apacheHttpdPackages.mod_python)
...

2 packages built successfully:

apacheHttpd (apacheHttpdPackages.apacheHttpd ,apacheHttpdPackages_2_4.apacheHttpd ,apacheHttpd_2_4)
apacheHttpdPackages.mod_perl (apacheHttpdPackages_2_4.mod_perl)


Add date and time filter Criteria to configuration

Currently configuration  doesn't allow filtering by specific criteria. The criteria can be added in two levels:

At the parent object level that is applied to all child objects
At the specific child object level.

Closed in https://github.com/anandbn/timeline-lwc/commit/8f8bcf90f065796aae8dd6de2e020814ebf34a4c


Update aws-java-sdk-core to 1.12.744

About this PR
📦 Updates com.amazonaws:aws-java-sdk-core from 1.12.459 to 1.12.744
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.amazonaws", artifactId = "aws-java-sdk-core" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.amazonaws", artifactId = "aws-java-sdk-core" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #781.


Confusion about or new specification of module-level logging setup [re: crypto]?

Recently a logging.crypto key was added to openchain.yaml, which is interpreted in the file
obc-peer/openchain/crypto/crypto_settings.go

"crypto" here is what the logging package refers to as logging module. The other logging.* keys in openchain.yaml refer to subcommands of the obc-peer command. Module-level logging control in obc-peer subcommands is already implemented and documented - it is not necessary to add new configuration keys to control module-level logging.
Currently a module-level override specification is potentially required for each peer subcommand. However if the desire is to add a way to specify module-level overrides that apply to all obc-peer subcommands that would also be easy to do in a way that would not require defining a unique configuration key for each logging module.

I'm not sure I follow @bcbrock's phrasing here as closely as I'd like to (sorry Bishop...), and apologies in advance if I'm derailing this thread, but the issue that I'm seeing is this:
If you launch the membersrvc binary, the config file that viper reads is the membersrvc/membersrvc.yaml one. Then when we call into the crypto package, then Init() function attempts to read the key logging.crypto which is NOT present in membersrvc.yaml. (It is however present in core.yaml, but that config file is not loaded when running the membersrvc binary.)
As a result, we get a "Log level not recognized" error.
Would it be easy to have the membersrvc binary run on the same logic as the peer one when it comes to control its logging modules? (I believe that this is what Bishop is referring to.)
If not, can we at least add a logging.crypto key to membersrvc.yaml to make the error message go away?
Copying @adecaro who may be able to help on this.

I think I see the other issue that Bishop noted.
When running the peer binary, the crypto package does not respect the value set via the logging-level flag, and instead logs at the level set by the logging.crypto key.
This behavior is inconsistent with the other modules. Any way we could fix this?

@kchristidis Yes, your second point was my point. The YAML entries were supposed to provide default logging levels for 'peer' subcommands, and the syntax of the entry then allowed individual module level control. After this mechanism was introduced the "crypto" system was subsequently modified to take its logging level directly from the YAML file.

@kchristidis @bcbrock I can fix this (i.e. Make the crypto package follow the defined logging level).

@juliancarrivick-ibm Hi Julian, it would be great if you could fix this. Even better, if you have the time, would be to fix #653.

I'm currently having some environment issues which I'll hopefully resolve today, but yes I can look at #653 too.
As per @GrapeBaBa suggestion, I will probably just remove the explicit retrieval of the logging level in crypto_settings.go and core.yaml. If we want an override for module level logging then we can always add it later but this will make it consistent for now.

@juliancarrivick-ibm Thanks for looking into it!

There also seems to be some Docker logging which is not respecting the logging level set:
2016/07/12 03:57:08 Load docker HostConfig: &{Binds:[] CapAdd:[] CapDrop:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:host IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:0 MemorySwap:0 MemorySwappiness:0 OOMKillDisable:false CPUShares:0 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:0 CPUPeriod:0 BlkioWeight:0 Ulimits:[]}
I am going to look into that too.

@juliancarrivick-ibm that can be fixed if #2140 merged.

Ah, great! I was just about to say that it would probably be best fixed in another pull request so that saves me the trouble!


Removed Project with Dead Link

Removed Binary Machine project as repository link results is dead.
Fixed link to applications block.

Makes sense
I cannot see what change has been made here.

The link on line 13 is currently #applications, the section title is application (non plural). Have just dropped the s on line 13 to correctly link to the application section of this document.

Ah, makes sense. What is the change to line 42?

No edits where made on line 42, this is just a whitespace change

Sounds good. Thanks for the sharp eye!


Create a separate directory parallel to unit for version tests

This test fails all the time in development and other repos, since dev versions aren't >=0.6.0.

Is the idea that this test will run prior to release but after we've made the next tag? And if that isn't happening already do we need to change something about the github actions workflow?

Yes, something needs to change somewhere in the release process, but I'm not sure where.


[stable/mariadb] Feature/allow for existing pvc

What this PR does / why we need it:
Allows for users to add existing PVs/PVCs
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Doesn't so much fix an issue, but does allow for users to use the chart with an existing PV/PVC
Special notes for your reviewer:

@jldeen please bump the chart version and sign the DCO

/assign

@cpanato Chart version bumped and DCO signed

/ok-to-test

/retest

Believe the failing is related to open issue #148 on the bitnami/bitnami-docker-mariadb repo and is not related to the proposed changes in this PR.

Hi @juan131
I have made the proposed changes.

/retest


allow claimrewards after 86100 seconds

producer_pay.cpp, line 79:
check( ct - prod.last_claim_time > microseconds(useconds_per_day), "already claimed rewards within past day" );

this condition is a source of constant headache because one cannot establish a simple cron job to run every 24 hours. Please make it >= microseconds(useconds_per_day) - 300000000, so that it gives 300 seconds room for possible delays or microforks.
The rewards calculation does not rely on the fact that it's more than 24 hours. it will work fine for shorter periods too.

Moving this to the new reference system contract repo.
https://github.com/EOSIO/eosio.system/tree/v2.0.0-rc1 introduces a change to rewards which may make this issue a non-issue. We should revisit in a bit.


T5 out of memory

System Info

GPU 4 x A10G (EC2 g5.12xlarge) - memory 24GB
TRTLLM v0.12.0
torch 2.4.0
cuda 12.5.1
tensorrt 10.1
triton 24.04
modelopt 0.15

Who can help?
No response
Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
When running Flan-T5-XL, I get the following error:
Error:  2024-10-26 17:30:28 PyProcess - W-17921-test-stderr: [1,1]<stderr>:E1026 17:30:28.741998 17929 model_lifecycle.cc:638] failed to load '1c9ef0632367b6977384ee6179c5e0db60788391' version 1: Internal: unexpected error when creating modelInstanceState: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaMallocAsync(ptr, n, mCudaStream->get()): out of memory (/tmp/tensorrtllm_backend/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmBuffers.h:117)

In previous versions (e.g. v0.11.0), this problem did not happen. But in v0.12.0, it does.
To reproduce:
Convert checkpoint:
python3 convert_checkpoint.py --model_dir /tmp/flan-t5-xl/ --dtype bfloat16 --output_dir /tmp/trtllm_t5_ckpt/ --tp_size 4 --pp_size 1 --workers 4 --model_type t5

Build engine:
trtllm-build --checkpoint_dir /tmp/trtllm_t5_ckpt/encoder/ --log_level info --gemm_plugin bfloat16 --output_dir /tmp/.djl.ai/trtllm/07a75caac904529386b7aab25b551e6bd5d48bb0/9c57fb18d49e66cd903a6c1382315cbecd306463/1/encoder --workers 4 --gpt_attention_plugin bfloat16 --paged_kv_cache disable --context_fmha disable --max_beam_width 1 --remove_input_padding enable --use_paged_context_fmha disable --use_fp8_context_fmha disable --max_batch_size 256 --max_seq_len 1024 --max_num_tokens 16384 --enable_xqa disable --moe_plugin disable

trtllm-build --checkpoint_dir /tmp/trtllm_t5_ckpt/decoder/ --log_level info --gemm_plugin bfloat16 --output_dir /tmp/.djl.ai/trtllm/07a75caac904529386b7aab25b551e6bd5d48bb0/9c57fb18d49e66cd903a6c1382315cbecd306463/1/decoder --workers 4 --gpt_attention_plugin bfloat16 --paged_kv_cache enable --context_fmha disable --max_beam_width 1 --remove_input_padding enable --use_paged_context_fmha disable --use_fp8_context_fmha disable --max_batch_size 256 --max_seq_len 1024 --max_num_tokens 16384 --enable_xqa disable --moe_plugin disable --max_input_len 1 --max_encoder_input_len 1024

Then runtime:
mpirun --allow-run-as-root -np 4 python3 run.py --engine_dir /tmp/.djl.ai/trtllm/07a75caac904529386b7aab25b551e6bd5d48bb0/9c57fb18d49e66cd903a6c1382315cbecd306463/1/ --tokenizer_dir google/flan-t5-xl --max_output_len 64 --num_beams=1 --input_text "translate English to German: The house is wonderful."

The error will be:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 22.19 GiB of which 1.50 MiB is free. Process 641210 has 22.18 GiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Expected behavior
Expected behavior is no OOM like v0.11.0. I tested with v0.11.0 with similar commands and the error did not happen (the only difference  in the commands is following the breaking changes of v0.12, where tp_size and pp_size are set during checkpoint conversion, multi block mode during runtime, etc. I checked that all of these settings did not affect by also testing if adding --multi_block_mode true --enable_context_fmha_fp32_acc to run command would affect the result -- the OOM still happens.
actual behavior
Actual behavior is the OOM as described above. Please see the log attached in the 'additional notes' section for more information
additional notes
Full log when running TRTLLM

@ydm-amazon Which convert_checkpoint.py do you use?

@hello-11 Thanks for the quick response - I use the example one in examples/enc_dec.

Some more context, we specifically noticed that the gpu memory usage is about 2x higher with v12 compared to v11. I believe @ydm-amazon is observing something similar for v13, but i'll check and confirm. I reduced the batch size from 256 to 128, and with that we can successfully deploy, but memory is much higher.
For v11 vs v12 difference, i'm not sure if this behavior change is expected, or whether we are missing a configuration.
In v12, these are the commands we used to build the engine
# encoder
trtllm-build
--checkpoint_dir /tmp/trtllm_t5_ckpt/encoder/
--log_level info
--gemm_plugin bfloat16
--output_dir /tmp/trtengine/google-flan-t5-xl/1/encoder
--workers 4
--gpt_attention_plugin bfloat16
--paged_kv_cache disable
--context_fmha disable
--max_beam_width 1
--remove_input_padding enable
--use_paged_context_fmha disable
--use_fp8_context_fmha disable
--max_batch_size 256
--max_input_len 1024
--max_num_tokens 16384
--enable_xqa disable
--moe_plugin disable

# decoder
trtllm-build
--checkpoint_dir /tmp/trtllm_t5_ckpt/decoder/
--log_level info
--gemm_plugin bfloat16
--output_dir /tmp/trtengine/google-flan-t5-xl/1/decoder
--workers 4
--gpt_attention_plugin bfloat16
--paged_kv_cache enable
--context_fmha disable
--max_beam_width 1
--remove_input_padding enable
--use_paged_context_fmha disable
--use_fp8_context_fmha disable
--max_batch_size 128
--max_input_len 1
--max_num_tokens 16384
--enable_xqa disable
--moe_plugin disable
--max_encoder_input_len 1024
--max_seq_len 1024

In v11 we used:
# encoder
 trtllm-build
--tp_size 4
--pp_size 1
--checkpoint_dir /tmp/trtllm_t5_ckpt/encoder/
--log_level info
--gemm_plugin bfloat16
--output_dir /tmp/trtengine/google-flan-t5-xl/1/encoder
--workers 4
--gpt_attention_plugin bfloat16
--paged_kv_cache disable
--context_fmha disable
--max_beam_width 1
--remove_input_padding enable
--use_custom_all_reduce disable
--use_paged_context_fmha disable
--use_fp8_context_fmha disable
--max_batch_size 256
--max_input_len 1024
--max_num_tokens 16384
--enable_xqa disable
--moe_plugin disable

# decoder
trtllm-build
--tp_size 4
--pp_size 1
--checkpoint_dir /tmp/trtllm_t5_ckpt/decoder/
--log_level info
--gemm_plugin bfloat16
--output_dir /tmp/trtengine/google-flan-t5-xl/1/decoder
--workers 4
--gpt_attention_plugin bfloat16
--paged_kv_cache enable
--context_fmha disable
--max_beam_width 1
--remove_input_padding enable
--use_custom_all_reduce disable
--use_paged_context_fmha disable
--use_fp8_context_fmha disable
--max_batch_size 256
--max_input_len 1
--max_num_tokens 16384
--enable_xqa disable
--moe_plugin disable
--max_encoder_input_len 1024
--max_seq_len 1024

In both cases, we're using tp4 deploying on L4 gpus.
v12 output from nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:38:00.0 Off |                    0 |
| N/A   34C    P0             27W /   72W |   13426MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L4                      On  |   00000000:3A:00.0 Off |                    0 |
| N/A   30C    P0             27W /   72W |   13426MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L4                      On  |   00000000:3C:00.0 Off |                    0 |
| N/A   33C    P0             26W /   72W |   13426MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L4                      On  |   00000000:3E:00.0 Off |                    0 |
| N/A   32C    P0             27W /   72W |   13426MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    133162      C   python3                                     12650MiB |
|    0   N/A  N/A    133163      C   python3                                       248MiB |
|    0   N/A  N/A    133164      C   python3                                       248MiB |
|    0   N/A  N/A    133165      C   python3                                       248MiB |
|    1   N/A  N/A    133162      C   python3                                       248MiB |
|    1   N/A  N/A    133163      C   python3                                     12650MiB |
|    1   N/A  N/A    133164      C   python3                                       248MiB |
|    1   N/A  N/A    133165      C   python3                                       248MiB |
|    2   N/A  N/A    133162      C   python3                                       248MiB |
|    2   N/A  N/A    133163      C   python3                                       248MiB |
|    2   N/A  N/A    133164      C   python3                                     12650MiB |
|    2   N/A  N/A    133165      C   python3                                       248MiB |
|    3   N/A  N/A    133162      C   python3                                       248MiB |
|    3   N/A  N/A    133163      C   python3                                       248MiB |
|    3   N/A  N/A    133164      C   python3                                       248MiB |
|    3   N/A  N/A    133165      C   python3                                     12650MiB |
+-----------------------------------------------------------------------------------------+

v11 output from nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:38:00.0 Off |                    0 |
| N/A   38C    P0             27W /   72W |    7086MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L4                      On  |   00000000:3A:00.0 Off |                    0 |
| N/A   35C    P0             26W /   72W |    7086MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L4                      On  |   00000000:3C:00.0 Off |                    0 |
| N/A   37C    P0             26W /   72W |    7086MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L4                      On  |   00000000:3E:00.0 Off |                    0 |
| N/A   37C    P0             27W /   72W |    7086MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    151463      C   python3                                      6314MiB |
|    0   N/A  N/A    151464      C   python3                                       248MiB |
|    0   N/A  N/A    151465      C   python3                                       248MiB |
|    0   N/A  N/A    151466      C   python3                                       248MiB |
|    1   N/A  N/A    151463      C   python3                                       248MiB |
|    1   N/A  N/A    151464      C   python3                                      6314MiB |
|    1   N/A  N/A    151465      C   python3                                       248MiB |
|    1   N/A  N/A    151466      C   python3                                       248MiB |
|    2   N/A  N/A    151463      C   python3                                       248MiB |
|    2   N/A  N/A    151464      C   python3                                       248MiB |
|    2   N/A  N/A    151465      C   python3                                      6314MiB |
|    2   N/A  N/A    151466      C   python3                                       248MiB |
|    3   N/A  N/A    151463      C   python3                                       248MiB |
|    3   N/A  N/A    151464      C   python3                                       248MiB |
|    3   N/A  N/A    151465      C   python3                                       248MiB |
|    3   N/A  N/A    151466      C   python3                                      6314MiB |
+-----------------------------------------------------------------------------------------+

Hi @ydm-amazon @siddvenk,
I cannot get access to 4xL4, but I tried with 4xA30, I cannot reproduce your error on either TRT-LLM v0.12 or main branch.
There is tiny memory size diff between 4xL4 (23034MiB) and 4xA30 (24576MiB) but this is the closest I can have.
Currently the usage of the gpu memory is determined by --kv_cache_free_gpu_memory_fraction argument in run.py. The memory usage different might be a change of design in this feature. For enc-dec, it's more complicated, it's the value but divided by 2 (for enc and dec). See code here for reference.
I'm assuming you're using Tensorrt_LLM_backend to do inferencing. Please also check the value of kv_cache_free_gpu_memory setting in config.pbtxt.
I've attached my nvidia-smi running the engine.
Tue Nov  5 07:11:22 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A30                     On  |   00000000:81:00.0 Off |                    0 |
| N/A   30C    P0             30W /  165W |   23532MiB /  24576MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A30                     On  |   00000000:A1:00.0 Off |                    0 |
| N/A   31C    P0             35W /  165W |   23532MiB /  24576MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A30                     On  |   00000000:C1:00.0 Off |                    0 |
| N/A   31C    P0             32W /  165W |   23532MiB /  24576MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A30                     On  |   00000000:E1:00.0 Off |                    0 |
| N/A   30C    P0             31W /  165W |   23532MiB /  24576MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2995      C   python3                                     22828MiB |
|    0   N/A  N/A      2996      C   python3                                       224MiB |
|    0   N/A  N/A      2997      C   python3                                       224MiB |
|    0   N/A  N/A      2998      C   python3                                       224MiB |
|    1   N/A  N/A      2995      C   python3                                       224MiB |
|    1   N/A  N/A      2996      C   python3                                     22828MiB |
|    1   N/A  N/A      2997      C   python3                                       224MiB |
|    1   N/A  N/A      2998      C   python3                                       224MiB |
|    2   N/A  N/A      2995      C   python3                                       224MiB |
|    2   N/A  N/A      2996      C   python3                                       224MiB |
|    2   N/A  N/A      2997      C   python3                                     22828MiB |
|    2   N/A  N/A      2998      C   python3                                       224MiB |
|    3   N/A  N/A      2995      C   python3                                       224MiB |
|    3   N/A  N/A      2996      C   python3                                       224MiB |
|    3   N/A  N/A      2997      C   python3                                       224MiB |
|    3   N/A  N/A      2998      C   python3                                     22828MiB |
+-----------------------------------------------------------------------------------------+

Hi @jtchen0528,
Setting the kv_cache_free_gpu_memory fraction does not seem to fix the issue - I have tried values 0.9, 0.8, and 0.4 but they all give an OOM error.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 22.19 GiB of which 1.50 MiB is free. Process 885415 has 22.18 GiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I have also double checked the config.pbtxt file in the artifacts to make sure it's set correctly.

``Hi @ydm-amazon,
I see that you managed to reproduce the issue with TensorRT-LLM only. Let's focus on debugging TensorRT-LLM for now.
My step:

checked out tags/v0.12.0 here,
do a build from source on systems without GNU make section.
Followed the exact script in reproduction step. See no error after run.py

Since the examples/run.py script does not stop after launching the executor, so I put a time.sleep(60) before model.generate() to log the gpu allocation with nvidia-smi. Saw a 23532MiB /  24576MiB for each GPU since the allocation is based on the kv_cache_free_gpu_memory_fraction parameter. You should see a similar number with your L4 23034MiB.
Another possible cause is, I wonder if you set your mpi setting when allocating nodes is correct. I see the following log messages in the log you uploaded:
[TensorRT-LLM][WARNING] Device 3 peer access Device 0 is not available.

Setting the gpu id with CUDA_VISIBLE_DEVICE=0,1,2,3 before run.py, and try echo $CUDA_VISIBLE_DEVICES to ensure that the correct GPU is running.

I put the time.sleep(60) before the model.generate(), and this is the nvidia-smi:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A10G                    On  | 00000000:00:1B.0 Off |                    0 |
|  0%   20C    P0              53W / 300W |  22684MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A10G                    On  | 00000000:00:1C.0 Off |                    0 |
|  0%   20C    P0              56W / 300W |  22684MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A10G                    On  | 00000000:00:1D.0 Off |                    0 |
|  0%   21C    P0              56W / 300W |  22684MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A10G                    On  | 00000000:00:1E.0 Off |                    0 |
|  0%   21C    P0              56W / 300W |  22684MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    892976      C   python3                                   22676MiB |
|    1   N/A  N/A    892977      C   python3                                   22676MiB |
|    2   N/A  N/A    892978      C   python3                                   22676MiB |
|    3   N/A  N/A    892979      C   python3                                   22676MiB |
+---------------------------------------------------------------------------------------+

It seems that there isn't enough space to hold the three other processes per gpu when generate starts. In your memory usage output it is 22828 MiB + 224 MiB * 3, but for mine, 22676 MiB + 224 MiB * 3 = 23348 MiB which is larger than the 23028 MiB the GPU can hold.
For CUDA_VISIBLE_DEVICES, it seems that setting it does not prevent the "...peer access ... is not available" warning. Below I have the log up till the point of time.sleep(60), in case it helps:
root@9fcc5d585e64:/opt/ml/model# CUDA_VISIBLE_DEVICES=0,1,2,3
root@9fcc5d585e64:/opt/ml/model# echo $CUDA_VISIBLE_DEVICES
0,1,2,3
root@9fcc5d585e64:/opt/ml/model# mpirun --allow-run-as-root -np 4 python3 run.py --engine_dir /tmp/.djl.ai/trtllm/1d7f7fa222cc2d5726132bb27e06c64d65b384ec/google-flan-t5-xl/1/ --tokenizer_dir google/flan-t5-xl --max_output_len 64 --num_beams=1 --input_text "translate English to German: The house is wonderful."
[TensorRT-LLM] TensorRT-LLM version: 0.12.0
[TensorRT-LLM] TensorRT-LLM version: 0.12.0
[TensorRT-LLM] TensorRT-LLM version: 0.12.0
[TensorRT-LLM] TensorRT-LLM version: 0.12.0
[11/06/2024-17:35:01] [TRT-LLM] [W] This path is an encoder-decoder model. Using different handling.
[11/06/2024-17:35:01] [TRT-LLM] [W] This path is an encoder-decoder model. Using different handling.
[11/06/2024-17:35:01] [TRT-LLM] [W] This path is an encoder-decoder model. Using different handling.
[11/06/2024-17:35:01] [TRT-LLM] [W] This path is an encoder-decoder model. Using different handling.
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 1
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 3
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 2
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 0
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.12.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1024 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 2
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1024 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 3
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1024 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1024 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][WARNING] Device 3 peer access Device 0 is not available.
[TensorRT-LLM][WARNING] Device 3 peer access Device 1 is not available.
[TensorRT-LLM][WARNING] Device 3 peer access Device 2 is not available.
[TensorRT-LLM][WARNING] Device 1 peer access Device 0 is not available.
[TensorRT-LLM][WARNING] Device 1 peer access Device 2 is not available.
[TensorRT-LLM][WARNING] Device 1 peer access Device 3 is not available.
[TensorRT-LLM][WARNING] Device 2 peer access Device 0 is not available.
[TensorRT-LLM][WARNING] Device 2 peer access Device 1 is not available.
[TensorRT-LLM][WARNING] Device 2 peer access Device 3 is not available.
[TensorRT-LLM][WARNING] Device 0 peer access Device 1 is not available.
[TensorRT-LLM][WARNING] Device 0 peer access Device 2 is not available.
[TensorRT-LLM][WARNING] Device 0 peer access Device 3 is not available.
[TensorRT-LLM][INFO] Loaded engine size: 681 MiB
[TensorRT-LLM][INFO] Loaded engine size: 681 MiB
[TensorRT-LLM][INFO] Loaded engine size: 681 MiB
[TensorRT-LLM][INFO] Loaded engine size: 681 MiB
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 16961.00 MiB for execution context memory.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 16961.00 MiB for execution context memory.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 16961.00 MiB for execution context memory.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 16961.00 MiB for execution context memory.
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 677 (MiB)
[TensorRT-LLM][INFO] TRTEncoderModel mMaxInputLen: reset to 1024 from build config.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 0
[TensorRT-LLM][INFO] Rank 0 is using GPU 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 677 (MiB)
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 677 (MiB)
[TensorRT-LLM][INFO] TRTEncoderModel mMaxInputLen: reset to 1024 from build config.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 3
[TensorRT-LLM][INFO] Rank 3 is using GPU 3
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[TensorRT-LLM][INFO] TRTEncoderModel mMaxInputLen: reset to 1024 from build config.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 2
[TensorRT-LLM][INFO] Rank 2 is using GPU 2
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 677 (MiB)
[TensorRT-LLM][INFO] TRTEncoderModel mMaxInputLen: reset to 1024 from build config.
[TensorRT-LLM][INFO] MPI size: 4, MPI local size: 4, rank: 1
[TensorRT-LLM][INFO] Rank 1 is using GPU 1
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 256
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 256
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 1024
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 1024
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 16384
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 1 = max_input_len (in trtllm-build args)
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[TensorRT-LLM][INFO] Loaded engine size: 922 MiB
[TensorRT-LLM][INFO] Loaded engine size: 922 MiB
[TensorRT-LLM][INFO] Loaded engine size: 922 MiB
[TensorRT-LLM][INFO] Loaded engine size: 922 MiB
[TensorRT-LLM][INFO] Detecting local TP group for rank 0
[TensorRT-LLM][INFO] Detecting local TP group for rank 2
[TensorRT-LLM][INFO] Detecting local TP group for rank 3
[TensorRT-LLM][INFO] Detecting local TP group for rank 1
[TensorRT-LLM][INFO] TP group is intra-node for rank 1
[TensorRT-LLM][INFO] TP group is intra-node for rank 3
[TensorRT-LLM][INFO] TP group is intra-node for rank 2
[TensorRT-LLM][INFO] TP group is intra-node for rank 0
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1328.64 MiB for execution context memory.
[TensorRT-LLM][INFO] cudaDeviceCanAccessPeer failed for device: 2 peerDevice: 0
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1328.64 MiB for execution context memory.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1328.64 MiB for execution context memory.
[TensorRT-LLM][INFO] cudaDeviceCanAccessPeer failed for device: 0 peerDevice: 1
[TensorRT-LLM][INFO] cudaDeviceCanAccessPeer failed for device: 3 peerDevice: 0
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1328.64 MiB for execution context memory.
[TensorRT-LLM][INFO] cudaDeviceCanAccessPeer failed for device: 1 peerDevice: 0
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1594 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1594 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1594 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1594 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.04 GB GPU memory for runtime buffers.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.04 GB GPU memory for runtime buffers.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.04 GB GPU memory for runtime buffers.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.04 GB GPU memory for runtime buffers.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 176.93 MB GPU memory for decoder.
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.21 GiB
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 176.93 MB GPU memory for decoder.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 176.93 MB GPU memory for decoder.
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.21 GiB
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 176.93 MB GPU memory for decoder.
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.21 GiB
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.21 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 33
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 33
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 33
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 33
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.10 GiB for max tokens in paged KV cache (2112).
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.10 GiB for max tokens in paged KV cache (2112).
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.10 GiB for max tokens in paged KV cache (2112).
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.11 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 18
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.10 GiB for max tokens in paged KV cache (2112).
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.11 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 18
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.11 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 18
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 22.19 GiB, available: 0.11 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 18
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 16
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.05 GiB for max tokens in paged KV cache (1152).
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.05 GiB for max tokens in paged KV cache (1152).
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.05 GiB for max tokens in paged KV cache (1152).
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.05 GiB for max tokens in paged KV cache (1152).
[11/06/2024-17:35:04] [TRT-LLM] [I] Load engine takes: 2.3786461353302 sec
[11/06/2024-17:35:04] [TRT-LLM] [I] Load engine takes: 2.377972364425659 sec
[11/06/2024-17:35:04] [TRT-LLM] [I] Load engine takes: 2.3568196296691895 sec
[11/06/2024-17:35:04] [TRT-LLM] [I] Load engine takes: 2.362577199935913 sec

Hi @ydm-amazon,
Good catch! In that case, would you mind trying --kv_cache_free_gpu_memory_fraction 0.5 or other smaller values in run.py?
If I set 0.9, the engine process has 22828MiB, after I set to 0.5, it goes to 22270MiB, adding 3x224 would be 22942 < 23028.
Let me know if this solves your problem, thanks!
mpirun --allow-run-as-root -np 4 python3 ../examples/run.py \
    --engine_dir $ENGINE_DIR \
    --tokenizer_dir $HF_DIR \
    --max_output_len 64 \
    --num_beams=1 \
    --input_text "translate English to German: The house is wonderful." \
    --kv_cache_free_gpu_memory_fraction 0.5

Hi @jtchen0528, thanks for your response:
With free gpu memory fraction of 0.5, I'm getting the following error:
Assertion failed: maxTokensInPagedKvCache (896) must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width (1) * tokensPerBlock (64) * maxBlocksPerSeq (16))

Please see the log below for more information
Log
Looks like there is not enough space to store the kv cache tokens if we give it to the other 3 processes. And even if tokensPerBlock and/or maxBlocksPerSeq are changed, I think 896 is still a low amount of tokens
Is there any other setting I can use to change the allocation of gpu memory? As far as I know, flan-t5-xl is smaller than llama-2-7b which fits okay, so there must be lots of room somewhere.

Hi @ydm-amazon ,
My best suggestion is to configure the --kv_cache_free_gpu_memory_fraction until engine size fits your hardware, or try setting --tokens_per_block 64 during decoder build trtllm-build so that maxTokensInPagedKvCache (896) can be larger than beam_width (1) * tokensPerBlock (32) * maxBlocksPerSeq (16).
For Llama and Enc-dec comparison, there are architectural differences between a decoder-only and enc-dec model. There are also cross attention + self attention in T5 decoder, which requires 2 sets of kv cache, whereas llama only has an attention.
Thanks!


The default objectManagerLoader doesn't work when the entity has an embedded field.

Since https://github.com/phpstan/phpstan-doctrine/pull/253, the metadata is resolved without the need of an objectManagerLoader.
It works well, but I have the following issue:
$this->getEntityManager()->getRepository(Expedition::class)->getListOrders($context, $date);

report an error
Call to an undefined method Doctrine\ORM\EntityRepository<App\Entity\Expedition>::getListOrders().

getRepository is considered as an EntityRepository instead of ExpeditionRepository.
Seems like that as soon as I remove the field
/**
 * @ORM\Embedded(class="App\Entity\Embeddable\Slot")
 */
private ?Slot $slot = null;

from the Expedition entity, it works well.

For my Expedition entity this line https://github.com/phpstan/phpstan-doctrine/blob/master/src/Type/Doctrine/ObjectMetadataResolver.php#L181 returns null. (but not for others entities)
After some dump, I discovered I ends here https://github.com/phpstan/phpstan-doctrine/blob/master/src/Type/Doctrine/ObjectMetadataResolver.php#L120
With the exception message
No identifier/primary key specified for Entity "App\Entity\Embeddable\Slot". Every Entity must have an identifier/primary key.

But there is no need for identifier https://www.doctrine-project.org/projects/doctrine-orm/en/2.11/tutorials/embeddables.html
I think there is something to improve in the PHPStan\Doctrine\Mapping\ClassMetadataFactory.

Adding
	protected function validateRuntimeMetadata($class, $parent)
	{
		return;
	}

to the Phpstan metadatafactory solve the issue.
Would you agree with this @ondrejmirtes ?

That just seems like a workaround.
Does this work when you provide objectManagerLoader? Can you check why?

Does this work when you provide objectManagerLoader? Can you check why?

I use both ORM and ODM on my project.
I used my own fork with https://github.com/phpstan/phpstan-doctrine/pull/218 with this objectManagerLoader https://github.com/phpstan/phpstan-doctrine/pull/218/files#diff-7be597212047526284ecdf31e9ea0e3cd38415a178b2ebb71af7a7204f388de9 and it was working.
To debug this, I simply run
public function __construct(EntityManagerInterface $entityManager)
    {
        $entityManager->getMetadataFactory()->getMetadataFor(Expedition::class);

        (new \PHPStan\Doctrine\Mapping\ClassMetadataFactory())->getMetadataFor(Expedition::class);

        parent::__construct();
    }

in my symfony project.
The first getMetadataFor work, but the second doesn't.
I ended up in the Doctrine\ORM\Mapping\ClassMetadataFactory::doLoadMetadata which call validateRuntimeMetadata, in which the call $class->validateIdentifier() is made.
    public function validateIdentifier()
    {
        if ($this->isMappedSuperclass || $this->isEmbeddedClass) {
            return;
        }
        // do things

When the call is made by the phpstan factory, $this->isEmbeddedClass is false (but it's true with the doctrine factory).
So I find out that the Doctrine\ORM\Mapping\Driver\AnnotationDriver::loadMetadataForClass is supposed to do the job.
It's called with Expedition and Slot for Doctrine.
It's only called with Expedition for Phpstan.

Ok, so I found why.
Everything is done by the line
$this->driver->loadMetadataForClass($class->getName(), $class);

here: https://github.com/doctrine/orm/blob/2.11.x/lib/Doctrine/ORM/Mapping/ClassMetadataFactory.php#L133
In the MappingDriverChain, the code is
	public function loadMetadataForClass($className, ClassMetadata $metadata): void
	{
		foreach ($this->drivers as $driver) {
			if ($driver->isTransient($className)) {
				continue;
			}

			$driver->loadMetadataForClass($className, $metadata);
			return;
		}
	}

and $driver->isTransient($className) return true for embeddable entities.
Doctrine is doing $driver->loadMetadataForClass($className, $metadata) without any transient check.
I tried changing the code to
	public function loadMetadataForClass($className, ClassMetadata $metadata): void
	{
		foreach ($this->drivers as $driver) {
			$driver->loadMetadataForClass($className, $metadata);
		}
	}

but it doesn't work, I get another error. But changing it to
	public function loadMetadataForClass($className, ClassMetadata $metadata): void
	{
		foreach ($this->drivers as $driver) {
			$driver->loadMetadataForClass($className, $metadata);
                         return;
		}
	}

solve the issue for me, because I use Annotation IMHO ; it will just break for people relying on Attribute since the driver won't be use.

This should work: https://github.com/phpstan/phpstan-doctrine/commit/5fa8fefdf7d3b41ada096c0271250e2faf1d6adc
Thanks.

This should work: 5fa8fef
Thanks.

Indeed, it works for me.


Packman game

Is your feature request related to a problem? Please describe.
game addition
Describe the solution you'd like
game addition
Describe alternatives you've considered
game addition
Additional context

What problem is this feature trying to solve?
How do we know when the feature is complete?

Closing this as exists #511


Add config file support

Suggestion: Use toml with https://github.com/alexcrichton/toml-rs

Investigate: https://github.com/mehcode/config-rs

config-rs did looked like it was a bit nicer to use; gets my vote.

Sounds good toml-rs seems to integrate nicely with https://serde.rs/ as well


Adds documentation about arguments starting with a dash. Closes #4257

Adds documentation about arguments starting with a dash. Closes #4257
Added Values starting with a dash (-) section to the Use the CLI  page.

Thank you @nanddeepn
Will review it ASAP!

Merged manually. Thank you! 👏


internal/poll: stop using dedicated threads on Windows

In the internal/poll package, on Windows, if the function CancelIoEx is not available on the system, we use dedicated threads for I/O so that requests can be canceled using the older CancelIo function.
According to https://docs.microsoft.com/en-us/windows/win32/fileio/cancelioex-func, CancelIoEx is available on Windows Vista, Windows Server 2008, and newer.
According to https://golang.org/wiki/MinimumRequirements, as of the Go 1.11 release we require at least Windows 7, or Windows 2008R2, or higher.
According to https://en.wikipedia.org/wiki/List_of_Microsoft_Windows_versions, Windows 7 is newer than Windows Vista.
Therefore, I believe that all supported Windows systems support CancelIoEx, and we can remove the (rather complex) code that works using CancelIo.
@alexbrainman @mattn Do you agree?

@ianlancetaylor I agree. We should remove all code that calls CancelIo, because it should not be needed now.
Do you want me to do the honors?
Alex

I can do it, unless you would like to.  Thanks.


Suggestion: reuse same instance within rate-limit interval

For something like Farside, I feel like pointing to the same instance of a given service if receiving multiple requests to that service might be a good idea. I say this partly for usage with RSS, which is a very common usecase for me with many of these services as they offer RSS where official means do not.
However, this rate limiting can often mean updating a feed may fail. I think giving the same instance again might be a good way to mitigate the chances of this happening while also still reducing the load on the Farside server some. I may be wrong, but requests being outright denied in more programmatic situations like this is a little frustrating, especially with there only being a single Farside instance.

Thanks for the suggestion. It's been added and will be deployed shortly.