Theming Discussion

@PhrozenByte don't feel the need to read or answer this all at once. I just didn't see the point of making multiple issues on very similar topics.
I've been laser focused on updating an old theme this week (Freelancer).  It's gonna be pretty good. 😉
But, in the process I've had a few issues I wanted to discuss, especially considering the pico-theme.yml standard (and our as far as I can tell lack of documentation on it). (Okay, I lied here, it was like one question. 😂)
I've had a bunch of random thoughts and questions related to theming as I've been going, but I figured I'd hold them until the end and see what I couldn't just answer myself.
This is just meant to be a thread for a general discussion, not a singular issue.  So, let's get into it.

The pico-theme.yml's meta-headers section doesn't seem to support arrays for having multiple levels of depth.  Not sure whether this is an oversight or by design.  The use case I'd describe here would be:

I have several meta-headers named portfolio_title, portfolio_divider, portfolio_nav and portfolio_disabled.
These could be written:
Portfolio Title: Recent Projects
Portfolio Divider: clock
Portfolio Nav: Projects
Portfolio Disabled: false

But I feel they'd look better as:
Portfolio:
  Title: Recent Projects
  Divider: clock
  Nav: Projects
  Disabled: false

It's fine if you've decided you just don't want Pico to have multi-level arrays of metadata, but I have been using them to simplify the YML on my themes for quite a while, lol. 😂
What are your thoughts on this?  Is it an idea you absolutely hate or might it have potential?
Obviously in the meantime I can still accomplish this without registering them in pico_theme.yml (that's how I've been developing it anyway), they'll just have mandatory lower-case names like portfolio > title.
On a related side note, I almost opened this issue a few days ago to discuss how weird and inconsistent it feels when, if you don't register meta headers, they behave somewhat opposite of the built-in ones:
We typically write the meta headers with the first letter capitalized.  When you do this, it then only presents itself as lowercase in Twig (eg Title becomes title). But if you have an unregistered custom header, the opposite becomes true.  You write it with the first letter capitalized and it is only available the way you wrote it (Image stays Image).
I know why this is the case... it just feels odd and like it could be something of a trap for new users.

I've noticed that on some of my old themes (that I haven't got around to fixing yet), whitespace has started being removed by Twig.  This has led to text beingcrushedtogether in some cases.  Just curious if you knew when and why this behavior changed (besides the obvious "sometime between Pico 1 and 2 😂).

Is Twig's raw filter generally "safe" to use in Pico?  I've been using it on this project more than I intended too because if I do anything to manipulate strings before printing them they become escaped, which doesn't work so well for HTML tags. 😅

It doesn't really seem like an issue considering our usual attitude toward security (that any access to Pico's files is the same level of security, and if someone had access to put something malicious in the markdown anyway, then they could probably already do a lot worse without caring about the Twig template, etc).

Pico's markdown filter doesn't seem to throw errors.  This is a very different behavior to Twig's built in filters (which complain. A lot.).  I accidentally fed it an array a couple times and it just silently fails.  Compare this to Twig throwing an "Array to string conversion" notice every time I accidentally forget to json_encode my debug code (🤦🏻♀), and I just feel like it should say something when you do it wrong.  No idea if Pico's other filters are like this or not.

Do you think they should print errors for debugging?

It seems like having a _meta.md file is now an "officially endorsed" concept (at least in the docs).  A lot of themes (Freelancer included) ignore index.md in favor of giving a custom front-page (or single-page) experience.

I like the idea of having theme-related/site-wide metadata in _meta.md (and it's what I've used this time), however, it does leave an awkward index.md-sized hole to be filled. 😉
Specifically, I mean you still require an empty index.md in this case, even if you're not using it, both for Pico's content folder detection and for preventing 404's (with the content dir manually specified in config.yml).
I guess this is mostly an opinion thing, but it leaves me wondering what approach is better in this case.  If I use index.md for the metadata, it's at least used for something.  The instructions wouldn't need a step that says "make sure you have an index.md, even if it's empty".  What do you think?  Would you rather themes suggest a blank index or that they forgo _meta.md in favor of sticking everything in index.md?

404-ing on _meta.md behaves weird.

So, this theme rewrite is heavily dependent on _meta.md containing all its variables (don't worry, I've hard-coded defaults, so it all looks very professional even with an empty _meta.md 😉).  I decided to test that Pico supposedly denies access to _ files, and I discovered an odd behavior.  When it 404's on trying to access _meta.md... it also denies access to all the metadata I was accessing on _meta.md, causing my very customized site to revert to the demo data I've included.
Presumably this is happening because there's a shortcut somewhere in Pico that goes "I need to access this page.  Oh, good, I'm already on it, I'll just look at current_page instead."  Except its current page has been redirected to the 404 page. 🤦🏻♀
Just kind of an odd quirk.  I don't need it fixed or anything, just thought I'd bring it to your attention. 😂

Kind of surprised that using pages.pageid instead of pages["pageid"] still works.  I'm assuming this is still just a legacy, compatibility behavior, since I remember you telling me a long time ago that it was preferred to use pages["pageid"] instead.  I guess that's all, it was just kind of a "Huh... neat" moment. 🤷🏻♀

This one should probably be a separate discussion entirely.  Do you have any good suggestions for maintaining a large patch set against an upstream?  The Twig bits I've written for Freelancer are exhaustively extensive.  I've spent over a week on it now (while the Themes Gallery project hangs in limbo 😒).  It just keeps going, lol. I need to stop adding things to it.

The original theme is written with Bootstrap, in SCSS and PUG.  Since the Pico port is based on the compiled version, I can't exactly just grab the original repo as an upstream (though, maybe I could potentially pull from their compiled dist folder somehow).
At the moment, I plan to remove the upstream on my existing port (it was based on an out-of-date Jekyll version), and just replace the entire repo with the new version and sort this out later.
For future updates of Bootstrap and Freelancer though, it'd be nice to have a solution in place to be able to bring the changes forward without essentially starting from scratch every time.
At the moment, the best idea I'd have would be to diff the old and new Freelancer versions and simply use that to get a rough idea of what changed between versions.  Or, you know, see if I could somehow make a patch file based on the diff between my version and theirs, but I think the changes might be too extensive for that (or at the very least, I might have to merge all my includes back into one mega-template).  Either way, I don't see it being as easy as "fetch and merge upstream". 😩
So, if there's a fairly obvious solution you think I should look into, let me know.  Not looking for a full solution, just a nudge in the right direction if you have any ideas. 😅

Alright, this has gone on long enough.  I'm so ready to move on from this theme, but I keep finding one more thing to do.
And I promise the next one will be a "regular" theme.  This one's just been needing some attention. 😉

Oops, rolled the Issue tracker over to 600. 😅
Damn, should have come up with something better for that. 😂
Hey, at least it wasn't an Nginx issue this time! 🤣

The pico-theme.yml's meta-headers section doesn't seem to support arrays for having multiple levels of depth. Not sure whether this is an oversight or by design. The use case I'd describe here would be:

Yes, it doesn't support arrays right now. That's mostly because I'd rather consider this being sort of data validation, what is out-of-scope. The idea behind registering meta values is to allow aliasing meta values - like the mentioned Portfolio vs. portfolio (but not limited to, you can also alias Portfolio to foobar if you want to).
Anyway, I totally agree that this is inconsistent - and it's quite hard to fix this inconsistency. To be honest I'd rather go for removing aliasing altogether. If a theme expects portfolio, users should use portfolio and not Portfolio. We shouldn't mess with a user's meta values...

I've noticed that on some of my old themes (that I haven't got around to fixing yet), whitespace has started being removed by Twig. This has led to text beingcrushedtogether in some cases. Just curious if you knew when and why this behavior changed (besides the obvious "sometime between Pico 1 and 2 joy).

No idea, AFAIK they didn't change anything there. Did you check the HTML sources or just how it is being displayed? Maybe it's just an browser issue?

Is Twig's raw filter generally "safe" to use in Pico? I've been using it on this project more than I intended too because if I do anything to manipulate strings before printing them they become escaped, which doesn't work so well for HTML tags. sweat_smile

Yes and no. One should avoid using Twig's raw filter whenever possible, because it might introduce security issues with Pico CMS for Nextcloud (which I'd consider equally important - not for Pico itself though). Usually there's a better solutions than Twig's raw filter.

Pico's markdown filter doesn't seem to throw errors. This is a very different behavior to Twig's built in filters (which complain. A lot.). I accidentally fed it an array a couple times and it just silently fails. Compare this to Twig throwing an "Array to string conversion" notice every time I accidentally forget to json_encode my debug code (🤦🏻♀), and I just feel like it should say something when you do it wrong. No idea if Pico's other filters are like this or not.

Indeed. Can you open a separate issue for this? I'll add some checks later then.

It seems like having a _meta.md file is now an "officially endorsed" concept (at least in the docs). A lot of themes (Freelancer included) ignore index.md in favor of giving a custom front-page (or single-page) experience.

What's the reason for these themes to ignore index.md? Even if it's a single-page theme, why isn't index.md used to manage contents? Just take clean-blog for an example: It doesn't show the contents of index.md either, but honestly I don't really see a reason why. The theme could still show the contents right above the article list.

404-ing on _meta.md behaves weird.

Indeed, definitely a bug. Not sure why this is happening. Can you open a separate issue for this? I'll check this later then.

Kind of surprised that using pages.pageid instead of pages["pageid"] still works. I'm assuming this is still just a legacy, compatibility behavior, since I remember you telling me a long time ago that it was preferred to use pages["pageid"] instead. I guess that's all, it was just kind of a "Huh... neat" moment. 🤷🏻♀

Hmm... Not sure what you mean? This is Twig syntax - and Twig supports both and AFAIK there are no plans to remove this. It just might cause confusion, because the dot-syntax has some limitations regarding the keys supported (e.g. no whitespaces, one can't use a variable for the key)

This one should probably be a separate discussion entirely. Do you have any good suggestions for maintaining a large patch set against an upstream? The Twig bits I've written for Freelancer are exhaustively extensive. I've spent over a week on it now (while the Themes Gallery project hangs in limbo unamused). It just keeps going, lol. I need to stop adding things to it.

I'd recommend to let Git do this job for you. Use the original SCSS as a basis and add your changes. Choose solutions with which you don't have to edit the original SCSS sources too often. Download a SCSS compiler and also commit the compiled CSS to your repo. Rename index.html to index.twig first and then apply your changes step by step, so that Git still knows that index.twig is index.html and tracks changes. If you later wanna update from upstream, simply do a pull. You'll likely end up with some conflicts, but it should remain manageable - e.g. you can simply ignore conflicts in the compiled CSS.

Sorry in advance for not addressing these in order, but I'm gonna bounce around a little as I get to each issue today.

No idea, AFAIK they didn't change anything there. Did you check the HTML sources or just how it is being displayed? Maybe it's just a browser issue?

I found an example of this to share.  From the old Freelancer port:
Copyright © {{ site_title }} {{ pages.index.meta.date | date('Y') }}
The output, currently displayed on my personal site w/ Maya McDougall as site_title, and no pages.index.meta.date (so it's just using the current date):
Copyright ©Maya McDougall2021
For some reason, all the whitespace directly surrounding Twig's print tags is removed (And yes, it's actually written in the HTML like that, it's not a browser or style issue). This didn't happen back when I wrote the theme (otherwise, I wouldn't have left it like that!), and I've seen it exhibited in a couple of others too.
This is likely a Twig issue, not a Pico one.  I was just curious if you had any insight into it.  The only info I can find involves Twig changing how it handled newlines at one point, but nothing about spaces.
Obviously this can be worked around in a couple ways (concatenating a space into the string, using an nbsp instead, etc etc).
It's just odd that it changed at some point. 🤔 🤷🏻♀

Issues made. 😉
Sorry for giving you a todo list. 😂

I'd almost wonder if it'd be better to just standardize on lowercase names and convert all the keys to lowercase at runtime. I honestly don't care that much about the ability to alias things, but lower vs upper case feels like a usability issue. To a newer user, the idea that some meta properties can be upper or lower case (the internal ones), while some must be a specific case feels like a paper cut.

The problem is that this is going to be inconsistent with arrays (i.e. {{ meta.some_array.SomeValueThatRequiresUppercase }}). For Pico 3.0 (probably Pico 4.0 due to the dependency issues...) I was thinking about switching to objects to represent pages. We could do the same for meta data, allowing us to access all meta values no matter the depth case insensitive by default... I'll add this to the ToDo list for Pico ~3.0~ 4.0.

One place I couldn't find a good workaround though was adding bootstrap classes to my markdown content paragraphs:
{{ content|replace({"<p>": '<p class="mb-4">'})|raw }}

You shouldn't. Rather leave it a regular <p> and wrap it inside a literal <div> in your theme. You can then either apply the mb-4 class to that div instead or add additional CSS/LESS/SCSS rules to interpret all <p> below this <div> as mb-4.

In general, I'd say it's a design decision. Especially when working with an existing theme or template, shoehorning in a content section just to have it feels sloppy. Obviously, it depends on the situation, but many of these one-page themes or themes with dedicated front-pages just didn't originally have a space for general content.

Agreed. If there's really no place for {{ content }}, leave it.
We might add a new config section to pico-theme.yml to configure "static" pages (added this to the ToDo list) that don't require a Markdown file to exist to not yield a 404 page. However, requiring index.md to exist is a valid exception in my opinion.

I think I'm going to move all the site metadata from _meta.md to index.md. I mean, it is essentially the "content" of the index page, even if it's not the content of the index page. :wink:

:+1:

I've also tried to use index's content in the past to fill out certain specialty blocks of text, but it always feels clunky. The user experience is like "Okay, so this block of arbitrary text is your index's content, while this one comes from a Meta header and this one is your site_title. It gets confusing quickly.

I often use separate files for this, e.g. the page foo.md comes not just with the main foo.md, but also includes a _foo/box1.md, _foo/box2.md, etc. to represent pre-defined containers in my theme. I often restrict meta values to short single lines (e.g. headlines). For contents spanning multiple pages (e.g. a footer) I often use e.g. a _footer.md.

It's new, fresh code. It's highly documented. And I'm always around answering questions here anyway, so it's not like an extra burden or anything. I would still plan to redirect any issues that arise to its own repo. It would still be considered a separate project, even if it's "Officially" supported.
What do you think?

Unfortunately I don't have much time to do a "deep-dive" here, but everything I saw looks very, very good. Nothing tripped me. Please don't interpret my notes above (regarding the use of the raw filter, or using the markdown filter for multiple lines, etc.) as criticism, there's no single "right" answer, there are always multiple valid and absolutely equal solutions. Very extensive docs, great! :+1: I really love your work. Awesome! :heart:
If you choose to add an "Official" tag to themes I'm totally fine with it, I'm very confident in your considerations :smiley: :+1:

I was thinking about switching to objects to represent pages.

Interesting 🤔
I don't know enough about PHP to really comment on it on a technical level, but from an "any other language I know" perspective, that does sound like it'd be a lot easier to work with on the backend. 😉

You shouldn't.

lol. I was just being really OCD about keeping the output HTML the same as the source. 😅
I did end up styling it with extra CSS instead.
Don't worry, all raw filters were removed by the end. 😂

Agreed. If there's really no place for {{ content }}, leave it.

I didn't expect you to budge on that opinion. 😱

However, requiring index.md to exist is a valid exception in my opinion.

I agree.  I liked the idea of putting things in a special file like _meta.md.  It felt less "hacky" that way.  But without another use for index.md, it would never have been a good user experience.  I'll just have to use _meta.md for something next time.

I often use separate files for this

Wait wait wait, hold up.  You use Pico? And here I thought you just hacked on it once in a while. 😂
I'm kidding obviously, but you do very seldom talk about how you actually use Pico.
I like that idea of having chunks like  _foo/box1.md and  _foo/box2.md.  For a bigger, non-one-page Theme, I could see that being a really powerful way to keep things organized.  Might even be a good strategy for the Pico site in the future.  Keep all the modularity it has with Jekyll right now, but clean out all the clunky Jekyll cruft. 😉

Unfortunately I don't have much time to do a "deep-dive" here

No worries, I didn't expect you to.  I didn't actually think you'd continue the earlier discussion here either.  I appreciate it, it's just a little obsolete now.

Please don't interpret my notes above as criticism, there's no single "right" answer

No worries again.  I was actually able to make the code much cleaner by removing the raw usage anyway.
I had gone down a bit of a rabbit hole* of using some clever string manipulation to change which tags I was writing.  It was super super clever... and entirely unreadable.  You know you've gone in the wrong direction when you have trouble re-reading your own code.
(* Hey, another of those expressions. See what I said about the Docs rewrite comment.)
I did still end up with a clever markdown_by_paragraph macro that I used to turn multi-line YML strings into separate markdown paragraphs (without having to double-space the paragraphs because that makes the YML look bad in my opinion, lol).  Totally over-engineered, but it also allowed me to customize those <p> tags depending on the situation. 😂

If you choose to add an "Official" tag to themes I'm totally fine with it, I'm very confident in your considerations 😃 👍

Thanks. 😁
It's a future concern anyway.  It would be odd if the only two "Official" themes were the Default Theme and Freelancer.  There's a LOT of difference between the two (you know, like, everything).  So until there's more options, I'd rather not push new users toward a theme that works so differently from the Default Theme.


projects card is not good aligned

curious how you even got npm start to work. I had so many deprecations. When I run npm start I simply get this site can't be reached. If I update the packages I end up with multiple dependency errors. It looks like you were able to get passed this error. Curious about how you managed that.

@ghost can you provide detailed info on this?


HTTP request took too long to complete - magento-1

Hi Mark as I told you via Slack, I'm trying to setup a M1 environment/
[behavior]
running ./bin/start receiving ERROR:
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
[Expected]
download imges and create containers.

Are you on Windows or Mac? I haven't ever experienced this issue. It almost sounds like your Docker for Mac instance needs to be restarted or isn't properly setup. There's no reason Docker Compose should be timing out here. Please try to re-setup and re-build from latest docker-compose.yml file and reopen this ticket if you are still having issues.


Add autogenerated TypeDoc docs

This should help a bit with documentation woes. We'll have to iteratively go through and add comments to functions for this to be really helpful.
One thing to note is that the titles of the "modules" in the sidebar are wrong and can't be overridden without a comment in the file header (has to be at the top of the file). We have a lint rule that was silently overwriting all my changes in the pre-commit 🙄  so I'll need to fix that to support the module name override eventualy.

Closes #1517 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1527


Use Scala 2.12.9 on Travis CI

I think this will clean up this error.

#2763 is also doing this. There's still a scalafix issue, that I will try to get fixed.

#2763 is for 0.20 though. This will need the same scalafix changes to work.

#2763 is better, and we'll merge upward.


How to delete a folder without causing partial sync errors

Hello. After removing a folder that's not even scanned by plex, autoscan stopped working with this error:
Jan  4 08:59:25 ERR Consecutive errors occurred while syncing drive, drive has been stopped... error=["drive_id: performing partial sync: getting parents: removed file: could not get folder: random_deleted_folder_id: random_deleted_folder_id: folder not found: sql: no rows in result set","drive_id: performing partial sync: getting parents: removed file: could not get folder: random_deleted_folder_id: random_deleted_folder_id: folder not found: sql: no rows in result set","drive_id: performing partial sync: getting parents: removed file: could not get folder: random_deleted_folder_id: random_deleted_folder_id: folder not found: sql: no rows in result set","drive_id: performing partial sync: getting parents: removed file: could not get folder: random_deleted_folder_id: random_deleted_folder_id: folder not found: sql: no rows in result set","drive_id: performing partial sync: getting parents: removed file: could not get folder: random_deleted_folder_id: random_deleted_folder_id: folder not found: sql: no rows in result set"]

Any way to solve this without doing a full database rebuild?

I'd advice you to switch over to A-Train. The Google Drive implementation that's currently inside Autoscan unfortunately has multiple logic errors which we were not able to fix. A-Train uses a completely different approach, which has worked flawlessly since its introduction. I'd definitely recommend you to check it out :)!

Thanks will check it out.
Does switching from bernard to A-train require database file recreation? Will I still have to delete autoscan.db to allow it rebuild or does changing to a-train in autoscan config will resolve errors somehow?

A-Train builds its own database file inside its container, so there's no need to delete the current autoscan.db file. Except for removing the Bernard section in Autoscan's config file, there's no need to change anything there either, as a-train is automatically enabled in recent versions of Autoscan.

Does it not make sense still to delete the database? My library is pretty big & autoacan.db is ~127 mb.


Update examples

In one of the latest versions adapter_name was moved from Controller into ControllerManager

👍

Any news?

This was addressed in: #32


Fix farm withdrawal

Farm page is not showing data, thus it is impossible to withdraw
Issue identified

wrong abi format, and network request failure due to request limit

The Coingecko price api is not yet fixed, a good solution to this is probably having a global state management, so to avoid duplicated network request.


Allow variable ploidy for ancestor/descendant lines

Will require

[x] User specified ploidy variables for anc/desendant lines
[x] Haploid prior on genotypes
[ ] `MutationAccumulation Matrix and MAprobability functions need to take variable sized matrices

OK. At this point we have MutationMatrix object for all possible experimental designs.
We need to work out how to use them. With the current design it might make sense to have different ReadDataVisitor classes for each experimental design, each with a different MutationMatrix set as a member function and each with a different Visit function to calculate the mutation probabilites

Now working for functions alone, just need to incorporate them into the VariantVisitor in main


Update readme & min-imp plugins

Updated settings guide & Updated plugins guide
Plugin Odoo 12  tests success
Track logs /tmp/gnome-pomodoro-tracking.log
Added new methods utils

Closes #14

Is required tokes(plugins) for run tests

Hi @gnome-pomodoro/devs  Can you review this ?

flake8 gives these hints:
./gp_tracking.py:250:17: E722 do not use bare 'except'
./plugins/clockify.py:34:5: E731 do not assign a lambda expression, use a def
./plugins/clockify.py:35:5: E731 do not assign a lambda expression, use a def
./plugins/clockify.py:87:13: E741 ambiguous variable name 'l'
./plugins/clockify.py:148:9: E722 do not use bare 'except'
./plugins/clockify.py:159:9: E722 do not use bare 'except'
./plugins/clockify.py:172:9: E722 do not use bare 'except'
./plugins/clockify.py:176:13: E722 do not use bare 'except'
./plugins/clockify.py:181:9: E722 do not use bare 'except'
./plugins/clockify.py:209:13: E722 do not use bare 'except'
./plugins/gpt_utils.py:51:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:7:1: F401 '.gpt_utils.make_request' imported but unused
./plugins/odoo.py:104:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:106:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:198:61: E271 multiple spaces after keyword
./plugins/odoo.py:226:13: E722 do not use bare 'except'
./plugins/toggl.py:58:5: E731 do not assign a lambda expression, use a def
./plugins/toggl.py:80:13: E741 ambiguous variable name 'l'
./plugins/toggl.py:140:9: E722 do not use bare 'except'
./plugins/toggl.py:151:9: E722 do not use bare 'except'
./plugins/toggl.py:214:13: E722 do not use bare 'except'
./tests/test_clockify.py:14:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'
./tests/test_gpt_plugin.py:20:20: E711 comparison to None should be 'if cond is None:'
./tests/test_gpt_plugin.py:28:20: E711 comparison to None should be 'if cond is None:'
./tests/test_odoo.py:16:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'
./tests/test_toggl.py:14:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'

Running tests shouldn't require env variables. Even running $ GP_TRACKING_ENV=Test pytest gives errors...

You could use mocks to fake responses from these services
I would drop pytest in favor for builtin unittest, as TestGPTPlugin is clearly a base class / mixin.

The use /tmp/gnome-pomodoro-tracking.stdout...
You could log debug info and in the test case use self.assertLogs() or in pytest use caplog.
Or in tests redirect stdout to a buffer https://stackoverflow.com/a/39320622
println - you can specify the ouput file with with builtin print(..., file=...). But it would be better to use logging or to redirect stdout/stderr to file if there is --debug flag. See contextlib.redirect_stdout. To me GPTracking.cli() looks like good place to add it.
printlg - it's more common to expose logger at a module level, and setup handlers according to commandline args / config. And as I wrote before, you can easily tap into it during execution of each test (self.assertLogs() / caplog).
def make_request(method, url, **kwargs):
        response = requests.request(method, url, **kwargs)
        if response.ok:
            return response.json()

I recommend using requests library directly. Here you assume every call is returning a json, end you lose access to response.status_code.

flake8 gives these hints:
./gp_tracking.py:250:17: E722 do not use bare 'except'
./plugins/clockify.py:34:5: E731 do not assign a lambda expression, use a def
./plugins/clockify.py:35:5: E731 do not assign a lambda expression, use a def
./plugins/clockify.py:87:13: E741 ambiguous variable name 'l'
./plugins/clockify.py:148:9: E722 do not use bare 'except'
./plugins/clockify.py:159:9: E722 do not use bare 'except'
./plugins/clockify.py:172:9: E722 do not use bare 'except'
./plugins/clockify.py:176:13: E722 do not use bare 'except'
./plugins/clockify.py:181:9: E722 do not use bare 'except'
./plugins/clockify.py:209:13: E722 do not use bare 'except'
./plugins/gpt_utils.py:51:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:7:1: F401 '.gpt_utils.make_request' imported but unused
./plugins/odoo.py:104:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:106:5: E731 do not assign a lambda expression, use a def
./plugins/odoo.py:198:61: E271 multiple spaces after keyword
./plugins/odoo.py:226:13: E722 do not use bare 'except'
./plugins/toggl.py:58:5: E731 do not assign a lambda expression, use a def
./plugins/toggl.py:80:13: E741 ambiguous variable name 'l'
./plugins/toggl.py:140:9: E722 do not use bare 'except'
./plugins/toggl.py:151:9: E722 do not use bare 'except'
./plugins/toggl.py:214:13: E722 do not use bare 'except'
./tests/test_clockify.py:14:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'
./tests/test_gpt_plugin.py:20:20: E711 comparison to None should be 'if cond is None:'
./tests/test_gpt_plugin.py:28:20: E711 comparison to None should be 'if cond is None:'
./tests/test_odoo.py:16:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'
./tests/test_toggl.py:14:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'

Running tests shouldn't require env variables. Even running $ GP_TRACKING_ENV=Test pytest gives errors...

You could use mocks to fake responses from these services
I would drop pytest in favor for builtin unittest, as TestGPTPlugin is clearly a base class / mixin.

The use /tmp/gnome-pomodoro-tracking.stdout...

You could log debug info and in the test case use self.assertLogs() or in pytest use caplog.
During test redirect stdout to a buffer https://stackoverflow.com/a/39320622

println - you can specify the ouput file with with builtin print(..., file=...). But it would be better to use logging or to redirect stdout/stderr to file if there is --debug flag. See contextlib.redirect_stdout. GPTracking.cli() looks like good place to add it.
printlg - it's more common to expose logger at a module level, and setup handlers according to commandline args / config. And as I wrote before, you can easily tap into it during execution of each test (self.assertLogs() / caplog).
def make_request(method, url, **kwargs):
        response = requests.request(method, url, **kwargs)
        if response.ok:
            return response.json()

I recommend using requests library directly. Here you assume every call is returning a json, and you lose access to response.status_code.

@kamilprusko  Thanks your feedback.
I'm working for improvement every suggestions.

[X] flake8 based code refactoring
[X] gh-action added flake8
[X] Replaced pytest by unittest
[X] Using mock third-party API response
[X] Added flag --debug for redirect stdout,  stderr  and logs in a  file


fix: increase toucharea of table links

Little optimisation, make the touch areas inside the table nice and big.
(Red background used to see the difference)

good one, LGTM :+1:

Taking @jsimplicio 's :+1: as a LGTM :)


Include squids icap_service_failure_limit configuration

This commit seems to be the final solution for https://github.com/opnsense/plugins/issues/3875.

I'd prefer to skip the GUI addition. It makes life simpler.

Okay, did it!

Merged, thanks!


documentation with more sophisticated examples

Hi,
it would help a lot if there were examples of nested objects like array and how to select them.
I have added a simple example of WikiData below.
maybe pick $.entities.Q5639868.labels.en.language.value
{
  "entities":{
    "Q5639868": {
      "pageid":5400904,
      "ns":0,
      "title":"Q5639868",
      "lastrevid":184473449,
      "modified":"2014-12-29T07:41:31Z",
      "type":"item","id":"Q5639868",
      "labels": {
        "en":{
          "language":"en",
          "value":"Haji Ali Mohammad"
        }
      },
      "descriptions":{},
      "aliases":{},
      "claims":{},
      "sitelinks":{}
    }
  }
}

Can you clarify what you are trying to do?
What's the result that you would want to see?
That seems like a single element not an array. So not sure how that fits into a CSV file.

Closing since there hasn't been any response.
Feel free to reopen if needed.


AKS sysbox nodes sporadically enter bad state (stat failed on /var/lib/containers/storage/overlay/...)

I'm attempting to port our CI kubernetes cluster to use sysbox to enable building and running docker containers inside our CI jobs.
We're finding that after a number of CI builds, nodes sporadically get into a bad state and are unable to start new containers (pods with status CreateContainerError). For pods/containers that are already running, their local docker daemon gets stuck and the CI build jobs running on them (mostly just docker builds) time out.
We are running roughly 14 pods per node, and we restart our pods (including their local dockerd process) after every job. The cluster is hosted by AKS on Azure. My sample size isn't enormous, but this issue seems to reproduce more frequently on sysbox 0.6.3 (after ~30 builds per node) vs 0.6.2 with SYSBOX_ALLOW_TRUSTED_XATTR set to FALSE (after ~120 builds per node), however it does reproduce on both. The node's kernel version is 5.15.
kubectl describe node doesn't show anything out of the ordinary, the node is ready and hosting pods (though they cannot start) and doesn't have any errors flagged on it.
journalctl -u kubelet does contain error log lines that healthy nodes do not contain, namely many lines that look like:
W0129 09:05:08.825295   17183 container.go:589] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753ba6ae_0b8f_47c8_992c_0f9eac91476a.slice/crio-761d1ec685b47ab5c8df26d44bb00c748610eb39c834c7d43383f749ff841835.scope": unable to determine device info for dir: /var/lib/containers/storage/overlay/2e13b53c75be18eb8c3f3773de0c861efe4135b1298157f5d981b3fea38c25ce/diff: stat failed on /var/lib/containers/storage/overlay/2e13b53c75be18eb8c3f3773de0c861efe4135b1298157f5d981b3fea38c25ce/diff with error: no such file or directory, continuing to push stats

journalctl -u crio also contains error log lines that healthy nodes do not have, but it seems just a result of the containers not being able to start:
time="2024-01-29 05:09:24.451212422Z" level=warning msg="Unable to delete container b01fadc160a9a10c4da6ac84c1d6e6082a30119de55eec23a3b9e855d71f6e04: `/usr/bin/sysbox-runc --root /run/runc --systemd-cgroup delete --force b01fadc160a9a10c4da6ac84c1d6e6082a30119de55eec23a3b9e855d71f6e04` failed: time=\"2024-01-29T05:09:24Z\" level=error msg=\"container init still running\"\n : exit status 1" id=3d1dc20d-694b-46b3-a7e4-4f38ffa6939c name=/runtime.v1.RuntimeService/CreateContainer

crictl stop $STUCK_CONTAINER_ID gives the following message:
E0129 09:10:39.944756  458151 remote_runtime.go:505] "StopContainer from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to stop container 7f5d0d7dbffa70f5a34a07a62ed25975a1910596819e29803c2b75c413af48fc: context deadline exceeded" containerID="7f5d0d7dbffa7"
FATA[0002] stopping the container "7f5d0d7dbffa7": rpc error: code = DeadlineExceeded desc = failed to stop container 7f5d0d7dbffa70f5a34a07a62ed25975a1910596819e29803c2b75c413af48fc: context deadline exceeded

I don't see anything out of the ordinary in journalctl -u sysbox-mgr or journalctl -u sysbox-fs on the bad node.
Are there more places that I can find potentially helpful information here?

Hi @top-oai,
Thanks for giving Sysbox a shot, hope you find it helpful (after resolving the issues you are having).
A few questions to narrow the problem space:

Are regular pods (i.e., non sysbox) working fine in the cluster?

If you launch a regular privileged pod with Docker inside (without sysbox), does that work fine?

Are the K8s nodes healthy (i.e., disk space & memory)?

For pods/containers that are already running, their local docker daemon gets stuck and the CI build jobs running on them (mostly just docker builds) time out.

Can you exec into one of those sysbox pods and try accessing /proc/uptime and any directory under /proc/sys? Those are emulated by sysbox-fs inside the pod, so I want to see if sysbox-fs is responding or if it got stuck somehow.
Also, inside the pod, create a dir such as /root/testdir and mount tmpfs on it: mount -t tmpfs tmpfs /root/testdir? Since sysbox intercepts that mount syscall, I want to see if that's working OK too.

My sample size isn't enormous, but this issue seems to reproduce more frequently on sysbox 0.6.3 (after ~30 builds per node) vs 0.6.2 with SYSBOX_ALLOW_TRUSTED_XATTR set to FALSE (after ~120 builds per node), however it does reproduce on both.

Interesting, that could be a clue; will keep that in mind as we debug.

The node's kernel version is 5.15.

Can you check if shiftfs is installed on the node? (lsmod | grep shiftfs)
Thanks!

Hi @ctalledo , I really appreciate the thoughtful response.

Thanks for giving Sysbox a shot, hope you find it helpful (after resolving the issues you are having).

Me too! :)

A few questions to narrow the problem space:

Are regular pods (i.e., non sysbox) working fine in the cluster?

Yes they are fine. And to be clear, other sysbox pods on other nodes continue to run fine also (until they eventually hit the same issue).

If you launch a regular privileged pod with Docker inside (without sysbox), does that work fine?

I'm not sure. It takes quite a few runs and time to get the confidence on whether it works or not, but it's probably worth the sanity check so let me try it out and get back to you.

Are the K8s nodes healthy (i.e., disk space & memory)?

Yes, all nodes in the cluster are healthy. Even the ones in the bad state do not indicate any disk pressure or memory pressure, and I've verified with htop/df.

Can you exec into one of those sysbox pods and try accessing /proc/uptime and any directory under /proc/sys? Those are emulated by sysbox-fs inside the pod, so I want to see if sysbox-fs is responding or if it got stuck somehow.

Also, inside the pod, create a dir such as /root/testdir and mount tmpfs on it: mount -t tmpfs tmpfs /root/testdir? Since sysbox intercepts that mount syscall, I want to see if that's working OK too.

Oh, hmm, I tried to exec in and I see the following message:
FATA[0000] join_namespaces:591 nsenter: failed to open /proc/1090344/ns/ipc: No such file or directory
FATA[0000] nsexec:809 nsenter: failed to sync with child: next state: Invalid argument
ERRO[0000] exec failed: container_linux.go:427: starting container process caused: process_linux.go:119: executing setns process caused: exit status 1
command terminated with exit code 1

and in sysbox-fs logs I see:
2024/01/30 05:27:22 FUSE: -> [ID=0xbe] Getattr valid=2562047h47m16.854775807s ino=5837941533507063541 size=0 mode=dr-xr-xr-x uid=231072 gid=231072
2024/01/30 05:27:22 FUSE: <- Lookup [ID=0xc0 Node=0xb Uid=231072 Gid=231072 Pid=1373073] "kernel"
time="2024-01-30 05:27:22" level=debug msg="Requested Lookup() operation for entry kernel (req ID=0xc0)"
time="2024-01-30 05:27:22" level=debug msg="Executing Lookup() for req-id: 0xc0, handler: ProcSysKernel, resource: kernel"
time="2024-01-30 05:27:22" level=debug msg="Executing Lookup() for req-id: 0xc0, handler: PassThrough, resource: kernel"
time="2024-01-30 05:27:22" level=debug msg="Executing nsenterEvent's SendRequest() method"

I can't remember now if I was ever able to exec into these stuck pods (even though their k8s status is "Running").

My sample size isn't enormous, but this issue seems to reproduce more frequently on sysbox 0.6.3 (after ~30 builds per node) vs 0.6.2 with SYSBOX_ALLOW_TRUSTED_XATTR set to FALSE (after ~120 builds per node), however it does reproduce on both.

Interesting, that could be a clue; will keep that in mind as we debug.

I think my sample size really wasn't big enough to conclude anything there after all, I switched back to 0.6.3 again and needed quite a few attempts to reproduce it there just like I did with 0.6.2. So, I think this is not a clue after all.

Can you check if shiftfs is installed on the node? (lsmod | grep shiftfs)

Yes shiftfs is installed.
Today I managed to get the problem to reproduce while --log-level debug was set. I see a few interesting things:

I see that sysbox-fs logs repeatedly show the lines:
Requested Lookup() operation for entry kernel (req ID=0xc0)
Executing Lookup() for req-id: 0xc0, handler: ProcSysKernel, resource: kernel
Executing Lookup() for req-id: 0xc0, handler: PassThrough, resource: kernel
Executing nsenterEvent's SendRequest() method
Container pre-registration started: id = 5702dfa6e839
Container 5702dfa6e839 will share sysbox-fs state with [0xc000420500 0xc000420a00]
Created fuse server for container 5702dfa6e839f30a576cc15a694ef0297b2b945483d2ff811fc442790ea742ac
Fuse server for container 5702dfa6e839f30a576cc15a694ef0297b2b945483d2ff811fc442790ea742ac shares state with container<PHONE_NUMBER>1ba743f2890f7740b6d6e24f9e7d6fabfdf4bc949d780857c77d2a
Container pre-registration completed: id = 5702dfa6e839

In our setup pods are frequently restarting as part of their normal operation. I'm not sure if the shares state with container line is a red herring or not, but it appears as though a new container is starting up before the previous container has shutdown?
Note that these shares state with container appear throughout the logs, both before and after the nodes have entered these bad states.
Note also that the logs show the container<PHONE_NUMBER>1ba743f2890f7740b6d6e24f9e7d6fabfdf4bc949d780857c77d2a gets shutdown shortly after this, and when it was created similar logs existed indicating that it shares state with an older container and so on.

I now recall that at one point I had tried mounting /var/lib/docker to a Kubernetes EmptyDir, and noticed the mount source at <some/host/dir> should be mounted in one container only, but is already mounted in containers <ids> error appearing in the logs because it looked again as if new containers were being created before the old containers were removed. But I'd expect the EmptyDir mount to only ever be mounted to one container at a time. I stopped mounting /var/lib/docker to an EmptyDir because I realized that sysbox already takes care of that mount for me, but perhaps this overlapping containers issue is the problem?
Note also that my Kubernetes cluster version is 1.26.10.

In the sysbox-mgr logs, for every container creation (both before/after becoming unhealthy) I see:
"containerdVolMgr: created volume for container 5702dfa6e839"
"dockerVolMgr: created volume for container 5702dfa6e839"
"autoRemoveCheck: Docker query start for 5702dfa6e839"
"FIXME: Got an status-code for which error does not match any expected type!!!: -1" module=api status_code=-1
"autoRemoveCheck: Docker connection failed for 5702dfa6e839: failed to retrieve Docker info%!(EXTRA errdefs.errUnknown=Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?)"

I'm a bit puzzled by the logged docker errors, the Kubernetes cluster is using containerd. Docker daemons are indeed running within each sysbox pod though, perhaps is this referring to those, then?

If you launch a regular privileged pod with Docker inside (without sysbox), does that work fine?

I'm not sure. It takes quite a few runs and time to get the confidence on whether it works or not, but it's probably worth the sanity check so let me try it out and get back to you.

Okay, I ran a whole bunch of tests in priviledged: true pods. It was easiest to do this by just replacing my sysbox deployment with a "regular" deployment. I ran these pods on the same nodes that the sysbox pods were running on. The "regular" pods were performing the same docker builds as the sysbox pods, but appear to not have the same issue as the sysbox nodes, no nodes entered into a failed state. Not only that, but the regular nodes were able to be scheduled and run successfully on the "bad nodes" from before when the sysbox pods were running. Additionally, after I ran this test, I re-deployed the sysbox pods and they went back to being unable to get start up on the "bad nodes".

Thanks for all the info @top-oai.
I think the fact that the kubectl exec into the sysbox pod failed is problematic and related to the failure you are seeing, so this is what we should focus on.
I need to dig into the code to understand how the exec can cause the errors you saw:
FATA[0000] join_namespaces:591 nsenter: failed to open /proc/1090344/ns/ipc: No such file or directory
FATA[0000] nsexec:809 nsenter: failed to sync with child: next state: Invalid argument
ERRO[0000] exec failed: container_linux.go:427: starting container process caused: process_linux.go:119: executing setns process caused: exit status 1
command terminated with exit code 1

I don't see anything suspicious in the sysbox-fs or sysbox-mgr logs.

Container 5702dfa6e839 will share sysbox-fs state with [0xc000420500 0xc000420a00]: sysbox-fs emulates /proc/sys state per pod, so this simply means that the pod has two containers sharing that state (i.e., the pause container and the main container).

autoRemoveCheck: Docker query start for 5702dfa6e839: sysbox-mgr checks if the higher level runtime is Docker by querying the Docker socket; if it's not there, then it will move on. In this case the higher level runtime is K8s + CRI-O, so there's no Docker and that should be fine. The "FIXME: Got an status-code for which error does not match any expected type log is coming from the Docker API client library used by sysbox-mgr to query Docker (it's harmless).

The mount source at <some/host/dir> should be mounted in one container only, but is already mounted in containers <ids> error occurs when Sysbox detects that the same host-dir is mounted into two containers at /var/lib/docker. It issues this warning because it knows that two Docker engines can't share the same /var/lib/docker (they'll collide on it). Not sure why you saw it with empty-dir, but let's park this since you've removed the mount (given that Sysbox will implicitly mount a host-dir into /var/lib/docker to avoid the overlay-on-overlay problem that occurs if it didn't).

Let me dig into the Sysbox code to see why kubectl exec would fail that way.
In the meantime, could you share your pod spec?

Thanks for reading through and providing background for everything.

Let me dig into the Sysbox code to see why kubectl exec would fail that way.

Sounds good, thank you!

In the meantime, could you share your pod spec?

Sure, you can find it here.

Hi @top-oai,
Question: when the sysbox pods enter a bad state, what happens if you launch a new Sysbox pod with an alpine image? Are you able to exec into that sysbox pod? Or does it also enter a bad state?

Okay, I've applied the following pod:
apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - image: alpine:latest
    command:
      - /bin/sh
      - "-c"
      - "echo hi && sleep 60m"
    imagePullPolicy: IfNotPresent
    name: alpine
  restartPolicy: Always
  nodeSelector:
    kubernetes.io/hostname: aks-sysboxcpu1-xbadx-xnodex
  tolerations:
  - key: sysbox-runc
    operator: Exists
    effect: NoSchedule

and I see the familiar results:
% kubectl describe pod alpine

...

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  10s   default-scheduler  Successfully assigned openai/alpine to aks-sysboxcpu1-20143122-vmss00000f
  Normal  Pulled     10s   kubelet            Container image "alpine:latest" already present on machine
  Warning  Failed     26s                  kubelet            Error: create container timeout

at this point I can get it to start by either deploying it to a good node, or by removing runtimeClassName: sysbox-runc from the spec.
I guess it's worth noting that the other images all have Ubuntu as their base.

Are you able to exec into that sysbox pod? Or does it also enter a bad state?

I am not able to exec into it because it can't get past the ContainerCreating state (and eventually lands in the CreateContainerError state after timing out). This is the same state as the pods from above enter into.
Let me see if I can get into this bad state while the pod is already successfully deployed.

Thanks for trying that.

I am not able to exec into it because it can't get past the ContainerCreating state (and eventually lands in the CreateContainerError state after timing out). This is the same state as the pods from above enter into.

The fact that you can't create an alpine pods means Sysbox is in bad shape then.
Any suspicious logs from sysbox-mgr, sysbox-fs, or CRI-O at the time the alpine pod is created?
When you launch a pod, the way it works is:

K8s talks to kubectl
Kubectl talks to CRI-O
CRI-O tells sysbox-runc to create the container
sysbox-runc sets up the container (namespaces, etc.)
sysbox-runc "registers" the container with sysbox-mgr and sysbox-fs
sysbox-runc exits and reports status to CRI-O (which in turn reports it to kubelet).
The container starts running; sysbox-fs monitors activities in the container (accesses to /proc and /sys, traps syscalls, etc.)

The pod creation must be failing in step (4) or (5) above, so look for sysbox-mgr and sysbox-fs logs at the time the alpine pod fails to be created.

Let me see if I can get into this bad state while the pod is already successfully deployed.

Okay I managed to start an alpine pod on a node, and then later have that node become bad. In this case, I've verified that I see the same thing I posted above (join_namespaces:591 nsenter: failed ...) when I try to exec into other pods on the node, however I am able to exec into the alpine pod without issue. Here's the output of a couple commands run on that pod: mount, df.

Any suspicious logs from sysbox-mgr, sysbox-fs, or CRI-O at the time the alpine pod is created?

I'm not seeing much unfortunately, it's not much different than what I've posted above (no smoking guns as far as I can see):

journalctl -u crio logs for the creation and subsequent container creation timeout of the alpine pod.
journalctl -u sysbox-mgr just has time="2024-02-01 16:55:45" level=info msg="registered new container 6e262f5d50d2", and I can't find a corresponding "unregistered". In fact it looks like I don't see any "unregistered" logs after the node has entered a bad state. Note that log-level is back to the default of info now, not debug like above.
journalctl -u sysbox-fs is similarly not very interesting, logging only the container creation:time="2024-02-01 16:55:45" level=info msg="Container pre-registration completed: id = 6e262f5d50d2"
time="2024-02-01 16:55:45" level=info msg="Container registration completed: id = 6e262f5d50d2, initPid = 2018155, uid:gid = 624288:624288"

So, since registration is being logged as completed, it seems that the issue is arising somewhere around step (5), or maybe between (5) and (6)? As far as I can tell from the pod status (e.g. it never gets past ContainerCreating ) and logs, it doesn't look like the container never actually starts running.
Also, maybe an aside, I'm not sure if it's benign or not, but throughout the sysbox-fs logs I see a lot of:
time="2024-02-01 09:58:56" level=error msg="No namespace file found /proc/48929/ns/user"

as well as a few:
time="2024-02-01 09:56:21" level=info msg="Unexpected error during NotifReceive() execution (no such file or directory) on fd 59, pid 40624, cntr 99578746acba"

and also a few:
time="2024-02-01 16:49:32" level=warning msg="Sysbox-fs first child process error status: exit status 1, pid: 1965267"

Aha, okay, I think I found the cause.
These pods are running CI jobs, and a few of them were failing (consistently). I had originally planned to dive deeper into that problem after this was sorted, but I decided to take another look at them.
It turned out that they were hitting the pod's memory limit and getting oom killed. The node had plenty of memory this whole time, it was just the pods that were hitting their limit. After resolving that, I've run enough tests consecutively now without nodes entering a bad state that I'm quite confident that this was the trigger.
The nodes were compiling C++ code inside of a docker build step when they OOMed. This obviously caused the pod to fail, but it seems as though there was a chance that it would also leave the node in a bad state, causing all the issues from above.
The OOM didn't really show up in any of the logs I was looking at, but I didn't think to look at the node's kernel logs, which is where the pod OOM was reported.
Everything's working great now and I'm unblocked, but it's still a mystery to me how an OOM in the pod might corrupt the node (or at least sysbox-specific state, based on the fact that regular runc containers can still work as usual). Let me know if you'd like me to run other queries on that.
Thanks again for working through this with me, I really appreciate that.

Hi @top-oai,
Thanks for the detailed explanation, and very happy to hear you found the issue.

It turned out that they were hitting the pod's memory limit and getting oom killed.

I see; what must be happening then is that the container's init process is getting killed by the kernel's OOM killer. That likely explains the failure to exec into the pod (e.g., join_namespaces:591 nsenter: failed to open /proc/1090344/ns/ipc: No such file or directory) because it's trying to enter into the namespace of a process that is dead. And this error log from sysbox-fs points to the same: time="2024-02-01 09:58:56" level=error msg="No namespace file found /proc/48929/ns/user". And finally, this also explains why sysbox-mgr reports the container is registered, but never unregistered.
BTW, the kernel log in the K8s node should show the OOM kill events IIRC.
Having said that, when the pod's processes are killed by the OOM killer, you would expect that the pod as a whole would be terminated. So there's a bug somewhere, but I don't know if its at the level of Kubectl, CRI-O, or Sysbox. I would need to reproduce and dig deep to see what's going on.

The implication here is that adding a memory limit on a pod is worse than not adding one, since OOM in either case has a chance to take out a node

I thought it's not taking out the full node correct, because you are able to launch a new pod (e.g., the alpine pod experiment we did above) and that works fine. Nonetheless I understand that the OOM kill is creating instability in the node (by killing some pods), so it's best to not use it until we get to the bottom of why.

It turned out that they were hitting the pod's memory limit and getting oom killed.

I see; what must be happening then is that the container's init process is getting killed by the kernel's OOM killer.

Well, not quite. It looks like the container's docker daemon's subprocesses triggerd by a docker build, which in this case are C++ compiler processes (e.g. cc1plus), are both 1) the processes consuming all the memory and 2) the processes that the OOM killer is targeting. I see this by looking at the kernel logs (e.g. journalctl -k). The OOM killer does not appear to be targetting the container's init process and I do not see this being reported as OOMKilled status by Kubernetes, the container persists.
In fact, if I manually echo "1000" > /proc/1/oom_score_adj during container startup, to encourage the kernel's OOM killer to kill the container's init process first, things seem to work better and I'm unable to reproduce the issue.
I think that if instead of 1.26 which I am currently using, if I was using Kubernetes version 1.28 which sets memory.oom.group to 1, I would automatically see this desired behavior instead of the OOM killer killing container sub-processes but leaving the container alive.
Nonetheless, termination of random processes within a container still should not put the node into a bad state.

BTW, the kernel log in the K8s node should show the OOM kill events IIRC.

It does. In particular in the logs of at least one bad node the OOM killer only killed cc1plus subprocesses (spawned by a docker build, and executed by dockerd all within the pod's only container), yet still the node entered the bad state. It didn't for example kill any container init processes, or the docker daemon, or anything else. I actually see multiple OOM kills of these cc1plus processes, but on "good nodes" I see zero. In these cases the docker build seems to get stuck and the CI job reports a timeout.

Having said that, when the pod's processes are killed by the OOM killer, you would expect that the pod as a whole would be terminated. So there's a bug somewhere, but I don't know if its at the level of Kubectl, CRI-O, or Sysbox. I would need to reproduce and dig deep to see what's going on.

Hmm, that's certainly what I initially expected would happen, and it's what I want to have happen, but as far as I can tell from some recent reading, that's not necessarily what the kernel OOM killer will do, and it certainly doesn't look like it's what is happening according to the kernel logs described above. Instead it seems the OOM killer is killing a subprocess within the pod.

The implication here is that adding a memory limit on a pod is worse than not adding one, since OOM in either case has a chance to take out a node

I thought it's not taking out the full node correct, because you are able to launch a new pod (e.g., the alpine pod experiment we did above) and that works fine.

Not quite, to clarify and summarize from above:

I'm not able to launch a sysbox alpine pod on a bad node, I see the familiar issue in this case of k8s not being able to create the container.
I can start regular runc non-sysbox alpine pods on a bad node and they function normally. I'm also using kubectl debug to access system logs of bad nodes which is also creating new regular runc non-sysbox ubuntu pods.
If I start a sysbox alpine pod on a good node, and while the alpine pod is running the node becomes bad, then I can exec into it.

It's a bit of an aside, but I do notice that if I start a pod that runs stress-ng --vm-bytes 200M --vm-keep -m 1 --oomable with a memory limit of < 200Mi, then Kubernetes reports container status as OOMKilled when not running on sysbox, but Completed when running on sysbox. In both cases the exit code is 0 which is expected (oom killer kills child procss of stress-ng, at which point stress-ng exits 0), the difference is just that when not using sysbox, Kubernetes flags the status as OOMKilled, but not when sysbox is used.

Ahh, after scrounging through the kernel logs more, I now am noticing something else that is unique to bad nodes (I don't see this output in good node kernel logs), and this usually occurs after an oomkill:
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: INFO: task runc:[2:INIT]:1987683 blocked for more than 120 seconds.
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:       Not tainted 5.15.0-1053-azure #61-Ubuntu
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: task:runc:[2:INIT]   state:D stack:    0 pid:1987683 ppid:1987654 flags:0x00004006
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: Call Trace:
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  <TASK>
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  __schedule+0x243/0x5f0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  schedule+0x69/0x110
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  request_wait_answer+0x138/0x210
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? wait_woken+0x70/0x70
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  fuse_simple_request+0x1b5/0x360
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  fuse_lookup_name+0x105/0x250
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  fuse_lookup+0x6d/0x1b0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  __lookup_slow+0x88/0x150
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  walk_component+0x145/0x1c0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  link_path_walk.part.0.constprop.0+0x23f/0x3a0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? path_init+0x2c0/0x3f0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  path_openat+0xb5/0x2b0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? list_lru_add+0xeb/0x120
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  do_filp_open+0xb2/0x160
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? __check_object_size+0x1d/0x30
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? alloc_fd+0xb9/0x180
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  do_sys_openat2+0x9f/0x160
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  __x64_sys_openat+0x55/0x90
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  do_syscall_64+0x5c/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? __rseq_handle_notify_resume+0x2d/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? exit_to_user_mode_prepare+0x49/0x100
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? syscall_exit_to_user_mode+0x27/0x40
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? do_syscall_64+0x69/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? do_syscall_64+0x69/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? do_syscall_64+0x69/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  ? do_syscall_64+0x69/0xc0
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  entry_SYSCALL_64_after_hwframe+0x62/0xcc
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: RIP: 0033:0x40408e
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: RSP: 002b:000000c00069e9f8 EFLAGS: 00000206 ORIG_RAX:<PHONE_NUMBER>000101
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: RAX: ffffffffffffffda RBX: ffffffffffffff9c RCX:<PHONE_NUMBER>40408e
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: RDX:<PHONE_NUMBER>080000 RSI: 000000c00013a840 RDI: ffffffffffffff9c
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: RBP: 000000c00069ea38 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000000
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000206 R12: 000000c00013a840
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel: R13:<PHONE_NUMBER>000000 R14: 000000c0000061a0 R15: ffffffffffffffff
Feb 05 02:19:23 aks-sysboxcpu1-20143122-vmss00001L kernel:  </TASK>

I don't have a lot of experience with fuse, but this seems like useful information. It doesn't look like quite the same thing, but after some searching around this does look like it has a few things in common with https://github.com/moby/moby/issues/42883.

Okay, one more update.
Unfortunately I don't have any new information to help debug the root cause of the issue, however I am now working around it. By upgrading from Kubernetes 1.26 -> 1.28, it activated the memory.oom.group cgroups v2 option and because of this, when a OOM kill is triggered, all processes in the container are killed, and as documented above, this seems to workaround the issue and I can have pods be OOM killed many times without having the node enter into a bad state.
Unfortunately though, since this memory.oom.group behavior is difficult to configure, I think that post-upgrade it will now be difficult to reproduce the issue even if I wanted to. I'm sorry that I wasn't able to distill a minimal repro case here, but it wasn't for lack of trying; even with the particular docker build step known to OOM isolated and repeatedly run in multiple nodes, it would not trigger the bug. It was perhaps a funny combination of events brought about by running our unique set of CI jobs that usually ran together, I suppose.
For what it's worth, I feel like my previous post, https://github.com/nestybox/sysbox/issues/771#issuecomment-1926267508, was the most promising clue so far about the root cause of this particular issue.

Thanks @top-oai for the extra info.
It's so hard to tell though, this is the type of issue that I need a local repro on in order to debug.
For the kernel stack trace you provided, what's interesting is that is on task runc:[2:INIT], which tells me it likely corresponds to the failed kubectl exec because runc:[2:INIT] is the name of the process that enters the container during kubectl exec, right before it execs into the binary passed to kubectl exec (e.g., say you run kubectl exec <pod> -- sh, then the process is briefly named runc:[2:INIT] right before it execs into sh).
The fuse_* functions in that backtrace are likely accesses that are intercepted by sysbox-fs. That is, sysbox-fs is a FUSE-based filesystem, mounted on subdirs of the container's /proc filesystem (e.g., /proc/sys). When doing kubectl exec into a Sysbox-based pod, sysbox-runc creates a process that enters the container. When that process enters the container, it may sometimes accesses files under /proc/sys to setup things, and those accesses will now go to sysbox-fs.
Thus, I think that backtrace is pointing to a problem in sysbox-fs, but not sure what problem it is though.
Hope that helps.


image_types_tegra: Adds EXTRA_IMAGEDEPENDS to fix the image task do_populate_lic_deploy

This fix is needed to bring the complete dependency chain in order to guarantee the recursive runtime dependencies [1] of do_populate_lic_deploy in do_populate_lic will run as expected.
[1] openembedded-core/meta/classes-recipe/license_image.bbclass:do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
Fix the following image build issues:
ERROR: lmp-base-console-image-1.0-r0 do_populate_lic_deploy: Couldn't find license information for dependency tos-optee
ERROR: lmp-base-console-image-1.0-r0 do_populate_lic_deploy: Couldn't find license information for dependency edk2-firmware-tegra

Can you help me with reproducing this?  I'm not seeing these errors in the demo distro, and the dependencies appear to be there already:
$ bitbake -g -c populate_lic_deploy demo-image-base
$ grep -E 'do_populate_lic_deploy" -> "(edk2|tos).*lic"' task-depends.dot
"demo-image-base.do_populate_lic_deploy" -> "edk2-firmware-tegra.do_populate_lic"
"demo-image-base.do_populate_lic_deploy" -> "tos-optee.do_populate_lic"

Can you help me with reproducing this? I'm not seeing these errors in the demo distro, and the dependencies appear to be there already:
$ bitbake -g -c populate_lic_deploy demo-image-base
$ grep -E 'do_populate_lic_deploy" -> "(edk2|tos).*lic"' task-depends.dot
"demo-image-base.do_populate_lic_deploy" -> "edk2-firmware-tegra.do_populate_lic"
"demo-image-base.do_populate_lic_deploy" -> "tos-optee.do_populate_lic"

I started to see this is some BSP layers after [1] and to get the above result with using the bitbake -g tip, I need the EXTRA_IMAGEDEPENDS addiction otherwise and don't see at all the edk2-firmware-tegra.do_populate_lic nor do I see tos-optee.do_populate_lic.
It is always possible that this could only exist in my DISTRO because of something miss configured.
Anyway I will to replicate the issue with the tegra demo distro next week to see if I can better understand why this is happening in my DISTRO.
Thank you for your help.
[1] https://git.yoctoproject.org/poky/commit/?id=7ebbd68f0fb237f511cd2629ef9f6fc4d8488736

Can you help me with reproducing this? I'm not seeing these errors in the demo distro, and the dependencies appear to be there already:
$ bitbake -g -c populate_lic_deploy demo-image-base
$ grep -E 'do_populate_lic_deploy" -> "(edk2|tos).*lic"' task-depends.dot
"demo-image-base.do_populate_lic_deploy" -> "edk2-firmware-tegra.do_populate_lic"
"demo-image-base.do_populate_lic_deploy" -> "tos-optee.do_populate_lic"

I started to see this is some BSP layers after [1] and to get the above result with using the bitbake -g tip, I need the EXTRA_IMAGEDEPENDS addiction otherwise and don't see at all the edk2-firmware-tegra.do_populate_lic nor do I see tos-optee.do_populate_lic. It is always possible that this could only exist in my DISTRO because of something miss configured.
Anyway I will to replicate the issue with the tegra demo distro next week to see if I can better understand why this is happening in my DISTRO.
Thank you for your help.
[1] https://git.yoctoproject.org/poky/commit/?id=7ebbd68f0fb237f511cd2629ef9f6fc4d8488736

Thanks @quaresmajose for the clarification. Please let us know if this happen only with your distro.


[URGENT] Upgrade jackson databind

This upgrades the jackson-databind package. According to Github, the current version is vulnerable.

Coverage remained the same at 100.0% when pulling 93b3d43270cec73c2c7dc87341beb5c9a032d14c on yuce:upgrade-jackson-databind into a69d42235f74bbe81785f84deeb0bc0b11f8e3e1 on pilosa:master.

Coverage remained the same at 100.0% when pulling 93b3d43270cec73c2c7dc87341beb5c9a032d14c on yuce:upgrade-jackson-databind into a69d42235f74bbe81785f84deeb0bc0b11f8e3e1 on pilosa:master.


Is this plugin dead? Any alternatives?

Is this plug-in still actively maintained? If not, what are the alternatives or is a fork more current? There haven't been any comments from the developer on open issues. And PR are not merged either. This really is too bad, since it is a very helpful plugin!

It also doesn't seem to work on Gradle 2.6.

I ran into the 2.6 issue too. I think the only major thing that's necessary to get this working with 2.6 is to upgrade the gradle wrapper (there were also some minor issues with checkstyle after the upgrade). I've got a pull request that I think does everything necessary here: https://github.com/bluepapa32/gradle-watch-plugin/pull/24.

That fix looks like it modifies the build of the plugin itself to work with gradle 2.6  What @antony is talking about is that the plugin doesn't work with a project that is using gradle 2.6.

Right. I don't entirely understand why upgrading the plugin's build system to 2.6 causes it to work on other projects that use gradle 2.6, but that has been my experience so far.

Stop using the plugin, gradle 2.5 supports this
http://gradle.org/feature-spotlight-continuous-build/

Thanks for the link, I didn't know about that feature. Reading through the docs it seems like that will cover a majority of use cases, but unfortunately not mine (yet). I've implemented a hot-reload feature with this plugin that works with jettyRunWar. They touch on that in the "Future Improvements" section at the end. Since these release notes are for 2.5, I searched through the 2.6 and 2.7 release notes as well, but I didn't see anything about non-Play hot-reload features (I would be happy to be corrected if I missed something).
Here's the relevant bit from the link you posted:

In the next release (2.6), Play support is coming to Gradle, and with that you’ll be able to start Play applications in a separate JVM for local development. With continuous build enabled, Gradle will hot-reload the Play application whenever classes or assets are changed. The Play plugin accomplishes this by registering the Play JVM with Gradle in a way that survives between builds.
We want to eventually evolve this Play specific reload functionality into a general feature, so plugins can have their own “hot-reload”-like behavior.


Pretrained Model

Hi, everyone! I am running the code. Due to the limitation of my computer, I cannot train the AVOD model. May I ask if you can share the pretrained model with me? Your reply is pretty appreciated. My email box is<EMAIL_ADDRESS>Please help me, many thanks!

Hi everyone,
I am also looking for AVOD model, my gmail is<EMAIL_ADDRESS>


judge can use telephone_field form

I want to use telephone_field in form for.
please check my commit.

Great, thanks for the PR! Happy to merge – however could you also add a spec for the field, e.g. https://github.com/MatsumotoHiroko/judge/blob/feature/use_telephone_field/spec/form_builder_spec.rb#L72-L74

Thank you very  match. I will try to add code in spec. please waiting.
Sorry, my English isn't better.

Excellent, thank you so much! :rocket:

very thanks you!!!
my first gem commit.

@MatsumotoHiroko congrats!


Failed to send IMAP email

Hi.  I have added a new IMAP email address to K9 and although everything else on the account is working I can't send email.  The email smtp verified ok. debug log attached.
k9log.txt

It doesn't look like the log contains all log messages generated by K-9 Mail. See https://github.com/k9mail/k-9/wiki/LoggingErrors
The log does contain the following error message:
Transaction failed. Please report this error to your system administrator (223E)

It is most likely the text sent by the SMTP server when it declined to accept the message for delivery. I suggest you contact your provider to ask for help.


Docs issue 4143: minor edits to the subscription address type description

Type of change

Documentation

Description
Docs: Minor edits to the subscription address type concept file to make it more clear
Checklist

[ ] Update/write design documentation in ./documentation/design
[ ] Write tests and make sure they pass
[X] Update documentation
[ ] Check RBAC rights for Kubernetes / OpenShift roles
[ ] Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
[ ] Reference relevant issue(s) and close them after merging
[ ] Update CHANGELOG.md

Can one of the admins verify this patch?

@k-wall, can you please review these edits to make sure the meaning is still technically correct? Thanks!


This addon appears to break fastboot server start up

In my application, I was able to isolate this addon as the culprit by uninstalling in my fastboot app and having the app start successfully.  Could we update this addon to be fastboot compatible?

Going ahead to close this issue.  Have learnt the best practise is to always trigger geolocation apis only on direct user interaction which can only happen client side.  As such, a trigger in fastboot should not be possible.


cw_geodata.vector_label.polygon.geojson_to_px_gdf im_path

cw_geodata.vector_label.polygon.geojson_to_px_gdf
im_path : str
Path to a georeferenced image (ie a GeoTIFF) that geolocates to the
same geography as the geojson(s). If a directory, the bounds of each
GeoTIFF will be loaded in and all overlapping geometries will be
transformed.
Based on my reading of this I assumed I could feed a directory containing multiple tif images, which fails.

Thanks! Resolved with #45.


Creating a modal dialog with Tailwind CSS

Proposed title of article
Creating a modal dialog with Tailwind CSS
Introduction paragraph (2-3 paragraphs):
A modal is a dialog box/popup window that appears on top of the main content requiring user interaction. A modal box disables the main content until the user explicitly interacts with the modal dialog. Modal dialogs are helpful because they give critical warnings to the user as a way to prevent errors.
In this tutorial, we are going to create our modal dialog box with Tailwind CSS. We will use various Tailwind classes to style up our modal dialog. We will also use JavaScript to add functionality to our modal dialog.
Key takeaways:
The reader will learn how to create a simple modal dialog
The reader will learn how to style up their modal dialog
The reader will learn how to add functionality to their modal dialog with JavaScript
References:
N/A

@ianmasae
Good afternoon and thank you for submitting your topic suggestion.
Your topic form has been entered into our queue and should be reviewed (for approval) as soon as a content moderator is finished reviewing the ones in the queue before it.

Sounds like a great topic @ianmasae - approved🚀.


Test RTF for online websocket server

Here are the results from GitHub actions for this PR
https://github.com/k2-fsa/sherpa/actions/runs/3529134025/jobs/5919902244
Greedy search
50 connections

100 connections
RTF: 0.0671
total_duration: 19452.481 seconds
(5.40 hours)
processing time: 1305.643 seconds (0.36 hours)

%WER = 3.35

Errors: 192 insertions, 143 deletions, 1424 substitutions, over 52576 reference words (51009 correct)

200 connections
RTF: 0.0767
total_duration: 19452.481 seconds
(5.40 hours)
processing time: 1492.854 seconds (0.41 hours)

%WER = 3.35

Errors: 192 insertions, 143 deletions, 1424 substitutions, over 52576 reference words (51009 correct)

Modified beam search
50 connections
RTF: 0.0703
total_duration: 19452.481 seconds
(5.40 hours)
processing time: 1366.988 seconds (0.38 hours)

%WER = 3.31

Errors: 206 insertions, 120 deletions, 1414 substitutions, over 52576 reference words (51042 correct)

100 connections
RTF: 0.0741
total_duration: 19452.481 seconds
(5.40 hours)
processing time: 1441.297 seconds (0.40 hours)

%WER = 3.31

Errors: 206 insertions, 120 deletions, 1414 substitutions, over 52576 reference words (51042 correct)

200 connections
RTF: 0.0793
total_duration: 19452.481 seconds
(5.40 hours)
processing time: 1542.081 seconds (0.43 hours)

%WER = 3.31

Errors: 206 insertions, 120 deletions, 1414 substitutions, over 52576 reference words (51042 correct)


Clpeak termux problem...

git clone https://github.com/krrishnarraj/clpeak.git
ln -sf /system/vendor/lib64/libOpenCL.so /data/data/com.termux/files/usr/lib/libOpenCL.so
cd
git clone https://github.com/KhronosGroup/OpenCL-Headers
cd OpenCL-Headers
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$PREFIX
cmake --build build --target install
cd ~
cd clpeak
cmake . -DCMAKE_CXX_COMPILER=clang++
make

And solution
Termux proot-distro
Ubuntu 20.04
Copy symbolic link /system/vendor/libOpenCL.so to usr/lib..
I got make error
Any solution.. ?

Your playstore version Clpeak Android apk working fine..
But have problem with clpeak termux..
MALI GPU
Mali G-610 MC6
Can you help please?
ALREADY tried with
export LD_LIBRARY_PATH="/system/vendor/lib64:/system/lib64"
export LD_LIBRARY_PATH=$PREFIX/lib:/vendor/lib64:/vendor/lib64/egl

@krrishnarraj please help wanted..
Clpeak termux given issue
Platform not found..


[BUG] EfficientNets not working in v0.6.7?

After upgrading to v0.6.7, I noticed that when training EfficientNets, the training loss would not budge, and the model was predicting the same thing no matter what the input was.
I tracked the output of each layer, and it seems like after the last BN layer, the model just outputs the same thing for everything (my data are scaled from -1 to 1).
Other model architectures (e.g., ResNeSt) did not have this issue.
At first, I thought it was an issue on my end, but then I downgraded to v0.5.4, and everything was working again- no other modification to my code.
Was there a change to the EfficientNet codebase in v0.6.7 that could explain this? I was using ImageNet pretrained weights across a variety of EfficientNets: efficientnet_b0, tf_efficientnet_b0, tf_efficientnet_b4, tf_efficientnetb6_ns, tf_efficientnetv2_m, etc.
Interestingly, when I loaded weights from a model that was trained before I upgraded, it worked fine at inference time even using v0.6.7.

@i-pan you'll have to provide specifics on your training setup... this python train.py /magenet --amp --model efficientnet_b0 --channels-last -j 8 -b 256 --drop-path 0.1 --drop 0.1 --opt adamw --weight-decay .05 --lr 3e-4 works fine for me, signs of life after first warmup epoch
Biggest change to the model since 0.5.4 was

switch to BatchNormAct layers through the  network so that BN + Act can be swapped for EvoNorm or other norm + act combo layers
Tweaks to the forward fn for a separate head method (should have no impact in normal use)
Some extra bits for grad checkpointing, layer name helpers to setup param groups, etc

The BN + Act change could have impact if you use normal sync bn helpers, those won't work and will render the model untrainable,  you must use the sync bn helper in timm.

SyncBN was the issue, thanks for the quick response!


6375 migrate role part3

Issue
https://github.com/gravitee-io/issues/issues/6375
Description

finish migration for role
Improve consistency between each new angular screen

Additional context
Pr link : https://github.com/gravitee-io/gravitee-ui-particles/pull/12

Gravitee.io Automatic Deployment
📚 View the storybook of this branch here

Gravitee.io Automatic Deployment
Congrats, all the CI steps were 🟢 . The Bot was able to deploy the build of this PR, so you can now try it directly here! 🚀
Notes: The deployed app is linked to the management API of the Element Zero team's environment.

Gravitee.io Automatic Deployment
📚 View the storybook of this branch here

Gravitee.io Automatic Deployment
Congrats, all the CI steps were 🟢 . The Bot was able to deploy the build of this PR, so you can now try it directly here! 🚀
Notes: The deployed app is linked to the management API of the Element Zero team's environment.


[WIP] Add try_fold, try_for_each, try_reduce

There are six variations here, matching the existing non-try suite:

try_fold and try_fold_with
try_for_each and try_for_each_with
try_reduce and try_reduce_with

All of them operate on Try::Ok values similar to the exiting non-try
methods, and short-circuit early to return any Try::Error value seen.
This Try is a pub-in-private clone of the unstable std::ops::Try,
implemented for Option<T> and Result<T, E>.
TODO and open questions:

[ ] Needs documentation, examples, and tests.
[x] Should we wait for Iterator::try_fold and try_for_each to
reach rust stable?  They were stabilized in rust-lang/rust#49607,
but there's always a chance this could get backed out.

Resolved: they're stable in 1.27

[x] Should we wait for stable std::ops::Try?  We could just keep
ours private for now, and change to publicly use the standard
trait later (on sufficiently new rustc).

Resolved: keep our pub-in-private Try for now.

[x] Should try_fold and try_fold_with continue to short-circuit
globally, or change to only a local check?

When I first implemented try_reduce_with, I use a try_fold +
try_reduce combination, like reduce_with's implementation, but
I didn't like the idea of having double full: AtomicBool flags
in use.
If try_fold only errors locally, then other threads can continue
folding normally, and you can decide what to do with the error
when you further reduce/collect/etc.  e.g. A following try_reduce
will still short-circuit globally.
Resolved: changed to just a local check.

Closes #495.

I'm torn about the proper behavior of try_fold; that's a very interesting question. So let me summarize to make sure I understand:

Today, try_fold yields up an iterator over Option<T> values (or whatever);
But once it encounters a None anywhere, it makes a best effort to stop producing further values anywhere;
So typically you would have at most one None in the sequence (though it's probably possible to have to more than one, if the timing is poor).

The alternative being proposed is that, once fold has settled on a sequential list of things to process, it will always finish, and hence you can very easily get multiple None in the sequence, and fold avoids global communication via an AtomicBool.
Given these two options, I suspect that we want the current behavior. It seems to me that the are only reason to want the second behavior is if you somehow wanted to ensure that some elements were processed even though a None would occur at some other point; but given that you don't control where the splits occur in a fold operation, you are not actually given that assurance.
The only property that you do get, afaict, is that if a None occurs later in the sequence, then the side-effects of earlier operations will occur (the side-effects of later operations may occur). That seems pretty thin and unlikely to be what you wanted.
In contrast, what we give up is that we potentially do a bunch of extra work, producing Some(_) values that will later be discarded.

The alternative being proposed is that, once fold has settled on a sequential list of things to process, it will always finish, and hence you can very easily get multiple None in the sequence, and fold avoids global communication via an AtomicBool.

I'm proposing that try_fold would still short-circuit its local work, and just lose the global check.  My thought is that you'd most often combine with try_reduce, which has its own global check.  But that won't know there's any Err/None until the preceding try_fold passes it through, so we still need that to happen ASAP.
I don't know if there's an actual useful semantic here, beyond avoiding redundant AtomicBools.  This is a small optimization for the try_fold().try_reduce() case, but the change would be noticeable if anybody did try_fold() followed by something that doesn't short circuit.  I don't know if that's realistic.  You're right that folks don't have control over splitting in general, but with_min/max_len do offer some control.

I guess what you thought I was proposing was that all sequential runs would be processed up to an Err/None, then resume that local sequence with a new accumulator?  So a local fold might actually produce multiple accumulations for the next consumer... that's interesting too.  I don't know if that's useful either.

bors r+


test ethernet: Remove xfail on MTU changes of VLANs

The NetworkManager patch has been included in 1.22.0 release.
Removed the xfail and fix the test case.

Coverage remained the same at ?% when pulling be9db27b52f8ed73780c056cf9bb957356235325 on cathay4t:fix_mtu_test into 9d41e49f4e612251b0527f3225dc00d9bc728a7b on nmstate:master.


Improve manifest public key loading

Remove the last user of the old Base58 functions and the functions along with them.

Current coverage is 57.89%

Merging #1499 into develop will increase coverage by +0.11% as of 65fcda9

@@            develop   #1499   diff @@
=======================================
  Files           952     950     -2
  Stmts         66405   66269   -136
  Branches          0       0
  Methods           0       0
=======================================
- Hit           38372   38366     -6
  Partial           0       0
+ Missed        28033   27903   -130

Review entire Coverage Diff as of 65fcda9

Powered by Codecov. Updated on successful CI builds.

This change looks correct to me.
:+1:

:+1:  Yay, less code!

Merged as 767d2535934f9390b12e458cd245fb36a3cf85cb


Where is the plugin folder

where is this folder : albertowd-oraclegrafana-datasource-bundle-1.0.0.tar.gz

Hello, the link is in the release page: https://github.com/albertowd/oracle-grafana/releases/tag/1.0.0

Thank you albert for your response.
however still didn't work for me

i extracted the linux amd bundle as attached in the a directory inside the plugins directory, added the environment variable and edited the grafana.ini.
restarted the service and still nothing.
Can you please help ?

Thank you albert for your response. however still didn't work for me
i extracted the linux amd bundle as attached in the a directory inside the plugins directory, added the environment variable and edited the grafana.ini. restarted the service and still nothing. Can you please help ?

Have the same problem.

same problem here, the plugin is not detected

For others having issue with this plugin not showing up.   I was having the same issue until I did this.
Navigate to /var/lib/grafana/plugins/
create folder named albertowd-oraclegrafana-datasource
Unpack contents of albertowd-oraclegrafana-datasource-bundle-1.0.0.tar.gz into
/var/lib/grafana/plugins/albertowd-oraclegrafana-datasource
Double check your grafana.ini file to make sure there is an entry in there with the same folder name for allow_loading_unsigned_plugins.
allow_loading_unsigned_plugins = pcp-redis-datasource,albertowd-oraclegrafana-datasource


External Auth/Adding Claims

Pluggable auth is a current industry trend for handling authentication/authorization. Being able to add Claims in JWT tokens is a way for users to securely pass user, role, and other information into persisted queries in a way that does not rely on a bespoke auth implementation.
While Stargate appears to have the ability for pluggable auth, the current implementation track seems to be predicated on the idea that users will be using it in conjunction with c* row-level authorization and user accounts. It isn’t a common paradigm for frontend applications to create a database user for each application user and they may not want to take advantage of row level security. Additionally, the current graphql schema also doesn’t support the ability to define role permissions so it is unclear how a user would set this up without a backend service or manual intervention.
The current track for pluggable auth seems to disallow the ability for frontend developers to do their own user/role management. Developers may also want to utilize a different provider such as Facebook auth or Auth0 for auth. This may lead to a brittle second system of credential handling in a frontend application.
We can support external authentication mechanisms and custom user handling with only needing to support JWT claim parsing on the graphql endpoint. This would allow a bring-your-own auth with a clear separation of concerns for the underlying layers.
Graphql Schema Example:jwt:
{
  "user_id": "a4cd2479-9915-4b03-9f78-dca71dc81dac",
  "iat":<PHONE_NUMBER>
}

mutation InsertCartEntry($cartId: TimeUuid, $quantity: Int) {
 insertCart(value: { cartId: $cartId, quantity: $quantity, _jwt_claim: {userId: "user_id"} }) {
  value {
   cartId
   quantity
  }
  applied
 }
}

query getCartByUser($pageState: String) {
 cart(options: {
     pageState: $pageState
 }, filter: {
    _jwt_claim: {userId: "user_id"}
}) {
   value {
     product {
        name
        price
     }
     quantity
   }
   pageState
 }
}

How does this relate to #339?

Actually #339 closes this issue.


⚠ Bofrost.be has degraded performance

In 163b0c6, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 855 ms

Resolved: Bofrost.be performance has improved in 65813bf after 6 minutes.


I want to pass params to route and handle them in custom Formatter

Hello guys, Is that possible to pass params to route like.
get '/', param1: 'param1' do
end

and handle them in formatter call
class Formatter
  def self.call(object, env, options={})
  end
end

Can you please help me with that? How can I archive that stuff?

The formatters have access to env, so I think env[Grape::Env::GRAPE_REQUEST] may have the actual request which has params.
I'll close this. Please use the Grape mailing list for questions.


Add ContactModel1D

Implementation of a contact model 1D that constrains only the translation along z-axis . It allows to apply a normal force while moving tangentially, which is useful for e.g. scraping or sanding tasks, but not allowed by current contact models (2D, 3D, 6D).
More specifically :

implementation of ContactModel1D
completion the contacts unittests factory
implemention of python binding
completion of ResidualModelContactForce to allow normal force cost

I'm not sure if things are missing, please let me know !

In addition, the flow of commits should not contain merged commits, so far and so on, which drastically reduce the overall readability of the history.
It is better to rely on a git rebase strategy.

Thanks for your feedback, I cleaned up the commits history . I'll fix the CI for this PR and open a new PR for contact models unification as discussed

Thanks for the review, and sorry for the delay. I tried to address all the requested changes, and I just fixed the CI. Please let me know if I can do anything else for this PR !

I merge this as I don't want to wait for gitlab-ci issues (it passes the other tests).
@skleff1994 thank you so much for your patience.

After this PR, Crocoddyl fails with clang-10/clang-12 but works with g++-9 under Ubuntu 20.04
I am getting build failures in the Python bindings from contact-1d.hxx using clang-12 on Ubuntu 20.04 (Eigen 3.3.7):
FAILED: bindings/python/crocoddyl/CMakeFiles/crocoddyl_pywrap.dir/multibody/contacts/contact-1d.cpp.o
/usr/bin/clang++-12  -DBOOST_MPL_LIMIT_LIST_SIZE=30 -DBOOST_MPL_LIMIT_VECTOR_SIZE=30 -DPINOCCHIO_WITH_URDFDOM -Dcrocoddyl_pywrap_EXPORTS -I. -Iinclude -I../include -I../bindings -isystem /usr/include/eigen3 -isystem /home/wxm/dev/install_ws/install/include -isystem /usr/include/python3.8 -isystem /usr/local/lib/python3.8/dist-packages/numpy/core/include -pedantic -Wno-long-long -Wall -Wextra -Wcast-align -Wcast-qual -Wformat -Wwrite-strings -Wconversion  -O2 -g -DNDEBUG -fPIC   -Wall -Wpedantic -Wextra -Wno-sign-conversion -Wno-conversion -Wno-deprecated-declarations -MD -MT bindings/python/crocoddyl/CMakeFiles/crocoddyl_pywrap.dir/multibody/contacts/contact-1d.cpp.o -MF bindings/python/crocoddyl/CMakeFiles/crocoddyl_pywrap.dir/multibody/contacts/contact-1d.cpp.o.d -o bindings/python/crocoddyl/CMakeFiles/crocoddyl_pywrap.dir/multibody/contacts/contact-1d.cpp.o -c ../bindings/python/crocoddyl/multibody/contacts/contact-1d.cpp
In file included from ../bindings/python/crocoddyl/multibody/contacts/contact-1d.cpp:10:
In file included from ../include/crocoddyl/multibody/contacts/contact-1d.hpp:206:
../include/crocoddyl/multibody/contacts/contact-1d.hxx:37:18: error: 'row' following the 'template' keyword does not refer to a template
  d->Jc.template row(0) = d->fJf.template row(2);
        ~~~~~~~~ ^
/usr/include/boost/python/object/make_ptr_instance.hpp:61:23: note: in instantiation of member function 'crocoddyl::ContactModel1DTpl<double>::calc' requested here
            type_info(typeid(*x))
                      ^

It passed the buildserver which uses g++ by default. I'll add another config with clang to catch this in the future.
Any advice on how to resolve this / which compiler settings did you use to get it to work?

There is a mistake in the code. The template keyword should not be there. clang, as usual, is correct.

Thank you for the quick confirmation @jcarpent. I found a similar issue & fix in caffe2 and am currently preparing a PR with a fix


Support wildcard matching in server.fs.allow list

Description
The issue arises when we are following the workaround in https://github.com/vitejs/vite/pull/15957#issuecomment-1952012252 for in the original issue https://github.com/vitejs/vite/issues/15945 that the fonts cannot be rendered correctly
Our developers install the packages on local environments and each developer has a different username. The current server.fs.allow does not support wildcard folder names in the path, which makes it difficult for us to synchronize the vite.config.ts file on git. We also don't want to switch the allowed path to the root folder or disable server.fs.strict for security reasons.
Suggested solution
The server.fs.allow should match patterns with * in the path string.
This will make it consistent with server.fs.deny
Alternative
No response
Additional context
To reproduce the issue, clone the repo in https://stackblitz.com/edit/github-tmpxuz-g4gr11?file=src%2FApp.jsx and install with yarn pnp.
Change the server.fs.allow to in vite.config.ts
[searchForWorkspaceRoot(process.cwd()), "/Users/<your user name(replace with<EMAIL_ADDRESS>to reproduce the issue.
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

I'll close this issue for now because the original usecase has now disappeared by #15957.


Support React 18

Support React version 18.
Problem;
Currently peer dependency will fail when using this in a React 18 project.

@Aeolun Please take a look if you can

Sorry, missed this. Let me have a look.

:tada: This PR is included in version 3.2.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

@all-contributors please add @ericmorgan1 for code

Thanks!


Add DV Chain to the list of merchants

What's the name of your business?
DV Chain (OTC)
What's the URL of the website?
https://dvchain.co
Write a brief description of your business/service (max 50 characters)
DV Chain is a cryptocurrency OTC desk that offers buying and selling of Monero.
Enter the URL where a Monero mention can be found
Monero logo in the "deep global liquidity" icon.
https://dvchain.co
How do you accept Monero? (choose only one option)

[ ] Monero integrations (github.com/monero-integrations)
[ ] Globee
[ ] Native wallet
[x] Other payment gateaway (Coinpayments, Morphtoken, etc. Please specify): third party custodian

How would you categorize your business? (choose only one option)

[x] Exchange
[ ] Block Explorer
[ ] Payment Gateway
[ ] Library and Helper
[ ] Tool
[ ] Service
[ ] Goods
[ ] Entertainment

Comment
No further comments

#1159