Use terraform-docs native saving to file

In terraform-docs v0.12.0, we've added ability to insert generated output directly into a target file (https://github.com/terraform-docs/terraform-docs/issues/286)! This can replace all the sed/awk/perl and drastically simplify the process to the point of only executing:
terraform-docs --output-file README.md --output-mode inject .

Check out all the available option to configure this.
We've just cut a v0.12.0-beta.1 for folks to start experimenting. Please feel free to start using and file any issues or feedback.

Does it make sense to deprecate terraform_docs_without_aggregate_type_defaults and terraform_docs_replace?
Why not just support native terraform-docs usage similar to how tflint is setup? It seems like the other terraform_docs_* options were there because of the lack of features out-of-the-box with terraform-docs at the time, and that's no longer the case.

This fix would be very helpful to me! 😸

Any chance to get this fixed and remove all those crazy dependencies like perl?
terraform-docs is already at version 16 (for a long time btw) and with correct .terraform-docs.yml I'm ending up with my README.md corrupted by the hook:
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
README.md updated successfully
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

While when I run terraform-docs from cli README.md is updated as expected.
Basically, terraform-docs is currently unusable if we want to preserve possibility to run terraform-docs manually from CLI...
Just for the reference, my .terraform-docs.yml looks like this:
formatter: "markdown table"
output:
  file: README.md
  mode: inject
  template: |-
    <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
    {{ .Content }}
    <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Would be also great to get rid of custom <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK --> and <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK --> markers and use default <!-- BEGIN_TF_DOCS --> and <!-- END_TF_DOCS -->...

Any chance to get this fixed and remove all those crazy dependencies like perl? terraform-docs is already at version 16 (for a long time btw) and with correct .terraform-docs.yml I'm ending up with my README.md corrupted by the hook:
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
README.md updated successfully
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

While when I run terraform-docs from cli README.md is updated as expected. Basically, terraform-docs is currently unusable if we want to preserve possibility to run terraform-docs manually from CLI...
Just for the reference, my .terraform-docs.yml looks like this:
formatter: "markdown table"
output:
  file: README.md
  mode: inject
  template: |-
    <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
    {{ .Content }}
    <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Would be also great to get rid of custom <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK --> and <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK --> markers and use default <!-- BEGIN_TF_DOCS --> and <!-- END_TF_DOCS -->...

@dzavalkin-scayle I'm coming across this behaviour too - there's some kind of conflict between terraform-docs which now writes to the README.md natively instead of printing the rendered text to stdout and the current pre-commit hook which takes stdout and does its own replacement into the README.md
I've worked around this by adding an argument into my hook which restores the previous behaviour of making terraform-docs return the parsed output on stdout
        args:
          - --args=--output-file=""

Maybe this helps you too? In the long term I agree that the entire wrapper should just be simplified and require a recent version of terraform-docs to do this natively; but I don't have time for that at the moment so this was a quick band-aid for me.

@dzavalkin-scayle @gchappel

--output-file="" set by default starting from https://github.com/antonbabenko/pre-commit-terraform/releases/tag/v1.92.2
Starting from https://github.com/antonbabenko/pre-commit-terraform/releases/tag/v1.93.0 <!-- BEGIN_TF_DOCS --> and <!-- END_TF_DOCS --> markers used by default (with under-the-hood replacement of old-one markers).

Previously, you just can specify - --hook-config=--use-standard-markers=true (starting from https://github.com/antonbabenko/pre-commit-terraform/releases/tag/v1.86.0)

Please let me know if 1.93.0 can be perceived as breaking change for your workaround with
  template: |-
    <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
    {{ .Content }}
    <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

We can add replacement of <!-- -->  in specified --args=--config= file to hook, but I am not sure is it worth it, as we never promised to support such workarounds.
I see that it used in 32 public repos
With 1 quite problematic case like in this TFLint Azure CI setup:
Is pointing all folks which use such workarounds to
https://github.com/antonbabenko/pre-commit-terraform/blob/99fceb87b07cdbb97111c3b4eb17d3f70d36e13c/README.md#L584-L590
will be enough?

Also, changes in v1.93.0 open the path to implementation of  Use terraform-docs native saving to file without needs of breaking changes, as used markers now are the same

This issue has been resolved in version 1.94.0 :tada:


When disconnecting an external monitor, relative desktop 1 on both monitors condense to Desktop 1

I work often with an external screen, but sometimes I end up taking my laptop to a meeting. When disconnecting the external screen, Desktop 1 on the external monitor has all of its windows shifted to Desktop 1 on the internal monitor. When plugging the external monitor back in again, the windows that were originally on Desktop 1 on the external monitor all shift back to their respective places. I would like yabai to help by putting the windows from the external monitor onto a new desktop, instead of merging them onto the same screen (I think it's actually Mac doing this right?).
So, in the following example:

Monitor
Desktop
Window

1
1
1, 2

1
2
3

2
3
4

2
4
5, 6

This is what currently happens on disconnecting the external monitor:

Monitor
Desktop
Window

1
1
1, 2, 4

1
2
3

1
3
5, 6

But what would be preferable is:

Monitor
Desktop
Window

1
1
1, 2

1
2
3

1
3
4

1
4
5, 6

And ideally it would go back to its original state when plugging in the external monitor.
Right now, my current work around to avoid merging windows onto Desktop 1, is to just leave (in the example above) Desktop 3 empty.
Also, thanks so much for your work here. This is the first, and likely the last tiling window manager I'll use, and it's great to see there's active development!

(Sorry for closing, misclicked on mobile)
This has been previously discussed, and is a hard to solve issue. Yabai isn’t doing anything here—you’re observing the default behavior of macOS. You would need to serialize all spaces and their windows on almost every event to try to cache their state, just to create one new space (the one that’s missing when compared to the cached spaces) and move all windows back to it after a Display was disconnected.

I've experienced your pain as well @nzchicken! I took @koekeishiya's advice from this issue and stopped using the first desktop on my external display. I just leave it there empty and everything else works as expected. To be fair, I'm not in the habit of navigating to desktops using prev/next so I don't notice the extra desktop in the middle.

Thanks @dominiklohmann and @johnallen3d! I suspected that it was macOS just making life harder. I'll just that workaround of keeping the desktop empty, as I can imagine that the payoff for doing this stuff internally in yabai is going to be a super pain.


[CELEBORN-1240][FOLLOWUP] Introduce Husky Configuration to Celeborn Web

What changes were proposed in this pull request?
Removes the pre-commit of husky.
Why are the changes needed?
Husky pre-commit is not necessary for git commit. There is contribution guide and CI check for the frontend code style.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Local test.

LGTM. Thanks. Merged into main(v0.5.0).


Added pet-projects block

Added something individul. This is the pet-projects block

it is my own block here


Refactor/remove react router dom

Change links from react router dom for next links
Remove react router dom package

close #161

@heronlancellot can you review this one pls

hey @diegosano  ! Niceee Let's go !


Introduced XTypes and reworked the type graph infrastructure

This PR includes several issues we talked about during our last meetings.

introduced PrimitiveType, ClassType, ... which extend Type
introduced concrete types for edges and made TypeEdge abstract
fixed bugs
reduced the role of PrimitiveKind, ClassKind, ... accordingly
TypirProblem is an interface to extend now instead of a type union in order to enable additional problems by users
conversion: added structures to deal with transitivity and cycles, actual implementations of the algorithms are still missing (and will be added in another PR)
first support for bottom types
improved validation API
introduced type.getName() to have a short string to show to users
improved some details for FunctionTypes
unified and extended logic to compare types with each other
enforce uniqueness of new types, fixed calculation of uniqueness of functions
refactorings, simplified code, comments

For the review, it might be easier to check the single commits.

Thank you @insafuhrmann and @Lotes a lot for your reviews!
I resolved most of your comments and pushed the changes.


Remove Outdated Lib path Files

sfmlLibNames.txt and tguiLibNames.txt are useless after the switch to cmake.
Lets clean up these files.

Merged into Dev


Fixed many association and added notNull/unique constraints to mysql

Your newest update broke the many association by the saveRecord Callback by adding in the idProp paramerter.
Also this is not very tested but I added in a notNull constraint so that ever variable is not required to be not null, and a unique constraint both with
"unique" = true
and
"unique" = {"scope" = "second_column"}
possibly this can be added  and if these are well accepted I can work on adding the validations like in issue 40

I'll take a closer look on this later this week, thank you for posting.

I made the fix on the many associations. I'll bring more updates to have those kinds of features.

I've made the fixes to many associations. I want to add notNull but perhaps reverted. As usually people tend to use NOT NULL, I'm thinking of an option called allowNull or just null. The unique key is also interesting but I want to keep mysql/postgresql in sync, so I have to see how to do it also in pg.


Allow custom decorators to overwrite Bolt arguments injection

Hi,
Opening this issue as a continuation of the conversation in #688, as I believe there's some design decisions that need to be made prior to reviewing and implementing the suggested changes.
Suggested Changes
My suggestion is changing the get_arg_names_of_callable to:
def get_arg_names_of_callable(func: Callable) -> List[str]:
    # Check if the func is decorated and if the decorator requires specific data injection
    # in the case of a decorator that handles *args and **kwargs, the length of inspect.getfullargspec(func).args is 0,
    # so we can defer to the unwrapped func
    if hasattr(func, '__wrapped__') and inspect.getfullargspec(func).args:
        return inspect.getfullargspec(func).args
    return inspect.getfullargspec(inspect.unwrap(func)).args

Checking if the listener function contains a __wrapped__ attribute (populated by functools), allows the function to to instead inspect the wrapper itself.
In the cases where the wrapper has not defined specific args injection (ie, wrapper signature is def wrapper(*args, **kwargs)), the length of inspect.getfullargspec(func).args is 0, in which case the function defaults to unwrapping and inspecting the listener itself.
What does this achieve?
This change would allow users to have decorators that overwrite the arguments injection as defined by the listener functions, to illustrate;
With the current behavior, having a listener defined as:
@app.event("message")
def message_listener(event, say):
    say(f"hi! You said {event.get('text')}")

tells Bolt to inject the event and say arguments when calling the listener function:
>>> from test_file import message_listener
>>> from slack_bolt.util.utils import get_arg_names_of_callable
>>> get_arg_names_of_callable(message_listener)
['event', 'say']

Implementing the changes, would instead result in:
def custom_decorator(function):
    @wraps(function)
    def wrapper(event, context, say):
        # Do something here
        return function(event)
    return wrapper

@app.event("message")
@custom_decorator
def message_listener(event):
    print(f"Got {event}")

>>> from test_file import message_listener
>>> from slack_bolt.util.utils import get_arg_names_of_callable
>>> get_arg_names_of_callable(message_listener)
['event', 'context', 'say']

Or a custom decorator with additional arguments
def custom_decorator_with_value(custom_variable):
    def decorator(function):
        @wraps(function)
        def wrapper(event, context, say):
            # Do something here
            # Only pass the custom variable to the listener
            return function(custom_variable)
        return wrapper
    return decorator

@app.event("message")
@custom_decorator_with_value("test value overriding the args injection")
def message_listener(user_var):
    print(f"Got {user_var}")

While still retaining the fix of #689:
def general_decorator(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        # Wrapper hasn't defined specific args injection,
        # Bolt will inject whatever the listener func has requested.
        # It's up to the user to manipulate the kwargs and pass them onwards
        return function(*args, **kwargs)
    return wrapper

@app.event("message")
@general_decorator
def message_listener(event, client, say):
    print(f"Event: {event}")
    print(f"Client: {client}")
    print(f"Say: {say}")

>>> from test_file import message_listener
>>> from slack_bolt.util.utils import get_arg_names_of_callable
>>> get_arg_names_of_callable(message_listener)
['event', 'client', 'say']

Assumptions and Downsides
Unfortunately, this change operates based on a few assumptions, each of which comes with downsides, and this is the reason why I believe it's best to talk through it first before submitting a PR straight off the bat; perhaps the shortcomings will outweigh the benefits and implementing this would not be a good fit for the Bolt's design principles and long-term maintenance.

The request relies entirely on the assumption that any and all decorators would have been declared via the functools @wraps functionality as that wrapper is the one that populates the __wrapped__ argument;
The request, if implemented, will only work in the case of one decorator override - unfortunately, functionality becomes very obscure in cases where users have two or more decorators applied to a listener function and it opens the door to human error in cases where one decorator has been defined to require event and client, while the next one expects resp and ack.
The solution to this, of course, would be on the end user - they must be consistent in their decorators' definition and ensure that either all of them require the exact same args, or let the listener function be responsible for args injection while the decorators themselves only access the kwargs and modify them as required (in other words, the last code block above);
This does not account for frameworks such as the one mentioned in #709 , I have no experience with it and I can't provide feedback as to how it'd work, if at all, in that case.

Category (place an x in each of the [ ])

[x] slack_bolt.App and/or its core components
[x] slack_bolt.async_app.AsyncApp and/or its core components
[ ] Adapters in slack_bolt.adapter
[ ] Others

Requirements
Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

Hi @dimitarOnGithub, thank you so much for taking the time to write this.
First off, apologies for any confusion, but when I read your comment on the closed issue, I was imagining slightly different use cases (that being said, I was fully unsure, so I asked you to share more details though).
The flexibility described here is something the bolt-python project would like to avoid, because, as you mentioned, customizations of listeners/middleware utilizing decorators could cause unexpected behavior and be difficult for humans to debug. Additionally, future enhancements on the bolt-python side may create conflicts.
If I understand your intention correctly, customization before running a listener is achievable with custom global (@app.use/middleware) or listener-specific (a parameter to @app.event) middleware. We would like to provide a single approach, fully controlled by the framework, to handle this.
I apologize again if this was a waste of your time, but I hope my response here clarifies things.

Hi @seratch ,
Please do not apologize! In fact, I should be the one doing that considering I didn't make my intention in the original comment clear enough, it'd have surely saved you the time you had to put aside to review this instead.
I completely understand, hence why I wanted to first provide all the details before jumping the gun and submitting a PR, I appreciate you taking the time to review my suggestion and provide feedback.
In my case, I'm already using some middleware to filter out and manipulate incoming requests, but as I use Bolt as a part of a bigger application, I wanted to be able to further abstract chunks of the requests and only provide listener functions with the bare minimum they need to perform a given request. Either way, it's not the end of the world for me and given the flexibility of the library, I'm sure I'll be able to find workarounds.
Again, thank you so much for your time and I can't wait to see what the future holds for Bolt!


Please cherry pick dart2js packagespec implementation

Please cherry pick 63681d5b59a26d00233ee582e8f9b9fa63c0bd80 and fee40ca9e494cf6e854d81de3a53e37f70247241
This adds support for this DEP to dart2js.

:bowtie:

cc @kasperl @ricowind @whesse

Merged into 1.11.0-dev.5.2, which is now built, but not yet promoted.

promoted


intel-bootc image fixes

Containerfile remade to use packages from habanalabs repo.

/hold
Need extra work to make it run from repo.

rebase is needed

@enriquebelarte When you rebase, if https://github.com/containers/ai-lab-recipes/pull/489 is not merged, please add it to this PR so we will blow each other up.

It's rebased yet.


tvg-logo=

Hello there. first of all i like to thank you for the hard work . i have question regarding the   the channel logo ,how can we activate to show the logo in the in the screen?   Thank you
REM #EXTINF:-1 tvg-logo="http://myserver.com/euro-news.png" group-title=news",euro-news

Hi sorry for long reply, was it you, who wrote me on my email?
As I said, when the uri is parsed, you need to set the fields described here
I was about to make these changes, but too busy with other projects right now, if you would implement that, this would be great. this is not a hard task to-do.

thank you so much for your update . i my self dont have roku device at the moment .i will let my friend tested i will update you.Greetings

provide the size you need in the .m3u
or use this: https://sdkdocs.roku.com/display/sdkdoc/MarkupList
вс, 23 сент. 2018 г. в 0:22, zebiden<EMAIL_ADDRESS>
Thank you for your hard work .yes we can see the logo for the channels
now. but we can not size the logo .ether is to big or to small .how can we
adjust the size of to icon.
Thank you
Regards
Richard
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sudo97/Roku-IPTV/issues/6#issuecomment-423774376, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADjhTMtiYBdq9d54GMbxY11JI2050K5tks5udqoMgaJpZM4WdmqE
.

Hello and thank you for your email the icon size 100X100 png
Thank you
Richard

Hi sudo97  ,please any update for the icon .i try it any way to set the icon size the 100x100 .i didn't succeed .i will appreciate your help.
Thanks

Hi sudo97 yes sure , there is 3 things i like to add to it, icon ,category and favourite. please contact me via skype :martinex.ltd


Support thematic breaks

Bug/issue #: rdar://124275218
Summary
Markdown supports thematic breaks (aka horizontal rules) that are represented by placing three or more hyphens, asterisks, or underscores on a separate line, surrounded by blank lines. This PR adds support for thematic breaks within Swift DocC, adding a thematicBreak element in the render JSON.
[Minor fix]: also fixed a consistent typo in RenderContentCompilerTests.
Dependencies
DocC render needs to add support in order to render this thematic break from the render JSON.
Testing
Create a thematic break by add three or more hyphens, asterisks, or underscores on a separate line, surrounded by blank lines, and compile the documentation. The render JSON for the data type should now include an element of type "thematicBreak".
Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.

[x] Added tests
[x] Ran the ./bin/test script and it succeeded
[x] Updated documentation if necessary - N/A

Works perfectly for me 👍 - although as you say DocC Render doesn't support this yet but I see the same block content type appear in Render JSON.

It does now as of this PR: https://github.com/apple/swift-docc-render/pull/791 :)

@swift-ci Please test

@swift-ci please test


Add syntax for supplying multiple fastq files to docs

see https://groups.google.com/d/msg/transrate-users/eCHEJKzhwO0/o9e5a5CVJQwJ

Added to command-line help: 1a1404748180a88cf99bdce1ca698c396cd4762b


objects.font: fix iter method raising AttributeError

as reported by @mashabow, Layer object only has 'keys' method, and no 'values'.
https://github.com/typesupply/defcon/pull/48/files/a9c0c9f961fcc6064e14c5d629db09fd20b67ca7#r63487919
We can simply yield the values by iterating on the layer's keys().
Or we could add a values() method to Layers, like this:
E.g.:
def values(self):
    return [self[name] for name in self.keys()]

Your patch is a good solution.


It is better to translate 'Break Container Lease' to '中斷容器租用' in zh-TW

Storage Explorer Version: 1.11.1
Build: 20191203.1
Branch: master
Language: Chinese(zh-TW)
Platform/OS: Windows 10/ Linux Ubuntu 18.04/macOS High Sierra
Architecture: ia32/x64
Regression From: Not a regression
Steps to reproduce:

Launch Storage Explorer.
Open 'Settings' -> Application (Regional Settings) -> Select '繁體中文' -> Restart Storage Explorer.
Expand one storage account -> Blob Containers.
Create a blob container -> Acquire lease for it.
Right click the leased blob container -> Select '中斷租用'.
Check the title of the prompt dialog.

Expect Experience:
Show '中斷容器租用'.
Actual Experience:
Show '中止容器租用'.

On review.
File: loc\en\src\Standalone\resources\Azure.json
ID: 0; "TitleContainerLeaseBreak"
SRC: Break Container Lease
TGT: 中止容器租用

Implemented suggestion.

@johnliu2016 Is this change checked in yet? I don't see any changes to zh-Hant/Azure.json yet. To be clear, we're changing TitleContainerLeaseBreak tp "中斷容器租用", yes?

@craxal, the translation is changed to '中斷容器租用', please expect it in newer build, it's through loc pipeline.

This issue has been fixed on build 20191212.4. So we close it.


fix(gtk): menubar allocation

What kind of change does this PR introduce? (check at least one)

[x] Bugfix
[ ] Feature
[ ] Code style update
[ ] Refactor
[ ] Documentation
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes. Issue #___
[x] No

The PR fulfills these requirements:

[ ] When resolving a specific issue, it's referenced in the PR's title (e.g. fix: #xxx[,#xxx], where "xxx" is the issue number)
[ ] A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

Before merging: we need to test on linux;

Launch a window without allocated menu
SetMenu and check the menu show correctly
SetMenu again to make sure menu got OVERWRITTEN and not appended to existing menu's

Toggling now works great on Linux. A window without menu is also rendering correctly.


Official Docker image on Docker Hub

Would be great to have an official Docker image on Docker Hub, since there are lots of them and finding one built from this very same repo Dockerfile would be more trustable and would help focusing contributors Dockefile enhancements.
Thus, documentation could easily include a canonical "docker run" statement to quickly spin up a ME instance with just a single command.
There are also "automated builds" available that can be triggered on each commit automatically:
https://docs.docker.com/docker-hub/builds/

I personally don't use Docker Hub, but maybe @knickers has some insight to the feasibility of this request?

I would be willing to look into building a Docker official image. Until then @pataquets, you can use knickers/mongo-express as an easy "docker run" to quickly spin up an instance. I build it straight from the latest npm version as often as I can (which is usually within a day or two of the latest version).

Manual builds? Ouch. I'll look into getting Travis set up to automagically build Docker? I don't use it myself much so not to sure, but I'll have a wee look https://docs.travis-ci.com/user/docker/

@wulfsolter I think what @knickers meant by manual is bumping the version number that's embedded in the Dockerfile when there's a new release -- the build itself is managed by Docker's Automated Builds, so should always be up to date :+1:

I was going to write a quick wee pre-commit test to make sure I don't update package.json version without updating Dockerfile, but I can't see a version string in the Dockerfile

Oh sorry, @tianon is talking about a separate repo with my own Dockerfile
here https://github.com/knickers/mongo-express. This one pulls a specific
version from npm.
The Dockerfile in https://github
com/andzdroid/mongo-express doesn't need a version number because it builds
from the source code already in the directory.
On Feb 17, 2016 11:02 AM, "Wulf Sölter"<EMAIL_ADDRESS>wrote:

I was going to write a quick wee pre-commit test to make sure I don't
update package.json version without updating Dockerfile, but I can't see
a version string in the Dockerfile
https://github.com/andzdroid/mongo-express/blob/master/Dockerfile
—
Reply to this email directly or view it on GitHub
https://github.com/andzdroid/mongo-express/issues/181#issuecomment-185327206
.

Gotcha. Big fan of automating manual tasks and avoiding double handling. Let me know if there's anything I can do to help - Travis is hooked in https://travis-ci.org/andzdroid/mongo-express so if there's some way to get Travis to update version numbers that might save the hassle.

@wulfsolter, I created a new github organization for mongo-express and invited you as an owner. What do you think about moving development there?

Good call! @knickers

Hey @wulfsolter. The docker developers are taking forever getting around to reviewing the pull request for our official mongo-express image. I think we should include a link to my knickers/mongo-express image in the README.md so people will know that it is officially supported. What do you think?

Good plan :+1:

Awesome, done :smile:

The official docker image was approved today! :birthday: It is now available here: https://hub.docker.com/_/mongo-express/

Finally gave a try to the new official image. It works beautifully.
Thank you very much everyone!


[Improvement]: The optimizer adds a cache of eq delete files to reduce repeated IO cost of eq delete files

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?
For large tables written by Flink, each commit will submit an EQ DELETE file associated with all previous data files. Most of the generated optimize tasks will repeatedly read this EQ DELETE file, causing duplicate IO cost.
How should we improve?
Each JVM(taskmanager, executor) in the Optimizer generates a Cache to cache the EQ DELETE File.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Subtasks
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi, it is a good idea to cache the delete file data.
However, before proceeding with the actual implementation, we may need to clarify some details like:

Do we only cache data for equality-deleted files?
How to limit the size of the cache?
What kind of cache eviction strategy to use?
When allocating tasks in AMS, is it necessary to consider cache hits?

IMO,

Do we only cache data for equality-deleted files?
Yes, the position delete files only match limited insert files, so it seems to be unnecessary to cache them for now.

How to limit the size of the cache?
If we cannot fully cache a delete file, the cache will be ineffective. To better protect memory and cache as many files as possible, we may only want to cache delete files that are relatively small in size. Additionally, for delete files that are too large, they are now filtered during reading using the bloom filter of insert files, and they are not suitable for caching either.
Besides, we should add a configuration to control the maximum size of the cache, preferably in bytes.

What kind of cache eviction strategy to use?
LRU seems to be good enough.

When allocating tasks in AMS, is it necessary to consider cache hits?
Considering cache hits during scheduling can better utilize the cache. However, initially, we can overlook this and consider adding this optimization later.

Thanks for your reminder on the details.
The idea is from Iceberg community,  Introduced delete file cache in PR #8755 for spark executors.

IMO,

Do we only cache data for equality-deleted files?

Yes, the position delete files only match limited insert files, so it seems to be unnecessary to cache them for now.

Yes.

How to limit the size of the cache?

If we cannot fully cache a delete file, the cache will be ineffective. To better protect memory and cache as many files as possible, we may only want to cache delete files that are relatively small in size. Additionally, for delete files that are too large, they are now filtered during reading using the bloom filter of insert files, and they are not suitable for caching either. Besides, we should add a configuration to control the maximum size of the cache, preferably in bytes.

Iceberg community imported spark.sql.iceberg.executor-cache.max-total-size to limit memory usage.
The cache is mainly on the Optimizer side, we can add a parameter, like -msz, which requires an upper limit on memory usage to stabilize the operation of the optimizer.
For the part using the bloom filter, we might be able to skip caching.

What kind of cache eviction strategy to use?

LRU seems to be good enough.

How about LRU + active expiration (when optimizer table changed)?

When allocating tasks in AMS, is it necessary to consider cache hits?

Considering cache hits during scheduling can better utilize the cache. However, initially, we can overlook this and consider adding this optimization later.

I also think configuring it on the optimizer side at first is enough.

@zhongqishang Thank you for providing detailed information on specific improvements to the Iceberg community. I'm delighted to see that this improvement has been merged into the Iceberg Spark connector.

How about LRU + active expiration (when optimizer table changed)?

I understand your point that if the executor thread switches to a new process, it can evict the existing cache. This seems like a good strategy, but it requires AMS to always schedule tasks for a process together, which is currently the case. However, it seems difficult to determine the timing of switching to a new process in situations like multiple threads sharing a JVM environment.

@majin1102 Thanks for your reply.
I just want to cover some tables with many data files. The newly added eq files will all be associated. The eq delete in the optimizer will read the number of tasks repeatedly. If we cache it, we will speed up the merge and reduce the IO cost of the file.
There will be some adverse effects on tables that are not in this scenario, because cache will involve the materialization of data. Like @zhoujinsong said, we can consider cache hits during scheduling to reduce the effect.


how can i export Efficient_ns to onnx?

Can tf_efficientnet_b3_ns convert to onnx for using in tensorrt?

@letmejoin it's easier to do that with my other version of that model in https://github.com/rwightman/gen-efficientnet-pytorch ... it includes an export script and a special export friendly pad layer that requires passing through one batch at your export resolution before doing the export (to set the padding layers). I haven't moved support for that layer here yet.

@letmejoin it's easier to do that with my other version of that model in https://github.com/rwightman/gen-efficientnet-pytorch ... it includes an export script and a special export friendly pad layer that requires passing through one batch at your export resolution before doing the export (to set the padding layers). I haven't moved support for that layer here yet.

thanks，I have done！


Changed to Jekyll

So what we should do, is set up a clone of the gh-pages directory in as a subfolder.
Change the page to be generated by yaml so we can have the header and footer static and the generated table get included.
This way the whole process can be automated and updated daily.
Also that way the landing page can be styles a little nicer and not require a single manual interaction.

Good idea! Include the template of the landing page to add some JS an styles.

This is now fixed.


Docker build fails when downloading mdm-ui-7.3.0.tgz download from jenkins.cs.ox.ac.uk

Receiving the following error several times when trying to run a docker compose build this morning:
 => CACHED [mauro-data-mapper stage-1 1/3] FROM docker.io/maurodatamapper/tomcat:9.0.71-jre17-temurin-m  0.0s
 => [mauro-data-mapper internal] load build context                                                      0.0s
 => => transferring context: 206B                                                                        0.0s
 => CACHED [mauro-data-mapper mdm-build  2/16] RUN cd "/opt/mdm-application-build" && git fetch && git   0.0s
 => CACHED [mauro-data-mapper mdm-build  3/16] RUN cd "/opt/mdm-ui" && git fetch && git checkout "7.3.0  0.0s
 => CACHED [mauro-data-mapper mdm-build  4/16] RUN mkdir -p /opt/mdm-explorer/dist                       0.0s
 => CACHED [mauro-data-mapper mdm-build  5/16] COPY build_scripts /usr/local/bin/                        0.0s
 => CACHED [mauro-data-mapper mdm-build  6/16] RUN chmod a+x /usr/local/bin/*                            0.0s
 => CACHED [mauro-data-mapper mdm-build  7/16] COPY config/build.yml /opt/mdm-application-build/grails-  0.0s
 => CACHED [mauro-data-mapper mdm-build  8/16] COPY mdm-ui /opt/mdm-ui/                                  0.0s
 => ERROR [mauro-data-mapper mdm-build  9/16] RUN build_frontend.sh                                    135.8s
------
 > [mauro-data-mapper mdm-build  9/16] RUN build_frontend.sh:
0.392 Fetching and checking out commit
0.706 HEAD is now at 8e9cfac1 Merge branch 'release/7.3.0' into main
0.718 Using pre-compiled source
0.722 Downloading precompiled sources https://jenkins.cs.ox.ac.uk/artifactory/artifacts/mauroDataMapper/mdm-ui/mdm-ui-7.3.0.tgz
0.728   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0.728                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:15 --:--:--     0
135.8 curl: (28) Failed to connect to jenkins.cs.ox.ac.uk port 443 after 135057 ms: Connection timed out
135.8 tar (child): mdm-ui-7.3.0.tgz: Cannot open: No such file or directory
135.8 tar (child): Error is not recoverable: exiting now
135.8 tar: Child returned status 2
135.8 tar: Error is not recoverable: exiting now
135.8 cp: cannot stat 'mdm-ui-7.3.0/*': No such file or directory
------
failed to solve: process "/bin/bash -c build_frontend.sh" did not complete successfully: exit code: 1

@jonnyry Thank very much for the error report.  Our artefact server was hurriedly end-of-lifed last week (nothing to do with our use of it!), and we're working on rebuilding the images to point at the new location. We'll hopefully have the docker set-up all working again in the next day or two.

Hi @jonnyry,
If you pull this branch of mdm-docker, it should point to some updated images with a corrected URL: feature/update-repository-url
We'll push that as a patch release to this repo soon.

The build has now completely sucessfully with the latest version of main.
Thanks @jamesrwelch @joe-crawford for the super quick turnaround.


No way to disable STDOUT logger when supplying a formatter with log_location

Description
No way to disable STDOUT logger when supplying a formatter with a log_location.
Some formatters (nyan cat as referred to from the documentation being one), assume/interact with a terminal, and so are less log formatters and more custom log sinks.  Adding a default STDOUT logger into that mix really messes up the output of these formatters.
In Chef 12 force_logger false (the default for that option) could incidentally be used to address this, but refactorings along the way have removed this option.
https://github.com/chef/chef/pull/5851
https://github.com/chef/chef/pull/6035
Please re-add a way to inhibit the additional STDOUT default logger.
Chef Version
Chef 13+.  Replication case logs are from master@01ae5aac80444d6e849bb44ba0f5798e46978c7f aka v14.0.59.  Not present in v12.21.31
Replication Case
Given the following configuration:
# bug-config.rb
gem 'nyan-cat-chef-formatter'
require 'nyan-cat-chef-formatter'

log_location '/tmp/chef.log'

add_formatter :nyan
once true

Run chef-client -z -c bug-config.rb -o ''
We hope to see no lines in the output that look like [2018-02-21T10:12:08+00:00] WARN: Bad things ahead! but we do.
Client Output
$ chef-client -z -c bug-config.rb -o ''
[2018-02-21T10:12:08+00:00] WARN: No cookbooks directory found at or above current directory.  Assuming /Users/richardc/src/chef.
Starting Chef Client, version 14.0.59
resolving cookbooks for run list: []
Synchronizing cookbooks
done.
Compiling cookbooks
done.
[2018-02-21T10:12:14+00:00] WARN: Node penge.local has an empty run list.
Converging 0 resources
0/0: _,------,
0/0: _|  /\_/\
0/0: ~|_( - .-)
0/0:  ""  ""

System converged.
no resources updated
chef client finished, 0 resources updated

What's relevant are these lines:
[2018-02-21T10:12:08+00:00] WARN: No cookbooks directory found at or above current directory.  Assuming /Users/richardc/src/chef.

[2018-02-21T10:12:14+00:00] WARN: Node penge.local has an empty run list.

That's a logger connected to STDOUT.
Stacktrace
N/A

We have a workaround in our current configuration, which is to be awful and monkey-patch Chef::Application#want_additional_logger?.  Sharing it here in case it's of use to other nyan cat fans:
# config.rb
gem 'nyan-cat-chef-formatter'
require 'nyan-cat-chef-formatter'

log_location "/tmp/chef-log"

add_formatter :nyan
once true

# Icky monkey-patch - we never want an additional logger as the one in `log_location` is fine.
class ::Chef::Application
  def want_additional_logger?
    false
  end
end

does --force-formatter work and does force_formatter true in client.rb work in chef-client master past https://github.com/chef/chef/pull/6916 ?

Checked at 19608ab7e7a5904c0917686835b8e96babe12386 and no, that doesn't make any difference either on the CLI or in the client.rb.

closing as needing replication and will not be applicable after #9126