Explore gix APIs, experiment with gix-blame API

This is a draft PR that is supposed to help me get familiar with gix’s APIs as well as to experiment with possible APIs for gix-blame. Any feedback that helps me not go down the wrong path, is much appreciated! :-)
Review Tasks

[x] try gix blame for fun

not yet implemented: passing blame to more than one parent is not implemented yet on README.md :/

[x] rewrite commits to have one per crate that is touched (simplify handling)
[x] remove all todo!()
[x] replace .unwrap() with error handling
[x] Outcome with statistics
[x] tracing spans for perf info
[x] rough docs
[x] sort out blamed_file and original_file once and for all.
[ ] CLI improvements

[ ] make it easier to use a worktree-path through a pathspec (and use that in gix merge-file as well)

[x] avoid re-splitting of content, otherwise it's too easy to get the visualization wrong, or using the wrong content (use tokenized input)

[x] review TODOs
[ ] sort out assertion failures

Performance Opportunities

add pathspec (or even: single-path) support to tree-diffing to ideally allow to skip a lot of the work that is otherwise done, or stop the diff earlier.

@Byron I think I’ve now gotten to a point where I’ve learned enough about gixoxide’s APIs in order to hopefully be able write the first meaningful test. Would you mind looking at the PR’s current state? I’m not sure whether I it would be better to try and find a good API for struct Blame or to continue adding to the already quite large test it_works.
My goal now is to write a test that runs a complete blame for the sample repo that is part of this PR.

Thanks, that’s very helpful already! I think I wanted to mostly make sure I’m still on the right track and not missing anything. I’ll continue working on the test, then, and worry about the public API later.

Yes, I think it's great to keep up the explorative approach using cargo test and then put that behind an API for abstraction, along with the first simple 'actual' test. And from there, one can iterate to validate more complex cases, while slowly getting to the kind of observable and interruptible API that will eventually be needed for gitui.

@Byron This is a quick status update! The algorithm is now able to generate a blame for the sample repo. I extended the test to compare this blame against git blame’s output. Next, I will review the algorithm in more detail (the loop count, for example, is still hard-coded), then I would start adding more cases, trying to abstract/extract pieces of code on the way.

@Byron Sooner than expected, there seems to have been quite a bit of progress! I extended the test repo so that it now also contains commits that don’t touch the file being blamed. I also added a file that contains multiline hunks. There’s now 2 test cases that cover one file each. Also, most of the logic is now inside the loop, so I was able to delete a couple of lines. I was thinking of maybe trying to tackle merge commits next. In its current state, the algorithm only diffs against a single parent. What do you think? (Also, I’ll address your comment above.)

That's great news!

In its current state, the algorithm only diffs against a single parent.

That's interesting - I just know that Git usually ignores merge-commits, but I don't know if it includes them in blames nonetheless. I never thought about how that would work, and would probably see what Git would do. It's something I do in general, that is, try to understand what Git can do and ask if this would (eventually) be possible with the gitoxide implementation. When it comes to diffing, I definitely have a lot to learn.
Besides that, I definitely recommend to keep clippy happy and run it locally at least occasionally - ideally CI can stay green even during early development, and I found it worth it as it typically leads to cleaner code. Often I find myself doing a push&run, just to return one more time to quickly fix CI if it breaks so I can continue 'clean' in the morning :D.

@Byron I think I’m now at a point where I’d appreciate a brief sync or a bit of feedback on how to best continue. Overall, I’m happy with the results so far. I’ve tested the implementation on linear commit histories with up to 90 commits, and for the files that I’ve blamed, the results match git’s results. At this point, my next goals are centered around gaining more confidence in the implementation and to adding parts that are still missing compared to git’s implementation. Also, I had a lot of fun working on this PR, so thanks for giving me the space! :-)
These are some observations and remaining issues:

Merges still need to be taken into account, in the sense that while git indeed seems to ignore them, my implementation currently only follows one parent instead of all parents.
There seem to be differences between imara-diff and git diff. I want to do more research on this topic as it directly affects the result of a blame.
Running the current implementation on even slightly larger repositories (I ran some tests inside gitoxide) is slow. I suspect this is due to the part where two trees are diffed to find out whether the file under blame was changed. I might also just have missed an API that does exactly what I want.
Some of the tests are quite verbose, there’s probably opportunities for some de-duplication using macros.
Most of the recent fixes didn’t add test cases to make_blame_repo.sh, but I instead added very specific test cases like process_change_works_added_hunk that only test a single iteration of the basic building block.
The implementation doesn’t follow renames.
I even found a difference between git and libgit2 when it comes to coalescing blame entries before returning them to the caller. I’ve documented that fact in a comment.

This is a bit of a braindump and it probably is missing some important context. I hope that is’s useful anyway and am happy to elaborate on anything as needed. :-)

Also, there’s errors in the CI pipeline, but they don’t seem to be related to any change in this PR.

Thanks for the review and the feedback, that’s very helpful! I’ll have a look and continue working or ask follow-up questions. :-)

@Byron I think I was able to identify a case where imara-diff differs from git diff.
Before:
  line 1

  line 2

  line 3

After:
  line 1

  line in between

  line 2

  line in between

  line 3

imara-diff breaks this up as [Unchanged(0..2), Added(2..4, 0), Unchanged(4..5), Added(5..7, 0), Unchanged(7..9)] (ranges are exclusive at the end) while git diff comes to a different result: [Unchanged(0..2), Added(2..4, 0), Unchanged(4..6), Added(6..8, 0), Unchanged(8..9)]. This is with diff.algorithm = histogram and Algorithm::Histogram.
The difference is with respect to the newlines surrounding the second line in between. Interestingly enough, I was not able to observe this behaviour when there was only one block containing line in between added.

That's great! Could this be reproduced with a baseline test, maybe also producing versions with Meyers and Histogram respectively?
It's notable that the diff takes newlines into account, and it's something that Git might have special behaviour around, I didn't check.
Before reeling in Pascal who can most certainly enlighten us, I'd love to have a failing test along with all the facts that we can collect.
Thanks again for your help with this.

I just added two failing test cases. Since blame_file doesn’t take the algorithm as an argument yet, there’s possibly a mismatch for the Myers case (because the algorithm is hard-coded to Algorithm::Histogram currently), but I wanted to add the test case anyway.

I’ve also moved most of the setup out of blame_file. Is that what you had in mind? There’s still dependencies left on gix-ref and gix-traverse, but they seem to be a bit harder to remove.

Great, thanks, I will take a closer look later.

I’ve also moved most of the setup out of blame_file. Is that what you had in mind? There’s still dependencies left on gix-ref and gix-traverse, but they seem to be a bit harder to remove.

Yes, that's exactly it. I thought traversal can be reduced to an iterator yielding object-ids, but it's nothing I validated and I am probably missing an important detail.
Generally, I am optimistic that both gix-ref and gix-traverse can go away eventually.

Great, thanks, I will take a closer look later.

I’ve also moved most of the setup out of blame_file. Is that what you had in mind? There’s still dependencies left on gix-ref and gix-traverse, but they seem to be a bit harder to remove.

Yes, that's exactly it. I thought traversal can be reduced to an iterator yielding object-ids, but it's nothing I validated and I am probably missing an important detail. Generally, I am optimistic that both gix-ref and gix-traverse can go away eventually.

The issue might be that we need to take an Iterator that yields Items that themselves have commit_iter() which yields items that have parent_ids(), so there’s a bit of nesting. I imagine it’s certainly possible to abstract that into a trait or something, but it felt less like a low-hanging fruit than the previous changes. :-)
The dependency on gix_ref is only for BStr as far as I can see which can also be imported from gix_object I found out (gix-diff does that, but it also directly imports from bstr).

at first glance this looks like the slider problem (https://github.com/mhagger/diff-slider-tools/blob/master/README.md).
Basically the algorithm can say that your diff is:
"Line 1\n"
+"\n"
+"  line in between\n"
\n

or it can say that the diff is:
"Line 1\n"
"\n"
+"  line in between\n"
+ \n

Both are correct (minimal) difss so there is nothing wrong with the diff algorithm/results it's just that there are multiple different possible choices a diff can make. Git has some postprocessing heuristics to make a choice but from an objective point of view one choice isn't wrong just that some choices feel more intuitive to humans so it's not a bug in the diff algorithm just a missing feature.
Git resolves those ambiguities by a heuristic. Basic summary:

slide sliders as far down and up as possible to check if multiple hunks can be merged
finally slide the slider as far down as possible
a fairly complex indentation based heuristic developed and described in that repo I linked (semi-recent addition to git)

I have an implementation of the first two coupled with some (necessary) API change that was aimed to be the first breaking imara-diff change. I started porting helix to the new API but got a bit stuck timewise and didn't get around to finishing that.
The third heuristic is very involved implementation wise and also difficult to add from an API perspective since it only makes sense for (line-segmented) text but imara-diff operates on abstract tokens and allows diffing arbitrary data (segmented in arbitrary ways).
I think we could aim to get the basic heuristic in, the more advanced indentation based code I am not sure I have time to work on at the moment. For this PR I would suggest to just update the fixtures to match whatever imara-diff produces and then update them as imara-diff improves in that regard.

Quick side note to my last commit: I’ve had to add another parameter to blame_file, needed to associate the first UnblamedHunk with an ObjectId. This feels like duplication, though, as that information should, in theory, also be accessible through traverse. Since I did not quickly find a way to get start_id from traverse, I decided to add it as a parameter to blame_file.

at first glance this looks like the slider problem (https://github.com/mhagger/diff-slider-tools/blob/master/README.md).

Thanks so much for sharing!
I also do remember now that you already shared this with me, probably in person, so it's clear that these heuristics are a tuneable and there isn't really a right answer.
After reading at the link location really makes me think that imara-diff will one day outperform Git (if it offers new heurstics):

An implementation of an alternative slider positioning heuristic that significantly outperforms both git diff and git diff --compaction-heuristic as of Git 2.9.0.

Thus, this is likely going to be a place where gitoxide will differ from Git and hopefully at some point it will differ to the better.
Until then, we go with your suggestion and just keep these examples in our baselines and mark them accordingly. Maybe one day they fit, or one day we mark ours as preferred.

@Byron This is another quick status update: I’ve now refactored and simplified the implementation a bit, mostly to make the concepts used more easy to discern. I’m also quite far already in my approach to being able to blame commits with more than one parent. So, from my side, the point gets closer where I would call the first implementation done. For context, I just wanted to ask for feedback, in particular on how and when you think this should be merged. I’m mainly asking because there’s now more than one area I’d like to focus on, and I can imagine it might be easier to spread the work into PRs that are more specific to certain aspects, instead of keeping everything in this one. I can still follow it, but it’s already very layered in the sense that there’s quite a few discussions and threads overlapping. :-) What are your thoughts? I’m also fine with continuing in this PR, so whatever works best for you!

Thanks for the update :).
Generally I am happy to follow your preference when it comes to merging, so what follows is just my preference, which isn't the deciding factor here.
What I'd do before merging is to 'quickly' wire it up into gix blame to be forced to reproduce the lines with the blame information. The last time I checked, this would have been difficult or at least easy to get wrong.
And since you probably have some more areas of work already in your mind, I think it would be good to collect them in a task list or tracking issue, so we both can keep an overview. It's probably best if you'd create a tracking issue for that so you can edit it primarily.
In summary, a tracking issue we should have, and everything else is up to you then - you can ask me to review and merge, and that's what I will do, with or without gix blame.
I hope that helps.

Thanks, that helps a lot! I think I’ll give wiring it up into gix blame a try, and I’ll create a tracking issue.

Quick question: did I put the code for wiring up gix blame in the right locations? (Feel free to change as necessary!)

Another quick update: I was able to successfully run gix blame <file> in a repo whose history is linear. Next, I want to create the tracking issue, probably tomorrow.
Among other known issues, the implementation still does not handle commits with more than one parent because I need to dig deeper into git’s implementation before I can continue in this area.
Is there a way to pass GIT_DIR or GIT_WORK_TREE to gix when running through cargo run? I tried env GIT_WORK_TREE=<work_dir> cargo run -- blame <file>, but it appears the value was not picked up. This might be an issue with the blame implementation, so if there’s nothing obvious wrong with how I ran cargo run, I’m going to double-check what I might have missed.

@Byron There’s an error in just check related to imara-diff. From what I understand the check fails because imara-diff is part of the dependency tree despite --no-default-features being specified. It looks like I need to make the dependency on imara-diff in gix-blame optional so that it can be controlled by a feature flag. Is that correct?

Yes, that's what the test is trying to assert. New crates are typically gated, while it's easy to know which capability is powered by which plumbing crate, at least in gix.
gitoxide-core can just do everything, no need for a toggle there.

Yes, that's what the test is trying to assert. New crates are typically gated, while it's easy to know which capability is powered by which plumbing crate, at least in gix. gitoxide-core can just do everything, no need for a toggle there.

In this instance, would the proper solution be to put gix-blame behind a flag in gix or move the dependency to gitoxide-core?

My approach would be to put gix-blame behind a feature toggle in gix, assuming that it is made available there with a higher-level API. A recent example for this could be the blob-merge feature toggle, which was added to gix along with an API that is then used to implement gix merge file.

@Byron I’ve put gix blame behind a flag and marked the PR as ready to review. It is certainly a bit rough around the edges, but I’m at a point where I’d appreciate detailed feedback as well as the feeling of having completed a first milestone a lot! :-) Also, I will be pretty busy the next couple of days with other things, so I won’t be able to work on it much anyway. Let me know if you’ve other thoughts and if this works for you! I haven’t created the tracking issue yet, but it’s still on my list.
Now that gix blame is available, I will also try and verify the existing implementation by writing a script that would allow me to compare the output of gix blame and git blame on a larger scale, e. g. by running both commands on all files in a single repository.

That's great to hear, I share your excitement :)!
I will take a look soon and probably push some refactors (if needed) to get it into a mergeable state, so I wouldn't think you have to get back to it.

Now that gix blame is available, I will also try and verify the existing implementation by writing a script that would allow me to compare the output of gix blame and git blame on a larger scale, e. g. by running both commands on all files in a single repository.

That's a great idea! It will definitely be good to lure out 'hard' bugs, like panics, if these can exist, but also false-negatives with changes due to the difference in diff implementation. When setting up the 'merge' baseline, I noticed quite a few of these, and I suppose they happen more often in real-world blames. So I wouldn't worry too much about differences.
Something I think could be really helpful for you to have heard about is ein tool query - it builds a database of expensive information, which is exactly the kind of information you'd want to do blames real fast - want to know which commits touch a path with rename tracking? It's right there and basically instant.
I am thinking that eventually the API could be made abstract enough (if it isn't already) to support feeding this information from the outside and basically avoid interacting with the diff machinery directly.
As possible application, gitui could enable this database and use that to feed the blame, for a true 'insta-blame' even for the biggest repositories. That would be an alternative to a possibly more difficult to implement streaming blame implementation. Pros and cons, I know, just wanted to put it out there as option, without indicating it's a good one :D.

I will take a look soon and probably push some refactors (if needed) to get it into a mergeable state, so I wouldn't think you have to get back to it.

Sounds good! I’m certainly interested in learning about how to make the code as idiomatic as possible.

That's a great idea! It will definitely be good to lure out 'hard' bugs, like panics, if these can exist, but also false-negatives with changes due to the difference in diff implementation. When setting up the 'merge' baseline, I noticed quite a few of these, and I suppose they happen more often in real-world blames. So I wouldn't worry too much about differences.

Exactly! At this point, I’m mostly interested in finding the last remaining bugs and in getting a feel for how much variation there is in the real world when it comes to diffs and blames.

Something I think could be really helpful for you to have heard about is ein tool query […].

That sounds really interesting, thanks for the hint, I’ll have a look!

I made a plan and took the liberty to dump a task-list into the body of this PR.
My first encounter with gix blame was tricky though as I couldn't find a repository where it wouldn't choke on a merge commit (probably):
small ( master)
❯ /Users/byron/dev/github.com/Byron/gitoxide/target/release/gix blame hello
thread 'main' panicked at /Users/byron/dev/github.com/Byron/gitoxide/gix-blame/src/lib.rs:803:13:
not yet implemented: passing blame to more than one parent is not implemented yet

Do you think I should force this into submission by only walking the first parent, or is this something that you think could be easily addressed more correctly?

Do you think I should force this into submission by only walking the first parent, or is this something that you think could be easily addressed more correctly?

My first hunch is that this needs a proper solution. In order to facilitate testing, I think first-parent-only traversal will work if the file being blamed is only touched by the first parent’s ancestors. For a proper solution, I want to study git rev-list which I think backs git blame. I have already found a non-trivial case where the current diff-to-parent approach falls short.

Also, I understand if you don’t want to merge without the ability to follow merge commits. It will then just take longer until I get to that part. 😀

Thanks for offering, I take it :)!
Then I think it's best to take the time it takes to make gix blame useful, which probably means no more than figuring out what to do with merge-commits. Since there is a Git baseline, that should help to see what should be done.
I also hope that rev-list has nothing to do with how merge-commits are handled - I hoped that blame could rely solely on 'check diffs to see which one touches a portion of the file' - the ordering can then be left to the caller, with suggestions of what yields the best results.
Please also ake the above paragraph with a grain of salt, as it's just a birds-eye view on things which might well miss all the important details. Once I get to the thorough review phase, I will get into it properly for the first time.
PS: Please ignore the tasks I set for myself entirely I will take care of them once you give me a green light. I also will stop working on the PR now, it's all yours.

@Byron I’m now at a point where passing blame to more than one parent partially works. gix blame STABILITY.md already works well (apart from one instance of the slider problem), but gix blame README.md is only partially correct. I suspect this has to do with how blame is currently naively passed to parents without taking into account potential shifts in line numbers, but I’m not sure yet.
This has become increasingly harder to debug, though, for two reasons:

mostly, because the implementation is still quite slow (git blame README.md takes almost a minute on my old machine),
but also because we’re dealing with more than 13000 commits for README.md and it is a bit difficult to find the signal in that history.

I think I need to first make debugging easier before it makes sense to continue. Would you mind taking a look at the code to see whether there’s any low-hanging fruit with respect to performance or general debuggability? Thanks as always!

That's great, as it also means that it now runs without panic. I could imagine to add a note that it's not currently correct in all instances.
Also, I would think that the issue with README.md should be reproducible as part of the baseline, so it can more easily be run and fixed. 1 minute per run certainly isn't acceptable. I also did profile it once and noticed that most of the time is spent in traversing commits - there I thought it should not be possible that all of them are relevant, but I also didn't look further.
One major issue for diff-tree-performance right now certainly is the lack of pathspec support - we'd want to avoid looking at unrelated paths entirely which could save a lot of time. For now, I wouldn't worry about that and try to get it to be correct.
Speaking of correctness, I never really thought about how one would implement this, and trying to think about it just revealed that it's not at all as easy as I thought it was :D. However, libgit2 or git source code probably reveals its secrets when probed, something I also didn't do yet.
I will see to finding some time to work on this PR to get it merged in a first version.
Restarting a difficult problem with a clean slate and clear focus will be nice, too :).

That's great, as it also means that it now runs without panic. I could imagine to add a note that it's not currently correct in all instances.
Also, I would think that the issue with README.md should be reproducible as part of the baseline, so it can more easily be run and fixed. 1 minute per run certainly isn't acceptable. I also did profile it once and noticed that most of the time is spent in traversing commits - there I thought it should not be possible that all of them are relevant, but I also didn't look further.

Sounds good! I’ll try to create a minimal reproducible example.

One major issue for diff-tree-performance right now certainly is the lack of pathspec support - we'd want to avoid looking at unrelated paths entirely which could save a lot of time. For now, I wouldn't worry about that and try to get it to be correct.

Pathspec support would be awesome, as that is, from what I can tell, also used by git to speed things up (this function https://github.com/git/git/blob/6258f68c3c1092c901337895c864073dcdea9213/tree-walk.c#L1221-L1224 is used under the hood). My first idea was to apply .filter to the tree iters here: https://github.com/cruessler/gitoxide/blob/eb6f4207e43fbfce9af55b5a3acf4bf8fbd5da21/gix-blame/src/lib.rs#L659-L661, but I haven’t gotten very far yet.

Speaking of correctness, I never really thought about how one would implement this, and trying to think about it just revealed that it's not at all as easy as I thought it was :D. However, libgit2 or git source code probably reveals its secrets when probed, something I also didn't do yet.

Absolutely, I’ve immersed myself into git’s source code quite a bit, but I still find it hard to understand, compared to Rust. :-)

I will see to finding some time to work on this PR to get it merged in a first version. Restarting a difficult problem with a clean slate and clear focus will be nice, too :).

That sounds great! I’ll be on stage in a theatre play this weekend, so I will probably not spend a lot of time on gix blame, but I’m already looking forward to getting back to it. :-)
Also, I’ve created this (currently almost empty) tracking issue: #1619.

Thanks a lot for the rebase!
It will still take me a while until I get to do the in-depth review this PR needs though - I have to get a basic merge working first.

Thanks for letting me know, and no worries!
I think I’ve made a bit of progress with respect to merge commits. One observation that I need to double-check is that git seems to traverse the history differently than gix::traverse::commit::Simple. There doesn’t seem to by a Sorting that can be used to replicate git’s behavior exactly. I’ll provide more context soon-ish.

I think I’ve made a bit of progress with respect to merge commits. One observation that I need to double-check is that git seems to traverse the history differently than gix::traverse::commit::Simple. There doesn’t seem to by a Sorting that can be used to replicate git’s behavior exactly. I’ll provide more context soon-ish.

Thanks for sharing!
I agree that a difference in commit order will yield different solution which is a problem for baseline tests, but it's still a valid solution. And for the algorithm in gix-blame itself which isn't coupled with the iteration algorithm that's fine.
Something that could work is the Topo walk, which simulates what git log does. My hope is that this is exactly what's needed. Otherwise, one would have to figure out exactly how the walk is performed to have a chance to fit the baseline. In theory, all the primitives are there to do so and the Topo walk is by far the most complex, so it's definitely possible and certainly something to get more used to as graph walks are the bread and butter of many algorithms.

I was able to isolate a test case where the order of traversal matters in a way that makes the test fail (file_only_changed_in_branch is the failing test). The history looks like this:
(1, merge commit)
   |  \
   |  (2)
   |  /
   (3)

Using Simple sorting, blame_file traverses in this order: 1, 3, 2 while git blame uses a different order: 1, 2, 3. I would argue that git blame is correct and will try to get the same result using Topo, so thanks a lot for the hint!
Also, I ported a nice performance improvement from git which resulted in a significant speedup.

Good news: using Topo instead of Simple seems to have solved most of the remaining issues! The previously failing test case now passes and gix blame STABILITY.md as well as gix blame README.md now return blames that are very close to what git returns. It looks like the remaining differences could all be due to the “slider problem”, but I still have to confirm that.
Also, it seems there is no gix log yet. I would like to create a PR adding gix log if this sounds useful to you!

As long as the result is correct, I'd always think it can only be the slider problem - maybe the output of gix blame can be made to be similar to git blame so one can diff it. Maybe that's not worth the effort though.

You can already diff the two outputs and get a rather meaningful signal which was sufficient for what I wanted to do.

Also, it seems there is no gix log yet. I would like to create a PR adding gix log if this sounds useful to you!

If done right, that would be a beast, but I think something simple would be useful to visually investigate the traversal order of the offered algorithms. For now, only Simple is supported from gix anyway, but eventually Topo could be added as new sorting mode.

I was indeed thinking of something simple to start with, mostly to get familiar with the traversal order and later to investigate what ways there are to speed up traversal.

And I know, I am slow with this PR, so gix log would probably be merged earlier even 😅.

No worries, I’m in no particular rush. :-)

I was indeed thinking of something simple to start with, mostly to get familiar with the traversal order and later to investigate what ways there are to speed up traversal.

Sounds good! I recommend starting with 'oneline' mode without anything else.
Regarding traversal performance, there isn't anything one can do except for setting an object cache. There are a couple of tricks as well, all of the ones I could find are used with ein t hours if you want to have a look. For gix log I would be more interested in simple code though.

Regarding traversal performance, there isn't anything one can do except for setting an object cache.

P.S. if it is mostly I/O, then maybe io-uring will help

This is the script I use to compare the output of gix blame and git blame for batches of files in a given repository: https://gist.github.com/cruessler/54fdf1379e03e86be2b7b748750c181e.

@Byron I just updated the PR by rebasing on the latest main. I’ll fix any errors that might arise on CI tomorrow.
If there’s anything that I can already do to facilitate the review, let me know! I’ll have a bit of spare time to spend over the course of the next two to three weeks. :-)

The refactoring is going well and it's business as usual. I like the implementation more and more, now that I am more familiar with it. And interestingly, the UnblamedHunk riddle seems to be solved, as the meaning of Original File and Blamed File is simple swapped. This becomes evident in gitoxide-core/blame which displays the blamed_lines, not the original ones.
Also, I ran another profile run and besides the usual reasons for being slow, commit iteration with the topo-traversal, there was only one function that even as much as showed up: pass_blame. Probably this is called a lot as commits are traversed.

pass_blame in Git looks like an entirely different beast, so I am going to leave it for now.

That’s awesome news, glad to hear that! I’ll try to get profiling running on my machine (which runs Linux) to see whether there’s anything else to see.

And it's nearly there, all that seems left is a more standardized handling of the input path in the CLI.
When benchmarking/testing more, I came across this gem as it appears to be possible to try to an unblamed hunk into a blame entry without the suspect being present there.
This occurred in the Git repository itself.
git ( master) [?] via 🐍
❯ git rev-parse @
d882f382b3d939d90cfa58d17b17802338f05d66

git ( master) [?] via 🐍
❯ cargo run --manifest-path /Users/byron/dev/github.com/GitoxideLabs/gitoxide/Cargo.toml --bin gix -- --verbose --trace blame -s README.md
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
     Running `/Users/byron/dev/github.com/GitoxideLabs/gitoxide/target/debug/gix --verbose --trace blame -s README.md`
thread 'main' panicked at gix-blame/src/file/mod.rs:475:14:
Private and only called when we now `commit_id` is in the suspect list
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The .expect() was an unwrap() previously.
When 'looking for trouble' with git ls-files | xargs -n1 /Users/byron/dev/github.com/GitoxideLabs/gitoxide/target/release/gix --verbose --trace blame -s other files would also show this exact issue.
Your help would be appreciated, as I didn't really get to grasping how suspects really work. But I hope to get there.

I just 'fixed' it by assuming it's naturally possible that there is no entry for a suspect, so there is no need to fail. Now it runs through and produces the correct result.
In the background I am collecting more data and blame every file in the Git repository. It takes a while, but my hope is that it won't panic anymore.

I am still on solidifying the implementation, and running it on every file in the Git repository is a good way to run into the last remaining .expect()s.
It's also a great motivation to try improving performance, as it takes a while otherwise.
Just to share early, I added a counter to see if it's worth it to cache information, and by the looks of it, this really is the case.
Statistics {
    commits_traversed: 33439,
    commits_to_tree: 66357,
    trees_decoded: 66357,
    trees_diffed: 29,
    blobs_diffed: 27,
    odb_lookups: 67556,
    cacheable_odb_lookups: 74971,
}

Just need to figure out if it's trees or commits that should be cached additionally, hoping that it is trees mostly.

And a pretty good performance example (without commit-graph/bloom filter) where Git is 7.5x faster:
git ( master) [?] via 🐍
❯ /usr/bin/time -lp git blame t/t4013/diff.rev-list_--children_HEAD
15 years ago    Thomas Rast     fcbc6efc7c5│  1 │ $ git rev-list --children HEAD
                                           │  2 │ 59d314ad6f356dd08601a4cd5e530381da3e3c64
                                           │  3 │ c7a2ab9e8eac7b117442a607d5a9b3950ae34d5a 59d314ad6f356dd08601a4cd5e530381da3e3c64
                                           │  4 │ 9a6d4949b6b76956d9d5e26f2791ec2ceff5fdc0 59d314ad6f356dd08601a4cd5e530381da3e3c64
                                           │  5 │ 1bde4ae5f36c8d9abe3a0fce0c6aab3c4a12fe44 9a6d4949b6b76956d9d5e26f2791ec2ceff5fdc0
                                           │  6 │ 444ac553ac7612cc88969031b02b3767fb8a353a 1bde4ae5f36c8d9abe3a0fce0c6aab3c4a12fe44 c7a2ab9e8eac7b117442a607d5a9b3950ae34d5a
                                           │  7 │ $
real 0.48
user 0.43
sys 0.04
           335085568  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               22710  page reclaims
                  25  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   5  voluntary context switches
                  50  involuntary context switches
<PHONE_NUMBER>  instructions retired
<PHONE_NUMBER>  cycles elapsed
           139412224  peak memory footprint

git ( master) [?] via 🐍
❯

git ( master) [?] via 🐍
❯ /usr/bin/time -lp git blame t/t4013/diff.rev-list_--children_HEAD

git ( master) [?] via 🐍
❯ /usr/bin/time -lp /Users/byron/dev/github.com/GitoxideLabs/gitoxide/target/release/gix --verbose --trace  blame -s t/t4013/diff.rev-list_--children_HEAD
 16:46:34 tracing INFO     run [ 4.12s | 17.02% / 100.00% ]
 16:46:34 tracing INFO     ┝━ ThreadSafeRepository::discover() [ 656µs | 0.00% / 0.02% ]
 16:46:34 tracing INFO     │  ┕━ open_from_paths() [ 578µs | 0.00% / 0.01% ]
 16:46:34 tracing INFO     │     ┕━ gix_odb::Store::at() [ 381µs | 0.01% ]
 16:46:34 tracing INFO     ┝━ gix_index::File::at() [ 382µs | 0.00% / 0.01% ]
 16:46:34 tracing DEBUG    │  ┝━ gix::open_index::hash_index [ 181µs | 0.00% ] path: "./.git/index"
 16:46:34 tracing DEBUG    │  ┕━ gix_index::State::from_bytes() [ 144µs | 0.00% ] options: Options { thread_limit: Some(0), min_extension_block_in_bytes_for_threading: 0, expected_checksum: None }
 16:46:34 tracing INFO     ┕━ gix_blame::file() [ 3.42s | 82.96% ] file_path: "t/t4013/diff.rev-list_--children_HEAD" | suspect: Sha1(d882f382b3d939d90cfa58d17b17802338f05d66)
fcbc6efc 1 1 $ git rev-list --children HEAD
fcbc6efc 2 2 59d314ad6f356dd08601a4cd5e530381da3e3c64
fcbc6efc 3 3 c7a2ab9e8eac7b117442a607d5a9b3950ae34d5a 59d314ad6f356dd08601a4cd5e530381da3e3c64
fcbc6efc 4 4 9a6d4949b6b76956d9d5e26f2791ec2ceff5fdc0 59d314ad6f356dd08601a4cd5e530381da3e3c64
fcbc6efc 5 5 1bde4ae5f36c8d9abe3a0fce0c6aab3c4a12fe44 9a6d4949b6b76956d9d5e26f2791ec2ceff5fdc0
fcbc6efc 6 6 444ac553ac7612cc88969031b02b3767fb8a353a 1bde4ae5f36c8d9abe3a0fce0c6aab3c4a12fe44 c7a2ab9e8eac7b117442a607d5a9b3950ae34d5a
fcbc6efc 7 7 $
Statistics {
    commits_traversed: 57973,
    commits_to_tree: 114610,
    trees_decoded: 342268,
    trees_diffed: 1,
    blobs_diffed: 0,
}
real 4.13
user 4.09
sys 0.03
           206684160  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               13023  page reclaims
                   0  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                 118  involuntary context switches
<PHONE_NUMBER>5  instructions retired
<PHONE_NUMBER>5  cycles elapsed
            76841968  peak memory footprint

However, I think the problem is that we have tremendous uncached amounts of work to find the one commit that adds this file 55k commits ago. Git clearly manages to not redo much work.

The commit that will never be
commit 95569ac06cb53dabb675b9eae377e2e5b24b973d
Author: Sebastian Thiel<EMAIL_ADDRESS>Date:   Wed Dec 25 16:55:32 2024 +0100

    use a custom tree lookup implementation with our own cache.

    Maybe one day, this can be placed into `gix-object` as well.

diff --git a/Cargo.lock b/Cargo.lock
index 365fb927f..c84dff747 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1543,6 +1543,7 @@ dependencies = [
  "gix-filter",
  "gix-fs 0.12.1",
  "gix-hash 0.15.1",
+ "gix-hashtable 0.6.0",
  "gix-index 0.37.0",
  "gix-object 0.46.1",
  "gix-odb",
diff --git a/gix-blame/Cargo.toml b/gix-blame/Cargo.toml
index fc4baf3fe..f2c3478bf 100644
--- a/gix-blame/Cargo.toml
+++ b/gix-blame/Cargo.toml
@@ -20,6 +20,7 @@ gix-object = { version = "^0.46.0", path = "../gix-object" }
 gix-hash = { version = "^0.15.0", path = "../gix-hash" }
 gix-worktree = { version = "^0.38.0", path = "../gix-worktree", default-features = false, features = ["attributes"] }
 gix-traverse = { version = "^0.43.0", path = "../gix-traverse" }
+gix-hashtable = { version = "^0.6.0", path = "../gix-hashtable" }

 thiserror = "2.0.0"

diff --git a/gix-blame/src/file/function.rs b/gix-blame/src/file/function.rs
index f34749a41..bb9d48943 100644
--- a/gix-blame/src/file/function.rs
+++ b/gix-blame/src/file/function.rs
@@ -69,8 +69,11 @@ where

     let mut stats = Statistics::default();
     let (mut buf, mut buf2, mut buf3) = (Vec::new(), Vec::new(), Vec::new());
-    let blamed_file_entry_id = find_path_entry_in_commit(&odb, &suspect, file_path, &mut buf, &mut buf2, &mut stats)?
-        .ok_or_else(|| Error::FileMissing {
+    let mut trees = gix_hashtable::HashMap::default();
+    let blamed_file_entry_id = find_path_entry_in_commit(
+        &odb, &suspect, file_path, &mut buf, &mut buf2, &mut stats, &mut trees,
+    )?
+    .ok_or_else(|| Error::FileMissing {
         file_path: file_path.to_owned(),
         commit_id: suspect,
     })?;
@@ -129,7 +132,7 @@ where
             .filter(|(id, _)| *id == suspect)
             .map(|(_, entry)| entry);
         if entry.is_none() {
-            entry = find_path_entry_in_commit(&odb, &suspect, file_path, &mut buf, &mut buf2, &mut stats)?;
+            entry = find_path_entry_in_commit(&odb, &suspect, file_path, &mut buf, &mut buf2, &mut stats, &mut trees)?;
         }

         let Some(entry_id) = entry else {
@@ -138,7 +141,7 @@ where

         for (pid, parent_id) in parent_ids.iter().enumerate() {
             if let Some(parent_entry_id) =
-                find_path_entry_in_commit(&odb, parent_id, file_path, &mut buf, &mut buf2, &mut stats)?
+                find_path_entry_in_commit(&odb, parent_id, file_path, &mut buf, &mut buf2, &mut stats, &mut trees)?
             {
                 let no_change_in_entry = entry_id == parent_entry_id;
                 if pid == 0 {
@@ -215,6 +218,7 @@ where
     // I don’t know yet whether it would make sense to use a data structure instead that preserves
     // order on insertion.
     out.sort_by(|a, b| a.start_in_blamed_file.cmp(&b.start_in_blamed_file));
+    stats.tree_cache_size = trees.len();
     Ok(Outcome {
         entries: coalesce_blame_entries(out),
         blob: blamed_file_blob,
@@ -432,19 +436,62 @@ fn find_path_entry_in_commit(
     buf: &mut Vec<u8>,
     buf2: &mut Vec<u8>,
     stats: &mut Statistics,
+    cache: &mut gix_hashtable::HashMap<ObjectId, gix_object::Tree>,
 ) -> Result<Option<ObjectId>, Error> {
-    let commit_id = odb.find_commit(commit, buf)?.tree();
+    let tree_id = odb.find_commit(commit, buf)?.tree();
     stats.commits_to_tree += 1;
-    let tree_iter = odb.find_tree_iter(&commit_id, buf)?;
-    stats.trees_decoded += 1;

-    let res = tree_iter.lookup_entry(
-        odb,
-        buf2,
-        file_path.split(|b| *b == b'/').inspect(|_| stats.trees_decoded += 1),
-    )?;
-    stats.trees_decoded -= 1;
-    Ok(res.map(|e| e.oid))
+    Ok(lookup_entry_id_in_tree(file_path, tree_id, odb, cache, buf2, stats)?)
+}
+
+pub fn lookup_entry_id_in_tree(
+    path: &BStr,
+    mut tree_id: gix_hash::ObjectId,
+    odb: impl gix_object::Find,
+    cache: &mut gix_hashtable::HashMap<ObjectId, gix_object::Tree>,
+    buffer: &mut Vec<u8>,
+    stats: &mut Statistics,
+) -> Result<Option<ObjectId>, gix_object::find::existing_object::Error> {
+    let mut path = path.split(|b| *b == b'/').peekable();
+    while let Some(component) = path.next() {
+        let tree = match cache.get(&tree_id) {
+            Some(tree) => {
+                stats.tree_cache_hits += 1;
+                tree
+            }
+            None => {
+                // TODO(borrowchk): use entry-API once it's possible to mutate and return.
+                let mut tree = odb.find_tree(&tree_id, buffer)?.to_owned();
+                tree.entries.sort_by(|a, b| a.filename.cmp(&b.filename));
+
+                cache.insert(tree_id, tree);
+                stats.trees_decoded += 1;
+                stats.tree_cache_misses += 1;
+                cache.get(&tree_id).expect("just put")
+            }
+        };
+
+        use gix_object::bstr::ByteSlice;
+        match tree
+            .entries
+            .binary_search_by(|e| e.filename.as_bytes().cmp(&component))
+            .ok()
+        {
+            Some(entry_id) => {
+                let entry = &tree.entries[entry_id];
+                if path.peek().is_none() {
+                    return Ok(Some(entry.oid));
+                } else {
+                    if !entry.mode.is_tree() {
+                        return Ok(None);
+                    }
+                    tree_id = entry.oid;
+                }
+            }
+            None => return Ok(None),
+        }
+    }
+    Ok(None)
 }

 /// Return an iterator over tokens for use in diffing. These usually lines, but iit's important to unify them
diff --git a/gix-blame/src/types.rs b/gix-blame/src/types.rs
index e0c8843b4..4bfc3540d 100644
--- a/gix-blame/src/types.rs
+++ b/gix-blame/src/types.rs
@@ -33,6 +33,12 @@ pub struct Statistics {
     /// The amount of blobs there were compared to each other to learn what changed between commits.
     /// Note that in order to diff a blob, one needs to load both versions from the database.
     pub blobs_diffed: usize,
+    /// TODO
+    pub tree_cache_hits: usize,
+    /// TODO
+    pub tree_cache_misses: usize,
+    /// TODO
+    pub tree_cache_size: usize,
 }

 impl Outcome {

I just 'fixed' it by assuming it's naturally possible that there is no entry for a suspect, so there is no need to fail. Now it runs through and produces the correct result. In the background I am collecting more data and blame every file in the Git repository. It takes a while, but my hope is that it won't panic anymore.

From what I remember that can happen when we pass blame to a single parent only out of more than one. traverse will still yield the commits that we know can’t contain a relevant change, and those commits don’t have an entry in suspects.

I think I have got it, the proof that Git is just doing significantly less, magically… .
GIT_TRACE=true \
GIT_TRACE_PACK_ACCESS=true \
GIT_TRACE_PACKET=true \
GIT_TRACE_PACKFILE=true \
GIT_TRACE_PERFORMANCE=true \
GIT_TRACE_SHALLOW=true \
GIT_TRACE_SETUP=true \
/usr/bin/time -lp git blame t/t4013/diff.rev-list_--children_HEAD 2>&1 | rg packfile.c | wc -l
   27963

It reads only 28k objects in total, whereas we read a whopping 78553 unique objects, and fetch a total of 227k objects! This must be largely due to the commit graph traversal, which definitely has to be done internally to get the same performance.
That's 3x more objects fetched, which costs time to do but also costs extra time to parse and handle. So I am sure that if we woudl read not more objects than Git, we'd be as fast or faster.

One reason we do extra work is in the case where a commit has multiple parents and we pass blame to just one of them. Even if we didn’t have to follow the other ancestors, this is a fact that traverse does not know about, so it still yields commits that we know won’t contain relevant changes. It would be interesting to verify that hypothesis in a repository that has a strictly linear commit history.

With this patch one could extract additional performance information:

Now it's clear that Git asks for 41813 objects total, and 20864 of these are unique.
❯ GIT_TRACE=true \
GIT_TRACE_PACK_ACCESS=true \
/usr/bin/time -lp ./git blame t/t4013/diff.rev-list_--children_HEAD 2>&1 | rg object-file.c | awk '{ print $3 }' | wc -l
   41813

git ( master) +205 -237 [!?] via 🐍
❯ GIT_TRACE=true \
GIT_TRACE_PACK_ACCESS=true \
/usr/bin/time -lp ./git blame t/t4013/diff.rev-list_--children_HEAD 2>&1 | rg object-file.c | awk '{ print $3 }' | sort -u | wc -l
   20864

I am now running a script to count the occurrences of the type of each objects - that should make clear how many commits it sees, for example, and from there the trees are derived. Avoid full diffs by doing a path-based lookup first is exactly what Git does, even though Git has integrated that into the diff itself.
And here is the result:
   1 blob
   2 commit
20861 tree

Erm, probably I am missing commits, let's retry.

And I have another reason for the performance: Git can get the tree of a commit from the commit-graph itself. This is where we have to read each commit even though the traversal might get the data from the commit-graph.
Now that I have deleted the commit-graph, it also reads all commits like it should.
Now there are 35k unique objects:
❯ wc -l ./objects.list
   35664 ./objects.list

So I'd guess it's 15k commits overall that are part of the traversal. Quite some less compared to the 58k that we have to go through.
Once we can do what Git can, I think we will be just as fast, at least as long bloom filters aren't playing a role :D.
And here are the results:
❯ xargs -n1 git cat-file -t < objects.list | sort | uniq -c
   1 blob
14802 commit
20861 tree

Ok, so it's clear that it also very efficiently only reads the one blob that adds that file 15k commits ago. Very good, and something that we'd have to match to be competitive. Probably that's the top priority before implementing any additional features.

Congratulations! This is great work, and I really love how the core of the algorithm is basically perfect. All I could do here was to refactor what surrounds it, but the key to blaming is well-tested and well understood, implemented in a way that doesn't even show up in the profiler.
Thanks again for all your time - I sure hope there is more so gix blame can get as fast as Git, and faster :).

That is amazing! I’m really happy it turned out this way! Thanks a lot for the feedback, looking forward to my next PR! :grinning:

Woohoo! 🎉  I've been following this work from the sidelines and excited to see it land. Great collaboration @cruessler and @Byron! ✨


Mining_rig_config.json --- Support

How exactly does this config work? what does the number next to the tag represent? chance, output, richness? which?

would like to bump this as well, I am trying to add support for ores from the ad astra mod to my modpack and this mod seems to have no way currently to actually add other modded ores to the machine. It would be great to have more clear documentation or even just a wau to specify and add allowed ores for the mining drill as this would tie into modpacks nicely. I can look at the code and try making a PR soon if I can figure out how to override the default allowed ores.

I think it is related to energy use but I don't know how much this number impacts the usage.
Speaking from experience, I was trying to nerf this ore generation because it is ridiculously high


sfizz recognised as an effect plugin by fl studio

I don't think this is supposed to happen

Here a thread about how to fix it: https://github.com/sfztools/sfizz-ui/issues/3#issuecomment-1835450212

For a written response:

Scan the plugins with "verify plugins" enabled.
Load the plugin from FL Browser with the path Plugin Database -> Generators -> VST3

Hope it helps.


[SMALLFIX]Deprecate Throwables.propagate with RuntimeException in NetworkAddressUtils.java

replace throw Throwables.propagate() with throw new RuntimeException()

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/17212/
Test PASSed.


Comments in bib file are transformed into @comments

I come across two issues here:

[ ] Somehow the tool transforms my comments (ones starting with %) in bib file into @comment{} and places them at the head of the file;
[ ] All the bibs are by default organized in an alphabetic manner, is there a way (option) I can remain the order of bibs (and thus keep the comments where they are) .

Great tool by the way :)

I second this.
The most important to my eyes is to keep the git diff as small as possible, so that we can easily inspect was has been updated.

About the sorting, of the entries this patch solves it
Index: rebiber/normalize.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/rebiber/normalize.py b/rebiber/normalize.py
--- a/rebiber/normalize.py	(revision 3c46fd379a3da5414403d9ef5ffc695bafccd4b3)
+++ b/rebiber/normalize.py	(date<PHONE_NUMBER>058)
@@ -1,3 +1,4 @@
+from bibtexparser.bwriter import BibTexWriter
 import rebiber
 from rebiber.bib2json import normalize_title, load_bib_file
 import argparse
@@ -34,7 +35,7 @@
             return True
     return False

-def post_processing(output_bib_entries, removed_value_names, abbr_dict):
+def post_processing(output_bib_entries, removed_value_names, abbr_dict, sort=False):
     bibparser = bibtexparser.bparser.BibTexParser(ignore_nonstandard_types=False)
     bib_entry_str = ""
     for entry in output_bib_entries:
@@ -64,7 +65,10 @@
                     if re.match(pattern, output_entry[place]):
                         output_entry[place] = short

-    return bibtexparser.dumps(parsed_entries)
+    writer = BibTexWriter()
+    if not sort:
+        writer.order_entries_by = None
+    return bibtexparser.dumps(parsed_entries, writer=writer)

 def normalize_bib(bib_db, all_bib_entries, output_bib_path, deduplicate=True, removed_value_names=[], abbr_dict={}):

but the order within entries is not conserved so the diffs are still huge even if nothing changed.

About the sorting, of the entries this patch solves it
Index: rebiber/normalize.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/rebiber/normalize.py b/rebiber/normalize.py
--- a/rebiber/normalize.py	(revision 3c46fd379a3da5414403d9ef5ffc695bafccd4b3)
+++ b/rebiber/normalize.py	(date<PHONE_NUMBER>058)
@@ -1,3 +1,4 @@
+from bibtexparser.bwriter import BibTexWriter
 import rebiber
 from rebiber.bib2json import normalize_title, load_bib_file
 import argparse
@@ -34,7 +35,7 @@
             return True
     return False

-def post_processing(output_bib_entries, removed_value_names, abbr_dict):
+def post_processing(output_bib_entries, removed_value_names, abbr_dict, sort=False):
     bibparser = bibtexparser.bparser.BibTexParser(ignore_nonstandard_types=False)
     bib_entry_str = ""
     for entry in output_bib_entries:
@@ -64,7 +65,10 @@
                     if re.match(pattern, output_entry[place]):
                         output_entry[place] = short

-    return bibtexparser.dumps(parsed_entries)
+    writer = BibTexWriter()
+    if not sort:
+        writer.order_entries_by = None
+    return bibtexparser.dumps(parsed_entries, writer=writer)

 def normalize_bib(bib_db, all_bib_entries, output_bib_path, deduplicate=True, removed_value_names=[], abbr_dict={}):

but the order within entries is not conserved so the diffs are still huge even if nothing changed.

Thank you so much for the comment and providing your insight into this sorting issue. I incorporated your suggestion into our code, which now provides users with the option to either keep the original order of the input bib entries or sort them alphabetically in the output file (PR #32). This update will for sure be included in our next release.
We will look into the second problem mentioned above regarding the order within entries and work on it. Thanks again!

Happy to be useful


IMN-162 - descriptors archiver consumer job

[ To be merged after #304 ]
Closes IMN-162
This PR

Replaces the internal token generation we implemented, reusing (copy-pasting) the one from https://github.com/pagopa/interop-be-reports/tree/1.0.x/jobs/commons/src/services
Implements the EService Descriptors Archiver Job (Scala code here: https://github.com/pagopa/interop-be-jobs/tree/1.0.x/eservice-descriptors-archiver/src/main/scala/it/pagopa/interop/eservicedescriptorsarchiver)

Automated tests for the business logic
Not testing the event-consuming part, only the business logic after the archived agreement is retrieved from the event payload.

Manual tests
Case 1 - archiving Deprecated descriptor
Archiving Agreement f6c02190-eecf-4767-90b3-0cc7740f365c, all other Agreements for the same descriptor are also archived, and the Descriptor is Deprecated. Archiving descriptor.

Successful call to catalog-process:

Descriptor archived in readmodel:

Case 2 - archiving Suspended descriptor
Archiving Agreement f6c02190-eecf-4767-90b3-0cc7740f365c, all other Agreements for the same descriptor are also archived, the Descriptor Suspended and there exists a newer descriptor.  Archiving descriptor.

Successful call to catalog-process:

Descriptor archived in readmodel:

Case 3 - not archiving because not all other agreements are archived
Archiving Agreement f6c02190-eecf-4767-90b3-0cc7740f365c, but another non-archived Agreement exists for the same Descriptor. Not archiving descriptor.

Case 4 - not archiving because descriptor is not Deprecated or Suspended
Archiving Agreement f6c02190-eecf-4767-90b3-0cc7740f365c, all other Agreements for the same descriptor are also archived, but the Descriptor is not Deprecated or Suspended.  Not archiving descriptor.

Case 5 - not archiving, descriptor is Suspended and no newer descriptor exists
Archiving Agreement f6c02190-eecf-4767-90b3-0cc7740f365c, all other Agreements for the same descriptor are also archived, the Descriptor Suspended and no newer descriptors are found.  Not archiving descriptor.

Auth Updater smoke tests
Still working as expected.
Publishing a descriptor -- auth updater consumes the event and calls the auth management

Logs on auth management confirm

@galales ready for review again, addressed all comments!


Add UpgradeBehavior: uninstallPrevious to WindowsADK and WindowsSDK

This adds the uninstallPrevious behavior to WindowsADK and WindowsSDK packages. Both were throwing the error 2008, which when trying to apply the exe manually resulted in a popup error of "You must uninstall the previous version before installing this version".
For WindowsSDK, I had to manually uninstall all versions from Control Panel > Programs and Features before this fix would work. I did not have WindowsADK previously installed.
Before uninstalling the previous versions manually, winget uninstall threw an error (see below) and winget install and winget upgrade continued to fail.

(2/2) Found Windows Software Development Kit [Microsoft.WindowsSDK] Version 10.0.22621.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Successfully verified installer hash
Starting package install...
Installer failed with exit code: 2008

❯ winget --info
Windows Package Manager v1.3.2091
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.22000.856
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.18.2091.0

Error in log file:
[2D3C:5990][2022-08-10T08:16:57]i000: Setting string variable 'VS2022INSTALL' to value 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE'
[2D3C:5990][2022-08-10T08:16:57]i000: ERROR: MSI replacement servicing is not supported
[2D3C:5990][2022-08-10T08:16:57]i000: View is requesting shutdown
[2D3C:5990][2022-08-10T08:16:57]i199: Detect complete, result: 0x0
[2D3C:5990][2022-08-10T08:16:57]i500: Shutting down, exit code: 0x7d8
....
....
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: SystemFolder = C:\WINDOWS\SysWOW64\
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: UCRTEXISTS = 1
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: UCRTVERSION = 10.0.22000.1
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: VersionNT64 = <IP_ADDRESS>
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: VS2022INSTALL = C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleAction = 5
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleElevated = 1
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleInstalled = 1
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleLastUsedSource = C:\ProgramData\Microsoft\VisualStudio\Packages\Win11SDK_10.0.22621,version=10.0.22621.0,productarch=neutral\
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleLog = C:\Users\user\AppData\Local\Temp\windowssdk\Windows_Software_Development_Kit___Windows_10.0.22621.1_20220810081657.log
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleManufacturer = Microsoft Corporation
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleName = Windows Software Development Kit - Windows 10.0.22621.1
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleOriginalSource = C:\WINDOWS\Temp\{E8F15567-E078-42AF-A100-B3823D6B2D09}\.cr\winsdksetup.exe
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleOriginalSourceFolder = C:\ProgramData\Microsoft\VisualStudio\Packages\Win11SDK_10.0.22621,version=10.0.22621.0,productarch=neutral\
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleProviderKey = {fcb5e4cc-2739-4812-97fd-f3b7c5d151ae}
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleSourceProcessFolder = C:\Users\user\AppData\Local\Temp\WinGet\Microsoft.WindowsSDK.10.0.22621.1\
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleSourceProcessPath = C:\Users\user\AppData\Local\Temp\WinGet\Microsoft.WindowsSDK.10.0.22621.1\winsdksetup.exe
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleTag =
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleUILevel = 2
[2D3C:5990][2022-08-10T08:16:57]i410: Variable: WixBundleVersion = 10.1.22621.1

Unable to even uninstall the package via winget:
❯ winget uninstall --id Microsoft.WindowsSDK
No installed package found matching input criteria.

❯ winget upgrade
Name                                                    Id                   Version          Available    Source
-----------------------------------------------------------------------------------------------------------------
Windows Software Development Kit - Windows 10.0.20348.1 Microsoft.WindowsSDK < 10.0.22000.832 10.0.22621.1 winget

[ x ] Have you signed the Contributor License Agreement?
[ x ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ x ] Have you validated your manifest locally with winget validate --manifest <path>?
[ x ] Have you tested your manifest locally with winget install --manifest <path>?
[ x ] Does your manifest conform to the 1.1 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Currently you have to make a PR for each version changes. (No, they don't manually merge PRs)
And then i'll just throw in that when i tried to use uninstallPrevious the last time i got exit code<PHONE_NUMBER>
Logs over at https://github.com/microsoft/winget-cli/issues/1818#issuecomment-1000541439 if there was a issue with my VM

Ugh on the PR for each version change :(
As for the exit code, I tested both of these. I didn't have WindowsADK installed prior. I installed the lower verison first:
❯ winget install -m .\10.1.19041.1\
Found Windows Assessment and Deployment Kit - Windows 10 [Microsoft.WindowsADK] Version 10.1.19041.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://download.microsoft.com/download/8/6/c/86c218f3-4349-4aa5-beba-d05e48bbc286/adk/adksetup.exe
  ██████████████████████████████  1.84 MB / 1.84 MB
Successfully verified installer hash
Starting package install...
Successfully installed

m\Microsoft\WindowsADK on  windowssdk-adk-upgradebehave [!] took 42s
❯ winget upgrade -m .\10.1.22000.1\
Found Windows Assessment and Deployment Kit [Microsoft.WindowsADK] Version 10.1.22000.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://download.microsoft.com/download/1/f/d/1fd2291e-c0e9-4ae0-beae-fbbe0fe41a5a/adk/adksetup.exe
  ██████████████████████████████  2.08 MB / 2.08 MB
Successfully verified installer hash
Starting package uninstall...
Successfully uninstalled
Starting package install...
Successfully installed

No issues.
Did the same for WindowsSDK, but this one I had to uninstall all prior installed versions from Control Panel > Programs and Features as winget uninstall --id Microsoft.WindowsSDK returned with:
winget uninstall --id Microsoft.WindowsSDK
No installed package found matching input criteria.

Install + Upgrade of WindowsSDK:
❯ sudo winget install -m .\10.0.22000.832\
Found Windows Software Development Kit [Microsoft.WindowsSDK] Version 10.0.22000.832
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Successfully verified installer hash
Starting package install...
Successfully installed

m\Microsoft\WindowsSDK on  windowssdk-adk-upgradebehave [!] took 3m13s
❯ sudo winget upgrade -m .\10.0.22621.1\
Found Windows Software Development Kit [Microsoft.WindowsSDK] Version 10.0.22621.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Successfully verified installer hash
Starting package uninstall...
Successfully uninstalled
Starting package install...
Successfully installed

Hopefully it was just on my end, atleast the exitcode is searchable on this repo now.
As for WindowsSDK, it appears this is designed to be installed side-by-side https://github.com/microsoft/winget-cli/issues/2437 so i guess we need to split those instead of adding the uninstallPrevious switch, or is it good enough to pin a version? (future feature)


Make import suggestion dropdown more readable

The word "Add" in front of the line actually reduces readability.
If that dropdown somehow used the editor font (likely a monospace one), and ideally even syntax highlighting, it would be much easier to see what name is being imported from where.

Take an editor line like this for example - it's super clear what you are importing from where:

We've improved consistency between auto-imports and the text in the add import code action suggestions and added quotations to make it a little more clear.
Unfortunately, VS Code does not support monospaced font or syntax highlighting so we can't do much beyond that at this time. If you'd like to file an issue to their repo, I'm sure they'd welcome the suggestion.


Various docs fixes

Codecov Report

Merging #506 (24a0d70) into main (aa9563d) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #506      +/-   ##
==========================================
- Coverage   69.38%   69.37%   -0.02%
==========================================
  Files          79       79
  Lines        9192     9192
==========================================
- Hits         6378     6377       -1
- Misses       2814     2815       +1

see 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


All CI tests fail due to timeout when installing Epinio

Since 08/05/22 EKS-CI, EKS-LETSENCRYPT-CI, AKS-CI, AKS-LETSENCYPT-CI and also RKE-CI since 08/06/22 fail when installing Epinio.
We need to investigate why and fix it
Example with EKS-CI:
    STEP: Installing Epinio 08/08/22 05:01:53.275
  << End Captured GinkgoWriter Output

  Release "epinio" does not exist. Installing it now.
  Error: timed out waiting for the condition

  Unexpected error:
      <*exec.ExitError | 0xc00064a040>: {
          ProcessState: {
              pid: 57736,
              status: 256,
              rusage: {
                  Utime: {
                      Sec: 4,
                      Usec: 875937,
                      Pad_cgo_0: [0, 0, 0, 0],
                  },
                  Stime: {
                      Sec: 1,
                      Usec: 301678,
                      Pad_cgo_0: [0, 0, 0, 0],
                  },
                  Maxrss: 73805824,
                  Ixrss: 0,
                  Idrss: 0,
                  Isrss: 0,
                  Minflt: 34620,
                  Majflt: 0,
                  Nswap: 0,
                  Inblock: 0,
                  Oublock: 0,
                  Msgsnd: 4520,
                  Msgrcv: 6224,
                  Nsignals: 775,
                  Nvcsw: 523,
                  Nivcsw: 59030,
              },
          },
          Stderr: nil,
      }
      exit status 1
  occurred
  In [It] at: /Users/runner/work/epinio/epinio/acceptance/install/scenario4_test.go:117

Pasting also other cases:

AKS-CI
AKS-LETSENCRYPT-CI
GKE-CI
GKE-LETSENCRYPT-CI
RKE-CI

Note, while RKE-CI failed on 08/05/22 during the Metallb installation process (link), from next day onwards failed during Epinio installation as the rest; so it seemed a one-time issue
STEP: Installing MetalLB 08/04/22 08:14:15.234
  << End Captured GinkgoWriter Output

  Error from server (InternalError): error when creating "../../assets/tests/resources.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "[https://metallb-webhook-service.metallb.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s](https://metallb-webhook-service.metallb.svc/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s)": dial tcp <IP_ADDRESS>:9443: connect: connection refused
  Error from server (InternalError): error when creating "../../assets/tests/resources.yaml": Internal error occurred: failed calling webhook "l2advertisementvalidationwebhook.metallb.io": failed to call webhook: Post "[https://metallb-webhook-service.metallb.svc:443/validate-metallb-io-v1beta1-l2advertisement?timeout=10s](https://metallb-webhook-service.metallb.svc/validate-metallb-io-v1beta1-l2advertisement?timeout=10s)": dial tcp <IP_ADDRESS>:9443: connect: connection refused

I was able reproduce the original issue with RKE workflow (the second issue seems to be unrelated):
  Release "epinio" does not exist. Installing it now.
  Error: timed out waiting for the condition

It came from helm upgrade --install epinio ... --wait - the problem is, that for some reason epinio-server pod doesn't start and ends up in CrashLoopBackOff state with following error in logs:
kubectl logs epinio-server-6b4dc6f84c-vc7pr -n epinio

❌  error creating handler: SESSION_KEY environment variable not defined

Still needs to be investigated.

The issue is that latest epinio helm chart doesn't set SESSION_KEY variable anymore (shouldn't be needed for recent epinio binary). Due to that helm installation fails on timeout when using --wait even before building, patching and injecting the epinio binary into epinio-server pod.


result type of Result<Self,_> for inject macro

i think it make more sense to use result type of Result when using inject macro, as some service might fail to initialize, ex api service need to check whether the endpoint is accessible or not.
Edit : just read on how inject macro work and it turns out it just create InejctBuilder Instance so i can work around with that. but having it with inject macro might be useful imo

Using Result has many of the same challenges as function colorization. In fact, since Rust doesn't formally have constructors and the constructor is just a regular 'ol function and it is colored if it supports async. The crux of the problem occurs from a dependency chain of results:
C: Result<C, E3> → B: Result<B, E2> → A: Result<A, E1>

There is no guarantee nor requirement that all of the errors are the same or compatible. One approach would be to assume that the errors have an Into or simple conversion and render that. If there is no conversion or it's invalid, you get a compiler error. Things get considerably more complex if there are multiple possible results. Imagine two dependencies that produce two different errors. That should theoretically still be possible to code generate, but it's certainly more complex.
Although constructors in Rust are not the same as in other languages, a constructor should be infallible IMO. There are a few rare edge cases such as invalid arguments, but Rust does a pretty good job helping make that a non-issue (ex: no null, signed vs unsigned integers, etc).
All that being said, if you want to be able to use Result, then you should be able to. I fully expect there to be additional enhancements to the macro code generation and perhaps it should support Result. I think it just needs to flush out what that would look like and some of the obscure edge cases. I've tried to avoid those, but some crept up any; for example, impl Iterator<Item> only works with a constructor and non-injected types on bare struct definitions and tuple structs are assumed to implement Default.
The workaround for edge cases is meant to be solved by implementing Injectable yourself or using the builder functions. If you have some specific examples or additional thoughts about supporting Result, I'm open to them. A number of ideas and patterns have been flushed out by implementing by hand first and then reworking them into macro code generation.

It's also worth mentioning that I have a similar problem in the more-options crate. OptionsFactory::create has all the mechanics to perform validation and return Result<T, ValidationResult>, but there currently isn't a way to surface that in a graceful way.
Unfortunately, that means the only other option is to panic (a la Result::unwrap) when OptionsManager::get calls into the factory. In this particular use case, it's probably ok because that means the application is in a bad state and is unrecoverable. However, I'd prefer to propagate the Result and let the caller decide.
I'm not sure what throwing Result into the mix really looks like, but there is a use case for it. I'm certainly invested in thinking about how this issue might be addressed. 😉


Rancher UI refreshes the UI page with no relevant logs in the rancher server

Rancher Server Setup

Rancher version: 44bb5b4 master-head
Installation option (Docker install/Helm Chart): Docker install

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

Proxy/Cert Details: Self signed

Describe the bug
Rancher UI refreshes sometimes out of blue with no user intervention
To Reproduce
Create a rancher server on master-head
Create an rke2/rke1 cluster with 1 node all roles
Result
Clusters come active and with no intervention, the UI refreshes  randomly.
Noticed this while on the ember UI
Expected Result
UI shouldn't refresh randomly with no user intervention.
No relevant logs are found in the rancher server while the UI refreshed

The only time the UI reloads itself is when the WebSocket connection is connected, dropped for more than 30 seconds, and then reconnected.  Because it has no mechanism to know what events were missed during the time it was disconnected or to efficiently resync to get back to current.
This behavior has been around forever (since 1.x) and hasn't changed so if you're newly seeing it you may have connection problems or the server is now disconnecting when it shouldn't.
(The Vue UI uses a newer mechanism to resynchronize where it lost track instead of reloading)


Added an Example to :class:~.DashedVMobject

Changelog / Overview

Motivation

Explanation for Changes

Documentation Reference

Testing Status

Further Comments

Checklist

[x] I have read the Contributing Guidelines
[x] I have written a descriptive PR title (see top of PR template for examples)
[x] I have written a changelog entry for the PR or deem it unnecessary
[ ] My new functions/classes either have a docstring or are private
[ ] My new functions/classes have tests added and (optional) examples in the docs
[x] My new documentation builds, looks correctly formatted, and adds no additional build warnings

Reviewer Checklist

[ ] The PR title is descriptive enough
[ ] The PR is labeled correctly
[ ] The changelog entry is completed if necessary
[ ] Newly added functions/classes either have a docstring or are private
[ ] Newly added functions/classes have tests added and (optional) examples in the docs
[ ] Newly added documentation builds, looks correctly formatted, and adds no additional build warnings

Link to docs.
Another great contribution!
Suggestion:
Insead of
class Example(Scene):
    def construct(self):
        r = 0.5
        pos = 2.5 * UP + 6 * LEFT  # Tracker for position on screen
        # Top Row: Increases the number of dashes per iteration
        for dashes in range(2, 12):  # [2 , ..., 11]
            circ = DashedVMobject(Circle(radius=r, color=WHITE), num_dashes=dashes).move_to(pos)
            self.add(circ)
            pos += 1.25 * RIGHT

I would do:
class Example(Scene):
    def construct(self):
        r = 0.5
        circles= Group()
        # Top Row: Increases the number of dashes per iteration
        for dashes in range(2, 12):  # [2 , ..., 11]
            circ = DashedVMobject(Circle(radius=r, color=WHITE), num_dashes=dashes)
            circles.add(circ)
        circles.arrange().set_y(2.5)
        self.add(circles)


Instax Mini Link 3 compatibility issue

Thank you for your wonderful repository.
Recently, I bought Instax Mini Link 3 and I tested this repo to print images from my computer.
However, it does not work. My printer just blinking yellow light. According to manual, this means 'film is jammed'. (However, it is not jammed really because I can print image from my smartphone application.)
So, I captured packets between Instax Link app and Instax Mini Link 3.
I do not have previous version of Instax printer, but it seems same until the image download sequences.
And I found differences after PRINT_IMAGE_DOWNLOAD_END.
Current version of InstaxBLE send PRINT_IMAGE packet after the end of image download.
But in case of Mini Link 3, it sends packet beginning with 0x30 0x03 (LED_PATTERN_SETTINGS_DOUBLE) between PRINT_IMAGE_DOWNLOAD_END and PRINT_IMAGE.
(Full content is here: https://pastebin.com/8zYqZAry)
Is there any references or something similar about Instax Printer opcode?
If it exists, I want to make this repository compatible with Mini Link 3.
Thanks again

Before I take a look at the log (thanks for that!) my first question is always the same: are you sure you enabled printing in the script? By default the image will be send over, but the actual PRINT command will not be send (to prevent accidental prints during testing). See this part of the readme. It sounds like you might have, but just asking to make sure ;)
On my Link Mini the yellow light comes on when the image is sent, but printing is not enabled; it's basically a "something unexpected happened" signal.
To answer your last question; there isn't any official documentation for the opcodes. The whole script is based on reverse engineering the mobile app, which don't offer much info besides some variable names.
As this script was a side project I have limited time to work on it, but I'd love to make sure we get it to work with the Link Mini 3.

Thank you for your response.
First, I solved problem that I previously described. And LED_PATTERN_SETTINGS_DOUBLE was not related to printing.
In conlusion, there was a problem in my test code. I just copied and pasted test code from README.
But this code disconnects before download complete. So I append sleep(300) to wait for download sequences.
Below is my test code.
instax = InstaxBLE(device_name="INSTAX")
instax.connect()
instax.enable_printing()  # allow printing
instax.print_image("image.jpg")  # print image
sleep(300)
instax.disconnect()  # all done, disconnect

However, I faced to another problem.
When I execute the above code, progress stucks at Img packets left to send: 20 (or 30).
So, I edited max_size_kb=105 to max_size_kb=55 in print_image. (Line 376, 380)
(55 is a number I arbitrarily chose.)
Then, it works successfully without any problem.
In the other words, I can print my own image from my computer :)
By the way, I want to print my image in best quality and find origin of problem.
In my opinion, it seems there are some bugs in resizing algorithm.
When I tested above, size of my original image is 3.5MB, and resized image is 104,738B. (I saved resized image from save_img_with_quality)
However, body of sent packet is Ab\x00\x0f\x10\x00\x02\x00\x00\x00\x00\x01\xc1\xc2\xb7.
And this mean the size of image that sent is 115,138B which is greater than 105kB.
It is why the progress stucks I think.
Also, I checked value of len(imgData) in below code using debugger, and its value is 115138.
# Line 383-385, print_image in InstaxBLE
self.packetsForPrinting = [
    self.create_packet(EventType.PRINT_IMAGE_DOWNLOAD_START, b'\x02\x00\x00\x00' + pack('>I', len(imgData)))
]

I think it is some kind of overhead, and its size is 10400.
But I do not know meaning of this number and how to handle it.
Is there any idea about this?
Or that I missed something?
Thanks again

Ah so the yellow light was indeed the 'something went wrong' message: the printer would only receive part of the commands before the script would disconnect, leaving the printer in an error state. It just wasn't for the reason I thought it might be.
The built-in example (when you run instaxBLE.py directly) already had this pause, but I never realized it was missing from the example in the readme. I've updated those, thanks.

I wonder what LED_PATTERN_SETTINGS_DOUBLE actually is for, as I never used it in my scripts.

I wonder if the Link 3 needs some other settings, like how the Link 2 was different from the Link. If you look in Types.py you'll see a PrinterSettings object. Currently we get the requested image size from the printer and use that to determine the printer model but maybe this is overlapping with one of the existing models, even though the actual image we're sending can be made larger.

Could you try running instax.get_printer_info() and note what details it shows?

By the way, check out the first comment in issue #18. Maybe you can use it to find the right values for your maximum image size.
I figured the Instax app must have been updated since they released the Link 3, and do see some differences. One thing I'd be interested in is determine what printer we are using based on the model number (instead of the image size).
I'd love to improve InstaxBLE (make sure we support all printers, replacing the hard sleep() with a nicer threading system #11 ) but reverse engineering the app and implementing these things takes a huge amount of (unpaid) time which I can't really afford at the moment... :/

Thank you for your repsonse.
And #11 #18  is very helpful to me!
Below is result of instax.get_printer_info()
Printer details:
Model:               Instax Mini Link
Photos left:         6/10
Battery level:       95%
Charging:            False
Required image size: 600x800px
MTU:                 244

But I think there is no significant difference between PrinterSettings and this.
Now I am finding about maximum size of image from packets currently.


would this automigrate script be of use in upgrading this tutorial to tensorflow 2.1?

https://www.tensorflow.org/guide/migrate
This a great tutorial, wish it would run with tensorflow 2.1

I started to migrate to TF2.3.0 and hopefully, it will be done soon as I am still learning Tensorflow.


cmplxs: add Real, Imag and Complex functions

Please take a look.
Closes #1420.

I have filed #1422 to address the issues raised more generally.


Only specify major version for post-v1 actions

Description

In the repository we have conventionally only specified the major version for our actions (specifically for actions that are stable, v1+). Since the Slack actions are in that category, we can drop the minor and patch versions and let the newer releases be used automatically.
This will reduce the changes in future "Update workflows" PR (like this one).
Checklist

[x] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible errors.
[ ] I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
PRs) or the media properties generator (ov just catalog/generate-docs media-props
for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@dhruvkb, this does not work when the action repository does not have the major version tag. For instance, Renovate only has the tags for the full version qualifier, and suggests using Renovate to update the action version: https://github.com/renovatebot/github-action/issues/791

Oh no! I assumed it was GitHub that was determining the latest patch version based on the major version.
If it doesn't work, I guess we should revert it.


check-db init container fails on init_fs_encoiding using KubernetesExecutor

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version
8.8.0
Kubernetes Version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:24:23Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.15) exceeds the supported minor version skew of +/-1

Helm Version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}

Description
I originally ran into this issue trying to upgrade a working instance of 8.7.1 to 8.8.0. However, a fresh install of either version of the charts (8.7.1 using the helm --version arg) now fails with this error.
The install goes through, but none of the pods run, because the check-db init container fails with the error shown in Relevant Logs.
We are using a custom repo for the image, but the image we are using is a direct pull and push from the docker hub image. I've replaced some identifying data (the image repo and ingress domain) in the values with <> tags. The logs are the full log output of the check-db container in the db-migrations pod.
Relevant Logs
bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/usr/local/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/local/bin/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/local/bin/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python39.zip',
    '/usr/local/lib/python3.9',
    '/usr/local/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa8772f5b80 (most recent call first):
<no Python frame>

Custom Helm Values
########################################
## CONFIG | Airflow Configs
########################################
airflow:
  ## configs for the airflow container image
  ##
  image:
    # We are using a custom repo to mirror the image, but it is a direct pull and push of the airflow image
    repository: <custom repo>/airflow
    tag: 2.6.3-python3.9
    pullPolicy: IfNotPresent

  ## the airflow executor type to use
  ## - allowed values: "CeleryExecutor", "KubernetesExecutor", "CeleryKubernetesExecutor"
  ## - customize the "KubernetesExecutor" pod-template with `airflow.kubernetesPodTemplate.*`
  ##
  executor: KubernetesExecutor

###################################
## COMPONENT | Airflow Scheduler
###################################
scheduler:
  logCleanup:
    enabled: false

###################################
## COMPONENT | Airflow Workers
###################################
workers:
  ## if the airflow workers StatefulSet should be deployed
  ##
  enabled: false

###################################
## COMPONENT | Flower
###################################
flower:
  ## if the airflow flower UI should be deployed
  ##
  enabled: false

###################################
## CONFIG | Airflow Logs
###################################
logs:
  ## configs for the logs PVC
  ##
  persistence:
    ## if a persistent volume is mounted at `logs.path`
    ##
    enabled: true

    ## the name of the StorageClass used by the PVC
    ## - if set to "", then `PersistentVolumeClaim/spec.storageClassName` is omitted
    ## - if set to "-", then `PersistentVolumeClaim/spec.storageClassName` is set to ""
    ##
    # empty string means cluster-edfas
    storageClass: ""

    ## the access mode of the PVC
    ## - [WARNING] must be "ReadWriteMany" or airflow pods will fail to start
    ##
    accessMode: ReadWriteMany

    ## the size of PVC to request
    ##
    size: 1Gi

###################################
## CONFIG | Kubernetes Ingress
###################################
ingress:
  ## if we should deploy Ingress resources
  ##
  enabled: true

  ## the `apiVersion` to use for Ingress resources
  ## - for Kubernetes 1.19 and later: "networking.k8s.io/v1"
  ## - for Kubernetes 1.18 and before: "networking.k8s.io/v1beta1"
  ##
  apiVersion: networking.k8s.io/v1beta1

  ## configs for the Ingress of the web Service
  ##
  web:
    ## the path for the web Ingress
    ## - [WARNING] do NOT include the trailing slash (for root, set an empty string)
    ##
    ## ____ EXAMPLE _______________
    ##   # webserver URL: http://example.com/airflow
    ##   path: "/airflow"
    ##
    path: ""

    ## the hostname for the web Ingress
    ##
    host: test-airflow-web.<domain>

###################################
## CONFIG | Kubernetes ServiceAccount
###################################
serviceAccount:
  ## if a Kubernetes ServiceAccount is created
  ## - if `false`, you must create the service account outside this chart with name: `serviceAccount.name`
  ##
  create: true

  ## the name of the ServiceAccount
  ## - by default the name is generated using the `airflow.serviceAccountName` template in `_helpers/common.tpl`
  ##
  name: "test-airflow"

###################################
## DATABASE | PgBouncer
###################################
pgbouncer:
  ## if the pgbouncer Deployment is created
  ##
  enabled: false

  ## configs for the pgbouncer container image
  ##
  # image:
  #   repository: ctl-devops.aero.org/edfas/airflow/pgbouncer
  #   tag: 1.17.0-patch.0
  #   pullPolicy: IfNotPresent
  #   uid: 1001
  #   gid: 1001

###################################
## DATABASE | Embedded Postgres
###################################
postgresql:
  ## if the `stable/postgresql` chart is used
  ## - [WARNING] the embedded Postgres is NOT SUITABLE for production deployments of Airflow
  ## - [WARNING] consider using an external database with `externalDatabase.*`
  ## - set to `false` if using `externalDatabase.*`
  ##
  enabled: false

###################################
## DATABASE | External Database
###################################
externalDatabase:
  ## the type of external database
  ## - allowed values: "mysql", "postgres"
  ##
  type: postgres

  ## the host of the external database
  ##
  host: postgresql

  ## the port of the external database
  ##
  port: 5432

  ## the database/scheme to use within the external database
  ##
  database: test_airflow

  ## the username for the external database
  ##
  user: postgres

  ## the name of a pre-created secret containing the external database user
  ## - if set, this overrides `externalDatabase.user`
  ##
  userSecret: ""

  ## the key within `externalDatabase.userSecret` containing the user string
  ##
  userSecretKey: "postgresql-user"

  ## the password for the external database
  ## - [WARNING] to avoid storing the password in plain-text within your values,
  ##   create a Kubernetes secret and use `externalDatabase.passwordSecret`
  ##
  password: "admin"

  ## the name of a pre-created secret containing the external database password
  ## - if set, this overrides `externalDatabase.password`
  ##
  passwordSecret: ""

  ## the key within `externalDatabase.passwordSecret` containing the password string
  ##
  passwordSecretKey: "postgresql-password"

  ## extra connection-string properties for the external database
  ##
  ## ____ EXAMPLE _______________
  ##   # require SSL (only for Postgres)
  ##   properties: "?sslmode=require"
  ##
  properties: ""

###################################
## DATABASE | Embedded Redis
###################################
redis:
  ## if the `stable/redis` chart is used
  ## - set to `false` if `airflow.executor` is `KubernetesExecutor`
  ## - set to `false` if using `externalRedis.*`
  ##
  enabled: false

This issue is no longer occurring. We have not changed our docker image, nor the values file in any way, but the release now works as expected. I expect there is something that was changed or fixed in the Kubernetes architecture (managed by a separate team) that fixed this issue. If I find out what changed, I will comment here.


4.16 type not found HaxeModuleRules

Hi, I'm using the example PlatformerGame as testbed to see if this still works with UE 4.16.
I've taken the latest project from GitHub and updated Plugin/UnrealHx to use the 4.16 branch
While running the usual haxe init-plugin.hxml -D UE_VER=4.16 I get the following error. Can you point me in the right direction by any chance?
--macro:1: character 1 : Uncaught exception Type not found 'HaxeModuleRules'
/Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Haxe/BuildTool/Package.hx:43: characters 19-53 : Called from here

Update
This is the hxml generated by the build tool:
# use this to build the build tool. if you have made any changes to any .Build.hx files, run this
-cp /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Haxe/BuildTool/src
-cp /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Haxe/BuildTool
-cp /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx
-net-lib /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Haxe/BuildTool/lib/DotNETUtilities-4.16.dll
-net-lib /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Haxe/BuildTool/lib/UnrealBuildTool-4.16.dll
-cs /Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Intermediate/Haxe/BuildTool
HaxeInit

-D UE_VER=4.16
-D no-root
-D net_ver=45
-D analyzer
-D real_position
--macro Package.main("/Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Source", [{ name:"Generated", target:"/Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Plugins/UnrealHx/Source/HaxeInit/Generated.Build.cs" }], {"targetExternal":"/Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Source/PlatformerGame/HaxeExternalModule.Build.cs","files":["/Users/tanis/Documents/Unreal Projects/HaxePlatformerGame/PlatformerGame/Source/PlatformerGame/PlatformerGame.Build.hx"]})

UE4.16 me too.
--macro:1: character 1 : Uncaught exception Type not found 'HaxeModuleRules'
InitPlugin.hx:110: characters 21-26 : Uncaught exception Haxe BuildTool compilation failed

Sorry to take this long to answer. Are you still having trouble with the latest?

I'm a little bit left behind on this. Haven't upgraded to the latest UE nor this plugin yet. I'll let you know as soon as I try that :)

For UE 4.16 and later, you don't need to run haxe init-plugin.hxml anymore.  I meant to still support having Build.hx files, but it's been a while since I tested that, so that might be the issue