How to handle randomness in optimizer

Hi, I have prevously worked¹ on a gradient-free optimization algorithm called SPSA²-³, and I have Matlab/mex code⁴ that I can port to python easily. I am interested in benchmarking SPSA against other zero-order optimization algorithms using nevergrad.
I am following the instructions for benchmarking a new optimizer given in adding-your-own-experiments-andor-optimizers-andor-function. My understanding is that I can just add a new SPSA class in nevergrad/optimization/optimizerlib.py and implement

__init
_internal_ask
_internal_provide_recommendation
_internal_tell

functions and then add SPSA to the optims variable in the right experiment function in the nevergrad/benchmark/experiments.py module and then I should be able to generate graphs like
docs/resources/noise_r400s12_xpresults_namecigar,rotationTrue.png
However, SPSA itself uses an rng in the _internal_ask function. But the optimizer base class does not take any seed in the __init__ function. What will be a good way to make the experiments reproducible in such situation?
[1] Pushpendre Rastogi, Jingyi Zhu, James C. Spall (2016). Efficient implementation of Enhanced Adaptive Simultaneous Perturbation Algorithms. CISS 2016, pdf
[2] https://en.wikipedia.org/wiki/Simultaneous_perturbation_stochastic_approximation
[3] https://www.chessprogramming.org/SPSA
[4] https://github.com/se4u/FASPSA/blob/master/src/SPSA.m

@se4u not one of the authors here, but it seems the lib is relying on random and numpy.random for these matter. In both cases, you can set the seed globally by doing random.seed(1234) or np.random.seed(1234)` to make subsequent calls to rngs reproducibles.
You can see how the authors use this here https://github.com/facebookresearch/nevergrad/blob/7fb29affd443330d8b75936856f43182a2d3b233/nevergrad/benchmark/xpbase.py#L119-L124
So I guess if you are able to use base python or numpy random, you should be able to get reproducible experiments the same way.

Hi, we are happy to welcome new algorithms indeed!
Optimizer implementation
You are correct in your understanding: implementing these 4 functions should make it work. The only one that is absolutely necessary is _internal_ask though.
Randomness
As @tomjorquera has correctly seen (thanks for starting answering ;) ), we rely on seeding during the experiments to make the benchmark reproducible. However, this also means, we avoid seeding inside the optimizer, so that we can run several optimization with different seed. In a nutshell, as long as you do not seed inside the optimizer, everything should work fine. Also, please use np.random over the standard library random if you can, but this is not an absolute requirement (only for consistency)
Experiments
You have several options:

you update an experiment generator as you mentioned, just adding the name to the optims list should work
you create another experiment generator, you will be able to run it just as well with the nevergrad.benchmark command.
you can actually implement a benchmark generator in an external file and use the --import argument of nevergrad.benchmark to import the file at runtime. This is if you do not want to submit the experiment.

My preference go to the second option, so that everybody can use your experiment, and so that it does not modify existing experiments (for reproducibility of the existing experiments)
In any case, I'll try to help you in the process if I am available (Christmas / New year period may slow down my answer speed though :s) and don't hesitate to ask more questions if anything remains unclear!

Just mentioning that SPSA is very (very) welcome.
Due to complexity associated to the management of seeds in case of multiple runs and averaging, maybe it's better to not seed the optimizers, as previously discussed.

I added SPSA to my fork of nevergrad  and compared it on basic, noise and illcond* experiments. I didn't do any hyper-parameter tuning. The hyper-parameter come straight from my earlier SPSA Matlab code.
The basic SPSA with averaging never really outperforms TBPSA, but on the sphere function it is competitive with TBPSA on average. The basic first-order SPSA worked badly on illcond problems, so I am not showing those results here. However, there are some more complicated versions of SPSA that try to estimate the hessian and theoretically promise better behavior on ill-conditioned problems.
The results on noise and basic are as follows:

Fig 1. noise/fight_namesphere

Fig 2. noise/fight_all

Fig 2. basic/fight_all
The class for SPSA is the following, I hardcoded an rng inside the class so that experiments are reproducible without modifying anything outside the class.
@registry.register
class SPSA(base.Optimizer):
    def __init__(self, dimension: int, budget: Optional[int] = None, num_workers: int = 1) -> None:
        super().__init__(dimension, budget=budget, num_workers=num_workers)
        self.rng = np.random.RandomState(seed=1234)
        self.init = True
        self.idx = 0
        self.delta = self.ym = self.yp = None
        self.t = np.zeros(self.dimension)
        self.avg = np.zeros(self.dimension)
        return

    @staticmethod
    def ck(k) -> float:
        return 1e-1 / (k//2 + 1)**0.101

    @staticmethod
    def ak(k) -> float:
        return 1e-5 / (k//2 + 1 + 10)**0.602

    def _internal_ask(self) -> base.ArrayLike:
        k = self.idx
        if k % 2 == 0:
            if not self.init:
                self.t -= (self.ak(k) * (self.yp - self.ym) / 2 / self.ck(k)) * self.delta
                self.avg += (self.t - self.avg) / (k // 2 + 1)
            self.delta = 2 * self.rng.randint(2, size=self.dimension) - 1
            return self.t - self.ck(k) * self.delta
        return self.t + self.ck(k) * self.delta

    def _internal_tell(self, x: base.ArrayLike, value: float) -> None:
        setattr(self, ('ym' if self.idx % 2 == 0 else 'yp'), value)
        self.idx += 1
        if self.init and self.yp is not None and self.ym is not None:
            self.init = False
        return

    def _internal_provide_recommendation(self) -> base.ArrayLike:
        return self.avg

The code and plots are here - https://github.com/se4u/nevergrad

I'm happy you were able to obtain results so quickly!
If you have any feedbacks on what could be improved, let us know. It may still take some time to make it completely painless to run, but I hope we will get to it ;)
If you are willing to submit a pull request, you are most welcome to it.
Concerning the random seed generator you use, this is indeed a good option I forgot to mention.
Do you absolutely need to seed it though? Since we seed externally when running the benchmark, we would get reproducibility even without providing a seed to your random generator.

@jrapin I used my own rng because I didn't want to affect the external rng by executing my own code. Also, this makes it easy to test in future what is the effect of the randomness inherent in the SPSA algo. versus the random noise in the problem. Also, this makes the class self-contained. No matter when you initialize it, its behavior will be consistent.
Regarding feedback, I think the code structure is pretty intuitive so that's great. I guess some guidelines about how to submit new algorithms will be useful.
For example, I can just submit a pull request after adding the above class, but should I also update the experiments module?  Typically anyone will want to compare a new algorithm to the existing ones on some standard problem and if a new experiment function needs to be created for every algorithm that is added then the experiments.py module will become bloated. In my repo I have modified the noise and basic experiments, should I also submit those as part of the PR?  Some guidelines about these issues will make it easier to submit PRs.

I have kept thinking about how the algorithms should be seeded. The more I think about it, the more I think it should not be seeded:

for consistency with the current implementation (but this argument is not that important, we could update it
because we expect stochastic algorithms to be actually stochastic, if we set a seed inside the implementation this assumption is broken.
to obtain relevant statistics when benchmarking the algorithms on deterministic functions (we will provide some interesting test cases that are deterministic probably before the end of January).
because we can seed anyway when we need it, this is what happens in the benchmarks, because in this case we do want each independent run to be repeatable.

So, as soon as I have some time (this is not the best time of year for this :D) I will prepare:

guidelines on how to add an algorithm, as you pointed out this will become necessary. I'll try to detail why it should not be seeded (at least not by default).
unit tests ensuring that setting the seed outside will make computation repeatable (to make sure the benchmarks will be reproducible). As I mentioned, I already run these tests internally, I'll just need to port them to this repo.

I'll post back here when this is ready ;). Sorry for changing minds about it, it is indeed an important decision you are pointing out here!

I just wanted to mention that your results are super exciting!
The version of SPSA you are using is supposed to converge at rate simple-regret=O(1/n^alpha) for alpha=2/3 or ... ?
I guess some different noise models with a lot of dissymmetry in terms of variance could be more in favor of SPSA... not sure though.

@teytaud The asymptotic convergence rate using my hyper parameters (a=0.602, γ=0.101) is k^(-(0.602-2*0.101)/2) = 1/k^(1/5). So in your terminology alpha = 1/5.
There are some conditions which basically put restrictions on a, γ.¹ a ∈ [ 0.602, 1] and γ ∈ [a/6, a/2). The fastest rate that can be achieved is when a is highest, i.e. 1, and γ is lowest i.e. 1/6. With those hyperparameters the convergence rate will be alpha = 1/3. But Prof. Spall suggests to use the more conservative values and those are what I used. I think the analysis could probably be simplified and streamlined compared to his approach although I haven't tried it.
[1] Spall, James C. "Multivariate stochastic approximation using a simultaneous perturbation gradient approximation." IEEE transactions on automatic control 37.3 (1992): 332-341.  and theorem 7.2 on google books
Also, I wanted to mention that regarding submitting a PR I am waiting for

guidelines on how to add an algorithm to the experiments.
unit tests ensuring that setting the seed outside will make computation repeatable
On my end, I need to rerun things after adding the  no_parallelization flag and then remove the hardcoded rng.

I am getting the following two errors during testing. The first one is about my class not supporting parallelization. The second is about reproducibility. I am not sure why the second error is happening.
I get the error regardless of whether I set a fixed seed in my optimizer rng or not
self._rng = np.random.RandomState()
self._rng = np.random.RandomState(1234)

both of the above cause error.
prastog3@b19 ~/project/nevergrad Wed Jan 02 13:01:49 pr err=1
$ nosetests nevergrad/optimization/test_optimizerlib.py
..........capi_return is NULL
Call-back cb_calcfc_in__cobyla__user__routines failed.
............................................E.................S.....capi_return is NULL
Call-back cb_calcfc_in__cobyla__user__routines failed.
.............................................F...............
======================================================================
ERROR: test_optimizers(SPSA) (nevergrad.optimization.test_optimizerlib.OptimizerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/prastog3/.local/lib/python3.6/site-packages/genty-1.3.2-py3.6.egg/genty/genty.py", lin$ 364, in <lambda>
    *dataset
  File "/home/prastog3/project/nevergrad/nevergrad/optimization/test_optimizerlib.py", line 74, in t$st_optimizers
    check_optimizer(optimizer_cls, budget=300, verify_value=verify)
  File "/home/prastog3/project/nevergrad/nevergrad/optimization/test_optimizerlib.py", line 31, in c$eck_optimizer
    optimizer = optimizer_cls(dimension=2, budget=budget, num_workers=num_workers)
  File "/home/prastog3/project/nevergrad/nevergrad/optimization/optimizerlib.py", line 449, in __ini$__
    super().__init__(dimension, budget=budget, num_workers=num_workers)
  File "/home/prastog3/project/nevergrad/nevergrad/optimization/base.py", line 58, in __init__
    raise ValueError(f"{self.__class__.__name__} does not support parallelization")
ValueError: SPSA does not support parallelization

======================================================================
FAIL: test_optimizers_recommendation(SPSA) (nevergrad.optimization.test_optimizerlib.OptimizerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/prastog3/.local/lib/python3.6/site-packages/genty-1.3.2-py3.6.egg/genty/genty.py", lin$ 364, in <lambda>
    *dataset
  File "/home/prastog3/project/nevergrad/nevergrad/optimization/test_optimizerlib.py", line 89, in t$st_optimizers_recommendation
    err_msg="Something has changed, if this is normal, delete "
  File "/home/prastog3/.pyenv/versions/3.6.6/lib/python3.6/site-packages/numpy/testing/_private/util$.py", line 973, in assert_array_almost_equal
    precision=decimal)
  File "/home/prastog3/.pyenv/versions/3.6.6/lib/python3.6/site-packages/numpy/testing/_private/utils
.py", line 789, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 10 decimals
Something has changed, if this is normal, delete /home/prastog3/project/nevergrad/nevergrad/optimizat
ion/recorded_recommendations.csv and rerun to update the values
(mismatch 100.0%)
 x: array([-2.0152900820e-05, -2.0152900820e-05,  7.4004300435e-06,
        2.0152900820e-05])
 y: array([ 1.0604319978e-05, -2.3356762000e-05, -2.3356762000e-05,
        2.3356762000e-05])

----------------------------------------------------------------------
Ran 139 tests in 4.279s

FAILED (SKIP=1, errors=1, failures=1)

Thanks! Let's follow up on the PR ;)

Please close the issue if everything is fine for you.
And you're welcome to open new ones if need be ;)

I should mention that after some more work SPSA turns out to be very good.
A key point is to adapt the scale.
I'd try to do more, but I already got some good results in SPSA which were not visible in the early tests.
https://drive.google.com/file/d/1VOjzHtDuFtdc34SqohvEvZzQMwSdjvsQ/view?usp=sharing
(discussion in the pinned post at https://www.facebook.com/groups/nevergradusers/ )
Thanks to the authors of the SPSA code.

oh wow, really good to hear. I am trying to read the pdf but I must apologize, I wasn't quite able to figure out how to adapt the scale. Also I looked at section 4: "Statistics over all benchmarks" and Section 5: "conclusion" and I didn't see SPSA being competitive. How should I read the results ?


Replace 'static lifetime on the value in kv::set with parameterized

Requiring that the value of the bytes passed into kv::set makes this
API difficult to practically use, it need only outlive kv::set.
Consider a common example of serialization of an object prior to
inserting into the consul kv store.
let bytes = serialize_to_bytes(object); // Temporary lifetime
kv::set(&client, "key", &bytes, None).await;

Additionally (1) introduced tests for validating sending bytes roundtrip
and using recurse in the kv tests, (2) replaced test_env_log with
test_log as test_env_log was deprecated in favor of test_log.

LGTM, thanks for the PR.

Could you publish a new release containing these changes?

We also need this change. Can you publish a new release? thanks!


Reactivity broken for a rule dependency

Reactivity broken for a rule dependency
I have a rule that depends on someCondition ref. When someCondition changes, the rule should be re-evaluated. However, it is not. However if I change the value of the field then the rule is re-evaluated.
Steps to reproduce:

Enter weight value: 2
Toggle the condition to be OFF
The validation error should go away but it does not. == BUG 🐞 ==
Enter weight value: 3
Now the validation error is gone. (But it should have disappeared when we toggled the condition.)

Reproduction:
https://stackblitz.com/~/github.com/martinszeltins/regle-test-8
const extraWeightRule = (myArg: string, index: number) => {
    return withMessage(value => {
        return Number(value) > 1
            && someCondition.value === false
    }, t('totally_not_good', { name: myArg + index }))
}

const rules = computed(() => {
    return {
        shipmentItems: {
            $each: (item, index) => ({
                weight: {
                    extraWeightRule: extraWeightRule('weight banana', index)
                }
            })
        }
    }
})

Also, I think we should have automated tests for such cases.

withAsync works the same way, as Vue can't collect dependencies on async functions

This is not a problem from regle, but reactivity on your side, as your dependency someCondition is inside a function, Vue can't collect the reference.
Use withParams specificaly for this case

@victorgarciaesgi Oh, I see. I think other people will run into the same "issue" and wonder why it is not working. Do you think we could add a separate section to the docs like "Reactivity caveats" or something like that to clearly explain this?

I think it's a good idea yeah!

👍🏻 closing this and opening a proper issue to track this in https://github.com/victorgarciaesgi/regle/issues/17


Porting to Godot 4

Started on porting this to Godot 4, did most of the code changes and renames needed however a few things in Godot 4 are still broken so I'm unable to test this.

Direct movement and teleport now pretty much work, still working on the other features.

Updated this to include all the new changes that were in the Godot 3 version. Still a few things left to fix, it doesn't like extending PhysicsBody3D and it doesn't like MovementProvider and PlayerBody being dependent on each other, so still figuring out workarounds for that...

Ok, I worked around some of the issues that Godot 4 had, I haven't tested all things yet but I think nearly everything is converted.
Vignette is broken because POSITION seems broken in the shader.

I've merged this into an official 4.0-dev branch now that Godot 4 XR is working.


Document Lit integration removal

Description (required)
Documents Lit integration removal
Related issues & labels (optional)
https://github.com/withastro/astro/pull/11680
For Astro version: 5.0.0-beta.

Thanks for this @matthewp ! For your reference, you would have been welcome to commit to this page directly! I will be reviewing the upgrade guide with a fine-tooth comb, so it's OK to reduce the friction on the platform team by just adding your content there.
On any other page, PRs are appreciated!


adding canonical and removing referrer

RE #641 cc: @jdorfman @XhmikosR

I can't check this thoroughly but the concept seems goods to me. As long as
the variables are properly set that is.
On a side note we should also look into adding social tags later.
On Jul 21, 2016 01:11, "Joshua Mervine"<EMAIL_ADDRESS>wrote:

RE #641 https://github.com/MaxCDN/bootstrap-cdn/issues/641 cc: @jdorfman
https://github.com/jdorfman @XhmikosR https://github.com/XhmikosR
You can view, comment on, or merge this pull request online at:
https://github.com/MaxCDN/bootstrap-cdn/pull/672
Commit Summary

adding canonical and removing referrer

File Changes

M routes/index.js
https://github.com/MaxCDN/bootstrap-cdn/pull/672/files#diff-0 (16)
M views/layout.jade
https://github.com/MaxCDN/bootstrap-cdn/pull/672/files#diff-1 (6)

Patch Links:

https://github.com/MaxCDN/bootstrap-cdn/pull/672.patch
https://github.com/MaxCDN/bootstrap-cdn/pull/672.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MaxCDN/bootstrap-cdn/pull/672, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVVtTdUfeKRhYXjdcOTKa3YpA7RNVH7ks5qXp0OgaJpZM4JRQqM
.

@XhmikosR You can see the result here: https://bootstrapcdn-dev-pr-672.herokuapp.com/
For example https://bootstrapcdn-dev-pr-672.herokuapp.com/bootlint/ has:
<link rel="canonical" href="http://bootstrapcdn-dev-pr-672.herokuapp.com/bootlint/">

Why not https?
On Jul 21, 2016 01:28, "Joshua Mervine"<EMAIL_ADDRESS>wrote:

@XhmikosR https://github.com/XhmikosR You can see the result here:
https://bootstrapcdn-dev-pr-672.herokuapp.com/
For example https://bootstrapcdn-dev-pr-672.herokuapp.com/bootlint/ has:

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MaxCDN/bootstrap-cdn/pull/672#issuecomment-234103491,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVVtarMdID-l50171hkVrfG4bA_-Od0ks5qXqD9gaJpZM4JRQqM
.

@XhmikosR That's actually a good question, it should be. Perhaps the Protocol isn't being passed on correctly through Heroku's routing layer? I'll need to investigate.

I think I found the answer here, testing it now... http://stackoverflow.com/a/16405622/832409

@jdorfman please look into adding me to BootstraCDN. We can always chat
with Justin via Hangouts.
On Jul 21, 2016 02:09, "Joshua Mervine"<EMAIL_ADDRESS>wrote:

I think I found the answer here, testing it now...
http://stackoverflow.com/a/16405622/832409
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MaxCDN/bootstrap-cdn/pull/672#issuecomment-234111479,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVVtQarjj0pe7IAIxnSlDNVFcaIi6Viks5qXqqegaJpZM4JRQqM
.

I still want to dig in to why this is occurring, but for now, I'm hard coding it to https as we force https in production anyway.

Better...
<link rel="canonical" href="https://bootstrapcdn-dev-pr-672.herokuapp.com/bootlint/">

It's I need strange since the protocol is https.
Might be better investigating this  before merging.
On Jul 21, 2016 02:17, "Joshua Mervine"<EMAIL_ADDRESS>wrote:

Better...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MaxCDN/bootstrap-cdn/pull/672#issuecomment-234112833,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVVtV3A-uoYAWEoLy-m7izxQA5mG1h3ks5qXqyEgaJpZM4JRQqM
.

Fair. I'll have to table it then. I'm fairly certain it's something behind the scenes with the Heroku deploy. It shouldn't be too hard to figure out.

Yeah, so that seems to have done the trick. Looks like Express isn't setting the header correctly on forward. Based on what I'm reading enable('trust proxy') should have worked.

🚢 'd


pve_disk_usage_bytes = 0.0

Hi!
Your pve-exporter is great! But it has only one bad thing:
There is "pve_disk_usage_bytes = 0.0" for any VM :((
Please fix that.

Duplicate of #77


internal/frontend: use Resources in ConfigurableVolume entries.

This allows us to consistently make use of them when e.g. generating config images etc.

LGTM


Link back from the CMIP6 Data Information View do not resolve anymore

The CMIP6 Data Information View provides a link.
Example:
https://handle-esgf.dkrz.de/lp/21.14103/17b771a4-e7cc-3f61-9a4d-df17caa60de2

Only one search item is transferred to the search. All other fields are empty and not applied.

Looks like the issue is that the CORDEX project is not fully supported.  We will need support from people involved in the project to complete the enhancement.  This is hosted at DKRZ...

Thanks for reporting. We will probably just remove the link.


Testing data availability

API which can give date wise testing data (number of tests done) state-wise and district wise

API which can give the consolidated number of tests done state-wise and district wise

This can be used to evaluate the relationship between the tests done and the cases detected.

It's already available here : https://api.covid19india.org/data.json
Please checkout the "tested" key. Thank you.

It's already available here : https://api.covid19india.org/data.json
Please checkout the "tested" key. Thank you.

How can we figure out form this which state tested data it has?


Add CODEOWNERS file

Required as part of company policy.

Codecov Report

Merging #106 (c3f75a3) into main (274c458) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #106   +/-   ##
=======================================
  Coverage   73.99%   73.99%
=======================================
  Files          20       20
  Lines        1496     1496
=======================================
  Hits         1107     1107
  Misses        293      293
  Partials       96       96

Flag
Coverage Δ

integration-test
59.17% <ø> (-0.68%)
:arrow_down:

unittests
47.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Could someone add Mac builds to https://github.com/leanprover/lean-nightly/releases ?

Could someone add at least Mac builds to https://github.com/leanprover/lean-nightly/releases ?
I am now stuck on a Lean binary and a Mathlib circa Feb 19 and cannot advance either. The Feb. 19 Lean does not build the latest Mathlib, nor does the latest lean-nightly. I have not learned how to build Lean myself yet.
Many thanks in advance!

This is an issue with lean or lean-nightly, not mathlib. I certainly am not in a position to fix this issue from my end because of permissions problems. I believe Sebastian is working on improving nightly tracking, but I don't know if Mac builds are likely to come out of that.

There will definitely be Mac (and Windows) builds as soon as I'm finished. The releases
linked above were just test releases I built locally.

I tried to build Lean on my Mac yesterday and found it to be easier than I imagined.  So, personally, I do not need this feature anymore, though I suppose it would still be useful to other Lean users.
Thanks!


Update zod to the latest non-vulnerable version

see https://github.com/colinhacks/zod/pull/2824

Yikes, quite a bad vulnerability, thanks a lot for this!

You're welcome but a new version hasn't been published on deno x yet

Hello, yeah, sorry about that, haven't really had time for any personal todo list tasks the last week as I've been sick!
Just released v0.1.5! :)

Sorry about your health. Thanks for the release but it looks like just
updating zod’s version brings up some type issues as seen in the screenshot
below. This makes response hint weird type definitions
On Sun, 15 Oct 2023 at 16:52, Skosh @.***> wrote:

Hello, yeah, sorry about that, haven't really had time for any personal
todo list tasks the last week as I've been sick!
Just released v0.1.5! :)
—
Reply to this email directly, view it on GitHub
https://github.com/skoshx/pentagon/pull/38#issuecomment-1763446106, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAM3775U347BJ7FQHHO4CY3X7QIGBANCNFSM6AAAAAA5XSK24I
.
You are receiving this because you authored the thread.Message ID:
@.***>


Add the stat command

What type of PR is this? (check all applicable)

[ ] ♻ Refactor
[x] ✨ New Chapter
[ ] 🐛 Bug Fix/Typo
[ ] 👷 Optimization
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
This PR adds the stat command chapter.
Related Tickets & Documents
Corresponds to issue #110
Added to documentation?

[X] 📜 readme
[ ] 🙅 no documentation needed

[optional] What gif best describes this PR or how it makes you feel?

Glad to help! :)


Fix argument of ExAllocatePool2

A couple of locations were missed by 3b4153b when porting to the new API.

Thank you :)


Make spoke work properly with Hardware GL rendering

right now spoke will freeze on startup if hardware opengl rendering is enabled in sw


Added fix for command line options in refine

Fixes #5776
Fixes #5773
Changes proposed in this pull request:

Allow JAVA_HOME to be set from the ini.
Give command line options precedence over the ini.
Clean up consistency in some wording between refine.bat and refine scripts.

Tested on Ubuntu Linux:
./refine
./refine -h
./refine run
./refine test
./refine clean
./refine build
./refine -c my.ini
./refine -c my.ini run
./refine -c missing.ini
./refine -m 8192m
./refine -c my.ini -m 8192m
./refine -m 8192m -c my.ini
./refine -c missing.ini -m 8192m
./refine -p 3344 -i <IP_ADDRESS> -m 8192m --debug --jmx
./refine -c my.ini -p 3344 -i <IP_ADDRESS> -m 8192m --debug --jmx

@elebitzero do you want to have this released in 3.7.3, or should it wait for 3.8?

I wouldn't do another release just for these fixes, but if we are doing 3.7.3 anyway, it can be included.
I have a few more fixes for refine/refine.bat, like for the standardization of the Google extension support.


:arrow_up: move to using a native go implementation of argon2

Coverage decreased (-26.7%) to 73.333% when pulling 92b2447df2bbab5359c2b99dfa69951ae92197da on feature/native-go-argon into b12e6aa9eba1c51bfa4ffa863b6a7fb7a757d110 on master.


Take CPU time into account when calculating FPS in the editor Frame Time

This is required to display accurate FPS measurements when the CPU is the bottleneck.
Note: Not cherry-pickable to the 3.x branch, as the View Frame Time panel is new in master (and relies on Vulkan-only functionality).

I wonder if we could make it clearer that this FPS estimated and not actually measured.
As to which estimation method is best, I'm not sure but yeah taking the highest of CPU and GPU time might be OK to account for different types of bottlenecks. CC @lawnjelly

Agree on possibly making it clearer the FPS figure is estimated.
Taking the higher of the two times probably makes sense for a rough guesstimate. However it could be a little confusing - the CPU figure normally in the editor viewport is for the editor, not the running game. So the CPU figure in this case is likely to give little information about the bottlenecks when running the project.

I don't really think this is useful at all since within the editor, all that really matters is the GPU. The CPU is not super useful and if its very high likely something is wrong.


Updated out link to point to git

Proposed changes
Changed url from relative redirecting to redirecting to third party site (https://git-scm.com)
Unreleased project version (optional)

Related issues (optional)

Deploy preview for docsdocker ready!
Built with commit 42c1d7feac84633509b3ef9a9fea625c3fc32acc
https://deploy-preview-9765--docsdocker.netlify.com


Fix the IsEmpty Property

The _data is never null
see https://github.com/OfficeDev/Open-XML-SDK/issues/759

Why we need the IsEmpty property? Because this property is used to indicate that the structure is not initialized.
The structure can be created without going through the constructor.


Pass through authentication to Windows hosts

I recognise that this is outside the scope of this project, but I'm yet to find a solution that works as I'd like.
I use this project as a front end to Windows RDP servers, and integrate Keycloak for OpenID authentication.
The authentication to the gateway works beautifully, and the username is passed to the RDP Server, but I am yet to find a way to have the authentication (done via OpenID) to persist to the client. Obviously the password is not known at this point, so it can't just be put into the RDP file, and the token doesn't seem to be meaningfully able to be verified by the client.
Are there any suggestions on how to address this? I don't believe we can OpenID auth and then issue a Kerberos token either?

Same scenario for me, if you find a way, let me know please :)
Sorry to ask but, i have a question how do you manage public certificate for RDP ? i have an nginx with public certificate for accessing rdpgw. It works great but all of my RDP servers are internal and i have a warning for "identity unknown" as they present the internal certificate
Is there something to do for not having this warning ?
Thanks


[Change Request] Github Data Digger - Split the Print of GitHub Data into Another Module

Ref: https://github.com/guibacellar/OSIx/commit/25f96dc6ad48d7f84c9ff14fd43dca398ef58fa1


Import Wallet - Missing Sub Accounts

Hey, Im hoping you can help. I had to restore my computer, i imported my Mani wallet using my seedphrase after this, but I can only see my main account and not the sub accounts. How can I also import the sub accounts?

Hey, just create new accounts and you should get your old sub accounts again.

I've the same issue and creating a sub-account didn't help


IX: update required site id field to be more flexible

Out of curiousity. Why? 😅
Was this an issue for other Prebid Server vendors?

Out of curiousity. Why? sweat_smile
Was this an issue for other Prebid Server vendors?

We did it just to match with go implementation.
https://github.com/prebid/prebid-server/commit/bab487e8a54b37dca75c5f86694170e60d638ef3


FileNotFoundError: [Errno 2] File or directory does not exist: '../data/berlin/undistorted/reconstruction.json' when trying to reconstruct 'berlin'

Hi everybody! hope you to be having a good day. Recently I've installed OpenSfM with all it's dependencies (I guess) so now I'm trying to do the test reconstruction with 'berlin' but when I try I get the next error:
2021-02-07 09:23:37,342 INFO: Loading existing EXIF for 02.jpg
2021-02-07 09:23:37,343 INFO: Loading existing EXIF for 01.jpg
2021-02-07 09:23:37,343 INFO: Loading existing EXIF for 03.jpg
2021-02-07 09:23:40,924 INFO: Skip recomputing ROOT_HAHOG features for image 02.jpg
2021-02-07 09:23:40,924 INFO: Skip recomputing ROOT_HAHOG features for image 01.jpg
2021-02-07 09:23:40,925 INFO: Skip recomputing ROOT_HAHOG features for image 03.jpg
2021-02-07 09:23:44,625 INFO: Matching 3 image pairs
2021-02-07 09:23:44,648 INFO: Computing pair matching with 5 processes
2021-02-07 09:23:44,674 DEBUG: Image 03.jpg matches: 0 out of 0
2021-02-07 09:23:45,130 DEBUG: Masking 0 / 4436 (0.00) features for 02.jpg
2021-02-07 09:23:45,134 DEBUG: Masking 0 / 4656 (0.00) features for 01.jpg
2021-02-07 09:23:45,548 DEBUG: Masking 0 / 4743 (0.00) features for 03.jpg
2021-02-07 09:23:47,974 DEBUG: Matching 01.jpg and 02.jpg.  Matcher: FLANN (symmetric) T-desc: 3.269 T-robust: 0.029 T-total: 3.298 Matches: 1288 Robust: 1251 Success: True
2021-02-07 09:23:48,350 DEBUG: Matching 02.jpg and 03.jpg.  Matcher: FLANN (symmetric) T-desc: 3.655 T-robust: 0.016 T-total: 3.672 Matches: 830 Robust: 761 Success: True
2021-02-07 09:23:48,350 DEBUG: Image 02.jpg matches: 1 out of 1
2021-02-07 09:23:48,570 DEBUG: Matching 01.jpg and 03.jpg.  Matcher: FLANN (symmetric) T-desc: 0.588 T-robust: 0.005 T-total: 0.593 Matches: 348 Robust: 321 Success: True
2021-02-07 09:23:48,570 DEBUG: Image 01.jpg matches: 2 out of 2
2021-02-07 09:23:48,573 INFO: Matched 3 pairs for 3 ref_images (perspective-perspective: 3) in 3.948230093003076 seconds (1.3160773143342037 seconds/pair).
2021-02-07 09:23:51,836 INFO: reading features
2021-02-07 09:23:51,958 DEBUG: Merging features onto tracks
2021-02-07 09:23:52,055 DEBUG: Good tracks: 1749
2021-02-07 09:23:55,573 INFO: Starting incremental reconstruction
2021-02-07 09:23:55,714 INFO: 0 partial reconstructions in total.
Traceback (most recent call last):
File "/home/user/Documentos/OpenSfM/bin/opensfm_main.py", line 15, in
commands.command_runner(commands.opensfm_commands, default_dataset_type)
File "/home/user/Documentos/OpenSfM/opensfm/commands/command_runner.py", line 31, in command_runner
command.run(data, args)
File "/home/user/Documentos/OpenSfM/opensfm/commands/command.py", line 12, in run
self.run_impl(data, args)
File "/home/user/Documentos/OpenSfM/opensfm/commands/compute_depthmaps.py", line 11, in run_impl
compute_depthmaps.run_dataset(dataset, args.subfolder, args.interactive)
File "/home/user/Documentos/OpenSfM/opensfm/actions/compute_depthmaps.py", line 19, in run_dataset
reconstructions = udataset.load_undistorted_reconstruction()
File "/home/user/Documentos/OpenSfM/opensfm/dataset.py", line 866, in load_undistorted_reconstruction
with io.open_rt(filename) as fin:
File "/home/user/Documentos/OpenSfM/opensfm/io.py", line 771, in open_rt
return open(path, "r", encoding="utf-8")
FileNotFoundError: [Errno 2] File or directory does not exist: '../data/berlin/undistorted/reconstruction.json'
It's the same with every data set I try, does anyone knows the reason of this error? thank you very much!
Actually I'm using Xubuntu 20.

Hi @luisEMEBA ,
The reconstruction.json is typically in the data/berlin folder and not in the undistorted subfolder.
The easiest way to run the whole pipeline is the following command:
bin/opensfm_run_all data/berlin

This runs all the feature detection, matching, reconstruction and a few more and puts everything into the berlin folder. After that, there should also be a reconstruction.json in the data/berlin/undistorted folder.

Good day to you, @fabianschenk. First than all I wish to thank you for checking my issue. Well, I run the pipeline through the line bin/opensfm_run_all data/berlin. Look, I've documented the process with some screenshots:

Figure1: This is the content of the test folder before I ran the command (I've erased some files but got the same error with or without them).

Figure 2: This is me in the main directory seconds before running the command.

Figure 3: This is the error I get after executing the command.

Figure 4: This is the data/berlin folder after running the command.
what have I tried? Well, till this moments I've tried to re-install the dependencies of the library many times, I also re-builded ceres solver and build from source OpenCV. I've tried too changing the directory to the principal partition, I've tried too build the OpenSfM pipeline as sudo, I'v tried too change the values of number of processes in config.yaml but still getting the same error. I've used the Open SfM pipeline many times before in Xubuntu 20, actually I'm trying it in Xubuntu 18. May I did some mistake in the installation process? Thank you very much for the atention.

Update: I've tried in Xubuntu 18 and got the same error I really don't know the reason of this error.

Hi @luisEMEBA ,
puh, this seems like a tough one.
Did you try to update the repo to the newest version, i.e. run a git pull, in the directory?
It's strange that it prints Starting incremental reconstruction, then does nothing and of course has 0 reconstructions then. This explains why the rest fails but I have no idea why it doesn't reconstruct. The matching and tracks are similar to what I get.
You can try going step-by-step through the incremental_reconstruction in reconstruction.py.

Hi there, @fabianschenk ! Thank you very much for your help, much apreciate it. I've repaired this error, more than repair I think is to find a way to solve this trouble, so if you or any other person is having the same issue I'll like to share the solution I found for Xubuntu 18.04, as is the same error in Xubuntu 20 I guess the way to fix it is too the same :) So let's go step by step.
This procedure consist into clone the latest version of OpenSfM at the time I'm writing this reply (0.5.1) and then download the source code from an older version (in my case I used 0.4.0 ) then go to the folder of this older version, copy and then paste all it's files into the latest OpenSfM version folder (the one you cloned) replacing the pre-existing files of the new version, then build the pipe line with python3 setup.py build and that's all, run bin/opensfm_run_all data/berlin and enjoy.

Figure 1: clone the latest version from the git repository.

Figure 2: Download an older version from https://github.com/mapillary/OpenSfM/releases

Figure 3: In my case I used the 0.4.0 version.

Figure 4: Copy all files in the older version's folder.

Figure 5: Paste and replace them in the latest version's folder.

Figure 6: Build Open SfM

Figure 7: Enjoy!


Data Exchange Field Mapping Bug

Hi, I've noticed a little Bug in Data Exchange Field Mapping.
If you go to a Data Exchange definition Card and use the action "Field Mapping".

If you mark the boolean "Use as Intermediate Table" and you try yo map a field with a large caption it throws an error (for example the field 93 from table 37).

It could be a mess with the table relation and the field length.
Thanks.

Thank you for reaching out. Your request could be reporting a product bug not related to the AL application extensibility. Please reach out to the relevant support channels:

For collaboration in preview or beta versions of AL Language in Visual Studio Code (aka ModernDev), use GitHub (https://github.com/microsoft/al).
For Microsoft partners, for versions already in general availability and supported, file requests through Microsoft Partnersource (https://mbs.microsoft.com/partnersource/).
For ISVs/VARs who have discovered a bug in a in preview or beta version, use MS Collaborate (https://docs.microsoft.com/en-us/collaborate/).
For Business Central customers who have issues or found bugs, please contact your Microsoft partner.
For the new features please submit a product suggestion on https://aka.ms/bcideas


IMATH_NOEXCEPT macro to make noexcept a compile-time option

Legacy code that depends on the IlmBase v2 ExcMath mechanism of signal
handlers that throw exceptions is incompatible with noexcept, since
floating-point overflow and underflow can occur in computations within
Imath functions now marked with noexcept. Legacy code that needs to
accomodate the exception-handling behavior can disable the noexcept
specifier by defining the IMATH_NOEXCEPT macro to be empty. Otherwise,
IMATH_NOEXEPT defaults to "noexcept".

FPU exceptions don't throw C++ exceptions, do they?
I'm not sure I understand how an overflow leads to a regular C++ exception.

I assume that this is was never turned on by default?

It's turned on via the mathExcOn(when) function, which I don't think is invoked by the OpenEXR library, but the Imath python bindings did previously enable them by default.

Added IMATH_USE_NOEXCEPT Cmake option, defaults to ON.


Explore cloud solutions for a 3D-CNN

[x] select suitable Cloud environment

start with Google Cloud -> does not allow to use GPUs in the free version
check out Azure

[x] create GCP account
[ ] create Azure account
[ ] set up notebooks
[ ] upload images to cloud
[x] how to collaborate in the cloud?
[x] Apply for free TPUs in Google Cloud

waiting for response to activate TPU in Cloud Project
TPU in Colab works only sometimes in free version
OASIS images are uploaded to Google Clopud bucket


Data in UserState not persisting between conversations

Issue
I have my bot set up on a website with multiple page, and whenever the user changes pages it starts a new conversation. Based on what I read here: https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-concept-state?view=azure-bot-service-4.0, the Users State should be able to be accessed across different conversations. So for example if a counter for the amount of pages the visited was stored in User State, it should go up every time they visit the page. However this is not the case, as the User State seems to be bound to the conversation, and will reset with each new conversation.
This is either bugged, or I am just doing something extremely wrong, as the User State is supposed to persist across multiple conversations.
Expected behavior
When the user switches pages, the User State and its data should persist to the next conversation.
Screenshots
How they are declared=============================

User State being saved=============================

Never mind, we just had a small error on our end


DEVIDE LINKED LIST IN RATIO OF  P:Q

Aim
We have been given a linked list and we need to devide it in the ratio of p:q
ie:
Given a linked list and two integers p and q, the task is to divide the linked list in the ratio p:q i.e. the first list contains first p nodes from the original list and the second list contains the rest of the q nodes. If the original list cannot be split in the given ratio then print -1.
For eg:
Input: 1 -> 3 -> 5 -> 6 -> 7 -> 2 -> NULL
p = 2, q = 4
Output:
1 3
5 6 7 2
Input: 1 -> 2 -> 4 -> 9 -> NULL
p = 3, q = 2
Output: -1
Details
Given a linked list devide it in the ratio  of p:q
Programming language

[ ] C
[x] C++
[ ] Java
[ ] Python

@aYUSH7614 BHAIYA PLEASE ASSIGN THIS TO ME


How to use UI Kitten Icons outside of a buttons accessory prop

💬 Question
I am trying to use  but the icon does not appear on the UI.
But when I add icon as accessoryLeft on a button it appears.
const ArrowLeft = (props) => <Icon {...props} name="arrow-left" />;

const HeaderBack = () => {
  const navigation = useNavigation();
  return (
    <Layout style={NativeHeaderStyles.HeaderLeftWrapper}>
      <Button
        status="ghost"
        accessoryLeft={ArrowLeft}
        style={{}}
        onPress={() => navigation.goBack()}
        appearance="ghost"
      ></Button>
    </Layout>
  );
};

How can I use icons without attaching it to a button?
UI Kitten and Eva version

Package
Version

@eva-design/eva
2.1.0

@ui-kitten/components
5.1.0

Hey :wave:
It works because of the props implicitely applied by spreading the props argument passed to it. Basically, there nothing special but the size of the icon and it's color.
<Icon name='arrow-left' style={{ width: 24, height: 24, tintColor: 'black' }} />


placeholder_id issue in admin page

In a new installation of django cms 2.2.post0 from git i have next error in /usr/local/lib/python2.7/dist-packages/django_cms-2.2.post0-py2.7.egg/cms/admin/pageadmin.py in add_plugin:
invalid literal for int() with base 10: 'undefined'
I change the js of admin and error was solved:
$ diff plugin_editor.js plugin_editor.js.old
8c8
      var placeholder_id = $(this).parent().parent().attr('id').split("-")[1];
---
      var placeholder_id = $(this).parent().parent().data('id');

via @Finalangel


Update functions_degs.R

Use "markers" not "degs" where applicable to be consistent

Not needed for now


Increase Jenkins job timeout from 100m to 120m

The e2e tests job is timeout quite frequently after 100m. The go test
timeout does not need to change: based on what I can see the tests
themselves take at most 20m to run.

/skip-all


Upgrade dependency to basic_utils ^2.6.3

I've version solving failed. I want to use the latest json_annotation. I use validators and it depends on vin_decoder that depends on old basic_utils (the latest basic_utils already upgrade the json_annotation)
This is the error example: https://github.com/dart-league/validators/issues/20
Thanks in advance :)
CC: @pmundt

@pmundt Please upgrade.
I cannot upgrade json_annotation, because vin_decoder 0.1.2 depends on basic_utils ^2.5.4


cmd/link: include declared constants in DWARF

Currently the DWARF information does not include constant symbols (such as const x = 1). This can make debugging more difficult, as constants available to the program are not available in the debugger. It also causes problems for debugging helpers. For example, our own runtime-gdb.py GDB helper hard-codes the G status constant values. This list gets out of date and means we can't cull now-unused G statuses.
We should include constants in the DWARF information. DWARF has a DW_TAG_constant for exactly this. The DW_AT_const_value attribute can be a string or any constant form, including DW_FORM_sdata, which is encoded as a LEB128, so in principle it can store arbitrary precision integers (though I don't know how well this is supported in practice). It's not clear to me how to represent floating point constants, though I suspect we'd have to truncate them to 64-bit precision to get them to actually work (despite DWARF's dizzying array of numeric type encodings; scaled packed decimal anyone?).
Doing this would be difficult right now because the compiler would have to emit constants in a way the linker could consume to produce DWARF information. It would probably be relatively straightforward if we switched to producing DWARF directly in the compiler.
/cc @ribrdb @derekparker

Doing this would be difficult right now because the compiler would have to emit constants in a way the linker could consume to produce DWARF information.

This is somewhat easier now that DWARF is being generated by the compiler. It would still require that we keep track of the constants, since currently they disappear pretty early in the process.
This came up again in a discussion of how heap analysis from a core dump could get access to runtime constants. At the moment they just have to be hard-coded.
/cc @khr @heschik


About the checkpoints of demo command

Are the checkpoints of the model that can be retrieved in bash fetch_demo_data.sh trained using all of New days, VISOR, and Ego4D of the HInt dataset?

The released model has not been trained on HInt.

Do you publish the models trained with HInt? Also, what are the demo models trained with?

We only release the demo model that we reference in this repo. Please check our config for the datasets used during training.

The released model has not been trained on HInt.

hi @geopavlakos @Kaszanas @ntvthuyen et al.
i tried to finetune the released ckpt but found performance on hint drops while performance on other eval datasets does not, which looks like a distribution shift. did u have any idea about the phenomenon? thx & rgds,


Allow generic camera streams to use authentication info

Description:
The Generic Camera component can have a stream_source url configured, but it currently does not take authentication into account. A way around it is to include the authentication info in the URL, but that means repeating information, and also that the url is best placed in secrets.yaml, which can be an unnecessary complication.
This PR adds the auth_in_stream parameter, which, if set to true, will add the authentication information from the username and password parameters, and embed them in the stream source url.
Pull request with documentation for home-assistant.io (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):
camera:
  - platform: generic
    name: Test
    still_image_url: http://<IP_ADDRESS>/jpg/image.jpg
    stream_source: "rtsp://<IP_ADDRESS>:554/"
    username: user
    password: super secret passw0rd
    auth_in_stream: true

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[x] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[x] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

@balloob since this is a core component, there's no code owner, and I'm not sure who should be reviewing. Once I get an approval, I'll also handle the documentation PR.

We should not merge this, this is making it overly complicated. Users should just put the url in secrets if it contains information that they want to keep secret.

In a large house, users potentially have dozens of cameras. I would argue that having to put all of those urls in secrets is overly complicated. My solution is inline with how authentication is being handled for still_image_url with http basic auth. Users could have just put auth info into that url as well.

@balloob any chance to get you to reconsider based on my reasoning above? I believe this is fully consistent with how auth works for the still_image_url.

Should we close here?

@MartinHjelmare if this isn't the right solution let's close, but I do think this needs addressing and we should keep https://github.com/home-assistant/home-assistant/issues/23064 open.

Ok, I'll close here.


Basic menu as toolbar

Feature Request
Basic menu that is similar to Menu as a Toolbar usage
Problem description
Whenever an item is clicked, it set it's state to active. Due to which I am unable to use them as normal buttons. If I set currentActiveIndex = -1 then I am unable to open submenu.
Proposed solution
A prop can be passed which makes them behave as a button and whenever it is clicked, it's active state is removed.

Toolbar component is now available for this use case.


Cannot "Add To Program" with DefLoader

Discussed in https://github.com/NationalSecurityAgency/ghidra/discussions/5673

Originally posted by RadWolfie August 17, 2023
I tried to look in known places in Ghidra to import .def file extension for Windows executable project. Yet, I came up with nothing other than "load pdb file" which isn't what I'm looking for. I have executable file has over 200 export names that need to be named from .def file in order to reverse engineer easier.
I do see DefLoader.java file do exist yet seems doesn't connect with Windows' executable interface/context menus or something? Or Ghidra doesn't have def file extension import feature support built-in yet?
Another thing I do see in "Supported Formats" info window is "Module Definition (DEF)" except only show as raw binary from import file option which I'm pretty sure is wrong way to do it.

Here's the sample files per requested by @ryanmkurtz. It contains .def, .exe, and .ord (from dumpbin tool) files. I am unable to remove underscore from the sample files. In our .def file does not have underscope in the API functions and does work fine.

Thank you.  What generated the .def file?  Its format seems to be not quite what we expect.

I'm not the founder of the project and we had it like that since beginning of the project many years ago. I don't recall anyone asked about what generated the def file. If I had to guess, it wasn't generated but added manually? Compilers doesn't care about whitespaces. I assumed the founder of the project preferred indent the columns for better visual. It's the similar way for dumpbin to have indent columns as well.
I'm using the actual reference of xboxkrnl.exe.def file to work with dumped xboxkrnl.exe file from xbox hardware console.

I've researched the DEF format a bit and the files you are working with seem like they are valid, so we should update our parser.  We also need to enable "Add To Program" support in the DefLoader...I'm not sure why that got turned off.

@RadWolfie if you are able to build Ghidra, you can test my changes here: https://github.com/ryanmkurtz/ghidra/tree/defloader

Steps took to test your defloader branch:

Launch Ghidra from bat script (after finish build it).
Create new project.
Add ghidra-exe-def-sample / xboxkrnl.exe file(s) to project.
Open either executable file within project tree.
From main menu, File > Add to Program > select .def file for the executable project to add into program.
Dialog shows Format: Module Definition (DEF) is listed as default (was Raw Binary). Click OK button.

No error was reported for either one. I can see given def's symbol names are now used for both of them. After selective ordinal number of functions and data pointers investigation. I can confirm they are a match with past reverse engineered documented.
Thanks! Can't wait to see it gets upstream. 🥳

Thanks for testing! I'll work on getting it upstream.


Added Power Operator using ** and nullish operator ??

Please add relevant test cases to support these features.

Please add relevant test cases to support these features.

Collaborator

I have added the test cases.

Please add relevant test cases to support these features.

Done

You haven't implemented this in parser, it won't even compile. Please take a look at parser first. Closing this.


Include squash commit merges in changelog

Fixes #5180.
It is not perfect, because we cannot extract the Github user name from squash commit merges. So we use the git author instead, and create a link with a Github user search. Better than nothing I'd say.

Or maybe a link like this: https://github.com/openlayers/ol3/commits/master?author=ahocevar
Thanks for fixing this!

Thanks for the review @tsauerwein. A link like you suggested won't work because I don't get the 'ahocevar' from the git log - only 'Andreas Hocevar' or the e-mail address.

The link like you suggested won't work...

Sorry, I totally missed the point with that one... :hushed:

No problem 😄 - is this good to merge @tsauerwein?

:+1:


Fixed small typos in README.md

Hi! Thank you for opening a great project.
During my installation, I realized some small typos so let me contribute.
Thank you.

"Currency". Must be autocomplete. 😄
Thanks, @imjn!


Adds RDO third party check job for whitebox neutron tempest plugin

This pr adds a 4 node (one controller, 2 compute , 1 extracted crc) multinode edpm zuul job to test whitebox neutron tempest plugin opendev patches.
It adds:

podified-multinode-edpm-deployment-crc-2comp parent job
centos-9-medium-2x-centos-9-crc-extracted-2-36-0-xxl nodeset
whitebox-neutron-tempest-plugin-podified-multinode-edpm-deployment-crc-2comp

This patch will run as a rdo third party job in rdo openstack-check pipeline[1].
Links:
[1]. https://review.rdoproject.org/r/c/config/+/53918
As a pull request owner and reviewers, I checked that:

[x] tested here: https://review.rdoproject.org/r/c/testproject/+/53773/38#message-bb481b85a56ee9e5a28f904e41ef62f2b1636744

Build succeeded.
https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b59f4057362e43acaaffb8dd2883955a

noop https://softwarefactory-project.io/zuul/t/rdoproject.org/build/12a8f3ad0d67496aae3d988bfc5d67f9 : SUCCESS in 0s
openstack-meta-content-provider https://softwarefactory-project.io/zuul/t/rdoproject.org/build/389e844486e74067b56c3704967173ab : SUCCESS in 3h 44m 01s
whitebox-neutron-tempest-plugin-podified-multinode-edpm-deployment-crc-2comp https://softwarefactory-project.io/zuul/t/rdoproject.org/build/6073cdb501424749898da85aca4fb718 : SUCCESS in 2h 48m 04s

recheck

recheck

/lgtm

/lgtm

/approve


fix: openzeppelin test execution timeout

change single-threaded execution tasks to  multi-threaded execution
failed tasks will be retried 3 times

manual build 4 times all pass

https://github.com/gpBlockchain/test-action/actions/runs/2826079648
https://github.com/gpBlockchain/test-action/actions/runs/2823330551
https://github.com/gpBlockchain/test-action/actions/runs/2818708539
https://github.com/gpBlockchain/test-action/actions/runs/2816973395


Car jittering back and forth when frozen

Just installed this plugin and I love it! Only issue I'm having is the car will jitter back and forth slightly when it's frozen. I tried turning bakkesmod variance off completely and I have variance for this plugin set to zero as well.
https://user-images.githubusercontent.com/2062388/121806735-754ae200-cc94-11eb-9d46-948a840dea31.mp4

This is just how the plugin works.  There is no way to actually "freeze" time as far as I'm aware, so what we do is rapidly load the same state over and over.  The game keeps running it forward and we keep setting it back.  Aside from some visual glitchiness, it should not cause any problems.
Variance is applied only when resuming the frozen state, to add randomness that forces you to adapt.  The settings for this are also completely separate from the other bakkesmod variance settings.


[FEATURE] Revisit Go imports

Is there an existing feature request for this?

[X] I have searched the existing feature requests

Is your feature request related to a problem or use-case? Please describe.
The imports we go in our Go sources are not consistent and not ideal for code reviews.
In particular, when reviewing a PR like this, it would be helpful if import changes come without alias changes.
Describe the solution that you would like.
In the community meeting, we concluded that we will use names like buildapi or pipelineapi going forward for API imports to make them independent of the imported version. Exceptions would still be allowed when we import more than one version (like we do in triggers for Tekton as we need v1 to work with Pipelines but also v1beta1 for Runs).
In addition, Matthias suggested to evaluate if a tool exists that verifies that imports are as expected, and can maybe autofix this.
Describe alternatives you have considered.
No response
Anything else?
No response

We can consider to use goimports if its features would be helpful for our ideas. Need to verify the feature set first.


Field in Error component

This component will contain an input with validation errors.

Enter something:
<input class="" type="text" id="field2" name="field2" aria-invalid="true"/>
A comment about the error

We should add links to W3C documentation :

aria-invalid
aria-required

@gericci do we need any specitial Aria attributes to make the link between the input and the error message ? aria-describedBy ? aria-live ?

Yes. I'd add aria-labeledby and aria-controls.

Personnaly, I will pref aria-describedBy instead of aria labelledBy.
I have just found this definition for the MDN website :
aria-labelledby is very similar to aria-describedby: a label describes the essence of an object, while a description provides more information that the user might need.

Here is the updated version

Enter something:
<input class="" type="text" id="field2" name="field2" aria-invalid="true"/>
A comment about the error

Why do you want to use the aria-controls attribute ?
For the aria-labelledby vs aria-describedby, I will prefer the description. What do you think ?


Adding orjson to all section in pyproject.toml

Adding orjson to be in line with the "Optional Dependencies" docs as it was missing.

Nice, thank you @michael0liver ! :rocket: :cake:


SSH experiences / braindump

So this issue is a bit of a mess but I needed a way to record some strange things I noticed while preparing materials for a xenon tutorial https://github.com/xenon-middleware/xenon-tutorial/tree/2b00c8d7625cf2966fdfe7c3e2aaeb6046fd7823. With the below items, we at least have concrete things to follow up on. From this collection, we should probably spawn new, singular issues.

ssh config options not used xenon-docker-image's Dockerfile because ssh(d) uses the first option it finds in a file even when there are repeated entries, and we echo options at the bottom. E.g. https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L10-L13. I'd prefer to copy a complete ssh_config file from the Docker context anyway, that way it's easier to see what the options are. Example setup here https://github.com/xenon-middleware/xenon-tutorial/tree/ed69661cdc7125ea15863fa2360fb8fd23309d10/containers

docker images for client side and server side, should be separate. e.g. slurm-ssh should be server side since it is a parent to all slurm images, but then the Dockerfile also has https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L10-L13

ssh client docker image should have different configurations, for example

with wrong permissions on directories, xenon should complain
with and without a ~/.ssh directory
with and without a ~/.ssh/config file
with various options in ~/.ssh/config, such as

hashKnownHosts
strictHostKeyChecking

with and without a known_hosts file
with a known_hosts file, with various types of entries in there

Xenon does not respect certain combinations of configuration parameters, specifically when inpuit argument knownHosts is false (see https://github.com/xenon-middleware/xenon-tutorial/pull/122), so by setting knownHosts to falseI can disobey input parameter stricHostCheck

my ssh understands strictHostKeyChecking=ask, xenon doesn't

in xenon, the config state is not assembled correctly from system config, user config, command line config

in xenon, you can't ask xenon what it thinks the current config state is, like you can with ssh -G xenon@localhost and sshd -T

in xenon, there is no way to control fallback behavior. ssh first tries public-private key, then interactive, then username-password (or something) this makes it more difficult to know that something is not working, and more difficult what is going wrong if you realize that something is going wrong

rethink options in SSHUtil https://github.com/xenon-middleware/xenon/blob/c24fba2c048c6d3266166d091c39aa1eac39aad8/src/main/java/nl/esciencecenter/xenon/adaptors/shared/ssh/SSHUtil.java#L187-L188

useAgentForwarding is not used and only in the way
stricHostCheck typo plus should be strictHostKeyChecking
stricHostCheck option is unused if I set useKnownHosts false
loadSSHConfig ambiguous which SSHConfig you're talking about: system, user, commandline, or the combination of these

xenon-docker-images have multiple key pairs, would be better to have one per image, then you know how you got inside a container, or if it doesn't work, you know why (see https://github.com/xenon-middleware/xenon-docker-images/tree/07db6c0f12621e51bd109a5325fe08b1c0e9123f/slurm-ssh/.ssh)

docker images client side ssh has authorized_keys like https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L27, not sure that's needed if it is only client

xenon docker images (slurm:17 image): not sure what the value of  PasswordAuthentication is):
from ``/etc/ssh/sshd_config``
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options change a
# default value.

and
#PasswordAuthentication no

seems to suggest password authentication is disabled by default. Stands to reason, since it
is not as secure as keypairs. However, if I do sshd -T, I see
passwordauthentication yes.  Not sure where that's coming from.

Regarding point 9. This is indeed the place where we need to figure out what to do and what not to do.
i. useAgentForwarding was supported in the past, but has not been implemented yet when we moved from JSch to SSHD. Some (advanced) ssh users use this to forward keys to other servers when using ssh to traverse multiple hops. Will be implemented again at some point. Lets ignore this for now.
ii. stricHostCheck will rename
iii. when not using known_hosts,  a strict host check does not make sense since there is nothing to check against. The command lines ssh always checks known_hosts, but ignores any problems it encounters when -o StrictHostKeyChecking=no. We could mimic that behavour if we wanted to (see below).
iv. loadSSHConfig tells SSHD to use the default action to read the config(s). This may actually do nothing, as we just repeat the default? Maybe we could drop this.
We can probably clean this up quite a bit.
If I comment out this code completely and don't tell SSHD to do anything special it seems to accept all connections and not add anything to known_hosts. This is the same behavour as setting useKnownHost=false.
It we set  useKnownHost=true and stricHostCheck=false, we get the same behaviour as -o StrictHostKeyChecking=no, which will not complain about new or different keys, but just add everything to known_hosts.
The alternative, useKnownHost=true and stricHostCheck=true is similar to -o StrictHostKeyChecking=yes. It will refuse to connect when a key has changed or an new server is
connected to. This is regarded as being annoying (it says so in the ssh documentation ;-) because it requires the user to properly maintain the known_host manually.
Unfortunately, setting it to ask is not really an option in a library like xenon, since we do not assume there is a user interactively doing stuff.
Additionally, the DefaultKnownHostsServerKeyVerifier of SSD which we use to implement this has a parameter boolean strict which tells it to be fussy about the access rights to the .ssh folder. Its currently set to true, which may explain the problems you had earlier.


Adding EOReader library

Adding EOReader library.
The documentation can be found here.
Introduction
EOReader is a remote-sensing opensource python library reading Sentinel data (and other optical and SAR sensors), loading and stacking bands, clouds, DEM and index in a sensor-agnostic way.
Sentinel data

Sensors
Product Types
Use archive

Sentinel-2
L1C & L2A
✅

Sentinel-2 Theia
L2A
✅

Sentinel-3 SLSTR
RBT
✅

Sentinel-3 OLCI
EFR
✅

Sentinel-1
SLC & GRD
✅

Other optical sensors
It aims to allow the user to load in a pythonic-way optical bands, index and clouds.
You can see how EOReader works in real by looking at this tutorial.

Sensors
Product Types
Use archive

Sentinel-2
L1C & L2A
✅

Sentinel-2 Theia
L2A
✅

Sentinel-3 SLSTR
RBT
✅

Sentinel-3 OLCI
EFR
✅

Landsat 8 OLCI
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 7 ETM
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 5 TM
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 4 TM
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 5 MSS
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 4 MSS
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 3 MSS
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 2 MSS
Level 1
Collection 1: ❌, Collection 2: ✅

Landsat 1 MSS
Level 1
Collection 1: ❌, Collection 2: ✅

PlanetScope
L3A & L3B
✅

Pleiades
SEN, PRJ, ORT & MOS
✅

SPOT 7
SEN, PRJ, ORT & MOS
✅

SPOT 6
SEN, PRJ, ORT & MOS
✅

GeoEye-1*
Standard & Ortho
✅

WorldView-2*
Standard & Ortho
✅

WorldView-3*
Standard & Ortho
✅

WorldView-4*
Standard & Ortho
✅

* Archived Landsat Collection-1 are not managed because of the tar.gz format, which is too slow to process. It is better to work on the extracted product.
* Other Maxar satellites (such as WorldView-1, QuickBird...) with the same file format should be supported.
Other SAR sensors
It aims to allow the user to load in a pythonic-way orthorectified SAR GRD data, by binding ESA SNAP gpt.
You can see how EOReader works in real by looking at this tutorial.

Sensor
Product Type
Handled

COSMO-Skymed 1st and 2nd Generation
SCS
✅

COSMO-SkyMed 1st Generation
DGM
✅

COSMO-SkyMed 2nd Generation
DGM
⚠

COSMO-SkyMed 1st and 2nd Generation
GEC, GTC
⚠

ICEYE
SLC
❌*

ICEYE
GRD
✅

ICEYE
ORTHO
💤

RADARSAT Constellation Mission
SLC
⚠

RADARSAT Constellation Mission
GRC, GCC, GCD
⚠

RADARSAT Constellation Mission
GRD
✅

RADARSAT-2
SLC
✅

RADARSAT-2
SGX, SCN, SCW,SCF, SCS, SSG, SPG
⚠

RADARSAT-2
SGF
✅

TerraSAR-X, TanDEM-X, PAZ SAR
SSC
✅

TerraSAR-X, TanDEM-X, PAZ SAR
MGD
✅

TerraSAR-X, TanDEM-X, PAZ SAR
GEC
⚠

TerraSAR-X, TanDEM-X, PAZ SAR
EEC
✅

*always given with a GRD image
✅: Tested
⚠: Never tested, use it at your own risk!
❌: Not handled
💤: Waiting for the release
Those sensors are planned in a near future:

SAOCOM-1

Context
This library has been created in the context of the Copernicus Emergency Management Service rapid mapping and risk and recovery.
In these activations, it is needed to deliver information (such as flood or fire delineations, landslides mapping, etc.) based on various sensors (optical and SAR). As every minute counts in production, it seemed crucial to harmonize the ground on which are built our production tools, in order to make them as sensor-agnostic as possible.

0.12.0 version just has been released.
It adds the support of 3 new sensors:

Pleiades-Neo
Vision-1
SAOCOM-1

The changes are here.
Feel free to make any feedback 😃

Thank you for the PR and please excuse the tardiness on the merge. ⌛

No problem! Thanks for the merge 😉


Disable geocoding and use the gem only for its distance calculations

Hi, I have this situation in which I already have all the coordinates I need so I don't need to geocode but I'd like to use the library just for its near method.
Is there a way I can do this?
Would be enough to just change the geocoder_init visibility to public so then I could just pass :geocode => false to its options?
Or maybe there is some other gem that is recommended just for mysql distance calculations?

I'd rather not change the accessibility of geocoder_init, and you still (might) need to specify which columns store coordinates. Would it be good enough to pass an empty block, like this?:
reverse_geocoded_by(:latitude, :longitude) { |result| false }


Need help improving tabline to allow custom ordering of tabs

Hi, I would like to modify the tabline extension to alloww custom ordering of tabs.
Specifically, I want to order tabs in most recently used order. I have created an extension that track that order vim-bufmru, and I would like to integrate it with the tabline.
I tried to hack around the code, and I modified the function airline#extensions#tabline#buflist#list to get the buffers ordered as I wanted them to. It sort of works. Calling ...buflist#invalidate() followed by ...buflist#get() returns the list of buffer numbers in the order I want.
But the tabline is not refreshed, and visually the order is not refreshed. Am I missing something ? Is the displayed buffer list not constructed from airline#extensions#tabline#buflist#list()
Thank you

I just needed a little fresh air, calling airline#extensions#tabline#buffers#invalidate refreshes the tabline.


Update source documentation and Wiki

Complete updating all source code documentation and Wiki

Source code and wiki https://github.com/symphonyoss/symphony-java-client/wiki updated to support 1.0.0 release.


[website][upgrade]feat: docs migration - 2.7.1 / sql

(If this PR fixes a github issue, please add Fixes #<xyz>.)
Fixes #
(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)
Master Issue: #
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Click to Preview
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[ ] doc-required
(If you need help on updating docs, create a doc issue)

[ ] no-need-doc
(Please explain why)

[x] doc
(If this PR contains doc changes)

Fix #11766


Clipboard corruption with clip.exe

ver
Microsoft Windows [Version 10.0.18363.592]
WSL distro
daniel@Daniel-Dell-XPS:~$ uname -a
Linux Daniel-Dell-XPS 4.4.0-18362-Microsoft #476-Microsoft Fri Nov 01 16:53:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

daniel@Daniel-Dell-XPS:~$ cat /etc/*release (trimmed)
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

Test case:
Attempting to copy string into clipboard using clip.exe.
For certain text (possibly depends on the length) clipboard seems to have corrupt data
Examples of the correct and corrupt clipboard (pasted content of the clipboard is on the second line, after the command)

This one fails
daniel@Daniel-Dell-XPS:~$ val="51b969e6-4a10-e700-5c4f-9783e0b5b0b0"; echo -n $val | clip.exe
daniel@Daniel-Dell-XPS:~$ ㄵ㥢㤶㙥㐭ㅡⴰ㝥〰㔭㑣ⵦ㜹㌸づ㕢ぢぢ

With "x" in front of it, it works OK
daniel@Daniel-Dell-XPS:~$ val="x51b969e6-4a10-e700-5c4f-9783e0b5b0b0"; echo -n $val | clip.exe
daniel@Daniel-Dell-XPS:~$ x51b969e6-4a10-e700-5c4f-9783e0b5b0b0

In command line window it looks like:

What's wrong / what should be happening instead:
Expecting text to be copied to clipboard regardless of the text length

Possibly related to:
https://github.com/microsoft/WSL/issues/3716
Update:
Added explanation of the second line on the command line (based on comment from @therealkenc)

Been using this tool for a bit and so far so good.
Install in Windows side:
dotnet tool install --global utf8clip
then in wsl side:
echo -n 51b969e6623423423 | utf8clip.exe | powershell.exe -noprofile -command Get-Clipboard

utf8clip works, and I thank @bluemarsh for providing it, but it's absolutely ridiculous that it's needed.
Just add a /UTF8 flag. Or respect locale. Or something. Anything but what you're doing. What encoding is even default? 🤦
https://twitter.com/HW_BEAT_THAT/status/1270200908417167363

If WSL is using this escape sequence "\x1b[?1h" probably is what is causing the clipboard corruption. I was using it before in the Linux to restore the terminal, but was causing the clipboard corruption. After I removed it the issue stopped. Only with WSL I still have this issue and perhaps is something like a escape sequence being causing all this.

Suddenly it start happening for me few weeks ago... I wonder if there is a working solution? I tried to install utf8clip but it is failing on Windows 11.

This also reproduces similarly with Set-Clipboard via powershell.exe, no matter which UTF-8 encoding parameters I set in PowerShell before invoking it - I don't think Set-Clipboard is just a wrapper around clip.exe, right?  (surely it's calling the .NET API for the clipboard or something?)
With WSLg, this is slightly better in that xclip inside WSL is able to read/write the clipboard successfully (xclip -selection clipboard -out, for example), but for some reason I haven't been able to figure out it unrelatedly hangs when you run xclip -selection clipboard -in over SSH (my use case involves setting clipboard contents remotely, and SSH directly to my WSL instance is the only way I've found so far to reliably do so without writing some custom bit of software).
I noticed with emoji, not any particular string (but was also able to reproduce with just 51b969e6 as noted above):
wsl-debian$ clip.exe <<<'🌸'; xclip -selection clipboard -out; echo
≡ƒî╕

What's interesting IMO is that running the same from Git Bash on my host yields a different result, so unless I've horribly misunderstood, it seems like it's WSL related somehow, but I can't figure out exactly how:
git-bash$ clip.exe <<<'🌸'; debian run xclip -selection clipboard -out; echo
🌸

More fun examples, to show it's clipboard-specific:
wsl-debian$ /mnt/c/Program\ Files/Git/usr/bin/cat.exe <<<'🌸'
🌸
wsl-debian$ powershell.exe '$Input' <<<'🌸'
🌸
wsl-debian$ powershell.exe '$Input | Set-Clipboard' <<<'🌸'; xclip -selection clipboard -out; echo
≡ƒî╕

I was able to circumvent this issue by appending a null character at the end.
# Bug: clipboard returns 'ㄵ㥢㤶㙥㐭ㅡⴰ㝥〰㔭㑣ⵦ㜹㌸づ㕢ぢぢ'
echo -n 51b969e6-4a10-e700-5c4f-9783e0b5b0b0 | clip.exe
# Fix: clipboard returns '51b969e6-4a10-e700-5c4f-9783e0b5b0b0'
printf "51b969e6-4a10-e700-5c4f-9783e0b5b0b0\x00" | clip.exe

https://github.com/microsoft/WSL/issues/11047 There was a solution here, setting  Beta: Use Unicode UTF-8 for global language support in Windows. Although xclip doesn't work for me at all ...


Fix MonthTuple's serialisation to conform to ISO 8601

#16 was merged without checking it was applied to MonthTuple. This needs to be rectified.

Fixed in #18


Which step would one take to integrate an existing Makefile with moulti?

Hello,
Thanks for creating this software!
Which step would one take to integrate an existing Makefile with moulti?
One possibility might be to use script scrip.txt make configure build to get the make output to go to a text fail, then tail that file to moulti.
However, how to differentiate Makefile steps ? Should I instrument the makefile to emit markers ?
Also, how to do all that in the same terminal instead of multiple ones?
Best,

Which step would one take to integrate an existing Makefile with moulti?
One possibility might be to use script script.txt make configure build to get the make output to go to a text file

Do you really need script for that? Regular output redirection should do, especially if you can convince your tools to keep outputting ANSI colors.

then tail that file to moulti.

Same: technically, you do not need an intermediate file (but feel free to generate one anyway if that makes sense in your workflow).

However, how to differentiate Makefile steps ? Should I instrument the makefile to emit markers ?

I see two main approaches:
Approach #1: editing the Makefile so it runs moulti step add whenever necessary; but then it becomes necessary to pipe all output to a moulti pass command until the next moulti step add. I'll have a look at the GNU make documentation but right now, that does not strike me as a simple and straightforward approach.
Approach #2: as you suggest, having markers in the output of make makes it easier to write a wrapper. Actually, one of my ideas for the future of Moulti is to provide a wrapper tool that reads a format like that:
===== divider1 =====
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
##### title1 #####
# moulti: classes=warning min_height=3 max_height=3 top_text="this format still" bottom_text="needs some work"
Curabitur ultricies dolor augue, id eleifend erat varius ut.
Etiam eget dapibus lectus, in gravida sapien. Aliquam nec diam mi.
Interdum et malesuada fames ac ante ipsum primis in faucibus.
##### title2 #####
# moulti: classes=success
Mauris in pellentesque lorem. Nam id libero at libero malesuada euismod non vel turpis.
Aenean dolor sem, suscipit at risus sed, tempus blandit nisl. Nullam ut tincidunt metus, non interdum urna.

... and turns each title into a Moulti step. That way, Moulti users could "simply" reformat the output of their existing scripts to integrate them with Moulti. This is essentially a less specialized variant of what moulti diff does already.

Also, how to do all that in the same terminal instead of multiple ones?

Although the quickstart makes you play with two terminals at first, the documentation also details moulti run in the Shell scripting section.

Do you really need script for that? Regular output redirection should do, especially if you can convince your tools to keep outputting ANSI colors.
Same: technically, you do not need an intermediate file (but feel free to generate one anyway if that makes sense in your workflow).

I would very much like to avoid doing so, I am happy to read that you think it can be avoided altogether.

I see two main approaches: Approach 1: editing the Makefile so it runs moulti step add whenever necessary; but then it becomes necessary to pipe all output to a moulti pass command until the next moulti step add. I'll have a look at the GNU make documentation but right now, that does not strike me as a simple and straightforward approach.

This is what I undertood I should do reading the Documentation.md, and it seemed unrealistic to create a whole new makefile system to accomodate for moulti.

Although the quickstart makes you play with two terminals at first, the documentation also details moulti run in the Shell scripting section.

See answer for Approach1.

Approach 2: as you suggest, having markers in the output of make makes it easier to write a wrapper. Actually, one of my ideas for the future of Moulti is to provide a wrapper tool that reads a format like that:
===== divider1 =====
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
##### title1 #####
# moulti: classes=warning min_height=3 max_height=3 top_text="this format still" bottom_text="needs some work"
Curabitur ultricies dolor augue, id eleifend erat varius ut.
Etiam eget dapibus lectus, in gravida sapien. Aliquam nec diam mi.
Interdum et malesuada fames ac ante ipsum primis in faucibus.
##### title2 #####
# moulti: classes=success
Mauris in pellentesque lorem. Nam id libero at libero malesuada euismod non vel turpis.
Aenean dolor sem, suscipit at risus sed, tempus blandit nisl. Nullam ut tincidunt metus, non interdum urna.

... and turns each title into a Moulti step. That way, Moulti users could "simply" reformat the output of their existing scripts to integrate them with Moulti. This is essentially a less specialized variant of what moulti diff does already.

That would be ideal! the moulti instructions could even be hidden behind a "erase current line" terminal command so that it is not visible when running on a system where moulti is not installed.
I had a great experience with the klogg app https://klogg.filimonov.dev/ where you can open any log, then search for things and it would display in a window above. You could also define regexes and styles to be applied (fg / bg color, bold etc).
I guess one direction moulti development could take is follow that approach where a user create some configuration file that corresponds to the way its existing scripts already work, and then wrap the launch of such script.
For example, launching make with -d flags yields some interesting strings to grep:
    Must remake target `build-Debug'.
ninja: Entering directory `output/MyApp/Debug'

[0/20] SOLINK libMyAppLib.dylib, POSTBUILDS
[1/20] SOLINK libMyAppLib.dylib, POSTBUILDS
    Successfully remade target file `build-Debug'.
   Finished prerequisites of target file `build'.
  Must remake target `build'.
  Successfully remade target file `build'.
 Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.

Here the steps are started by "Must remake target" and completed by "Successfully remade target", where "[X/Y]" is the progress of such target.
If moulti has some DSL to express that in the command line, or with some ad-hoc config file ,it would be easier to integrate with existing machinery.
Or I could create a filter that calls moulti appropriately when such strings appear, but I am not even sure how to do that.

Or I could create a filter that calls moulti appropriately when such strings appear, but I am not even sure how to do that.

Like this?
#!/usr/bin/env bash

# Instruct the Moulti instance to harvest any output not explicitly assigned to
# a step:
export MOULTI_RUN_OUTPUT=harvest
[ "${MOULTI_RUN}" ] || exec moulti run -- "$0" "$@"

function process_lines {
	local step_id=
	local step_counter=0
	while read -r line; do
		if [[ "${line}" =~ 'Must remake target' ]]; then
			# New target: create a new Moulti step:
			local title=$(sed 's/Must remake target `\(.*\)'\''./\1/' <<< "${line}")
			((++step_counter))
			local step_id="make_$$_${step_counter}"
			moulti step add "${step_id}" --title="${title}" --bottom-text=' ' < /dev/null
			# Redirect stdout to `moulti pass our_new_step`
			exec > >(moulti pass "${step_id}")
		fi
		[[ "${line}" =~ 'Successfully remade target' ]] && moulti step update "${step_id}" --classes=success
		[[ "${line}" =~ error ]] && moulti step update "${step_id}" --classes=error
		printf '%s\n' "${line}"
	done
}

# Instance title:
moulti set --title="$*"

# Run the command given on the command-line and process each line of its
# output:
"$@" 2>&1 | process_lines

Regarding the rest of your suggestions:  thank you, I will take all of this into account when designing the next features.
The thing with Moulti is: there will always be more that one way to do it. Right now, Moulti requires writing a controlling script like the one above, except for a couple of uses (like Ansible playbooks and unified diff) where it already provides wrappers out of the box. As time goes on, Moulti should provide more of such wrappers, including some matching your use cases.

It works!!! That’s so cool! I think that example that you just created can easily be adapted to other "filter" use cases, and should be put in the documentation somewhere close to the beginning. :)

It works!!! That’s so cool!

Glad you like it.

I think that example that you just created can easily be adapted to other "filter" use cases, and should be put in the documentation somewhere close to the beginning. :)

It will definitely serve as a basis for my next reflections. Specifically:

I can turn it into some kind of mini-framework for users who prefer imperative approaches.
it is a good tool to explore what a declarative approach should cover -- e.g. you can already notice a number of arbitrary decisions I took when writing this script: a good declarative format should provide enough flexibility to account for most, if not all use cases.

Glad you like it.

Yes! And it was easy now with this template to complete with some progress bar \o/
see grep and sed lines:
#!/usr/bin/env bash

# Instruct the Moulti instance to harvest any output not explicitly assigned to
# a step:
export MOULTI_RUN_OUTPUT=harvest
export MOULTI_INSTANCE="make"
[ "${MOULTI_RUN}" ] || exec moulti run -- "$0" "$@"

function process_lines {
	local step_counter=0
	while read -r line; do
		if [[ "${line}" =~ 'Must remake target' ]]; then
			# New target: create a new Moulti step:
			local title=$(sed 's/Must remake target `\(.*\)'\''./\1/' <<<"${line}")
			((++step_counter))
			local step_id="make_$$_${step_counter}"
			moulti step add "${step_id}" --title="${title}" --bottom-text=' ' </dev/null
			# Redirect stdout to `moulti pass our_new_step`
			exec > >(moulti pass "${step_id}")
		fi
		if echo ${line} | grep -q '^\[[0-9]*\/[0-9]*\]'; then
			local progress_status=$(echo ${line} | sed -e 's/^\[\([0-9]*\)\/\([0-9]*\).*/\1/')
			local progress_target=$(echo ${line} | sed -e 's/^\[\([0-9]*\)\/\([0-9]*\).*/\2/')
			moulti set --progress-bar --progress="${progress_status}" --progress-target="${progress_target}"
		fi
		[[ "${line}" =~ 'Successfully remade target' ]] && moulti step update "${step_id}" --classes=success && moulti set --no-progress-bar
		[[ "${line}" =~ error ]] && moulti step update "${step_id}" --classes=error
		printf '%s\n' "${line}"
	done
}

# Instance title:
moulti set --title="$*"

# Run the command given on the command-line and process each line of its
# output:
"$@" 2>&1 | process_lines

klogg has a follow mode where the window scrolls to the latest line. that mode is toggled by pressing f.
could moulti have that too ? (should I open new issues?)

It will definitely serve as a basis for my next reflections. Specifically:
* I can turn it into some kind of mini-framework for users who prefer _imperative_ approaches.

That would be my case I guess.

* it is a good tool to explore what a _declarative_ approach should cover -- e.g. you can already notice a number of arbitrary decisions I took when writing this script: a good declarative format should provide enough flexibility to account for most, if not all use cases.

When I see how concise this moultimake.bash is, I wonder if it is necessary at all. Do you have any idea how a declarative file would look like?

Right now, when some steps have happened, I cannot know what happens or happened until I scroll with the mouse over moulti.
klogg has a follow mode where the window scrolls to the latest line. that mode is toggled by pressing f.
While moulti already scrolls inside steps, could moulti also have something similar to scroll finished/inactive steps ? (should I open new issues?)

Moulti has programmatic scrolling too. Refer to the documentation for details but TL;DR:

add --scroll-on-activity=-1 to the moulti step add command
hit 'l' to suspend programmatic scrolling (the main scrollbar turns green)
hit 'l' again to restore it (the main scrollbar turns grey again)

When I see how concise this moultimake.bash is, I wonder if it is necessary at all.

That will be part of my reflections too.

Do you have any idea how a declarative file would look like?

The default approach would be a serializable Python dict written as JSON/YAML/TOML that describes:

named patterns (e.g. section_title: '^#{10} (?<title>.+) #{10}$')
pattern groups (typically optional, purely for the sake of convenience)
actions (e.g. new step with these parameters, new divider with those options, set instance title or progress bar status)
action groups (typically optional, purely for the sake of convenience)
pattern/action associations (e.g. "upon reading a pattern from this pattern group, run those action groups")
what time to trigger the coffee machine

support for using regex captures and environment variables in actions.

I could implement that by myself OR I could look for a tool/library/framework that already provides this kind of logic.

Moulti has programmatic scrolling too. Refer to the documentation for details but TL;DR:
* add --scroll-on-activity=-1 to the moulti step add command

That’s very neat! I wonder how I missed that when searching in the documentation. Thanks!

* hit 'l' to suspend programmatic scrolling (the main scrollbar turns green)
* hit 'l' again to restore it (the main scrollbar turns grey again)

I tried those, but I have troubles understanding if scroll stops, or if scripts takes time. I will use --scroll-on-activity for now anyway so I will figure this later.

The default approach would be a serializable Python dict written as JSON/YAML/TOML that describes:
* named patterns (e.g. `section_title: '^#{10} (?<title>.+) #{10}$'`)
* pattern groups (typically optional, purely for the sake of convenience)
* actions (e.g. new step with these parameters, new divider with those options, set instance title or progress bar status)
* action groups (typically optional, purely for the sake of convenience)
* pattern/action associations (e.g. "upon reading a pattern from this pattern group, run those action groups")
* what time to trigger the coffee machine

support for using regex captures and environment variables in actions.

I could implement that by myself OR I could look for a tool/library/framework that already provides this kind of logic.

Reading all of this, I wonder if a python API might be easier to implement (and more powerful) than JSON/YAML/TMOL.
This api would read like declarative, but really just call python functions, a bit like the cmake syntax.
import moulti as m

m.detector.add_new_step_rule("Must remake target `\(.*\)'\'", auto_increment_id=True, show_progress_bar=False)
m.detector.add_step_success_rule(".*Successfully remade target.*")
m.detector.add_step_error_rule(".*error.*")
m.detector.add_step_status_progress_rule("^\[\([0-9]*\)\/[0-9]*.*", force_show_progress_bar=True)
m.detector.add_step_total_progress_rule("^\[[0-9]*\/\([0-9]*\).*", force_show_progress_bar=True)

m.add_action(["make", "-d", "configure"], can_be_restarted=True, on_success="next_command")
m.add_action(["make", "-d", "build"], can_be_restarted=True, on_success="next_command")
m.add_action(["make", "-d", "test"], can_be_restarted=True)

m.run(run_output="harvest", instance="make", start_instance_if_needed=True, follow=True)

Later part can easily be made customizable using vanilla python:
for i in range(1,len(argv)):
  m.add_restartable_command("make", "-d", argv[i))
m.run(run_output="harvest", instance="make", start_instance_if_needed=True, follow=True)

I tried those, but I have troubles understanding if scroll stops, or if scripts takes time. I will use --scroll-on-activity for now anyway so I will figure this later.

They are complementary features: --scroll-on-activity provides the automatic scrolling you expect. But sometimes, you may want to read a part of the output while your job is still running. But how can you read step number 5 while Moulti insists on scrolling to display step number 78? The "lock scroll" action enables you to scroll by yourself without being disturbed by the programmatic scrolling.

Reading all of this, I wonder if a python API might be easier to implement (and more powerful) than JSON/YAML/TMOL.

It is planned to implement a "Python User API", i.e. a Python API exposed to users and that would adhere to semver. It will definitely be more powerful (e.g. it should integrate a class that keeps track of all created steps, and it should not need as many fork-exec-connect-send-receive-close-exit operations as a shell script) but this will likely wait until things get more stable (because a stable feature set makes it easier to design a perennial API).
My reflections so far:

re: the idea of an output format that Moulti would recognize out of the box: it still makes sense but ideally, this idea should be implemented using one of the ideas below, possibly serving as example/demo.
re: the idea of a framework for imperative-style parsing/translation: one issue is that there are too many languages and I am reluctant to introduce a new Domain-Specific Language. Considering how it quickly and concisely solved your own issue, I am tempted to make bash a priority: adding a few functions to moulti-functions.bash, demonstrating their use in the documentation and ending with a solution that is even more concise.
re: the idea of a declarative-style tool: the idea remains relevant precisely because it is independent of any programming language. It still needs exploration, but I already pinpointed a use case that requires a Finite-State Machine and thus brings complexity: parsing such titles:

==========================================================
This is a title only because it is surrounded with = signs
==========================================================
This line should go inside a Moulti step.
This line should go inside a Moulti step.
This line should go inside a Moulti step.
================================================================
This is another title only because it is surrounded with = signs
================================================================
This line should go inside a Moulti step.
This line should go inside a Moulti step.
This line should go inside a Moulti step.

By the way, real life use cases are most welcome to feed those reflections. Do you have any other use case in mind?