Cancel goal when no available fd is found

If the objective is in error, and the Metacontroller does not find any valid mode to solve it, the goal should be cancelled.
@chcorbato is this the correct way to go? should it be an fd_error to go when all other "modes" are not avaible

If the objective is in error, and the Metacontroller does not find any valid mode to solve it, the goal should be cancelled.

I agree, the objective for the mros_reasoner should be cancelled. I think this would require:

[ ] remove the objective instance from those the reasoner is trying to maintain/pursue. Maybe this requires to extend in the ontology the status of an objective. What do you think @estherag @rsanz ?
[ ] report back, to the client of the mros_reasoner metacontrol services, that the objective cannot be solved by mros_reasoner. Something for the interface to be discussed with URJC

@chcorbato is this the correct way to go? should it be an fd_error to go when all other "modes" are not available

This is an interesting idea. To define a safe mode/configuration/architecture when the system is in error. But note that the fact that an objective is in error, i.e. cannot be achieved does not mean that the system is in error. Maybe it is simply out of the scope of the available FDs.
I think the current definition of the status property of an Objective in the ontology is not expressive enough for all these cases.

Maybe we can have more status for the objective: IN_PROGRESS, IN_ERROR_NFR, IN_ERROR_COMPONENT, UNREACHABLE.
IN_ERROR_NFR may be set when the current FG doesn't meet objectives NFRs so a new FG should be selected
IN_ERROR_COMPONENT would be set when the current FG uses an unavailable component so a new FG should be selected
When there isn't a FD reachable, we could set the objective at UNREACHABLE, then we can delete this objective and switch to another one (i.e. switch from navigating to go to recharge). In this case, maybe we can miss NFR because is an emergency action and navigate to the deck with fewer restrictions so FD with IN_ERROR_NFR property on the log can be selected.

I like your proposal @estherag
In the case of UNREACHABLE, I think the metacontroller should notify the client of its action server to achieve that objective, typically the task/mission coordination logic, who would decide whether to cancel that objective (thus the reasoner would eliminate it from the ontology) and which new objective to request.
We have discussed with URJC and agreed that the logic and decision to instantiate a new objective to navigate to the recharge station should not me in the reasoner/metacontroller, but in the application/task/mission logic.
The reason is to make the metacontroller as independent from the application (e.g. navigation) and reusable as possible.
I hope you also agree @estherag @rsanz

I agree, having more status makes sense.

In the case of UNREACHABLE, I think the metacontroller should notify the client of its action server to achieve that objective, typically the task/mission coordination logic, who would decide whether to cancel that objective (thus the reasoner would eliminate it from the ontology) and which new objective to request.

Still, shoudl the ontology define a  FG to choose when the objective is UNREACHABLE? or should it just report that it cannot be achieved and let the processing for the metacontroller/task planner?

I also agree.
I think from the ontology side both approach are possible:
I understand this UNREACHABLE status is set by the metacontroller when there is not FD available. In that case, the mission logic should set a new objective.  I think we could go in this case with the objective and FG in error until a new objective and FG are set.
However, if it is easier for the metacontroller to have a processing_unreachable_objective, we could have a FD in unavailable status that will become available when the objective is UNREACHABLE. Once this FD is available, we can make its corresponding FG.

I have moved this question of having a fall-back or default FD/configuration/mode to a new issue #118
I close this one, since certainly the objective being UNREACHABLE should be reported by the reasoner to the wrapper and then to the mission logic.


Mutation testing

Mutation testing is used to design new tests and evaluate the quality of an existing test suite. In other words, it can tell you about the quality of your tests, where coverage only tells you about how much code you have not yet written a test for.
It works by changing the behavior of a program by changing parts of the code ("mutating it"), running the test suite with this new mutation and checking if the test suite passed. If the test suite passed, the mutation is considered to have survived - if not, it is considered to have died.
If dead / total is below some threshold, then there is room for improvement in the test suite. In other words, the more dead mutations you have, the better.
I think this would be a pretty cool tool to have in Foundry - and I think it is somewhat trivial to implement as well (simplified strategy):

Have a list of mutations we can do
Parse the source contracts
Insert mutation somewhere
Run tests
Repeat until we're satisfied

Here's a paper on mutation testing for Solidity that I found interesting: https://arxiv.org/pdf/2105.03626.pdf

More notes:

You can identify weakly tested pieces of code: mutations are not killed for these snippets
You can identify weak tests: they never kill mutants

Related DappTools PRs/Issues:

https://github.com/dapphub/dapptools/pull/829
https://github.com/dapphub/dapptools/pull/687

potential inspiration: https://github.com/JoranHonig/vertigo
the paper<EMAIL_ADDRESS>has some good tips in it like knowing what test hits a particular mutation
we could do this by:

running all tests, build up a mapping of op -> Vec
perform mutation
run all tests for a mutated op

Vertigo works on solidity, but we may want to do part solidity, part raw opcodes so we can be faster and not have to recompile for all mutation types

alternatively we could sacrifice some disk space by copying the source of the project into cache/tmp/mutation-{1,2,3,n}, performing the mutation and in parallel spawn forge test --src ./cache/tmp/mutation-{1,2,3,n} (you get the idea).
if the command exits with status 0 the tests passed, otherwise they failed. afterwards we clean up by deleting those directories (or keep them around for diffing). if our cache works on hashes we theoretically(?) would only recompile the mutated contract anyway

Yoyoyo, just seeing this thread now.
Not sure if this is possible already but if foundry could include the solc json ast in the compilation artifacts then it'll be relatively straightforward to create a Vertigo adapter for foundry.

Vertigo works on solidity, but we may want to do part solidity, part raw opcodes so we can be faster and not have to recompile for all mutation types

There are some papers that evaluate this approach. From the top of my head there are two benefits to doing source code mutation:

Trivial Compiler Equivalence (basically free equivalent mutant detection)
Easier manual inspection of mutants (specifically equivalent mutations)

alternatively we could sacrifice some disk space by copying the source of the project into cache/tmp/mutation-{1,2,3,n}, performing the mutation and in parallel spawn forge test --src ./cache/tmp/mutation-{1,2,3,n} (you get the idea).
if the command exits with status 0 the tests passed, otherwise they failed. afterwards we clean up by deleting those directories (or keep them around for diffing). if our cache works on hashes we theoretically(?) would only recompile the mutated contract anyway

Hey all,
We have been working on a mutation generator called Gambit. It is implemented in Rust so it might be possible to integrate it with Forge easily.
Gambit is a mutation generator only and it works by traversing the Solidity AST to identify where to perform the mutations, then performs the mutations directly at the source level, and dumps them in a designated directory.
We are actively working on adding more functionality and better mutation generation when possible so let us know if you have any suggestions. PRs are also welcome :)

The AST modification logic makes sense - cc @mattsse we could add that a mutate fn to the Project struct in ethers solc? Which would traverse all the tests and perform a random mutation? And we would run the the tests against the mutation inside of forge mutate?

There is also Universal Simulator, one of the engineers from Trail of Bits maintains it -
here is some usage:
https://github.com/sambacha/universalmutator/tree/new-solidity-rules

There is also Universal Simulator, one of the engineers from Trail of Bits maintains it -
here is some usage: https://github.com/sambacha/universalmutator/tree/new-solidity-rules

Indeed, it's very cool! The difference between Gambit and the universalmutator is that they are regex based whereas Gambit is guided by the AST. And on top of that, Gambit also runs the compiler to filter out any invalid mutants. On the other hand, universalmutator supports many more languages since their technqiue relies on regexes and they don't need any additional AST information.

Would be supportive of a draft PR or some kind of RFC proposing exactly how we'd integrate the two together, if you'd want to give it a shot @chandrakananandi!

Hey @chandrakananandi, congrats on the announcement today!
Figured this might be a good time to revisit what the UX for what an MVP forge integration might look like. Previously we came up something like this:

forge test --mutate first runs all regular tests and makes sure they pass

forge test --mutate skip-regular could skip regular tests for quicker iteration during development

If tests all pass, forge passes the remappings / config info to gambit, runs gambit against all src/ contracts, and saves mutation to a temp dir
For each mutation file in the tmp dir:

Apply the change then run all tests

Maybe run tests with a hardcoded small amount of fuzz runs for now to prevent runtime from blowing up, and skip invariant tests? In general mutating solidity and having to recompile feels like it might be slow anyway, perhaps better suited as a CI-only type of integration

We expect at least one test to fail for each applied mutation, so if a test failure happens the run is considered successful, if all tests still pass it's marked failed

Question here: What's the best way to handle benign mutations that are equivalent to the original?

For each failed run, print the mutation to the console, then delete the tmp dir
Like usual, exit with an error if tests fail (i.e. mutations were not detected by test suite)
None of gambit's config options are exposed to user, for simplicity

Follow up versions could:

Respect the [fuzz] seed like invariant tests do
Generate/link to the certora UI for visualizing results
Integrate better failure persistence (https://github.com/foundry-rs/foundry/issues/2551 and https://github.com/foundry-rs/foundry/issues/2552)
Expose gambit config options to user, perhaps via a [mutation] section the foundry.toml file so you can have different profile configs, like you can with [fuzz] and [invariant] sections

Hey @chandrakananandi, congrats on the announcement today!
Figured this might be a good time to revisit what the UX for what an MVP forge integration might look like. Previously we came up something like this:
* `forge test --mutate` first runs all regular tests and makes sure they pass

  * `forge test --mutate skip-regular` could skip regular tests for quicker iteration during development

* If tests all pass, forge passes the remappings / config info to gambit, runs gambit against all `src/` contracts, and saves mutation to a temp dir

* For each mutation file in the tmp dir:

  * Apply the change then run all tests

    * Maybe run tests with a hardcoded small amount of fuzz runs for now to prevent runtime from blowing up, and skip invariant tests? In general mutating solidity and having to recompile feels like it might be slow anyway, perhaps better suited as a CI-only type of integration
  * We expect at least one test to fail for each applied mutation, so if a test failure happens the run is considered successful, if all tests still pass it's marked failed

    * Question here: What's the best way to handle benign mutations that are equivalent to the original?

* For each failed run, print the mutation to the console, then delete the tmp dir

* Like usual, exit with an error if tests fail (i.e. mutations were not detected by test suite)

* None of gambit's config options are exposed to user, for simplicity

Follow up versions could:
* Respect the `[fuzz]` seed like invariant tests do

* Generate/link to the certora UI for visualizing results

* Integrate better failure persistence ([feat: fuzz failure persistence #2551](https://github.com/foundry-rs/foundry/issues/2551) and [feat: fuzz corpus saving and replay #2552](https://github.com/foundry-rs/foundry/issues/2552))

* Expose gambit config options to user, perhaps via a `[mutation]` section the foundry.toml file so you can have different profile configs, like you can with `[fuzz]` and `[invariant]` sections

Hi Matt @mds1!
Thanks for writing this out. The flow makes sense! One quick note: at the moment, Gambit actually already compiles the mutants to ensure that they are all valid. So the tmp directory you mentioned will only have mutants that compile.  It is definitely slower but since we have been using the mutants in the context of verification so far, it's not a bottleneck yet. But in any case, we do have ongoing work right now as part of which are trying to use type information from the AST to generate mutants that are more likely valid.
As for the benign / equivalent mutant question: this is a very interesting problem and full on equivalence checking for mutants is yet another undecidable verification task. So our current thinking is that we should avoid generating benign mutants in the first place. For now, Gambit allows the user to specify what functions and contracts to mutate which is an easy way to avoid some useless mutants because hopefully the user has knowledge about the codebase we can leverage. We are actively working on other light weight strategies as well. There is also TCE (Trivial Compiler Equivalence) which people have used for this. The Certora Prover itself can be used here too but like I said, that's yet another verification task on its own.
For now, I am most interested in gathering data (lots of mutants, the result of running tests on them, some potentially manual analysis of mutants to understand how often we get equivalent mutants) and guiding our strategy for equivalence detection based on that. For the scale of data we will need for this, I think integrating with a testing framework like foundry would be really great!
Is there any dataset of solidity contracts together with foundry tests that you could point me to?

at the moment, Gambit actually already compiles the mutants to ensure that they are all valid. So the tmp directory you mentioned will only have mutants that compile.

Ah interesting, wonder if there should be an option to disable that, since otherwise we'll be compiling twice (once for gambit to verify, then the forge compilation)? Then instead forge would need to discard a mutation that doesn't compile

Is there any dataset of solidity contracts together with foundry tests that you could point me to?

Are you just looking for repos that use foundry for building/testing? If so, the forge integration tests run against some external repos which could be a good starting point (some of those repos use dapptools, which foundry should be backwards compatible with).  There's also a bunch in the awesome-foundry repo. But let me know if I misunderstood! 😅

Ah interesting, wonder if there should be an option to disable that, since otherwise we'll be compiling twice (once for gambit to verify, then the forge compilation)? Then instead forge would need to discard a mutation that doesn't compile

Yes this is definitely possible and in fact something we might do in an upcoming PR.

Are you just looking for repos that use foundry for building/testing? If so, the forge integration tests run against some external repos which could be a good starting point (some of those repos use dapptools, which foundry should be backwards compatible with).  There's also a bunch in the awesome-foundry repo. But let me know if I misunderstood! 😅

This is wonderful, and exactly what I was looking for. I wanted to gather some benchmarks so that we can see how Gambit helps them.  Thanks very much!

Hey @mds1, thought I'd jump in! I'm currently hacking away on a giant refactor of Gambit, and I'm hoping to land a PR some time next week/the following monday.
WRT equivalent mutants: these are just hard! There are a few ways to handle them:

don't generate equivalent mutants begin with! This can be done a few ways:

mutation operator selection: For instance, don't mutate a to a++ because if a is going out of scope the infected state cause by the post increment will never propagate to a test-visible location
suppression rules: there are common patterns of equivalent mutants (e.g., int i = 0; if (cond) { i = 1; } else {i = 2;} where mutating the initialization value 0 will result in an unreadable write). These patterns depend on the language, coding styles within the language, and a number of other factors. I'm pretty new to solidity (I've mainly been researching in Java land), so I don't know what the best suppression rules are just yet!

detect equivalent mutants with an optimizing compiler: trivial compiler equivalence is a technique that just runs optimizing compilers on all the mutants and checks them for equality with the original program. Optimizing compilers tend to normalize or canonicalize programs, so that syntactically distinct programs with the same semantics often end up being written as the same binary. Depending on workload/if you are already compiling this can be feasible, but it blows up at scale.

There is also the problem of mutant redundancy. For instance, consider the boolean condition c and the following three mutants:

c
!c
true
false

true
false (X)
true (X)
false

false
true (X)
true (X)
false

I've marked all cells where the mutant differs from the original value with an (X). Note that !c is always different from c: this means that any test that can kill the 2nd mutant c -> true will also kill the first mutant c -> !c; however, the reverse is not true. This makes the second mutant strictly harder to kill than the first mutant, we say that true dominates !c, or that !c is redundant to the original program c.
Redundancy is bad: the more redundant the mutant is (that is, the easier it is to kill), the less utility it has for things like mutation score (killed/total) or as a testing goal for a developer.
As a trivial example, if a mutant always causes a crash it is very easy to detect: all tests that execute this mutant kill it! This is called a trivial mutant.
So avoiding equivalent mutants is really important, but trying to minimize redundancies is also critical.
All this to say, I'm hoping to improve Gambit's mutation routine to avoid many common patterns of equivalent mutants and to try to start reasoning about common mutant redundancies.
This brings me to Foundry! If we are able to integrate Gambit with Foundry we can use test data to compute the kill matrix (pretty badass, no?). This is just a binary matrix of "does test T kill mutant M?". We can use this data to approximate redundancy information and help to detect equivalent mutants. This is of course very expensive to compute, so I'd be using this to inspect manually to improve Gambit for future uses.
Anyway, that's my brain dump...excited to chat more once I land this PR!

Hey all, poking around a bit in forge test related code to see how easy it would be to add a --mutate arg. Here is a quick overview of my understanding of how this should work, as well as a few questions that maybe I can get some guidance on.

Run Tests On Project: Ensure we have a passing test suite before mutation; abort otherwise.

Mutate Code: Apply mutation operators, resulting a mutated file set.

Question:  In my opinion mutation should be configurable (e.g., which files/contracts/functions to mutate, which mutation operators to apply). Using forge test --mutate makes this a little weird because the test subcommand now needs conf arguments that only apply in the presence of the --mutate option (e.g., --mutation-operators ...). Is it better to create a new forge mutation-test subcommand to encapsulate this functionality?

Run Tests On Mutants: The mutants are AST mutants, and are written to disk as source files. We will need to swap the mutated source in for the project source for each mutant, run testing infra, and then restore the original file for the next mutation run. This means we probably just want to copy the entire project structure to a temp directory and clean it up afterwards (this seems like the approach folks are suggesting above). And as is suggested, we could even make a few copies of the project to run in parallel:

Question: Is it sufficient to recursively copy the directory tree rooted at project.paths.root to copy a project and point a new Project struct at it?

Report Mutation Run: There are various statistics we can report. The basic one is mutation score (detected mutants / generated mutants). The detection matrix is also useful for research purposes, and is just a T x M boolean matrix representation of the predicate test t detected mutant m

Test Coverage Data
Running tests is expensive, so avoiding test runs that don't ever touch mutated code is desirable.  To do this I'd want coverage information.

Is test coverage data collected during execute_tests()?
If not, how hard would this be to gather?
Can we get per test coverage information?

I'm not sure if it's worth using test coverage in the initial PR, maybe I'll just get a proof of concept going. But I think incorporating coverage will be worth it in the medium to long term.

Hey @BenTheKush! Here's some answers to your questions:
2: I think we might already have some conditional args like that and I believe clap supports that, so it should be ok. Deferring to @Evalir @mattsse to confirm. Note that we'll also need to support those options via the CLI and via the config file, I don't think they currently sync automatically (i.e. options need to be added in two places)
3: One downside here is that copying the whole directory might get costly (i.e. slow) because some deps can be quite large with a lot of unrelated files (e.g. when installing chainlink as a dep for their interfaces, you get tons of non-solidity code from the rest of the system). Since I don't think we need to generate mutants for deps, we might want to try only copying the src dir (i.e. set of files that may be mutated) and symlink to the rest. Not sure offhand if this will cause issues with remappings though, another @mattsse or @Evalir question. But otherwise yea I think that should be sufficient
Test coverage: Coverage data is only generated when running forge coverage, not when running forge test. Note that it also has some limitations right now (notably, requires no optimizer and no via-ir) and some bugs (can search the repo issues to learn more here), so ideally for now we'd leave mutant improvements due to coverage as an optional feature

Thanks @mds1, yeah this all makes sense. Having conditional args would be perfect. And I agree, copying an entire directory over for mutation is not ideal. I might start there for a proof of concept/MVP and then see if there is something clever we can do to not have to copy everything. There's probably some clever manipulation of the Project struct I can do to get away with some stuff?
I'm not super familiar with the Solidity ecosystem so not sure what the stumbling blocks are...but I guess this is a great way to learn :)
WRT Coverage: happy to make it conditional! And again, I probably won't be implementing any optimizations in the first pass.
I'm going to do some requirements engineering this week and try to come up with a design. I think a lot of the changes will be happening on the Gambit side anyway, and I'll set up an API that forge test --mutate can call into.

Hi there! I think mutation testing would be very useful to enhance testing suites. Did someone make progress on this topic since May?

@MerlinEgalite we had started investigating gambit + foundry but had other things come up. We have plans to resume this in early 2024. Will keep you posted.

Hi @chandrakananandi, I'd like to work on integrating Gambit and Foundry

@everyone This is the doc about integrating Foundry and Gambit. Please feel free to drop your comments
https://hackmd.io/@tICezjHsSaiehIn9jbcUAA/rJXZb_JS6
@chandrakananandi Does Certora have any plans on open-sourcing the UI you demoed at your talk at SoliditySummit?

@chandrakananandi I just realized you dropped some comments on the design doc. I have responded.
Are you ok with joining a Telegram group if I create one to sync on this?

If anyone would like to join a Telegram group to find out about progress on this or give feedback on the design doc here is a link https://t.me/+a_SIusNyUMkzNDI8
Right now I'm at the stage of testing the generated mutants and generating a mutation test report.

Progress updates on this task can be found here https://hackmd.io/@tICezjHsSaiehIn9jbcUAA/SkTEyvuHa
We now have an alpha working implementation of integrating Gambit with Foundry.  If you have any questions please feel free to join the TG group posted above and ask.
@chandrakananandi If you could join the Telegram group posted above that would be awesome.
Also, how open is the Certora team to allowing changes to Gambit?

@samparsky Thanks so much for the updates! I responded to all the points in the doc: https://hackmd.io/@tICezjHsSaiehIn9jbcUAA/SkTEyvuHa
I am not the best at keeping up with telegram and other apps but I will join it!
I will certainly open to accepting PRs for gambit for improving the tool!


Periodic check with api server to see if user should log out

I tested manually by changing api server to return logout: true always, and making the slowChecks timer duration 10s.


Not flagged Spam

Good afternoon, a question, how can I access the messages that are not marked as spam, I have tried this way but it does not return anything, and there are several emails.
$ folder-> search () -> not_flagged ('spam') -> get ();

I also try to assign it the spam flag and it always returns false.
$message->setFlag (['spam']);

Hi @lsbescorpion ,
please use "Flagged" instead of "Spam".
The supported "flag range" is limited by design. These are the "default" flags which are supported by the official rfcs.
You're welcome to prove me wrong :)
Best regards,


[BUG] Unable to convert medit mesh to SU2

Describe the bug
When I try to convert any medit mesh to SU2 I receive the following error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/meshio/_helpers.py", line 188, in write
    return writer(filename, mesh, **kwargs)
  File "/meshio/su2/_su2.py", line 340, in write
    for index, (cell_type, data) in enumerate(mesh.cells):
TypeError: cannot unpack non-iterable CellBlock object

The error occurs when writing the SU2 mesh. In particular, after having written all the elements of the internal volume, when it has to start writing the surface elements for each tag, the execution stops and gives the error.
I believe the error is not with converting the medit format to SU2 format, just with writing a SU2 mesh.
To Reproduce
Just write a SU2 mesh with any tags.
mesh = meshio.read("input.mesh")
meshio.write("output.su2", mesh)

Hi, please did you find a fix for this?


Add new command 'manpage'

This also adds CI tests that check for up to date man pages in doc/man.
Closes #697

Codecov Report

Merging #1147 into master will decrease coverage by 0.08%.
The diff coverage is 14.28%.

@@            Coverage Diff             @@
##           master    #1147      +/-   ##
==========================================
- Coverage   52.83%   52.75%   -0.09%
==========================================
  Files         121      122       +1
  Lines       11423    11452      +29
==========================================
+ Hits         6035     6041       +6
- Misses       4679     4704      +25
+ Partials      709      707       -2

Impacted Files
Coverage Δ

cmd/restic/cmd_backup.go
24.29% <11.11%> (-0.5%)
:arrow_down:

cmd/restic/cmd_manpage.go
15.38% <15.38%> (ø)

internal/archiver/archiver.go
64.73% <0%> (-0.17%)
:arrow_down:

internal/backend/s3/s3.go
61.03% <0%> (+1.37%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bf3141...14f5f62. Read the comment docs.


Support to register nested data types (avro/json/parquet in particular) in Wrangler

JPMC(CCB)  Data Lake side, Customer  are looking to expand their support to register nested data types (avro/json/parquet in particular). To do this, we need to support table creation using json/avro schema files.
Ideally, we would hope the Glue create_table() API would expand support for this. However, a faster alternative would be to expand AWS Datawrangler SDK methods like these -> https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.catalog.create_json_table.html#awswrangler.catalog.create_json_table
to also support AVRO and support creation of Glue tables via schema files, We don’t necessarily want to go down the Glue SchemaRegistry route.
Reaching out what the process \step to incorporate above feature request for below wrangler SDK.
https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.catalog.create_json_table.html#awswrangler.catalog.create_json_table

Is the ask to create a wr.catalog.create_avro_table? If so, it should be as simple as adding AVRO definitions to these two files:

https://github.com/aws/aws-sdk-pandas/blob/main/awswrangler/catalog/_definitions.py
https://github.com/aws/aws-sdk-pandas/blob/main/awswrangler/catalog/_create.py
Feel free to have a go at it in a PR if you wish


Server examples outdated?

Hey everyone,
I am trying extend Express with Horizon, to get stand alone server.
Here's my backend.js file
#!/usr/bin/env node
'use strict'

const express = require('express')();
const horizon = require('@horizon/server');

const app = express();
const http_server = app.listen(8181);
const horizon_server = horizon(http_server);

console.log('Listening on port 8181.');

But when I run this script, it fails:
(...)/node_modules/finalhandler/index.js:58
    var status = res.statusCode
                    ^

TypeError: Cannot read property 'statusCode' of undefined
    at (...)/node_modules/finalhandler/index.js:58:21
    at EventEmitter.handle ((...)/node_modules/express/lib/application.js:169:5)
    at app ((...)/node_modules/express/lib/express.js:38:9)
    at Object.<anonymous> ((...)/backend.js:7:13)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:160:18)
    at node.js:445:3

I am using latest node version - 6.1.0. I've also try to run it on 5.10.1, but error still appears.
Maybe examples are outdated? Or is it problem with my code?

Try
#!/usr/bin/env node
'use strict'
const express = require('express');
const horizon = require('@horizon/server');
const app = express();
const http_server = app.listen(3000);
const horizon_server = horizon(http_server,{ auth: { token_secret: 'my_super_secret_secret' } });

I found the issue. Here is the correct code:
#!/usr/bin/env node
'use strict'

const express = require('express');
const horizon = require('@horizon/server');

const app = express();
const http_server = app.listen(8181);
const horizon_server = horizon(http_server,{ auth: { token_secret: 'my_super_secret_secret' } });

console.log('Listening on port 8181.');

There are 2 issues:

It should be:
const express = require('express');
not
const express = require('express')();
it's missing auth secret as previously mentioned:
const horizon_server = horizon(http_server,{ auth: { token_secret: 'my_super_secret_secret' } });

If you want to get something in the browser you'll need to add something like this before the app.listen(8181):
app.get('/', function (req, res) {
  res.send('Hello World!');
});

I can make a pull request if the above is correct

Thanks @gianpaj and @triniwiz, now everything works just fine ;)

PRs welcome!


Cutting a new release soon?

Hey there,
I'm trying to selectively upload/deploy stemcells.
I need the "support floating flag when uploading a stemcell" feature added in this commit: https://github.com/pivotal-cf/om/commit/16d387b8beb661f20fc5cd5d8af5ff878b597282 (Without this, when I run upload-stemcell, the stemcell is automatically applied to all the corresponding products, which I do not want)
Do you have an idea of when the next om release might be cut with this feature included? In the mean time I can build from source, but I don't want to build from source for production use. Apologies, I don't mean to sound pushy or anything, just figured I'd ask
Thank you!

@aegershman - thanks for bringing this up. we've been trying to cut releases pretty frequently, but there isn't a "standard" cycle time at this time.
We'll likely cut a new release sometime within the next week.

Thank you! I appreciate it.


[protobuf] Fix protobuf missing semi colons #1409

@atkawa7,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

Thanks!


Added expert settings

@hliebscher
Added A_CYC_RH_LEVEL_MODE (Feuchtigkeit) and A_CYC_SUPPLY_HEATING_ADJUST_MODE (Methoden der Temperaturregelung) to config.js

Thanks for the contribution!
Please give me a ping if you are ready to merge, i assume the commented state is waiting to be completed with @hliebscher s info

The Added A_CYC_RH_LEVEL_MODE (Feuchtigkeit) and A_CYC_SUPPLY_HEATING_ADJUST_MODE works fine for 270MV
For A_CYC_OPT_TEMP_SENSOR_MODE we have to test with sensor.
without we get "-273" as return value

There is a similar solution within vallox-api. Perhaps it does calculate the Kelvin offset on any Temp value like I do with the unit.
I will look into it tonight tomorrow. I think we can fix that.
@danielbayerlein

Added new pull Request with all 3 States and without "not working workaround"
https://github.com/hacki11/ioBroker.valloxmv/pull/13


account: enable cloudtrail action

Enable = start logging?

Comment from @kapilt on a dead pull request:

yeah, my bad, it was intended to put the trail and recorder in place if it had been deleted, or to enable if it had been disabled.

I’m revisiting this. If we need to put one in place, we need to name it. We also need to create (and name) an S3 bucket for it. Any preferences for those names?

I'd defer to policy for any resource details, most folks set this up during account creation, we're effectively doing this so we can have realtime remediation if folks disable or delete an extant trail somehow.


[all] Switch to pedantic lint package

Description
Our analysis_options.yaml was supposed to be a mirror of
flutter/flutter but had fallen out of date. This will be an easier way
to keep it standardized and synced.
This also required fixing all the lint errors brought in by that lint
package. Most of them were missing awaits.
Related Issues
Replace this paragraph with a list of issues related to this PR from the issue database. Indicate, which of these issues are resolved or fixed by this PR. Note that you'll have to prefix the issue numbers with flutter/flutter#.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[ ] All existing and new tests are passing.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] The analyzer (flutter analyze) does not report any problems on my PR.
[ ] I read and followed the Flutter Style Guide.
[ ] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[ ] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[ ] I updated CHANGELOG.md to add a description of the change.
[ ] I signed the CLA.
[ ] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[ ] No, this is not a breaking change.

Since, pedantic is only used for analysis_options.yaml, isn't it best to set it as dev_dependency ,expect for package_info?

@sarbagyastha yeah, good catch. This PR is still just a work in progress, there's some more things I'd like to fix here before landing it.

I'm closing this for now in favor of a more gradual approach, #2300. I don't think it's practical to try to add and fix all these new rules at once.


Ethereum adapter has no tests

Context
A recent PR broke the Ethereum adapter in a way that would have been very apparent if we had some tests guarding even its most basic functionality. The PR changed the web3 dependency to a much older version.
Expected Behavior
CI would have caught the mistake.
Actual Behavior
The change was merged, and the ethereum adapter was completely broken.
Possible Fix
An end-to-end test that runs a very minimal benchmark against the headless version of ganache. Ganache is a tool from Truffle, written in JavaScript, that emulates an ethereum network.
(Disclosure: I used to be the maintainer of Ganache).
Steps to Reproduce

Submit a PR that changes the web3 dependency of the caliper-ethereum module to 0.22.x.
Observe that CI checks do not fail.

Context
I work with the PegaSys team now, and my primary focus is on improving the performance of Hyperledger Besu. My benchmarking automation was relying on master, as there were other in-flight bug fixes that I needed. When the PR in question was merged, my benchmarking automation broke, and I needed to pin it to an earlier commit hash.
Your Environment

Version used: master
Environment name and version (e.g. Chrome 39, node.js 5.4): N/A
Operating System and version (desktop or mobile): N/A
Link to your project: https://github.com/hyperledger/besu

Turns out that there are integration tests for the ethereum adapter - I just didn't see them, as they were in a different module. The PR in question didn't break things for HTTP RPC connections, but it did break things for websockets. With my changes in #829, were the same PR to be put through again, I'd expect CI to catch it.
As a result, I'm closing this.


Support include/exclude at file/block/next line/current comment scope for documentation generation tags

This seems super cool. Could I be assigned to it?


How do I handle sitemap.xml.gz (compressed sitemap)

I'm trying to index a Plone site, and the only format I can get the sitemap in is .gz. Can Norconex somehow handle a sitemap.xml.gz file?

Yes, it can.  By default, it looks for /sitemap.xml or /sitemap_index.xml for each new domains encountered.
To change that, you have a few options.  First, if you are dealing only with one site, you can define the sitemap as part of the start URLs.  Like this:
    <startURLs ... >
      <sitemap>http://www.somewhere.com/sitemap.xml.gz</sitemap>
     </startURLs>

If you have a mix of sites that can have their sitemaps in different locations, you can define alternate locations to check for each domains:
  <sitemapResolverFactory class="com.norconex.collector.http.sitemap.impl.StandardSitemapResolverFactory">
     <path>/sitemap.xml.gz</path>
     <path>/sitemap.xml</path>
     <path>/whatever-other-weird-name-for-sitemap.xml</path>
  </sitemapResolverFactory>


'Save as...' dialog windows bugged

Description
I wanted to save files from programs or a browser (for example exporting a file as a PDF etc.) on multiple occasions now and usually a window pops up from the windows explorer where you can specify how you want to safe a file.  Addditionally to that, a .txt file called 'save_dialog' gets created on my desktop folder, which confuses me why that happens. I then tried to uninstall Files from my computer and now no save window pops up on my system, instead an error pops up saying that files.exe could not be found. I tried to disable the 'set as default' option under the advanced settings and after uninstallation even tried to run a .reg file postet here on github to manually set the registry keys back to their default values. How can i revert the changes made by Files so a normal windows explorer save dialog does appear again so I can save my files as intended? Or alternativeley is there a way to get a save dialog in Files?
Steps To Reproduce

Install Files
Set Files as the default browser and then unset it (Not sure how relevant this is)
Open a program or web browser to save something ('save', 'save as', 'export' etc.)
3a. Now a Files window pops up but I can't save anything there
Uninstall Files
4a.  An error window pops up, saying files.exe cannot be found

Requirements

Implement a proper save dialog window for Files
Revert everything back after uninstallation so windows explorer can behave normally

Files Version
<IP_ADDRESS>
Windows Version
10.0.22631 Build 22631
Log File
save_dialog.txt

Thanks for the report, Files doesn't change the save/open dialogs. Have you built from the source code before and enabled it there?

The option is a dev only feature. You will have to built it from source and turn the option off in Files settings > Advanced

@gave92 I do have the same issue.
When I try to open Save As and Open dialogs in Notpad it seems to hang; the window becomes uninteractive just like disabled but Windows seems not to be aware.


test: update block committer & fuel core docker setup for integration tests

closes https://github.com/FuelLabs/fuel-bridge/issues/363
updates the committer address for local deployment due to the new block committer version upgrade

few nits

will add env vars in a separate pr, better that way in terms of visibility, what do you think ?


Support for flask >= 2.3.2 to fix CVE-2023-30861

Description
Currently the latest stable version of connexion 2.14.2 restricts flask to < 2.3. All versions < 2.3.2 contain a security vulnerability https://github.com/advisories/GHSA-m2qf-hxjv-5gpq which we would like to remove, while continuing to use connexion.
Expected behaviour
It should be supported to install flask==2.3.2  to fix the CVE.
Actual behaviour
flask < 2.3 is enforced.
Steps to reproduce
pip install connexion

Flask 2.2.5 was just released with a backport for that security advisory, so you can update to that for now. It's my understanding that Connexion already has plans to support 2.3.x as well.

@davidism Hey, thanks for pointing this out. Thanks for providing such a fast backport as well!
Enforcing flask >= 2.2.5 resolves this issue for us.


Backfill using Kraken results in timeouts

System information

OS:Linux Mint 20 Ulyana (Ubuntu focal)
Zenbot version : 4.1.0
Zenbot branch:  unstable
NodeJS version: v10.19.0
Python version (when using a python script): 3.7.7
Exact command to reproduce (include everything): ./zenbot.sh backfill kraken.XXBT-ZEUR -d 14
Did I make any changes to conf-sample.js?: Only exchange

Describe the problem
Backfilling using Kraken exchange results in timeouts
Source code / Error logs
(base) mmarre@mmarre-VirtualBox:~/zenbot$ ./zenbot.sh backfill kraken.XXBT-ZEUR -d 14

skipping 93 hrs of previously collected data

kraken.XXBT-ZEUR saved 1000 trades 11 days left

Trades error:
{ RequestError: Timeout awaiting 'request' for 5000ms
    at ClientRequest.request.once (/home/mmarre/zenbot/node_modules/kraken-api/node_modules/got/dist/source/core/index.js:785:25)
    at Object.onceWrapper (events.js:286:20)
    at ClientRequest.emit (events.js:203:15)
    at ClientRequest.origin.emit.args (/home/mmarre/zenbot/node_modules/kraken-api/node_modules/@szmarczak/http-timer/dist/source/index.js:39:20)
    at TLSSocket.socketErrorListener (_http_client.js:401:9)
    at TLSSocket.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
    at Timeout.timeoutHandler (/home/mmarre/zenbot/node_modules/kraken-api/node_modules/got/dist/source/core/utils/timed-out.js:35:25)
    at ontimeout (timers.js:438:13)
    at tryOnTimeout (timers.js:300:5)
    at unrefdHandle (timers.js:520:7)
    at Timer.processTimers (timers.js:222:12)
  name: 'TimeoutError',
  code: 'ETIMEDOUT',
  timings:
   { start:<PHONE_NUMBER>284,
     socket:<PHONE_NUMBER>284,
     lookup:<PHONE_NUMBER>285,
     connect:<PHONE_NUMBER>294,
     secureConnect:<PHONE_NUMBER>310,
     upload:<PHONE_NUMBER>284,
     response:<PHONE_NUMBER>569,
     end: undefined,
     error:<PHONE_NUMBER>287,
     abort: undefined,
     phases:
      { wait: 0,
        dns: 1,
        tcp: 9,
        tls: 16,
        request: NaN,
        firstByte: 285,
        download: undefined,
        total: 5003 } },
  event: 'request',
  bufferedData:
   <Buffer 7b 22 65 72 72 6f 72 22 3a 5b 5d 2c 22 72 65 73 75 6c 74 22 3a 7b 22 58 58 42 54 5a 45 55 52 22 3a 5b 5b 22 38 33 34 34 2e 33 30 30 30 30 22 2c 22 30 ... > }

download complete!

It's known that Kraken is having a crappy API which is happy to run into timeouts. However, I suggest to work around this by running in a loop for backfill, just printing a warning when a timeout occur and continue to download until all data requested is being downloaded or a real error occur.
In any case "download complete!" is wrong here as it has been aborted earlier.

Increase poll times to fit your needs. Works without timeouts with these settings:
// ms to poll order status
c.order_poll_time = process.env.ZENBOT_ORDER_POLL_TIME || 5500
// ms to wait for settlement (after an order cancel)
c.wait_for_settlement = process.env.ZENBOT_WAIT_FOR_SETTLEMENT || 5000

// defaults to a high number of lookback periods
c.keep_lookback_periods = process.env.ZENBOT_KEEP_LOOKBACK_PERIODS || 50000
// ms to poll new trades at
c.poll_trades = process.env.ZENBOT_POLL_TRADES || 30000
// amount of currency to start simulations with
c.currency_capital = process.env.ZENBOT_CURRENCY_CAPITAL || 1000


Revert removal of CRLF from Windows bat files

Summary
This reverts #31128 - For the reasons stated in the thread. Files should have the correct endings in the repo (i.e. Windows .bat CRLF). There is no reason to perform additional conversion with attributes and/or an editorconfig. It was originally fixed in #29792 in August 2020.
:warning: EDIT 2021-08-31
Commits 85249cafe88 and 13107fa3d0b accidentally converted the gradlew.bat files to LF again, resulting in modified files to appear in the working directory:
$ git status -s
 M gradlew.bat
 M packages/react-native-codegen/android/gradlew.bat
 M template/android/gradlew.bat

The reasons why this is happening are explained in detail in the two PRs linked above.
I've added an additional (new) commit to the PR head branch to fix the line endings in all three gradlew.bat files of the repo and rebased it. It should be ready for merge.
CC @cortinico
EDIT 2021-09-02
The additional commit was removed again, but the original one remains.
To test the scenario locally run the following commands on a clean main branch (currently 455433f481bf5f743a3330a9a46edda5fc69da69):
$ rm gradlew.bat
$ git status -s
 D gradlew.bat               # Git shows the file as (D)eleted, as expected
$ git checkout gradlew.bat   # This should restore the file
$ git status -s
 M gradlew.bat               # The file still shows up, now as (M)odified with all line endings changed

The modified file will remain in the working directory until they are committed, or a different branch is force checked out. gradlew.bat files are generated automatically by Gradle (with the correct line endings in the first place). There is no need to special case them and perform line ending conversion using Git and/or editorconfig.
Changelog
[General] [Fixed] - Line endings in Windows files, Git/EditorConfig related conversions
Test Plan
Verify files are stored correctly in the repository (e.g. using the file command).

Files should have the correct endings in the repo (i.e. Windows .bat CRLF).

Thanks for your PR @friederbluemle
Those files have been generated automatically by Gradle with the command:
./gradlew wrapper --gradle-version 7.0.2 --distribution-type=all

I'm unsure if removing .gitattributes will actually solve the root cause.
I'm wondering if we would have to manually edit this .bat file whenever we bump the Gradle version.

Thanks for the quick reply @cortinico! The command you posted is correct. There should be no need to manually edit the Gradle Wrapper scripts. Like you said, Gradle will automatically generate the files with the correct line endings.
Can you test the following real quick:
On a clean main branch:
$ rm gradlew.bat
$ git status -s
 D gradlew.bat               # Git shows the file as (D)eleted, as expected
$ git checkout gradlew.bat   # This should restore the file
$ git status -s
 M gradlew.bat               # The file still shows up, now as (M)odified with all line endings changed

Can you test the following real quick:

Just tested and as you mentioned, the last git status -s shows the gradlew.bat as modified.
I believe that part of the problem is caused by the fact that internally we use mercurial instead of git and phabricator is exporting the git commits to Github.
I'm fine merging your change, but does this mean we would have to do the same whenever we touch the gradlew.bat file?

I'm fine merging your change, but does this mean we would have to do the same whenever we touch the gradlew.bat file?

Hey @friederbluemle Thanks for your PR. I've done a couple of tests and we should be fine merging this 👍

Thanks @cortinico :+1:

I believe that part of the problem is caused by the fact that internally we use mercurial instead of git and phabricator is exporting the git commits to Github.

I think Mercurial does not read a .gitattributes files, so there should be no issue one way or the other.

I'm fine merging your change, but does this mean we would have to do the same whenever we touch the gradlew.bat file?

As you've mentioned previously, gradlew.bat is updated/created by Gradle's wrapper task. It will have the correct line endings. There should be no need to touch/edit it manually.

cc @pvinis if there are any concerns?

seems like we are just "fixing" each other's changes. I am on holidays until end of next week. I could check after. If you want to emrge before, I trust you all.

No rush here.. We can wait for @pvinis to be back from holidays if there are additional questions etc. :+1:

@cortinico merged this pull request in fc553ed.

Hey @pvinis @friederbluemle apologize for the confusion. The diff was merged overnight as it was approved internally. I haven't seen your message early enough to stop the pre-approval @friederbluemle
I believe we can keep it merged as it fixes the local environment for now (e.g. see #32160) but I'm happy to keep on the discussion and find a definitive solution for this 👍

FYI @yungsters just merged #32178 that updates the gradlew.bat files to use LF only.

I believe that part of the problem is caused by the fact that internally we use mercurial instead of git and phabricator is exporting the git commits to Github.
I think Mercurial does not read a .gitattributes files, so there should be no issue one way or the other.

True, but we use a bot to export from mercurial to git and I believe that bot is having problems handling whitespace chars.

I know its late, but only now I moved a project to 68. Removing the editorconfig file and instruction seems unnecessary. Say the files are perfectly set in the repo and on everyones inited project, great. Now imagine i need to do an edit on the .bat file, on my mac machine. Maybe it's because of an upgrade, maybe I want to just fix a typo, or anything else in that .bat file. Now I save, and the file and diff is all messed up, because im on a mac. the editorconfig will make sure that the line endings on these files stay the same. gitattrs, i guess we can remove. editorconfig is needed.
What do you think?

Hi @pvinis - Good question. What I don't fully understand - please explain more - why would saving a file have the side effect of converting all line endings in that file? Which editor are you using? I just did a quick test in a RN 68 repo, editing gradlew.bat using Vim, WebStorm, and VS Code. The project did not have an editorconfig, I was testing on macOS as well, and non of the line endings were changed.

Hi @pvinis - Good question. What I don't fully understand - please explain more - why would saving a file have the side effect of converting all line endings in that file? Which editor are you using? I just did a quick test in a RN 68 repo, editing gradlew.bat using Vim, WebStorm, and VS Code. The project did not have an editorconfig, I was testing on macOS as well, and non of the line endings were changed.

mostly vscode. there's a lot of times/tools/editors that save and "make sure" the endings are right for the os you are in. also, if I have an editorconfig in my home dir, then that will be applied, and if the config is for Mac by default, it will mess up the bat files. it can be fixed by fixing the home config to include bat, but it's just safer.
anyway, I see a lot of resistance, and I have it for my projects, so I don't mind it. I'm only trying to make it less error prone for others 🤷♂.

@pvinis

Now imagine i need to do an edit on the .bat file, on my mac machine

You should not need to edit that .bat file. That is a gradle generated file and should not be touched. The output of the wrapper tasks already generates files which are valid (from the line-ending pov).
Your argument could potentially hold if you might have other .bat file in your project, but then it's up to you to customize

I don't believe in "don't touch this file", and if an upgrade has a small change, I usually do it myself, rather than copy paste the whole file anyway, but I get your point. I don't agree, but I get it.

and if an upgrade has a small change, I usually do it myself,

This specific issue is related to the Gradle Wrapper which must be updated usign this command: https://docs.gradle.org/7.4.2/release-notes.html#upgrade-instructions
So it's an automated bump, there is no copy-n-paste involved here. It will always be an automated step regardless of the size of the updated.


torchvision.transforms.Resize((128,64))

in test.py
transform = torchvision.transforms.Compose([
torchvision.transforms.Resize((128,64)),
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
why  Resize((128,64)) ?  image shape in market1501 is 64 * 128

size arg matchs to (height, width).
class Resize(object):
    """Resize the input PIL Image to the given size.

    Args:
        size (sequence or int): Desired output size. If size is a sequence like
            (h, w), output size will be matched to this. If size is an int,
            smaller edge of the image will be matched to this number.
            i.e, if height > width, then image will be rescaled to
            (size * height / width, size)


some configuration files under make/config/ should be ignored

Need to go through and figure out what files are generated by prepare, and ignore those files.

fixed #1042

fixed #1039


kv-common vcpkg fix: Option A: revert to original constructor

Fix for the vcpkg failing to build azure-security-keyvault-common-cpp that we deletedfrom our repo a long ago, but it is still in vcpkg.
The build error is the following:
sdk\keyvault\azure-security-keyvault-common\src\keyvault_exception.cpp(35):
error C2664: 'Azure::Core::RequestFailedException::RequestFailedException(
  const std::string &,std::unique_ptr<Azure::Core::Http::RawResponse,
  std::default_delete<Azure::Core::Http::RawResponse>> &)':
    cannot convert argument 2
      from 'std::unique_ptr<Azure::Core::Http::RawResponse,std::default_delete<Azure::Core::Http::RawResponse>>'
      to 'std::unique_ptr<Azure::Core::Http::RawResponse,std::default_delete<Azure::Core::Http::RawResponse>> &'

Which was caused by us updating the constructor just recently. This PR reverts that.
But with this PR, there is not clear path to drop the constructor in the future, as we wanted (#3215).
Other options are #3238 and #3239.

Because of the time in which KeyVault common beta for released, this fix alone won't  be enough.
We also had the macro rename from AZURE_UNREACHABLE_CODE to AZURE_UNREACHABLE_CODE() prior to Azure Core, GA.

@ahsonkhan, are you sure, that AZURE_UNREACHABLE_CODE update has happened in prior releases, why is it only now that it is broken?


RPM, or automated installation of this plugin

Does anyone have a spec file for building an RPM of this plugin, or a good solution for installing this on Amazon Linux using puppet?

+1, need unattended install desperately

+1, Any new progress on this issue? Auto installation is a needed enhancement.

No way "just download the setup file and run sudo python setup.py -I" will ever be allowed in any of our environments.
Can you confirm when this will be properly packaged?


Remove dangling temporary directories from outputs/tensors at startup time

Summary
If the web app exits unexpectedly it may not run its shutdown routine, which removes temporary directories, among other things. This PR checks INVOKEAI_ROOT/outputs/tensors for dangling temporary directories at startup time and removes them.

Related Issues / Discussions
See #6242 for background on a bug that led to a large number of dangling temporary directories in outputs/tensors.

QA Instructions
If you have been running earlier versions of invokeai-web you probably have a lot of dangling temporary directories in outputs/tensors. When you run this PR, these directories should be removed.

Merge Plan
Merge when approved.

Checklist

[X] The PR has a short but descriptive title, suitable for a changelog
[X] Tests added / updated (if applicable)
[X] Documentation added / updated (if applicable)

My first thought is this should instead be the responsibility of the service that writes the files

My first thought is this should instead be the responsibility of the service that writes the files

Good idea. I'll modify the PR.


Nuttx file descriptor dup operation issue

Issue
The dup operation of nuttx is different from that of freebsd and opengroup.

Nuttx
The old file descriptor and the new file descriptor generated by dup and dup2 don't share the file entity and are independent of each other. They correspond to different struct file entities in the filelist.
A file descriptor is an index into an fl_files array of sruct file. The struct file associates  the file descriptor to the file state and to a set of inode operations.

  struct file
  {
    int               f_oflags;   /* Open mode flags */
    off_t             f_pos;      /* File position */
    FAR struct inode *f_inode;    /* Driver or file system interface */
    FAR void         *f_priv;     /* Per file driver private data */
  };

  /* This defines a two layer array of files indexed by the file descriptor.
   * Each row of this array is fixed size: CONFIG_NFILE_DESCRIPTORS_PER_BLOCK.
   * You can get file instance in filelist by the follow methods:
   * (file descriptor / CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as row index and
   * (file descriptor % CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as column index.
   */

  struct filelist
  {
    sem_t             fl_sem;     /* Manage access to the file list */
    uint8_t           fl_rows;    /* The number of rows of fl_files array */
    FAR struct file **fl_files;   /* The pointer of two layer file descriptors array */
  };

FreeBSD/OpenGroup:

Do you have good idea to fix this issue?
In view of the above problems, it will cause some problems in the application of using dup, dup2 and fcntl on nuttx.
Here is one of the methods:
I will use filenode to correspond to file descriptor one-to-one, and multiple filenodes can correspond to a unified file entity, When dup occurs, the file pointer corresponding to the old filenode can be copied to the new filenode to ensure sharing.

struct file
{
  int16_t           f_crefs;    /* References to file */
  int               f_oflags;   /* Open mode flags */
  off_t             f_pos;      /* File position */
  FAR struct inode *f_inode;    /* Driver or file system interface */
  FAR void         *f_priv;     /* Per file driver private data */
};

/* A file descriptor is an index into an array of such types. The type is used to
 * multiple file descriptor to share the same open file, include file offset and
 * file status flags, buf don't share file descriptor flags "close-on-exec flag".
 */

struct filenode
{
  int               fn_oflags;  /* Open or duplicate mode flags */
  FAR struct file  *fn_file;    /* The pointer representation of an open file */
};

/* This defines a two layer array of files indexed by the file descriptor.
 * Each row of this array is fixed size: CONFIG_NFILE_DESCRIPTORS_PER_BLOCK.
 * You can get file instance in filelist by the follow methods:
 * (file descriptor / CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as row index and
 * (file descriptor % CONFIG_NFILE_DESCRIPTORS_PER_BLOCK) as column index.
 */

struct filelist
{
  sem_t                 fl_sem;   /* Manage access to the file list */
  uint8_t               fl_rows;  /* The number of rows of fl_files array */
  FAR struct filenode **fl_files; /* The pointer of two layer file descriptors array */
};

And it also needs to modify file_openation:dup, directly delete or remove unnecessary features.
Refer to
https://www.freebsd.org/cgi/man.cgi?query=dup&sektion=2
https://pubs.opengroup.org/onlinepubs/009604499/functions/dup.html

@pkarashchenko @patacongo @acassis @masayuki2009 what do you think about this bug?

I see that FreeBSD indeed includes such description, but I failed to find any similar lines in https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html

@pkarashchenko

I see that FreeBSD indeed includes such description, but I failed to find any similar lines in https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html
Additionally I can find in description from OpenGroup: The dup() function provides an alternative interface to the service provided by fcntl() using the F_DUPFD command. The call dup(fildes) shall be equivalent to:
fcntl(fildes, F_DUPFD, 0);

But fail to find similar in description from FreeBSD

@pkarashchenko FreeBsd also have similar behavior in desctiption:
https://man.freebsd.org/cgi/man.cgi?query=dup2&apropos=0&sektion=2&manpath=FreeBSD+9.0-RELEASE&arch=amd64&format=html#:~:text=The object referenced,between the references.
The object referenced by the descriptor does not distinguish between oldd
and newd in any way.  Thus if newd and oldd are duplicate references to
an open file, [read(2)],  [write(2)] and [lseek(2)] calls all move a single
pointer into the file, and append mode, non-blocking I/O and asynchronous
I/O options are shared between the references.

So any good suggestions to solve this problem? @pkarashchenko @acassis If there are no good suggestions, I will use filenode to solve this problem in the near future

@Donny9 It's a bit find out from a description what is the issue from POSIX perspective, but finally I think I managed to understand what you are talking about. It is https://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html
The dup2() function shall cause the file descriptor fildes2 to refer to the same open file description as the file descriptor
fildes and to share any locks, and shall return fildes2.

Is that correct? If yes, then it depends on the interpretation if refer to the same open file description is equivalent to does not distinguish between oldd and newd in any way.
If I still didn't understood the issue correctly then please explain it in more clear manner like:

state the POSIX description lines
state FreeBSD description lines (that I think is already done)
show how current NuttX implementation violates POSIX

@pkarashchenko This examples can run with pass result on linux, but can't pass on nuttx, because this issue: newd and oldd can't share file offset, because they are different file handle in nuttx.
//#include <nuttx/config.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <syslog.h>
//#include "test.h"
#define TESTFILE "testDup2File1"
#define TEST_FAIL -1
#define TEST_PASS 0

/****************************************************************************
 * Name: dup
 * Example description:
    1. Open a file and get the file identifier fd.
    2. Copy the file identifier to newfd
    3. Open the file through newfd. And check the return results
 * Test item: open()  write() dup2()
 * Expect results: TEST PASSED
 ****************************************************************************/

int main(void)
{
    char buf[16] = {0};

    off_t currpos;
    int fd1 = open(TESTFILE, O_RDWR | O_CREAT, 0777);
    if (fd1 == -1)
    {
        printf("open file fail !\n");
        return TEST_FAIL;
    }

    int fd2 = open(TESTFILE, O_RDWR | O_CREAT, 0777);
    if (fd2 == -1)
    {
        close(fd1);
        printf("open file fail !\n");
        return TEST_FAIL;
    }

    int ret = dup2(fd1, fd2);
    if (ret == -1)
    {
        close(fd1);
        close(fd2);
        printf("dup2 error ,  return : %d!\n", ret);
        return TEST_FAIL;
    }
    syslog(LOG_INFO, "current fd = %d\n", ret);
    char *buf1 = "hello ";
    char *buf2 = "world!";
    write(fd1, buf1, strlen(buf1));
    write(fd2, buf2, strlen(buf2));

    fsync(fd1);

    /* reset file pos use fd2 */
    lseek(fd2, 0, SEEK_SET);

    /* Check if file pos is shared */
    currpos = lseek(fd1, 0, SEEK_CUR);
    if (currpos != 0)
    {
        printf("Do not shared file pos after dup2()\n");
        close(fd1);
        close(fd2);
        return TEST_FAIL;
    }

    ret = read(fd1, buf, 16);
    if (ret <= 0)
    {
        printf("read fail\n");
        close(fd1);
        close(fd2);
        return TEST_FAIL;
    }
    close(fd1);
    close(fd2);

    if (strncmp(buf, "hello world!", 12) == 0)
    {
        printf("test pass\n");
        return TEST_PASS;
    }

    printf("test fail\n");
    return TEST_FAIL;
}

@Donny9 what about other OSes? I will try to run it on MAC OS and come back with results

The MACOS works the same and test pass.
On your proposal I do not fully understand why struct filenode is needed. Could you please explain more. I see that struct filenode contains fn_oflags and struct file contains f_oflags, so what is the difference between those two?

xplain more. I see that struct filenode contains fn_oflags and struct file contains f_oflags, so what is the difference between those two?

oldd and newd need to share file offsets and other information, but they need to have their own flags. We can remove the flags in the file structure, but it will change a lot.
Therefore, we need to make oldd and newd point to the same file structure, so let the file be one level below and use filenode as the file opening context.

But leaving flags in both file and filenode structures will lead to a duplication? Or some flags will stay in file while other will reside in filenode?
BTW I think that there is a typo in your test example. I didn't try to run it on NuttX, but believe that line ret = read(fd1, buf, 16); should be ret = read(ret, buf, 16); otherwise I do not see where dup2 result is used


Remove the integration of fetching the metadata from external sources

Some information is fetched from external sources. This source is not managed anymore, we should remove that integration.

The fetching of project meta data is done/attempted here: https://github.com/acdh-oeaw/apis-rdf-devops/blob/main/webpage/utils.py#L24. PROJECT_MD_URL is put together manually at the top of the same file using PROJECT_STATIC_URL.
PROJECT_DEFAULT_MD sits here: https://github.com/acdh-oeaw/apis-core-rdf/blob/dev/apis/settings/base.py#L55


Zenlink: dex smart contract

Grant Application Checklist

[x] The application-template.md has been copied, renamed ( "project_name.md") and updated.
[x] A BTC address for the payment of the milestones is provided inside the application.
[x] The software of the project will be released under the Apache license version 2.0 as specified in the terms and conditions.
[x] The total funding amount of the project is below CHF 30k at the time of submission.

@mmagician @Noc2 Please take your time to have a look at it. Thanks!

Could you integrate the medium article/tutorial into the deliverables?
Could you specify that you are using ink! as part of the deliverables?
Could you reduce the price or add additional functionality like an interface? 0.5 btc seems quite expensive to me, given that you previously already implemented a similar functionality in the form of a pallet.

@mmagician
I've integrated the new changes of the application template into this pr.


Bluetooth connection timeout

Problem
After connecting to board over bluetooth, the subscription fails and hence disconnect happens.
Set up -
We are using custom board which has two chip configuration for BLE stack. The controller being Laired connectivity BL651 chip which has nRF52810 MCU, it runs zephyr with HCI ( hci_uart) controller stack.
zephyr version is 1.13.99
On central side we use Bluegiga BLE module with Python scripts on Windows host.
To Reproduce
Build the host side peripheral example zephyr image and hci_uart example for the controller side.
cd hci_uart
mkdir build
cmake -GNinja -DBOARD=nrf52_pca10040 .. ; ninja
For peripheral
cd peripheral
mkdir build
cmake -GNinja -DBOARD=myboard .. ; ninja
Expected behavior
The central should be able to read all the characteristics of the board. However, it turns out that it fails while fetching and receives a EVT_DISCONNECT.
Central console output
At Central (python on windows machine) error -
DEBUG:bgapi.py:Expecting a response of one of [<ResponsePacketType.attclient_find_information: 41>] within 1.000000s
DEBUG:bgapi.py:Received a ResponsePacketType.attclient_find_information packet: Success
DEBUG:bgapi.py:Expecting a response of one of [<EventPacketType.attclient_procedure_completed: 17>] within 1.000000s
DEBUG:bgapi.py:Received a EventPacketType.connection_disconnected packet: Success

Are you able to reproduce this using a different central, e.g. a mobile phone, or another instance of Zephyr?

Also, what is the exact version you're using? "1.13.99" doesn't help much, since that can be anything between 1.13 and the 1.14-rc1.

Yes. We have tried with BLE scanner app on Phone and with python scripts on the Windows machine. Unable to connect with both.
We had taken SHA  ID "32cbc42ce620783ae6f14d9affe68b1a5cbe21a1"
To check the exact tag I ran
git describe --tags 32cbc42ce620783ae6f14d9affe68b1a5cbe21a1
warning: tag 'zephyr-v1.13.0' is really 'v1.13.0' here
zephyr-v1.13.0-3320-g32cbc42ce6

Ok .. will add little more to the above description. BLE scanner app does discover our board. So when I click on the connect button ... After some time the app says "time out"
However on the board, we see "Connected and then Disconnected (reason 8)" prints.
Similarly with the python script. We do get the see the connect successful and then script sends fetch characteristics - where it fails and disconnect happens.

Hi,
Our further analysis -
At the central side, When the python script starts fetching the Characteristics it sends the Request as attclient_find_information to which the response is successful. This command is followed by Bluetooth attclient_procedure_completed to which the response is [128, 3, 3, 4, 0, 8, 2] which means disconnect.
Hence, it means that controller issued a response to disconnect instead of the version_ind_packet
Going through the version_ind_send code in blue tooth, it sends an empty packet and I think waits for the next connection interval to send the actual data. The central has predefined expectation of data and hence is not ack-ing? to which the peripheral sends the disconnect event.
Is this the right direction we are in  ??
Really looking forward to a reply on it. Thank you so much !

Another update -
Checked with other centrals also and tried to sniff the packet. What we understood is that moment after connect happens, in a while disconnect event with error 0x0208 is sent.
Also observed one thing, addr_type set in our board is Random. ( this thing we noticed while we were comparing with other logs).

HI, Any inputs on this please???

@neha0119 any chance to get the HCI traffic on either end? You can also enable GATT/ATT debug options if you thinh this is related to GATT/ATT layers, but it seems to be related to the ACL disconnecting for some reason, you could perhaps enable conn logs if you can't find a way to record the HCI traffic in Windows.

Zephyr_host.log
Thanks for replying ...
PFA the host logs
We could capture very minimum HCI traffic with the BLUEgiga BLE graphical Tool Interface.
Below is the BLUEGIGA log when I click connect.

2019.03.11 20:35:22.214 ble_cmd_gap_connect_direct address: e9 d4 b0 7a b9 58 addr_type: 1 (0x01) conn_interval_min: 40 (0x0028) conn_interval_max: 56 (0x0038) timeout: 1000 (0x03e8) latency: 0 (0x0000)
2019.03.11 20:35:22.221 TX: 000f0603e9d4b07ab9580128003800e8030000
2019.03.11 20:35:22.226 ble_rsp_gap_connect_direct result:0x0000 ['No Error'] connection_handle: 0 (0x00)
2019.03.11 20:35:22.227 RX:<PHONE_NUMBER>0000
2019.03.11 20:35:22.268 ble_evt_connection_status connection: 0 (0x00) flags: 5 (0x05) address:58:b9:7a:b0:d4:e9 address_type: 1 (0x01) conn_interval: 42 (0x002a) timeout: 1000 (0x03e8) latency: 0 (0x0000) bonding: 255 (0xff)
2019.03.11 20:35:22.275 RX:<PHONE_NUMBER>05e9d4b07ab958012a00e8030000ff
2019.03.11 20:35:32.341 ble_evt_connection_disconnected connection: 0 (0x00) reason:0x0208 ['Link supervision timeout has expired.']
2019.03.11 20:35:32.341 RX:<PHONE_NUMBER>0802

@Vudentz
One more observation
While the board boots, I get this message. Does this mean ACL flow control from controller to host not supported?
 bt_hci_core: Controller to host flow control not supported
 bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
 bt_hci_core: HW Variant: nRF52x (0x0002)
 bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 1.13 Build 99
 bt_hci_core: Identity: c3:91:9c:4f:d1:bb (random)
 bt_hci_core: HCI: version 5.0 (0x09) revision 0x0000, manufacturer 0x05f1
 bt_hci_core: LMP: version 5.0 (0x09) subver 0xffff

@neha0119 it means you've enabled CONFIG_BT_HCI_ACL_FLOW_CONTROL=y in your app for no good reason (it's neither needed nor supported with a combined host + controller build)

@neha0119 actually, for a combined build it may be useful, however the Zephyr controller doesn't support it. Either way, I think you can just disable that for your app.

DISABLED!

2019.03.11 20:35:32.341 ble_evt_connection_disconnected connection: 0 (0x00) reason:0x0208 ['Link supervision timeout has expired.']
2019.03.11 20:35:32.341 RX:<PHONE_NUMBER>0802

So you are getting a link supervision timeout, which is normally some sort of problem with hardware/antenna setup. Though contrary to the log you pasted I can see some GATT traffic, or perhaps the HCI logs doesn't show ACL traffic? Anyway I don't see any subscription just a discovery going on (something worth noting that the discover seems to be using Find Information request instead of starting discovering services):
[00:00:43.519,237]  bt_att.bt_att_recv: Received ATT code 0x04 len 5
[00:00:43.519,248]  bt_att.att_find_info_req: start_handle 0x0001 end_handle 0xffff
[00:00:43.519,285]  bt_att.find_info_cb: handle 0x0001
[00:00:43.519,294]  bt_att.find_info_cb: handle 0x0002
[00:00:43.519,313]  bt_att.find_info_cb: handle 0x0003
[00:00:43.519,322]  bt_att.find_info_cb: handle 0x0004
[00:00:43.519,333]  bt_att.find_info_cb: handle 0x0005
[00:00:43.519,352]  bt_conn.bt_conn_send_cb: conn handle 0 buf len 26 cb 0x0000b225
And then it disconnects after having received just one request, Find Information, and transmitting a response (which probably never arrived).

ConnLogs.txt
My apologies, I enabled the other debug config. FA the DEBUG_CONN enabled prints.
To your point-
And then it disconnects after having received just one request, Find Information, and transmitting a response (which probably never arrived).
The prints at the Windows machine does state that
"Received a ResponsePacketType.attclient_find_information packet: Success"
Then immediately the python script @windows says
"Expecting a response of one of [<EventPacketType.attclient_procedure_completed: 17>" to this, no response is seen. Instead an event -
ble_evt_connection_disconnected connection: 0 (0x00)  is observed.
So this is in sync with your observation. But the question is why connection_disconnection event is being generated by the board?

The working case Script logs on windows
Expecting a response of one of [<ResponsePacketType.attclient_find_information: 41>] within 1.000000s
Received a ResponsePacketType.attclient_find_information packet: Success
Expecting a response of one of [<EventPacketType.attclient_procedure_completed: 17>] within 10.000000s
Received a EventPacketType.connection_version_ind packet: Success
Received a EventPacketType.attclient_find_information_found packet: Success
Received a EventPacketType.attclient_find_information_found packet: Success
Received a EventPacketType.attclient_find_information_found packet: Success

Now I have reduced my setup with just my board and the BLuegiga BLE Graphical User Interface Tool (as Central). My board is discover able by Bluegiga Interface tool and now I click the CONNECT button. ( Logs below are the ones right after the CONNECT button is clicked)

2019.03.12 17:08:08.403 ble_cmd_gap_connect_direct address: bb d1 4f 9c 91 c3 addr_type: 1 (0x01) conn_interval_min: 60 (0x003c) conn_interval_max: 76 (0x004c) timeout: 100 (0x0064) latency: 0 (0x0000)
2019.03.12 17:08:08.410 TX: 000f0603bbd14f9c91c3013c004c0064000000
2019.03.12 17:08:08.414 ble_rsp_gap_connect_direct result:0x0000 ['No Error'] connection_handle: 0 (0x00)
2019.03.12 17:08:08.416 RX:<PHONE_NUMBER>0000
2019.03.12 17:08:08.451 ble_evt_connection_status connection: 0 (0x00) flags: 5 (0x05) address:c3:91:9c:4f:d1:bb address_type: 1 (0x01) conn_interval: 60 (0x003c) timeout: 100 (0x0064) latency: 0 (0x0000) bonding: 255 (0xff)
2019.03.12 17:08:08.457 RX:<PHONE_NUMBER>05bbd14f9c91c3013c0064000000ff
2019.03.12 17:08:09.574 ble_evt_connection_disconnected connection: 0 (0x00) reason:0x0208 ['Link supervision timeout has expired.']
2019.03.12 17:08:09.574 RX:<PHONE_NUMBER>0802
------------ This means that After one response and an event sent, board has sent disconnect event.

Or Host has asked to close the connection (for some reason) and controller sends that to the Windows machine to close it?

It never that error thought, so I guess this is not the host doing at least.

why connection_disconnection event is being generated by the board and not the expected response/event?

Well perhaps something is not right with the link so it timed out for some reason, you could perhaps try with another board or qemu on Linux to check if that is not a problem with the board or the controller.

@neha0119 could you please check if the BL651 module has a 32KHz external crystal? If you are using nRF52_pca10040 build on BL651, then it is expected to use external crystal for maintaining the BLE connection.
If BL651 does have a crystal, then you need to verify the crystal accuracy, and use the correct value in the Zephyr build, else the connections will drift leading to eventual disconnection.

Hello @cvinayak No we don't have external 32Khz external crystal. As per the BL651 module, the external crystal is optional. Hence we had not mounted it on out board.
We do have enabled the two configs in the zephyr.
CLOCK_CONTROL_NRF_K32SRC_RC and CLOCK_CONTROL_NRF_K32SRC_250PPM
But still there are disconnections all the time.
You mentioned that "it is expected to use external crystal with nRF52_pca11040 build" .. but we have not mounted it, is there any workaround?

@neha0119 as there is support for BL652 board, could you check the difference in Kconfig with that board? also, have you tired using the latest upstream/master hci_uart or v1.14.0-rc1? if you are on slack please ping me there for quicker discussions.

Hi @cvinayak
We tried those two patches which you had shared. But with that also we are getting disconnect.

@neha0119 with https://github.com/zephyrproject-rtos/zephyr/pull/11552 applied, you had the below or the default of 0?
CONFIG_BT_CTLR_MASTER_SCA_EXTRA=500

If our timezones match, we should have a screen sharing session wherein we can debug live.

@neha0119 have you managed to test the suggested changes from @cvinayak?

Hello @cvinayak and @carlescufi
By setting the CLOCK_CONTROL_NRF_K32SRC_500PPM solved our problem. Earlier it was set to 250PPM.

Thanks @neha0119. Closing this issue then.


ignore alternative channels; support private groups

This is pretty much my first time writing real Go code, so it's probably not pretty.
This changes two things:

Only accepts commands from specified channel.
Allows musicbot to work with private groups.

If this change make sense, I'll clean up the log statements / typos, etc. :+1: ?

Sure, go ahead


helm install when resources already exist will fail

This is a PR to fix the issue described here: https://github.com/kubernetes/helm/issues/3319

the intent here is to make the client smart enough to fail if a Create call is not able to complete successfully, due to a conflicting pre-existing resource.

I decided to implement this pre-existing resource check within the kubeclient. Reason being, if a Create call can not actually create the resource b/c the resource already exists, then one would expect the call to return an error.
This seemed more appropriate then implementing a check in the tiller package as part of the InstallRelease call, as tiller knows only about installing a release, and that it has a client interface upon which it can call Create. It shouldn't need to know anything about the expected behavior of the client Create call, and its handling of pre-existing resources.
The kubeclient Create implementation borrows the approach taken in the Update call on the kubeclient. Obvious difference being, update will skip pre-existing resources and create will now fail on pre-existing resources.

hi @bacongobbler, @technosophos,
I just wanted to follow up and see if you had any thoughts about this PR? Feedback is be welcome and much appreciated. Thanks.
-John

Hi @xchapter7x. Thanks for taking the time to write up a PR for this. At the time I did not have enough experience with pkg/kube to determine whether or not this was the right fix. Now that I'm more familiar with the codebase, this does seem like the right fix for the problem.
cc @thomastaylor312. What do you think about merging this one into Helm 2 and porting it over to Helm 3? Seems like it'd fix the issues we were discussing about the --atomic flag as well.

I implemented this using your handy function @xchapter7x in #6542 for v3. Based on what I see here with your PR, my one worry is that a release may still get created and cause issues when someone tries to delete it, but I've been going back and forth between the v2 and v3 codebase that I could be wrong.

@thomastaylor312 @bacongobbler
I'm glad this PR ended up adding some value for the community :)
Let me know if I can help with more contributions moving forward.
Keep up the great work 👍

Thanks @xchapter7x! Given the complexity in this change we aren't planning on moving this through for Helm 2, however thank you for catalyzing the movement on Helm 3!


a favorite song bubble added to profiles similar to the best girl bubble

Thank you for your suggestion, we will probably let people store their favorite songs, not necessarily just one!
Cheers,


How can I select which captions should be downloaded in get_transcript()?

If have a caption name string (example: "English (auto-generated)"), how can I select this specific caption to get downloaded?
How can I get the transcript's name from the API?
Thank you in advance!

Bump.

I'm more than happy to create an example for this - will look into this now for you

Ok so this is the first approach I've got working (this lists captions for all available languages):
from innertube import InnerTube
from pprint import pprint

# YouTube Web CLient
web = InnerTube("WEB", "2.20230920.00.00")

# Linus Tech Tips - I couldn't do my job without this. - PiKVM
player = web.player("232opnNPGNo")

for caption_track in player["captions"]["playerCaptionsTracklistRenderer"]["captionTracks"]:
    base_url: str = caption_track["baseUrl"]
    name: str = caption_track["name"]["simpleText"]
    language_code: str = caption_track["languageCode"]

    print(f"Captions for {name!r} [{language_code}]:")

    captions: dict = web.adaptor.session.get(base_url, params={"fmt": "json3"}).json()

    pprint(captions)
    print()

A few important points about this approach:

This uses YouTube's /api/timedtext endpoint, which is not a part of the InnerTube API
This makes use of the httpx.Client session the InnerTube client uses under the hood which is a bit hacky. If you want to take this approach, it's probably worth instantiating your own InnerTubeAdaptor.
The /api/timedtext endpoint returns XML by default, hence the addition of an fmt=json3 query parameter to get it to return JSON

I'll do some more digging to see if there's an easier/better way to do this using the InnerTube API

Here's a lashed-together example that pulls the data from the InnerTube API:
from innertube import InnerTube

PANEL_IDENTIFIER_TRANSCRIPT = "engagement-panel-searchable-transcript"

def extract_transcript_params(next_data):
    engagement_panels = next_data["engagementPanels"]

    for engagement_panel in engagement_panels:
        engagement_panel_section = engagement_panel[
            "engagementPanelSectionListRenderer"
        ]

        if (
            engagement_panel_section.get("panelIdentifier")
            != PANEL_IDENTIFIER_TRANSCRIPT
        ):
            continue

        return engagement_panel_section["content"]["continuationItemRenderer"][
            "continuationEndpoint"
        ]["getTranscriptEndpoint"]["params"]

def extract_transcript_languages(transcript_data):
    transcript_menu_items = transcript_data["actions"][0][
        "updateEngagementPanelAction"
    ]["content"]["transcriptRenderer"]["content"]["transcriptSearchPanelRenderer"][
        "footer"
    ][
        "transcriptFooterRenderer"
    ][
        "languageMenu"
    ][
        "sortFilterSubMenuRenderer"
    ][
        "subMenuItems"
    ]

    transcript_languages = []

    for transcript_menu_item in transcript_menu_items:
        title = transcript_menu_item["title"]
        continuation = transcript_menu_item["continuation"]["reloadContinuationData"][
            "continuation"
        ]

        transcript_languages.append(
            {
                "language": title,
                "params": continuation,
            }
        )

    return transcript_languages

def extract_transcript_segments(transcript_data):
    return transcript_data["actions"][0]["updateEngagementPanelAction"]["content"][
        "transcriptRenderer"
    ]["content"]["transcriptSearchPanelRenderer"]["body"][
        "transcriptSegmentListRenderer"
    ][
        "initialSegments"
    ]

# YouTube Web CLient
client = InnerTube("WEB", "2.20230920.00.00")

# Linus Tech Tips - I couldn't do my job without this. - PiKVM
data = client.next("232opnNPGNo")

transcript_params = extract_transcript_params(data)

transcript = client.get_transcript(transcript_params)

languages = extract_transcript_languages(transcript)

for language in languages:
    print(f"Transcript for {language['language']!r}:")

    language_transcript = client.get_transcript(language["params"])

    segments = extract_transcript_segments(language_transcript)

    for segment in segments:
        segment_key = next(iter(segment.keys()))

        if segment_key == "transcriptSectionHeaderRenderer":
            header_segment = segment["transcriptSectionHeaderRenderer"]

            snippet = header_segment["snippet"]["simpleText"]

            print(f"# {snippet}")
        elif segment_key == "transcriptSegmentRenderer":
            segment = segment["transcriptSegmentRenderer"]

            start_time = segment["startTimeText"]["simpleText"]
            snippet = segment["snippet"]["runs"][0]["text"]

            print(f"[{start_time}] {snippet}")
        else:
            raise Exception(f"Unknown segment renderer {segment_key!r}")

    print()

Here's a more concrete example that uses the timed-text API and parses the data using pydantic (highly recommend pydantic):
from typing import Final, Optional, Sequence

import httpx
from pydantic import AliasPath, BaseModel, ConfigDict, Field
from pydantic.alias_generators import to_camel

import innertube.api
from innertube import InnerTube
from innertube.config import config
from innertube.models import ClientContext

def get_client_context(client_name: str, /) -> ClientContext:
    client_context: Optional[ClientContext] = innertube.api.get_context(client_name)

    if client_context is None:
        raise Exception(f"No context available for client {client_name!r}")

    return client_context

WEB: Final[InnerTube] = InnerTube("WEB", "2.20230920.00.00")
WEB_CONTEXT: Final[ClientContext] = get_client_context("WEB")
WEB_SESSION: Final[httpx.Client] = httpx.Client(
    base_url=config.base_url,
    headers=WEB_CONTEXT.headers(),
)

class BaseCamelModel(BaseModel):
    model_config = ConfigDict(alias_generator=to_camel)

class CaptionTrack(BaseCamelModel):
    base_url: str
    name: str = Field(validation_alias=AliasPath("name", "simpleText"))
    vss_id: str
    language_code: str
    is_translatable: bool

class TimedTextEventSegment(BaseCamelModel):
    utf8: str
    t_offset_ms: Optional[int] = None
    ac_asr_conf: Optional[int] = None

class TimedTextEvent(BaseCamelModel):
    t_start_ms: int
    d_duration_ms: Optional[int] = None
    id: Optional[int] = None
    wp_win_pos_id: Optional[int] = None
    ws_win_style_id: Optional[int] = None
    w_win_id: Optional[int] = None
    a_append: Optional[int] = None
    segs: Optional[Sequence[TimedTextEventSegment]] = None

class TimedText(BaseCamelModel):
    wire_magic: str
    pens: Sequence[dict]
    ws_win_styles: Sequence[dict]
    wp_win_positions: Sequence[dict]
    events: Sequence[TimedTextEvent]

def get_caption_tracks(video_id: str, /) -> Sequence[CaptionTrack]:
    player: dict = WEB.player(video_id)

    caption_tracks: Sequence[dict] = player["captions"][
        "playerCaptionsTracklistRenderer"
    ]["captionTracks"]

    return [
        CaptionTrack.model_validate(caption_track) for caption_track in caption_tracks
    ]

def get_timed_text(url: str, /):
    response: httpx.Response = WEB_SESSION.get(url, params={"fmt": "json3"})

    response_data: dict = response.json()

    return TimedText.model_validate(response_data)

def print_timed_text(timed_text: TimedText, /) -> None:
    timed_text_event: TimedTextEvent
    for timed_text_event in timed_text.events:
        timed_text_event_segments: Sequence[TimedTextEventSegment] = (
            timed_text_event.segs or ()
        )

        composite_segment: str = "".join(
            timed_text_event_segment.utf8
            for timed_text_event_segment in timed_text_event_segments
        )

        print(f"[{timed_text_event.t_start_ms}] {composite_segment}")

# Linus Tech Tips - I couldn't do my job without this. - PiKVM
video_id: str = "232opnNPGNo"

caption_tracks: Sequence[CaptionTrack] = get_caption_tracks(video_id)

caption_track: CaptionTrack
for caption_track in caption_tracks:
    print(f"Fetching timed-text for {caption_track.name!r}...")

    timed_text: TimedText = get_timed_text(caption_track.base_url)

    print_timed_text(timed_text)
    print()

Thank you! I knew this solution existed, but didn't want to use 3 API calls. I guess there isn't a way around that.


JVM memory can't be tuned via BESU_OPTS parameters

Description
I am trying to improve the Besu performance (TPS, latency) through making better use of the available RAM resources. By default, JVM can only use up to 25% of RAM available to the Docker container. I wanted to make it to be 60% or higher by setting the BESU_OPTS parameters. But, it doesn't work.
Steps to Reproduce (Bug)

Configure a private Besu network on 4 VM instances each one with 8vCPU-30gbRAM. Use Docker to run a node on each VM, connecting via host network.

Run Docker container on each node in a bash file by setting the environment parameter of BESU_OPTS as following:
export HOSTNAME=$(hostname)docker run -d --rm \  --name $HOSTNAME \  --net host \  -v ${PWD}/data:/opt/besu/data \  -v ${PWD}/genesis.json:/config/genesis.json \  -v ${PWD}/config.toml:/config/config.toml \  -e BESU_OPTS='-Xms16G -Xmx24G' \  hyperledger/besu:latest \  --config-file=/config/config.toml

Check the JVM info in Docker container.
docker exec -it $(docker ps --format '{{.Names}}' | grep besu) java -XX:+PrintFlagsFinal -version | grep -Ei "maxheapsize|maxram"

Expected behavior:
Expected the MaxHeapSize to be around 24gb.
Actual behavior:
The MaxHeapSize is around 7.8gb, 25% of the total RAM of the VM.
size_t MaxHeapSize                              =<PHONE_NUMBER>                                {product} {ergonomic}  uint64_t MaxRAM                                   =<PHONE_NUMBER>72                           {pd product} {default}     uintx MaxRAMFraction                           = 4                                         {product} {default}    double MaxRAMPercentage                         = 25.000000                                 {product} {default}    size_t ShenandoahSoftMaxHeapSize                = 0                                      {manageable} {default}
Frequency: [What percentage of the time does it occur?]
Always.
Versions (Add all that apply)

Software version: [besu --version] 21.10/22.1/latest
Java version: [java -version] openjdk 11.0.11
OS Name & Version: [cat /etc/*release] Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-104-generic x86_64)
Docker Version: [docker version]. 20.10.8

I also tried to export the BESU_OPTS as an environmental variable, and use JAVA_OPTIONS instead.
None of them works as expected.
Any suggestions would be much appreciated. Thank you.

Step 3 shows the memory to a plain old java call whereas BESU_OPTS is used in the besu shell script.  Hence any settings to BESU_OPTS are not applied.
Try adding -XshowSettings:vm into the BESU_OPTS and java will show the memory settings prior to execution in step 2.  If you do not want memory settings in every startup you can alter step 3 to include java $BESU_OPTS -XX:+PrintFlagsFinal -version as the java command.

Thanks a lot, shemnon!


For the simulator, automate away getting compiled code of the contract

There's a bunch of accidental complexity involved when getting .wasm into the simulator:
https://github.com/near/near-sdk-rs/tree/31cba6213919ec4866bc0fab7f6963de241cc43a/near-sdk-sim#test-files

near_sdk_sim::lazy_static_include::lazy_static_include_bytes!

{ > // update contract.wasm for your contract's name > CONTRACT_WASM_BYTES => "target/wasm32-unknown-unknown/release/contract.wasm", > > // if you run cargo build without --release flag: > CONTRACT_WASM_BYTES => "target/wasm32-unknown-unknown/debug/contract.wasm", >}

Note that this means you must build before you test!

I think we should be able to automate this away.
I imagine API like
 pub fn read_rust_contract(path_to_crate: &Path, contract_name: &str) -> &'static [u8] {
 }

Internally, this thing would:

lock a global static hash-map of already loaded contracts
if the contract is not yet loaded, run cd path_to_crate && cargo build --release and std::fs::read(path_to_crate/target/release/contract_name.wasm)

I can imagne an excended version for when you want to tweak things:
 let contract = load_contract_opt()
 .package("./fungible-token")
 .bin("fungible.token") // required only for crates with many contracts
 .release(true)
 .load();

This problem is solved in workspaces by being able to get the wasm from compiling https://github.com/near/workspaces-rs/blob/7520d87d41d5f8ec6bcd6d8884dfd23b6623e1e5/README.md#updating-contract-afterwards