feat(assembly): Refactor Assembler to support incremental compilation

This is a draft implementation of the materialize_module(..) procedure for the Assembler.  This attempts to address the requirements of #894.
Introduce materialize_module(..) method in Assembler.

Not a review yet, just some initial thoughts about this.
I think there are 4 main use cases for why we need something like this:

We need to compute account code roots, and thus, we need to compile all of account methods.
We need to compute a script root for a given note.
We need to build an executable program for a given note.
We may want to check if a given note can be executed against a given account.

For the first point, we can use changes in this PR (though, I wonder if some other approach might be simpler).
The second and third points are more complex though, and require a bit more analysis. There are several cases to consider here, but in all of them we start out with a note script AST and need to get program MAST.
The AST could two types of external references:

procedure_id which is a hash of an absolute path for a given procedure.
mast_root this is what is described here (not yet implemented).

Let's consider procedure IDs, first. Procedures could come from two places:

Some well-known library (e.g., miden-lib).
Some account-specific module (i.e., a unique procedure defined in some account).

For the first case, things should be fairly straight-forward: we just need to load the library into the assembler (e.g., using with_library() method). Then, calling compile() against note script MASM should work, though we should probably expose something like compile_ast() as we won't have source MASM code for serialized notes. So, in this case, we don't actually need to have a materialized account module.
For the second case, we don't have a solution right now as we don't have account-specific procedure IDs. But I'm actually wondering if having account-specific procedure IDs is a good idea. A better solution may be to make calls to account-specific methods via mast_roots.
There are two was to handle mast root approach:

We load account modules into the assembler directly (basically, maintain a map similar to ProcedureCache in the assembler but for mast root |-> code block mapping). This could be done by adding something like load_module(module_ast) function to the assembler.
We build something like materialized module before hand, and then have something like compile_with_module(program_ast, mateiralized_module) method on the assembler.

My (very) preliminary thinking is that the first approach is better. If so, we may be able to get a way with adding just load_module() method which would look as follows:
pub fn load_module(&mut self, ast: ModuleAst) -> Result<Vec<Digest>, AssemblyError>

This method would compile procedures in the provided module, add them to this new cache structure I mentioned above, and return MAST roots of all procedures exported from the module.
Once a module is loaded into the assembler, we would be able to compile note scripts against it by simply calling compile() (or compile_ast()) method.

A few more thoughts on this.
1. Building account code root
The starting point for building account code root is a MASM file which defines account interface. For example, the file could be my_account.masm and it could look like so:
export.receive_assets
  ...
end

export.send_assets
  ...
end

export.auth_tx
  ...
end

If receive_assets and send_assets are from a standardized interface, then ideally, this would look something like this:
use.miden::abi::basic_wallet

export.basic_wallet::receive_assets
export.basic_wallet::send_assets

export.auth_tx
   ...
end

Once we have the file defining account interface, we need to convert it to an AST representation. This can be done via ModuleAst::parse() function. This will return a ModuleAst struct. This struct has a couple of purposes:

Serialized version of this struct would go on chain for public accounts.
It would be stored as part of the account code (e.g., here) locally on client devices (e.g., by wallets).
It would be used to compute account code root.

For this last point, we'd probably need to add load_module() method for the assembler I mentioned in the previous comment. This method would return a vector of MAST roots for all procedures exported by the parsed module AST. We could then sort these roots and build a Merkle tree out of them. The root of this tree would be the account code root.
One open question is which libraries should the assembler be initialized with. We should probably include stdlib and miden-lib, but maybe something beyond that would be useful as well.
2. Building note script root
The starting point for building note script root is also a MASM file. This MASM file will contain an executable program which will call some of account methods. For example, it could look something like this:
use.miden::abi::basic_wallet

begin
  ...
  call.basic_wallet::receive_assets
  ...
end

The above code could be executed against any account which exposes basic_wallet::receive_assets endpoint.
One open question is how scripts which need to call non-standardized account endpoints. One way to support this is to introduce a variant of call instruction which would work with raw MAST roots. This could look like so:
begin
  ...
  call.0x1234567
  ...
end

Where 0x1234567 would be a MAST root of a procedure exposed by the account interface. Note, that we could call standardized procedures in the same way too as all of them would have a specific MAST root (e.g., basic_wallet::receive_assets would reduce to some specific MAST root).
Once we have the MASM file, the first thing we need to do is to build an AST out of it. This could be done via the ProgramAst::parse() function. This function returns ProgramAst struct. This struct, similarly to ModuleAst has a couple of purposes:

Serialized version of this struct would go on chain for public notes.
It would be stored as part of the note script (e.g., here) locally on client devices (e.g., by wallets).
It would be used to compute note script root.

For the last point, we'd need to add a version of compile() method to the assembler to accept a ProgramAst struct rather than raw source string. Once the ProgramAst is compiled into a Program struct, we would be able to get the code root from it.
Note that if the note calls any non-standardized account methods, then account module would have be loaded into the assembler before we try to compile note script.
3. Building transaction program
To execute a transaction we first need to build a transaction program from note scripts and an optional tx script. Putting aside tx script for now (handling of which would be pretty similar to note scripts anyway), there are two was we can go about building a transaction program:

We could build individual script Program objects and then merge them together. This would probably be the simplest approach (since it would requite minimal changes in the assembler), but it would entail some inefficiencies. The main inefficiency would come from duplicate data in the cb_table contained in individual note programs.
We could modify the assembler to enable "incremental" compilation of multiple programs. This could be done by letting the user control the AssemblyContext. For this, we could add something like compile_with_context() method to the assembler which could look like so:

pub fn compile_with_context(
    &self,
    program: &ProgramAst,
    context: &mut AssemblyContext
) -> Result<CodeBlock, AssemblyError>

This method would compile the provided program in the specified context and return the MAST (represented by CodeBlock) of the program. But it would not reduce the context to cb_table. This way, the same context could be used to compile multiple programs, and in the end reduce the context to cb_table manually via AssemblyContext::into_cb_table().

One open question is which libraries should the assembler be initialized with. We should probably include stdlib and miden-lib, but maybe something beyond that would be useful as well.

We have discussed the potential of allowing the use of shared user defined libraries, I think this is an interesting idea.  A library should be uniquely defined by a root.  We can use this root as a key for libraries in a (root -> Library) map.  Maybe we can have a use declaration that allows the user to specify a root which identifies a shared library.  It would then be the responsibility of the operator to load this library into the assembler for compilation.

Note that if the note calls any non-standardized account methods, then account module would have be loaded into the assembler before we try to compile note script.

Could you elaborate a bit on this please?  What do you mean exactly when you refer to standard and non-standard account methods?   Another point that may be relevant to this conversation is private accounts where the account methods are private.  In this case the sender may only know the MAST root of the desired procedure of the receiver account code which has been provided to them out of band by the receiver. In this case we will not be able to load the account procedure into the assembler when compiling the note program.  I guess it would be the responsibility of the receiver to do this when they want to consume the note?

We have discussed the potential of allowing the use of shared user defined libraries, I think this is an interesting idea. A library should be uniquely defined by a root. We can use this root as a key for libraries in a (root -> Library) map. Maybe we can have a use declaration that allows the user to specify a root which identifies a shared library. It would then be the responsibility of the operator to load this library into the assembler for compilation.

Yep - the overall idea makes sense, though there are still some specific open questions in my mind. For example, where definition of such libraries live (i.e., on chain or off chain)? Who can update/create new libraries and how? How a particular note/account can specify upfront (i.e., without requiring parsing) which libraries it needs?

What do you mean exactly when you refer to standard and non-standard account methods?

By standardized methods I mean methods from interfaces which are well-know. For example, miden::abi::basic_wallet might be one such interface. It is well-known because it comes from miden-lib so, probably everyone will have it. I could imagine that in the future we might have something like an improvement proposal mechanism (e.g., MIP or PIP) to standardize more interfaces. These could go into miden-lib or maybe even some different library.

Another point that may be relevant to this conversation is private accounts where the account methods are private. In this case the sender may only know the MAST root of the desired procedure of the receiver account code which has been provided to them out of band by the receiver. In this case we will not be able to load the account procedure into the assembler when compiling the note program.

Good point. We should probably modify the assembler to not error out in case some entries in the cb_table are missing. These entries may be required for executing the program, but they may not be needed for computing the MAST root of the program. An example of this would be call.0x1234567 instruction where we have all the required info to compute MAST root in the instruction itself, but we won't be able to execute this instruction unless we know the program under 0x1234567 root.

I guess it would be the responsibility of the receiver to do this when they want to consume the note?

Yes, the receiver would need provide the implementation for any program which needs to be executed. One thing to note here is that we only need to know the path within the MAST which will be executed - we don't need to know the full program. This is not yet supported by the implementation (as we can't have "hidden" branches in the MAST just yet), but this is one of the reasons for something like Proxy block (which is currently unused).

I've updated the Assembler to support incremental compilation that should satisfy our requirements for building the transaction program and associated assets.
1. Building account code root
The account code module can be built using the compile_module(..) method.  This method returns Vec<RpoDigest> which are the roots of exported procedures.  We can then use this vector to construct the method tree and root for the AccountCode.  At the end of this method the account module methods will be added to the assembler procedure cache for downstream usage in note scripts.
2. Building note script root
To support the compilation of note scripts we have introduced the compile_in_context(..)  method.  This method takes a ProgramAst, compiles it and returns a CodeBlock which represents the root of the body of the note script.  We want to make the note script body available downstream such that the transaction program can call the note scripts.  To achieve this when we compile the program body we add it to the procedure cache.  The prior AssemblyContext enforced that all programs would have a path of #exec and the program body would have a procedure name of #main. This results in some challenges as we have many notes each containing their own body that we want to compile without naming conflict.  As such we have introduced a constructor to the AssemblyContext - AssemblyContext::new_program(..) such that we can specify the procedure name and path to avoid conflicts when many executable programs incrementally.
3. Building transaction program
The requirements here are to ensure that the Assembler has the appropriate procedures in it's procedure cache (including the note script bodies) - this is achieved by incrementally compiling the note scripts without consuming the Assembler.
A test case has been provided that shows the expected compilation pipeline for transaction program and associated assets.

Thank you! I left a couple of comments inline. One other thing I was thinking (though probably for a different PR): we should probably change ProcedureCache into a real struct. One reason for this is that we'd want to be able to look up procedures by two things: procedure ID and procedure MAST root.
The interface could look something like:
impl ProcedureCache {

    pub fn insert(&mut self, proc: Procedure) {
        todo!()
    }

    pub fn get_by_id(&self, id: ProcedureId) -> Option<&Procedure> {
        todo!()
    }

    pub fn get_by_hash(&self, root: RpoDigest) -> Option<&Procedure> {
        todo!()
    }

   pub fn contains_id(&self, id: ProcedureId) -> bool {
        todo!()
    }

    pub fn contains_hash(&self, root: RpoDigest) -> bool {
        todo!()
    }
}

This let us support call.<MAST root> instruction - thought, we'd probably also need to update how Callset works as well.

This makes sense to me.  Looking to the future we will also need to be mindful of being able to support the pcall operation.  The parser will not be able to infer the pcall target from the AST so it will be the responsibility of the client to specify the expected pcall targets to ensure they are included in the Callset.

Do we actually need to load account procedures into the assembler cache by name? It seems like it could lead to collisions. One example of such a collision I listed in the comments (and I think we could probably find a way how to fix that), but another example would be if we want to load multiple accounts into the assembler and they might have procedures with similar names but different implementations. One way to handle this would be to first address #904 and then come back to implement this using the call.0x1234567 syntax.

As noted in the reply to your comment, provided the fully qualified module path is unique then I don't think this will lead to collisions in the ProecedureCache (however it will do in the parser).  If we needed to load two accounts then we could specify non-conflicting module paths appropriately - potentially using account_id as part of the path (e.g. context::account::account_id).  I'm not convinced about the usage of using MAST roots for call targets in this context - it seems to me that it reduces readability.  Is there a hard requirement for their usage?

How do we build transaction programs? My initial thinking was that we build them manually by joining compiled note script CodeBlock's and CodeBlock's for prologue, epilogue etc. Under this approach, we won't need to modify the assembler as much (as we won't need to import note scripts via use.context::note1) and it would make transition to pcall a bit simpler. Though, the code for manual joining would probably be a bit more complex than using the assembler.

I think the first question we should answer here is should the Assembler be responsible for the construction of the transaction program.  In my opinion, the introduction of a TxCompiler indicates some limitation of the Assembler. The transaction program is a Program like any other and as such we should be able to compile it using the Assembler.  On the other hand I guess it could be argued that the compilation pattern for the transaction program is unique enough that it is out of scope for the Assembler to support.  However, I tend to lean on the side of the former opinion - the Assembler should be flexible enough to support this.

I'm not convinced about the usage of using MAST roots for call targets in this context - it seems to me that it reduces readability. Is there a hard requirement for their usage?

I don't think it is a hard requirement - but I do think it is a better overall approach. Here is how I'm thinking about it:
In my mind, procedures which an account can expose fall into two categories:

Standardized procedures - e.g., miden::abi::basic_wallet.
Non-standard, account-specific procedures (similar to the ones you have in the tests here).

Calling procedures from the first category should probably be done using the approach we have now - i.e., call.basic_wallet::receive_assets. This has nice advantage of not tying the call to a specific account - i.e., this call can be done against any account which exposes miden::abi::basic_wallet::recieve_assets procedure (the resolution will be done via MAST root, but procedure ID reference is also unambiguous).
For the second category, things are a bit more complicated.
First, if we use something like context::account we may run into collision issues at the parser level. Also, in the above context now has a special meaning (i.e., nobody should create a library with context as a root namespace) and having it as such feels a bit brittle. So far, for special names, I've tried to use something that couldn't be declared by the user (e.g., #sys or #exec). We could do the same for context - i.e., use #context - but this would require quite a few updates to the parsing logic (and wouldn't solve parser collisions mentioned above).
Another issue is that if we have a user wallet which manages several accounts (or a block producer which needs to execute public transactions against many accounts), we'd probably want to re-use the same instance of the assembler between them (to avoid re-compiling common procedures). With context::account we cannot do that because the account part would actually be different for different accounts. A way to avoid it is to use account ID somewhere in the path. For example, something like this:
use.context::0x1234567 # this is the account ID

begin
  ...
  call.0x1234567::procedure_1
  ...
end

This fully disambiguates procedure IDs, but has a couple of issues of its own:

Whoever crafts the note needs to know the account ID. This may be both difficult and undesirable (for privacy reasons).
We cannot execute such notes against multiple accounts who share the same (though non-standard) interface.

Overall, for calling non-standard procedures, we probably want to have the following properties:

Ability to call a procedure without having to specify an account ID.
Ability to call a procedure against multiple accounts which expose the same non-standard interface.
Ability to have multiple accounts loaded into the assembler cache while keeping references unambiguous.

I think that call.0x1234567 (where 0x1234567 is the MAST root of the procedure) approach achieves these goals. We basically say that we can execute this note against any account which exposes a procedure with the specified MAST root. Another nice benefit is that it binds the call very explicitly and avoids potential issues which we could run into for accounts with updatable code. For example, say our note script is written as:
use.context::0x1234567 # this is the account ID

begin
  ...
  call.0x1234567::procedure_1
  ...
end

If account 0x1234567 has updatable code, the implementation of procedure_1 could be changed to mean something completely different from the original procedure. Then, if we are not careful, we may compile our note against the new implementation without realizing that it is no longer what we want.
For standardized procedures this should be much less of an issue because, hopefully, these either don't change or change very infrequently with enough lead time for users to adjust (and if someone wants, they can also call standardized procedures by their hash rather than procedure name).
I do agree that readability suffers somewhat - though, I hope that this can be addressed via tooling / higher-level languages.

Calling procedures from the first category should probably be done using the approach we have now - i.e., call.basic_wallet::receive_assets. This has nice advantage of not tying the call to a specific account - i.e., this call can be done against any account which exposes miden::abi::basic_wallet::recieve_assets procedure (the resolution will be done via MAST root, but procedure ID reference is also unambiguous).

Wouldn't we want some compile time guarantees that the miden::abi::basic_wallet::recieve_assets procedure is actually exposed by the account interface?  A user may accidentally use some procedure which is not actually part of the target account interface.

For the second category, things are a bit more complicated.
First, if we use something like context::account we may run into collision issues at the parser level. Also, in the above context now has a special meaning (i.e., nobody should create a library with context as a root namespace) and having it as such feels a bit brittle.

Agreed, I do think this is a bit brittle.  Having said that maybe we could leverage the add_module method of the ModuleProvider which checks for conflicts.

So far, for special names, I've tried to use something that couldn't be declared by the user (e.g., #sys or #exec). We could do the same for context - i.e., use #context - but this would require quite a few updates to the parsing logic (and wouldn't solve parser collisions mentioned above).

I think there is a bit of nuance to this.  #sys and #exec are special in the context of the virtual machine.  context is special in the context of the application (the miden rollup).  I don't think it makes sense to push application specific requirements into the Assembler the add_module solution referenced above would probably be fine.

Another issue is that if we have a user wallet which manages several accounts (or a block producer which needs to execute public transactions against many accounts), we'd probably want to re-use the same instance of the assembler between them (to avoid re-compiling common procedures). With context::account we cannot do that because the account part would actually be different for different accounts. A way to avoid it is to use account ID somewhere in the path. For example, something like this:
use.context::0x1234567 # this is the account ID

begin
  ...
  call.0x1234567::procedure_1
  ...
end

This fully disambiguates procedure IDs, but has a couple of issues of its own:

Whoever crafts the note needs to know the account ID. This may be both difficult and undesirable (for privacy reasons).

For the purpose of privacy I was of the understanding that it would be the receiver that would construct the note script and provide the recipient to the sender.  The assumption would then be that the receiver knows the account id.

We cannot execute such notes against multiple accounts who share the same (though non-standard) interface.

Overall, for calling non-standard procedures, we probably want to have the following properties:

Ability to call a procedure without having to specify an account ID.
Ability to call a procedure against multiple accounts which expose the same non-standard interface.
Ability to have multiple accounts loaded into the assembler cache while keeping references unambiguous.

One point around ambiguity here is we do not know which procedures are associated with which accounts, we will just know that at least one of the accuonts loaded in the assembler cache has this procedure.

I think that call.0x1234567 (where 0x1234567 is the MAST root of the procedure) approach achieves these goals. We basically say that we can execute this note against any account which exposes a procedure with the specified MAST root. Another nice benefit is that it binds the call very explicitly and avoids potential issues which we could run into for accounts with updatable code. For example, say our note script is written as:
use.context::0x1234567 # this is the account ID

begin
  ...
  call.0x1234567::procedure_1
  ...
end

If account 0x1234567 has updatable code, the implementation of procedure_1 could be changed to mean something completely different from the original procedure. Then, if we are not careful, we may compile our note against the new implementation without realizing that it is no longer what we want.

Agreed but I would say this is an edge case.  The base case is worse for MAST calls as we do not know the context / namespace from which the MAST procedure originates - i.e. as mentioned before we do not know the MAST procedure is part of the target account interface.
I see pros and cons of both approaches here.  My conviction around either approach isn't particularly high so I'm happy to take your lead. I will back out changes related to tx program compilation.

In my opinion, the introduction of a TxCompiler indicates some limitation of the Assembler.

In my mind, TxCompiler is a bit more general. Specifically, I'm imagining a component which takes an account (or maybe just an account ID) and a set of notes, and outputs a CompiledTransaction object. Internally, it maintains an instance of the assembler to perform compilation of note scripts etc. but it may be responsible for other things (e.g., loading account modules into the assembler). For example, part of the interface could look something like this:
impl TxCompiler {
    pub fn load_account(&mut self, account_code: ModuleAst) -> AccountCode {
        todo!()
    }

    pub fn compile_transaction(
        &self,
        account_id: AccountId,
        notes: Vec<Note>,
        tx_script: Option<ProgramAst>,
    ) -> CompiledTransaction {
        todo!()
    }
}

If it can push more work onto the assembler, I think that's fine - but it probably still needs to do a few things to orchestrate the compilation process.

Wouldn't we want some compile time guarantees that the miden::abi::basic_wallet::recieve_assets procedure is actually exposed by the account interface? A user may accidentally use some procedure which is not actually part of the target account interface.

Yes, we would - but I'm not sure this could be guaranteed by the assembler (or via MASM code in general). I think we'll need to have something else that is aware of the state of the blockchain to perform these checks. (maybe it could be a part of the TxCompiler mentioned in the comment above).

For the purpose of privacy I was of the understanding that it would be the receiver that would construct the note script and provide the recipient to the sender. The assumption would then be that the receiver knows the account id.

This is true. I was thinking more about a use case with multiple potential recipients - but that's a separate concern.

One point around ambiguity here is we do not know which procedures are associated with which accounts, we will just know that at least one of the accuonts loaded in the assembler cache has this procedure.

Yeah, that's an issue - but I'm not sure how to get around this as in some cases we probably want to send a note to any account which can execute it (e.g., for a swap). So, we can't specify account ID upfront.

I see pros and cons of both approaches here. My conviction around either approach isn't particularly high so I'm happy to take your lead. I will back out changes related to tx program compilation.

Yeah - I also see pros and cons. I do think MAST-based calls are better than using context::account imports, but they are not ideal either. Maybe there is a better solution which accommodates all use cases we are thinking about.

I do think MAST-based calls are better than using context::account imports

And just to state the reason concisely: I think we should be able to send notes to multiple recipients (i.e., any account which implements a given interface, which may or may not have been standardized) and MAST-based calls support this while I'm not sure how it could work with context::account import approach.

One point around ambiguity here is we do not know which procedures are associated with which accounts, we will just know that at least one of the accuonts loaded in the assembler cache has this procedure.

To come back to this again: I think this is a real issue and we should avoid this for sure. I'll need to think a bit more about this.

When compiling notes (e.g., note x against account a), we'd like to achieve the following goals:

We may want to get compile-time guarantees that x can be executed against a and only against a.
If the above is not possible, we'd like to get guarantees that x can be executed against a (and maybe against other accounts).
If we cannot do the above, what can we do?

Let's look into these one-by-one.
Given the current structure of MASM, it is pretty clear that achieving the first point is impossible. If x calls a.foo, there is nothing preventing account b from also implementing foo. The only way we can ensure that x can be executed only against a is to check some properties of a dynamically. For example, something like this:
use.miden::sta::account
use.context.0x1234567 # this is the account ID

begin
  exec.account::get_id
  assert_eq
  call.0x1234567::foo
end

The above assumes that the first note input is set to account ID of a - so, what we do here is compare ID of the current account to the ID provided via note inputs, and if they match, we call a.foo. Without knowing what note inputs are and executing the code, we won't know whether the note can be executed against a and only against a.
Moving on to the second point, I would argue that this is also impossible to do at compile time. Consider a note with the following script (there are two accounts both implementing foo):
use.context.0x1234567
use.context.0x7654321

begin
  push.1
  if.true
    call.0x1234567::foo
  else
    call.0x7654321::foo
  end
end

This note would compile against both accounts, but clearly, it can be executed only against one. The above is just a simple illustrative example (which could be caught by dead code elimination), but real notes could be way more complex with branching conditions coming from note inputs.
There is one other thing that I find problematic with the above code: it gives a perception that foo does the same thing when called as 0x1234567::foo and as 0x7654321::foo - though, in fact these could be very different procedures. In fact, if we are compiling this note in the context of 0x1234567, we may not even know what 0x7654321::foo is (i.e., if 0x7654321 is a private account).
So, if we can't guarantee at compile time whether x can be executed against a - what can we do?
For one, we should be able to tell if x cannot be executed against a. Let's consider this script for example (here I'm using MAST root call notation):
begin
  call.0x123456789
end

At compile time, it should be pretty easy to tell if this note can be executed against a. To do this, all we need to do is check if a exposes a procedure with MAST root 0x123456789.
This also means that we should be able to perform a similar check against:
use.miden::abi::basic_wallet

begin
  call.basic_wallet::recieve_assets
end

This works because we can statically resolve miden::abi::basic_wallet::recieve_assets to its MAST root.
The situation is a bit more tricky if we have something like this:
begin
  <some code here>
  if.true
    call.0x123456789 # is expected to be called on account 0x123
  else
    call.0x987654321 # is expected to be called on account 0x456
  end
end

Without extra information, the best thing we can do is to check whether a exposes either procedure 0x123456789 or procedure 0x987654321, and if it does, we can assume that it may be executed. This is because we don't know which branch will be taken at compile time. There could be more complex situations but as long as all procedures in one branch can be executed against a, we assume that x may be executed against a and this can be checked using fairly simple static analysis.
This static analysis cannot be performed by the assembler (because the assembler doesn't know these rules) - but it can be performed by something like a TxCompiler. One way to do it would be this:

Compile the program with the assembler.
Traverse the MAST in DFS order.
Use CALL nodes to build a set of calls made in the same branch.
Check if all targets of these calls are present in target account's interface.

We could also improve on the above by providing the extra information in the program itself. For this, we should probably use new syntax. Maybe something like this:
begin
  <some code here>
  if.true
    call.0x123456789@0x123
  else
    call.0x987654321@0x456
  end
end

During compilation to MAST, the assembler could add these attributes to CALL nodes, and then TxCompiler would interpret them accordingly. Though it is worth noting again that with this type of static analysis we can can't give true guarantees that we'll be able to execute a note against a - but we may be able to catch more instances of when we wouldn't be able to do so.

Assuming my last comment was not too wrong, I think the way to proceed on this is:

Keep this PR open for now.
Implement simple MAST-based call instruction (without attributes). This will require #904.
Merge the result into this PR and update accordingly.
Merge this PR into next.
In the future, add attribute syntax to MAST-based call instruction.

@bobbinth if you could support with the failing test that would be appreciated.

I pushed a commit which fixes the compilation error. The test which uses this code is currently ignored (as it is failing and need to be updated to include latest changes from Winterfell). This is something @Al-Kindi-0 will address in future PRs.


tooltips

integratting tippy tooltips would also be nice
https://kabbouchi.github.io/vue-tippy/4.0/demo.html
maybe via some html or custom markdown ⁉ shortcut

I don't really get this one, what's the use case?

I really can't justify them, other than being cool.
I mean - you put emoji... would be nice to have tooltips.
Even wikipedia has them, when you hover. and it looks real nice.
So why the heck not? 😃


SDKV4 and OTA compatibility with existing SDKV3 products using NVS and flash encryption (IDFGH-6162)

We have a product using flash encryption, but no NVS encryption.
The (nvs) partitions are defined as:
otadata,   data,      ota,    0x18000,    0x2000,  encrypted
phy_init,  data,      phy,    0x1A000,    0x1000,  encrypted
nvs_keys,  data, nvs_keys,    0x1B000,    0x1000,  encrypted
nvs,       data,      nvs,    0x1C000,   0x14000,  encrypted
#factory,   app,  factory,    0x30000,   0x80000,  encrypted
ota_0,      app,    ota_0,   0x0B0000,  0x3A0000,  encrypted
ota_1,      app,    ota_1,   0x450000,  0x3A0000,  encrypted
coredump,  data, coredump,   0x7F0000,   0x10000,
The sdkconfig of SDKV3 is:

Bootloader config

CONFIG_APP_ROLLBACK_ENABLE=y
CONFIG_APP_ANTI_ROLLBACK=y
CONFIG_APP_SECURE_VERSION=0
CONFIG_APP_SECURE_VERSION_SIZE_EFUSE_FIELD=32
CONFIG_EFUSE_SECURE_VERSION_EMULATE=

Security features

CONFIG_SECURE_SIGNED_ON_BOOT=y
CONFIG_SECURE_SIGNED_ON_UPDATE=y
CONFIG_SECURE_SIGNED_APPS=y
CONFIG_SECURE_BOOT_ENABLED=y
CONFIG_SECURE_BOOTLOADER_ONE_TIME_FLASH=y
CONFIG_SECURE_BOOTLOADER_REFLASHABLE=
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=
CONFIG_SECURE_BOOT_VERIFICATION_KEY="XXX-SECUREBOOT-PUBLIC-V1.bin"
CONFIG_SECURE_BOOT_INSECURE=
CONFIG_FLASH_ENCRYPTION_ENABLED=y
CONFIG_FLASH_ENCRYPTION_INSECURE=

NVS

CONFIG_NVS_ENCRYPTION=

The sdkconfig of SDKV4 is:

Security features

CONFIG_SECURE_SIGNED_ON_BOOT=y
CONFIG_SECURE_SIGNED_ON_UPDATE=y
CONFIG_SECURE_SIGNED_APPS=y
CONFIG_SECURE_SIGNED_APPS_ECDSA_SCHEME=y
CONFIG_SECURE_BOOT=y
CONFIG_SECURE_BOOT_V1_ENABLED=y
CONFIG_SECURE_BOOTLOADER_ONE_TIME_FLASH=y
CONFIG_SECURE_BOOTLOADER_REFLASHABLE is not set
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES is not set
CONFIG_SECURE_BOOT_VERIFICATION_KEY="WIFI2LIN-SECUREBOOT-PUBLIC-V1.bin"
CONFIG_SECURE_BOOT_INSECURE is not set
CONFIG_SECURE_FLASH_ENC_ENABLED=y
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT is not set
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_RELEASE=y
end of Security features

NVS

CONFIG_NVS_ENCRYPTION is not set
end of NVS

When updating a SDKV3 device to an SDKV4 application via OTA, the new application is launched, but as soon as NVS initialization code is called:
// Initialize NVS
esp_err_t ret = nvs_flash_init();
if (ret == ESP_ERR_NVS_NO_FREE_PAGES) {
ESP_ERROR_CHECK(nvs_flash_erase());
ret = nvs_flash_init();
}
ESP_ERROR_CHECK( ret );
The bootloader reverts to old version.
Is it impossible to build a SDKV4 software, which is compatible with existing NVS content and partition setup, which worked in SDKV3 application?
Any clues are appreciated.

as soon as NVS initialization code is called: The bootloader reverts to old version.

You mean on the next boot the old app will run or...? Do you have logs?

Yes.
During the first run of new app, the NVS initialization fails with an error, that nvs partition cannot be encrypted:
ESP_ERROR_CHECK failed: esp_err_t 0x1119 (ESP_ERR_NVS_WRONG_ENCRYPTION) at 0x40095388
file: "../src/main/esp32init.c" line 20
That causes device to reboot, and the the old app runs again (due to bootloader feature, where new app must confirm successful boot).
But I have disabled NVS encryption, which was also disabled in old app.
The flash is, however, encrypted via the built-in hardware encryption.
I guess I need to add debug code and then figure out why the NVS partition cannot have the encrypted flag set...
But I find it strange, and not optimal, if I cannot update SDKV3 devices....

Nvs partition is never supposed to have encrypted flag set, only keys partition.  Encrypted nvs uses separate software encryption not hardware flash encryption.

I understand that - now. But it was not a problem in SDKV3. And did not report any compile error.
But when NVS encryption is not used, why did the SDK change behaviour?
It worked before...

Probably due to the unified partition api, it should be easy to patch it to ignore the encrypted flag on nvs partitions here
https://github.com/espressif/esp-idf/blob/417ef24b0641856ea71a7aeb44ccee62b11b5cc8/components/spi_flash/partition.c#L227-L237

https://github.com/espressif/esp-idf/issues/5747#issuecomment-956244068

@negativekelvin 's suggestion is probably better than ad-hoc patching of partition data structure, I think we'll implement it.

@negativekelvin @igrr Thank you so much for the help.
We have tried to make patching of partitions before, but I think we would need temporary storage of the NVS data as well. So not an easy task.
@igrr When do you think Espressif will decide, if the SDK is updated to allow secure "nvs" partition, if encrypted NVS is disabled? And do you think it is risky to follow this "incorrect" use of NVS? The more devices, we have out, the more risky the partition update is...

@BitProgress nvs partition is never hardware encrypted, so it is safe for the sdk to ignore the flag in the partition table.  That's all the patch would do, there is no rewriting of partitions.

@negativekelvin Thank you for the information.
I thought flash handling obeyed the "flash" encryption flag.
But to update existing devices with NVS encryption, we first need to store the NVS content somewhere else, then load a new ota image (with NVS encryption enabled) and then save the data in NVS again. Right?

Yes if you want to achieve "encrypt in place" you would have to copy the data, reinitialize the partition, and rewrite all the data.  You may be able to come up with a way to do it more efficiently one page at a time if you want to dig into low level stuff.

@BitProgress
Could you please help to check attach patch
0001-nvs-add-config-to-ignore-encrypted-flag-of-nvs-parti.patch.tar.gz?

This is added under new config option, please enable it before building the application
Patch is against release/v4.3 branch

@mahavirj I confirm, that the system now allows a new update built on SDK V4.3.1.
We will move forward to perform an update of partition table (also ad-hoc patching), and will try to update NVS content to use encryption.
Thanks a lot for the help.


Ask Password Option: Confusing and repetitive steps for the user

In [1] the expiry time is set in the first step and the email verifcation is enabled in a latter step. In order to get this working the user has to restart the server, log in to management console twice. This maybe tiresome for the user who wants to try out this particular flow. Hence to reduce effort on user side to enable configs, please provide the configs in one go.
[1] https://docs.wso2.com/display/IS540/Creating+Users+using+the+Ask+Password+Option

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


HBASE-29009 Depend on jaxws-rt instead jaxws-ri

This removes 12 unneccessary JARs but adds duplicate versions in the assembly for two.
I plan to deal with the duplicate dependencies in a separate JIRA.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 29s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 18s

master passed

+1 :green_heart:
compile
7m 55s

master passed

+1 :green_heart:
spotless
0m 44s

branch has no errors when running spotless:check.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 59s

the patch passed

+1 :green_heart:
compile
7m 56s

the patch passed

+1 :green_heart:
javac
7m 56s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
xmllint
0m  0s

No new issues.

+1 :green_heart:
hadoopcheck
11m 27s

Patch does not cause any errors with Hadoop 3.3.6 3.4.0.

+1 :green_heart:
spotless
0m 44s

patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 27s

The patch does not generate ASF License warnings.

43m 25s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6504

Optional Tests
dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile

uname
Linux b70256bde4dc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / a7f76173b3644eec976385a47bf81af5992221e2

Default Java
Eclipse Adoptium-17.0.11+9

Max. process+thread count
188 (vs. ulimit of 30000)

modules
C: hbase-it hbase-assembly . U: .

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 33s

the patch passed

+1 :green_heart:
compile
1m 51s

the patch passed

+1 :green_heart:
javac
1m 51s

the patch passed

+1 :green_heart:
javadoc
2m 20s

the patch passed

+1 :green_heart:
shadedjars
4m 54s

patch has no errors when building our shaded downstream artifacts.

_ Other Tests _

+1 :green_heart:
unit
283m 37s

root in the patch passed.

314m 51s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6504

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 08c9c4147471 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / a7f76173b3644eec976385a47bf81af5992221e2

Default Java
Eclipse Adoptium-17.0.11+9

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/1/testReport/

Max. process+thread count
8370 (vs. ulimit of 30000)

modules
C: hbase-it hbase-assembly . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/1/console

versions
git=2.34.1 maven=3.9.8

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

I'm holding this until we remove jax-ri from the assembly.
After the removal, this will only apply to the test classpath (but simplifying that is also worth doing).

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
2m 28s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 16s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 56s

master passed

+1 :green_heart:
compile
7m 41s

master passed

+1 :green_heart:
spotless
0m 42s

branch has no errors when running spotless:check.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 53s

the patch passed

+1 :green_heart:
compile
7m 41s

the patch passed

+1 :green_heart:
javac
7m 41s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
xmllint
0m  1s

No new issues.

+1 :green_heart:
hadoopcheck
10m 39s

Patch does not cause any errors with Hadoop 3.3.6 3.4.0.

+1 :green_heart:
spotless
0m 41s

patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 26s

The patch does not generate ASF License warnings.

43m 20s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6504

Optional Tests
dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile

uname
Linux 591dc38d4c93 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / ed88f8ef0e0d3bd9ba47a3a46e08976fe4a25351

Default Java
Eclipse Adoptium-17.0.11+9

Max. process+thread count
190 (vs. ulimit of 30000)

modules
C: hbase-it hbase-assembly . U: .

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 38s

the patch passed

+1 :green_heart:
compile
1m 55s

the patch passed

+1 :green_heart:
javac
1m 55s

the patch passed

+1 :green_heart:
javadoc
2m 22s

the patch passed

+1 :green_heart:
shadedjars
4m 52s

patch has no errors when building our shaded downstream artifacts.

_ Other Tests _

+1 :green_heart:
unit
306m 59s

root in the patch passed.

341m 51s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6504

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 81fa0cf60a44 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / ed88f8ef0e0d3bd9ba47a3a46e08976fe4a25351

Default Java
Eclipse Adoptium-17.0.11+9

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/2/testReport/

Max. process+thread count
9271 (vs. ulimit of 30000)

modules
C: hbase-it hbase-assembly . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/2/console

versions
git=2.34.1 maven=3.9.8

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m  0s

Docker mode activated.

-1 :x:
patch
0m  6s

https://github.com/apache/hbase/pull/6504 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/6504

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/3/console

versions
git=2.25.1

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m  0s

Docker mode activated.

-1 :x:
patch
0m  6s

https://github.com/apache/hbase/pull/6504 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/6504

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6504/3/console

versions
git=2.17.1

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.


Fix free space check

I must have tested this with a file that fit exactly. Another case of a single character breaking everything...

 I'll tag another release.

So, next up is flash-defrag? :)

Haha, think that's quite far down the list! :smile:

Is that also a fix for #470?


Adding pia wireguard to wiki

Not sure where to post this. But I found a rather easy way to setup PIA wireguard with help of the github below by kilegrantlucas.
Via the link below I was able to easely pull off a PIA-wg.conf file.
https://pkg.go.dev/github.com/kylegrantlucas/pia-wg-config#section-readme
Perhaps there can be a writeup for PIA wireguard in the wiki for those interested?

VPN_SERVICE_PROVIDER=custom
VPN_TYPE=wireguard
VPN_ENDPOINT_IP=<IP_ADDRESS>
VPN_ENDPOINT_PORT=51820
WIREGUARD_PUBLIC_KEY=wAUaJMhAq3NFutLHIdF8AN0B5WG8RndfQKLPTEDHal0=
WIREGUARD_PRIVATE_KEY=wOEI9rqqbDwnN8/Bpp22sVz48T71vJ4fYmFWujulwUU=
WIREGUARD_ADDRESSES=<IP_ADDRESS>/32

It can either be done by entering the enviromental variables from the conf fill required by following the link above.
Or... you can also bind mount this wireguard configuration file (ini format) to /gluetun/wireguard/wg0.conf.

Thank you! Is it possible to setup port forwarding with this?

seconded. is there a way to add port forwarding with this?

Haven't tried it but perhaps with
PIA_PF=true

doesnt seem to do anything...

https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/port-forwarding.md
Add environments

yeah I've tried those. when using VPN_PORT_FORWARDING=on, gluetun states that it cannot be used with "custom" service provider.

@Kieros Thanks, that's a great tool and I may even incorporate some of its code in Gluetun later on. I've documented it in 78b457bf66e02769de0d0fd96961b5c32f1a330f thanks!
@Srijal97 @xtinct101 @Kieros For port forwarding with Wireguard custom provider, subscribe to issue https://github.com/qdm12/gluetun/issues/2320 it's work-in-progress. I'm hiding your comments as off topic in this issue.

@Kieros still not able to configure this. i dont understand where to get this ip port and addresses.


mrkl and search-and-chat examples not working

just tried the above mentioned two examples with my own freshly created openai api key and got error messages. one was:
duckduckgo_search.exceptions.DuckDuckGoSearchException: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). [...]

some further remarks: i refered to the example implementation given in the readme file. search-and-chat is working flawlessly on my machine.

Search-and-chat does not work for me as well. It is for a different reason, but it might under this issue still. There error is this, and may be related to recent updates to Langchain, perhaps?
AssertionError: The input to RunnablePassthrough.assign() must be a dict.
Traceback:
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 542, in _run_script
exec(code, module.dict)
File "/home/user/streamlit/lc1/app.py", line 98, in
response = agent_with_chat_history.invoke(prompt,
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4525, in invoke
return self.bound.invoke(
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4525, in invoke
return self.bound.invoke(
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2499, in invoke
input = step.invoke(
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4525, in invoke
return self.bound.invoke(
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/passthrough.py", line 469, in invoke
return self._call_with_config(self._invoke, input, config, **kwargs)
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1626, in _call_with_config
context.run(
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/config.py", line 347, in call_func_with_variable_args
return func(input, **kwargs)  # type: ignore[call-arg]
File "/home/user/miniconda3/envs/shiny_py/lib/python3.10/site-packages/langchain_core/runnables/passthrough.py", line 450, in _invoke
assert isinstance(


Resolve Dev Issues Relating to Monorepo Integration

Please verify the version of auro-form you have installed
1.0.0
Please describe the bug
Bugs relating to the dev process were introduced after integrating Turbo and converting auro-form to a monorepo in #19.
Reproducing the error
Run npm run dev

Completed as part of #19 PR


OAuth Confusion

To reproduce with one Matrix user ID (using riot.im/develop -> room settings -> Manage Integrations -> Twitter)

OAuth using twitter account A using the "Sign in with Twitter" button on Riot, get a new timeline room for A
"Remove Credentials"
Repeat 1. but with twitter account B
See timeline tweets from account A being sent into the new timeline room B

This is not necessarily vulnerable to attack, because OAuth has to be done for account A in the first place. But now I can't use my (account B's) timeline room because it will presumably send tweets as account A.

Restarting the bridge fixes temporarily, and so presumably the handling around some inner cache is failing OR unOAuth needs to blow away some Twitter client state.


Update interval

I set up and kept all the settings default. And the data refresh every 5 or more seconds,but the demo page seems to be realtime. How to configure?

The default update interval is 2 seconds for the webpage.
The clients report data every seconds by default.
Maybe your webserver/browser/cloudflare is caching the json response?
Check the headers on /json/stats.json

Hi ,
I checked the header and found that there many 304 Not Modified. I compared your demo and all your status of stats.json are 200. I thought it was gzip to blame,but after I disabled it,the 304 still exist. My webserver is nginx.

Does your filesystem have modified date metadata turned off maybe?
You should check if the file modified actually changes (just spam ls -l every second in your terminal)

Date: Thu, 28 Jul 2016 08:27:35 GMT
Last-Modified: Thu, 28 Jul 2016 08:27:08 GMT

Looks like your file isn't being written to very often..
Though the server sleeps 1 second and then writes the new file: https://github.com/BotoX/ServerStatus/blob/master/server/src/main.cpp#L268

It's also possible that something in the script blocks for a few seconds. For example if you have a network file system mounted, each call to df can take a few seconds.

The server will write a new stats.json every second nonetheless.
https://github.com/BotoX/ServerStatus/blob/master/server/src/main.cpp#L202

Sorry for the late update. I changed the host server and the problem solved. I guess it maybe the network problem.


Let travis-ci push javadoc to the website

Should fix javaslang/javaslang#135

Coverage remained the same at 100.0% when pulling 080e3db867500f7d6c01378bc753bbc06ae574f7 on danieldietrich:master into 174f99ad389464be6c9999d3fec79798e38e68f9 on javaslang:master.


Can you show me some full examples

I try in my project, but it doesn't work. Error happen like below, it don't know why

You need to be a bit more detailed for me to give any good suggestions.
Here's an example where the plugin's used:
https://github.com/jouni-kantola/webpack-output-by-build-type/blob/master/webpack.config.js


Battery drain from YouTube Music due to Google Tag getting blocked

First reported by XDA:
https://www.xda-developers.com/psa-using-an-ad-blocker-a-bug-in-google-play-music-may-be-draining-your-battery/
It seems like this is still the case with YouTube Music:
https://twitter.com/arter97/status/1397973993529823233
The app retries the connection to Google Tag without any sleep, resulting in consuming a CPU core 100% (this is a 8 core phone, so anything higher than 12.5% is consuming an entire core) and causing massive battery drains on Android.
While it seems to be the case that Google Tag is used for analytics, causing a massive battery drain over it isn't worth it.
The issue is gone after removing:
www.googletagmanager.com
www-googletagmanager.l.google.com
googletagmanager.com
googletagservices.com
www.googletagservices.com

One thing I found weird is that the hosts file here already whitelists www.googletagservices.com for some reason, but is overridden with another non-commented entries:
# <IP_ADDRESS> www.googletagmanager.com #[affects ASUS]
#<IP_ADDRESS> www.googletagservices.com              #interferes with techrepublic

Well Park @arter97 the article suggest to whitelist it until google decide to fix their crap.It's a bug after all and the domain in question is a...one of the most popular tracking domains.
@jawz101 what do you think,it's in your list,still...i'm all 👎 for removing it.

The article dates back to 2017, so I'm not entirely sure if Google even wants to fix it.
I'm now told that GMS may also misbehave like this, which may mean all Android users with Google Play services may be affected.
https://twitter.com/luca020400/status/1397986103114842113 (LineageOS director)
IMHO, this repo should serve practical use-cases. If users are paranoid enough to take security and privacy to that extreme, they should have other means (e.g., VPN, Firefox with Facebook container, etc) to protect themselves.
As this repo is the simplest form of ad blocking, I believe that possible battery drains from this isn't worth it for practical users.

I don't want to drain my battery for the sake of blocking a domain so I'll remove it though it bugs me that blocking a domain would cause battery drain.

Most of those domains are included by several other curators.  So this has little or no practical effect on this list.
Which is fine by me. 👀

Which is fine by me. 👀

And ME 😉

I don't want to drain my battery for the sake of blocking a domain so I'll remove it....

@jawz101 this here....i believe is the first complaint since 2017,don't you find that strange?

@dnmTX not from @arter97.  I recognize the name from XDA.  They've been doing Android kernel stuff for 5+ years.

I am using Adaway to block ads, YT music was still taking 100% CPU after whitelisting the domains listed in the issue.
But, after whitelisting googleads.g.doubleclick.net YT music is taking 22% CPU.


integrating vowpal_wabbit's optimizers

The existing vw modular in shogun is outdated and partially buggy :(.
As suggested by @karlnapf , we should do another round of integration.
In my opinion, there are two main components of vw to be integrated into shogun.

distributed communication and IO
(parallel) stochastic optimizers

Due to time constraint, I am mainly working on these optimizers. I assume that data can be completely loaded into memory using StreamingFileFromDenseFeatures.

this is also related to issue
https://github.com/shogun-toolbox/shogun/issues/3316

Absolutely,
I think focussing on the optimizers first is fine!


Flutter build api --release issue

I have been gotten issue with retrieveCalendars when build release.
If I build project by code, device_calendar work well. But if I build release apk and then install by generated apk. this method cannot retrieveCalendars on device.
Please help me about this.
Thanks

We suspect this is related to the use of Proguard and we'll investigate

Thank for your response. I found problem myself but forgot update issue, it's really problem when release with Proguard, sorry for wasting your time.

@thanhhuan1990 can you elaborate more on what the issue is/way - if would help others if we can document what the issue was

I had the same issue due to Proguard. I fixed it by adding the following line to my Proguard file.
-keep class com.builttoroam.devicecalendar.**  { *; }

Might be worth adding it to the readme.

Thanks @britannio, we planned to do so

As an aside, turns out everyone is a proguard user now... This fixed my issue, even though I'm using a standard autogen flutter project with the latest release, etc. Finding it was a lucky stroke. Can how to create a proper proguard file be documented? I added the line from this issue to the file documented here https://flutter.dev/docs/deployment/android#step-1---configure-proguard .

This solution worked for me for the Android build, what about the iOS? Because this bug is happening on iOS as well.

Where is this line supposed to go exactly?

Where is this line supposed to go exactly? I know it says on the proguard file but, according to the docs it says it should be there in the project auto-generated, but in my case is not, do I suppose to create it?

android/app/proguard-rules.pro
Mine looks like this:
#Flutter Wrapper
-keep class io.flutter.app.** { *; }
-keep class io.flutter.plugin.**  { *; }
-keep class io.flutter.util.**  { *; }
-keep class io.flutter.view.**  { *; }
-keep class io.flutter.**  { *; }
-keep class io.flutter.plugins.**  { *; }

# You might not be using firebase
# -keep class com.google.firebase.** { *; }
-keep class com.builttoroam.devicecalendar.** { *; }

then in android/app/build.gradle add the following line to buildTypes { release  {}}
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
I also use minifyEnabled true and shrinkResources true in the same place.
Finally under android/gradle.properties I have android.enableR8=true, R8 still respects your proguard rules.

Now I can retrieve calendars in release mode by adding proguard. But for release mode deletion of event is not working.Can anyone  help me in this case.I have an important feature to release.

@britannio android/app/proguard-rules.pro - this would be inside the flutter app's android/app or inside the plugin's?

@britannio android/app/proguard-rules.pro - this would be inside the flutter app's android/app or inside the plugin's?

Inside your Flutter app.

I did not have the pro guard files before and I am creating the apk as follows
flutter build apk --release
I have added the solution that they mention but it does not work, it seems that when I generate the apk it does not take the proguard
in the gradle.properties located in android/gradle.properties add this line
android.enableR8=true
in android/app/build.gradle file add
buildTypes {
release {
minifyEnabled true
shrinkResources true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
}
}
and add the file
android/app/proguard-rules.pro
#Flutter Wrapper
-keep class io.flutter.app.** { ; }
-keep class io.flutter.plugin.* { ; }
-keep class io.flutter.util.* { ; }
-keep class io.flutter.view.* { ; }
-keep class io.flutter.* { ; }
-keep class io.flutter.plugins.* { *; }
You might not be using firebase
-keep class com.google.firebase.** { ; }
-keep class com.builttoroam.devicecalendar.* { *; }

I did not have the pro guard files before and I am creating the apk as follows
flutter build apk --release
I have added the solution that they mention but it does not work, it seems that when I generate the apk it does not take the proguard
in the gradle.properties located in android/gradle.properties add this line
android.enableR8=true
in android/app/build.gradle file add
buildTypes {
release {
minifyEnabled true
shrinkResources true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
}
}
and add the file
android/app/proguard-rules.pro
#Flutter Wrapper
-keep class io.flutter.app.** { ; }
-keep class io.flutter.plugin.* { ; }
-keep class io.flutter.util.* { ; }
-keep class io.flutter.view.* { ; }
-keep class io.flutter.* { ; }
-keep class io.flutter.plugins.* { *; }
You might not be using firebase
-keep class com.google.firebase.** { ; }
-keep class com.builttoroam.devicecalendar.* { *; }

As someone who just spent a full day on this thinking the problem came from somewhere else, you can follow the steps from @jurgenramirez but do use the proguard file of @britannio, with the endings in .** { *; }.

Where is this line supposed to go exactly? I know it says on the proguard file but, according to the docs it says it should be there in the project auto-generated, but in my case is not, do I suppose to create it?

android/app/proguard-rules.pro Mine looks like this:
#Flutter Wrapper
-keep class io.flutter.app.** { *; }
-keep class io.flutter.plugin.**  { *; }
-keep class io.flutter.util.**  { *; }
-keep class io.flutter.view.**  { *; }
-keep class io.flutter.**  { *; }
-keep class io.flutter.plugins.**  { *; }

# You might not be using firebase
# -keep class com.google.firebase.** { *; }
-keep class com.builttoroam.devicecalendar.** { *; }

then in android/app/build.gradle add the following line to buildTypes { release {}} proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
I also use minifyEnabled true and shrinkResources true in the same place.
Finally under android/gradle.properties I have android.enableR8=true, R8 still respects your proguard rules.

Works!


Can't input password.

Either If I use Visuals studio or CMD,
when I use instaloader --login {YOUR_USERNAME} or instaloader login, I can't Input my password,
Putting my username works, but the scanner for Password is dead, I tried many times, but It seems like either something interrupts it, or it's broken.

same, what do i do?

Can You Just Not see it or does it just not work?

Can You Just Not see it or does it just not work?

It does not work, I checked the output folders done with default or picked output

It won't let me input the password... For some reason I can't type in
anything in the space for password but can type all the other lines
Any ideas what I'm facing ?
On Sun, 12 Feb, 2023, 9:26 pm OMGerEDU, @.***> wrote:

Can You Just Not see it or does it just not work?
It does not work, I checked the output folders done with default or picked
output
—
Reply to this email directly, view it on GitHub
https://github.com/nathan-149/automated_youtube_channel/issues/54#issuecomment-1427066726,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5TJHNVGOFHLKFKHXYZZU7LWXEB3ZANCNFSM6AAAAAATT55LLM
.
You are receiving this because you commented.Message ID:
@.***>

You can't see the password while entering but the password is still being entered this is done for security reasons

but then why cant i proceed?
On Wed, Mar 29, 2023 at 2:36 AM Masoom darinda @.***>
wrote:

You can't see the password while entering but the password is still being
entered this is done for security reasons
—
Reply to this email directly, view it on GitHub
https://github.com/nathan-149/automated_youtube_channel/issues/54#issuecomment-1487591080,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5TJHNVDXCA4QJK6P2DZY6LW6NHD5ANCNFSM6AAAAAATT55LLM
.
You are receiving this because you commented.Message ID:
@.***>

Whats the problem tell me in detail

I am doing everything as per the guide. On the stage where I have to input
username and password of my Instagram account, there I am not able to type
in anything in the password space.
On Thu, Mar 30, 2023 at 1:32 AM Masoom darinda @.***>
wrote:

Whats the problem tell me in detail
—
Reply to this email directly, view it on GitHub
https://github.com/nathan-149/automated_youtube_channel/issues/54#issuecomment-1489228192,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5TJHNR4LEFILHLWSZEH2YDW6SIOXANCNFSM6AAAAAATT55LLM
.
You are receiving this because you commented.Message ID:
@.***>

The command prompt is taking the input but it is not showing the password on the screen. All u need to do is just type the password blindly and hit enter i am sure it will work

okay ill try that.
On Thu, Mar 30, 2023 at 1:38 AM Masoom darinda @.***>
wrote:

The command prompt is taking the input but it is not showing the password
on the screen. All u need to do is just type the password blindly and hit
enter i am sure it will work
—
Reply to this email directly, view it on GitHub
https://github.com/nathan-149/automated_youtube_channel/issues/54#issuecomment-1489235643,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5TJHNSSAGVTEORPQU22AFDW6SJCNANCNFSM6AAAAAATT55LLM
.
You are receiving this because you commented.Message ID:
@.***>

Let me know if it worked

After entering the password i get CRITICAL 'NoneType' object has no attribute 'group'. Do you happen to know why?

just enter the password hit enter and wait. it takes a bit.

now im stuck at opening main.py file:
Im getting this:
python: cant open file 'C:\Users\Dell\main.py' : [Error 2] No such file
or directory.
and sorry I couldn't continue this for months because my PC was broken.
On Sun, Apr 16, 2023 at 12:11 PM Uch1ha @.***> wrote:

just enter the password hit enter and wait. it takes a bit.
—
Reply to this email directly, view it on GitHub
https://github.com/nathan-149/automated_youtube_channel/issues/54#issuecomment-1510116095,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5TJHNRLKPTZWO4N7Y7ESLTXBOICTANCNFSM6AAAAAATT55LLM
.
You are receiving this because you commented.Message ID:
@.***>

Message https://instagram.com/boydtech761?igshid=NTc4MTIwNjQ2YQ== he’s the best on account recoveries

NEED URGENT HELP?!!
NOTE:MESSAGE ONLY THIS ACCOUNT
#🇬🇧Expert Here To Assist In Any Cyber Issues General_Hacking_Recovery_Spy_Blackmail_Stolen,Lost BTC_Hacked Phones ☎ 📥
🌎Facebook hacking - recovery
🌎Instagram hacking - recovery
🌎Snapchat hacking - recovery
🌎Lost money(investment scams) recovery
🌎Cheating partner spy (online)
🌎Crypto wallet hacking
🌎Phone hacks📱
🌎Message hacks & tweaks
For swift response send a message to my social media accounts, we are active 24/7 to solve all your Online ETHICAL HACKING issues
INSTAGRAM 📥 spygency.1
EMAIL 📧<EMAIL_ADDRESS>


Removing Podspec

Description
Removing the pod spec as we no longer need to support CocoaPods internally.

 I'm gonna try one more time, otherwise just merge since these won't have impacted the build process.


Bdb dev

Changes to defaults.
Changes to min/max ranges.
Changes to settings text
Changes to correct number of entries added based on the interval

You made your signing in this PR. And it's not synced with latest bdb_dev. I will close it now.


Feature/UI/UX: Change the color of the 'anonymously' status (title) display in posts/comments

Fixed by 2b2c49e commit


[Bug-7229] [Upgrade] fix postgre upgrade

close #7229

Codecov Report

Merging #7230 (2b9db2d) into dev (4743e8f) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #7230      +/-   ##
============================================
- Coverage     40.42%   40.40%   -0.02%
+ Complexity     3685     3682       -3
============================================
  Files           648      648
  Lines         27065    27065
  Branches       3035     3035
============================================
- Hits          10941    10936       -5
- Misses        15085    15091       +6
+ Partials       1039     1038       -1

Impacted Files
Coverage Δ

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...org/apache/dolphinscheduler/remote/utils/Host.java
37.77% <0.00%> (-2.23%)
:arrow_down:

...inscheduler/common/task/sqoop/SqoopParameters.java
74.00% <0.00%> (-2.00%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
52.11% <0.00%> (-1.41%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4743e8f...2b9db2d. Read the comment docs.


Search by group?

Is there a way to search groups in <multiselect>?
So if it finds a group I would see the group label and all the options for that group.

If you pass the groups to <multiselect>, then you have the control over them, right? :) So simply find the group you want in the options you're passing, it doesn't have to do anything with <multiselect> really.

What I think @Zip753 might mean is that you can use your own search function by disabling the :internal-search by passing false. This way you can react to @search changes yourself and adjust the :options. Ofc you will have to mimic the results structure as it is in multiselect itself to fit the grouping requirements but that shouldn’t be much of a problem.
Please look into the source code to see an example of such transfrom.

My bad, I thought of programmatic search somehow.

Why did you close ? It would be a nice enhancement to have this natively


Check if need to re-run FG R8 pipeline for distributions

same as #191 but for distributions

Data sent to @SaraKuitunen for integration in FR+FG R8 portal.


broken link to research-software.nl

hey @vsoch! happy new year :smiley:
seems like you're event has a broken link to https://www.research-software.nl (see https://sorse.github.io/programme/discussions/event-013/ and https://app.circleci.com/pipelines/github/SORSE/sorse.github.io/1024/workflows/842bd5f3-58fb-4655-a0ca-9abbaef932b4/jobs/1719 for instance). Any idea, why?

hey @Chilipp ! The site is maintained by the RSE group in the Netherlands, and the speaker from our event that you'd want to ask is @jspaaks.

hey @Chilipp ! The site is maintained by the RSE group in the Netherlands, and the speaker from our event that you'd want to ask is @jspaaks.

oops I just tried to assign @jspaaks and I messed it up :/

oops I just tried to assign @jspaaks and I messed it up :/

the website is up again so we can close this issue

the website is up again so we can close this issue

Hi All and happy new year.
There were some memory problems with research-software.nl over the holidays. For the time being we put a band-aid on it, hopefully we'll have time to do a more structural fix in the next couple of days.

Hi All and happy new year.
There were some memory problems with research-software.nl over the holidays. For the time being we put a band-aid on it, hopefully we'll have time to do a more structural fix in the next couple of days.

Thanks for the update @jspaaks! Happy New Year to you too :)

Thanks for the update @jspaaks! Happy New Year to you too :)


add libarchive

https://github.com/libarchive/libarchive
The libarchive project develops a portable, efficient C library that can read and write streaming archives in a variety of formats. It also includes implementations of the common tar, cpio, and zcat command-line tools that use the libarchive library.

Judging from libarchive's CMakeLists most (all?) of it's dependencies are optional, but the library is barely functional without them. IMHO it would be best to gather most important ones before adding libarchive's port.
Complete list of what buildsystem tries to find:

[x] zlib
[ ] bzip2, see #200
[ ] liblzma
[ ] lzo2 <- GPL licensed - better not link to it by default
[ ] lz4
[ ] nettle <- quite big and configure only, also license hell
[x] openssl
[ ] libmd <- used only if openssl not found
[x] iconv
[x] libxml2
[x] expat <- used only if libxml2 not found
[x] pcre <- only for samples/tools/tests

I think we could add libarchive once we add bzip2, lz4 and liblzma ports. All of them are quite small so this shouldn't be difficult.

libarchive is functional without dependencies, but it would basically support tar/cpio  without them.
compression libraries are interesting, what is this nettle thing for, I have no idea (or even openssl)

Does libarchive actually need nettle or can it get all the cryptography it needs from OpenSSL if Nettle isn't found?

Does libarchive actually need nettle or can it get all the cryptography it needs from OpenSSL if Nettle isn't found?

Buildscript isn't as clear in this case as in libxml2/expat situation but after searching for occurences of HAVE_LIBNETTLE  macro in source I believe no functionality will be lost. There is also a mode that definitely bears inverstigating in which winapi is used for cryptography, so maybe we could avoid any big dependency in this regard.

Closing this as fixed by #493.


graph package should be 'graphs'. This will be in sync with other packages convention.

To be corrected in next successful build.

Issue has been solved with the recent commit.


[RESOLVED] Unavailable by ICMP ping

Problem started at 18:29:29 on 2023.08.12
Problem name: Unavailable by ICMP ping
Host: HP_iLO
Severity: High
Operational data: Down (0)
Original problem ID: 35309
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23266&eventid=35309

Problem has been resolved in 2h 7m 0s at 20:36:29 on 2023.08.12
Problem name: Unavailable by ICMP ping
Host: HP_iLO
Severity: High
Original problem ID: 35309


check build reproducibility of cronet and envoy

There is a good chance that GF cronet and envoy are already reproducibly built since the Chromium team already works on reproducible builds, and Android/Java builds are generally easy to make reproducible.  If we have a reproducible process for cronet and envoy, then we have much less to worry about when making release builds for Maven Central.
I think the first step would be to run the Vagrant setup in #5 and the compare the result to
https://repo1.maven.org/maven2/info/guardianproject/envoy/cronet/<IP_ADDRESS>.4280.66/
@stevenmcdonald

FYI: I gave the VM 8GB of ram and 10 cpus, and it took about 4 hours (3:48:27elapsed according to time). It looks like it successfully built for all the target platforms, but crashed out on the app build.
The "release" aar it produced isn't identical to the one at https://repo1.maven.org/maven2/info/guardianproject/envoy/cronet/<IP_ADDRESS>.4280.66/, it's slightly smaller:
-rw-rw-r-- 1 vagrant vagrant 16123019 Dec 22 15:20 cronet-<IP_ADDRESS>.4280.66.aar
-rw-rw-r-- 1 vagrant vagrant 16193313 Feb 23 10:40 cronet-debug.aar
-rw-rw-r-- 1 vagrant vagrant 16090858 Feb 23 10:18 cronet-release.aar

Do you want me to look in to what's specifically different, or do you want my file to compare?

Seems worth throwing into diffoscope and seeing if there is anything simple we
can address. cronet might not build reproducibly upstream, so that's not
something we could really take on since it would probably be a big project.

Ah ha, I did a search for a tool to compare, but the only one I found was old and didn't work. It looks like some of the differences are just ordering, but there are some binary differences too. This probably means more to you than to me :)
cronet_diff.html.gz

diffoscope is the one true way!  It compares everything :)  That's a big diff,
looks like we can deal with reproducible builds later.

I ran the build a couple more times, because it was basically no effort, and I figured seeing if it didn't make the same binary on the same system, that might help track down some of the issues... It looks like it can silently fail to include some of the architectures in the aars it produces, I just took a quick look and saw that so far. I won't spend any more time on it now, but that issue is probably worth looking in to at some point.

I added a bunch of set -euo pipefail to the scripts (see http://redsymbol.net/articles/unofficial-bash-strict-mode/), and found a failure that was being ignored during the initial setup, but didn't seem to actually cause a problem, since the VM does work when it's done. It kinda looks like the x86-64 version is always left out of the release build on the first run (that's the first thing it builds, so maybe related to the failure). Running the full vagrant up takes hours though, so it's been a slow going, "fire it off and look at it later" process that's still ongoing.
Looking at repeated builds on the same machine, I think the only thing changing is the date (I'm not certain if the diff for the binary is just showing the date differences in binary, or if there are other subtle differences, I'm not that familiar with diffoscope):
stscm.zip
Are differing timestamps something we're worried about?  I guess we could change the mtime of the files included to match the source dates or something like that?
Comparing to the build on maven or builds I do on you (HCS's) machine look like they're using slightly different versions of the java classes... maybe a library version difference (maybe some security updates?). I think they're both set up from the same Vagrantfile... here's the diffs if you have any insight in to why they'd be different
hcsscm.zip


Port changes of [#11411] to branch 2.2

Cherry-pick #11411
Reduce logging on FileSystem instantiation
Motivated by issue https://github.com/Alluxio/alluxio/issues/11404, this
PR updates logging level for certain logging messages
After this PR, each query only introduced two info logging like the
following;
2020-05-22T13:58:05.120-0700 INFO hive-hive-6
alluxio.hadoop.AbstractFileSystem Creating Alluxio configuration from
Hadoop configuration Configuration: , uri configuration {}
2020-05-22T13:58:05.174-0700 INFO
20200522_205804_00001_nzuct.1.0.0-0-104
alluxio.hadoop.AbstractFileSystem Creating Alluxio configuration from
Hadoop configuration Configuration: , uri configuration {}

These two are required as currently each Presto query triggers a new
Alluxio FileSystem to be created.
pr-link: Alluxio/alluxio#11411
change-id: cid-c2395a720b64042699650f59459aeb229aea6edb

alluxio-bot, merge this please


receiving messages again and again on topic

Hello, I have a .net client which publish messages to another client which is nodemcu. I want the message to be received just once on the nodemcu. Following is the code I am using;

client = new MqttClient(BrokerAddress, 1883, false, MqttSslProtocols.None, null, null);
clientId = Guid.NewGuid().ToString();
client.Connect(clientId);
client.Publish(Topic, Encoding.UTF8.GetBytes(firmware.version),
MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
System.Diagnostics.Debug.WriteLine("Published");`

here is the nodemcu code;

void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
d[i] = (char)payload[i];
Serial.print(d[i]);
}
}
void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
client.subscribe(mqtt_topic);
}

So what is your question?

I want the message to be received just once. currently if I publish a message to the nodemcu it received many time. The following message was send from .net client.

You are subscribing in your main loop. Every time you subscribe, the broker will send any retained message in that topic.
Do not subscribe in your loop - subscribe once after you connect.

Do I need to reset the device everytime?

Why would you want to do that?

Here is my code. I want the message just once. can you suggest where to subscribe here and when to call callback?

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>
//=======================================================//
#define SLEEP_DELAY_IN_SECONDS 60
#define WAIT_BEFORE_SLEEP_IN_SECONDS 5
#define REFERENCE_VOLTAGE_ADC 3.3
//======================================================//
// initializes or defines the output pin of the LM35 temperature sensor
int outputpin = A0;
///ADC_MODE(ADC_VCC);
float readingIn = 0;
String ssid1, pass1;
String newv = "";
String versions = "001";
const char* ssid = "Akhtar";
const char* password = "akhtar1234";
const char* mqtt_server = "broker.hivemq.com";
const char* mqtt_topic = "ser";
char temperatureString[6];
byte mac[6];                     // the MAC address of your Wifi shield
String strMacAddress = "";  // String type of Mac address
String d[7];
String a = "";
int count = 1;
WiFiClient espClient;
PubSubClient client(espClient);
void setup_wifi() {
Serial.println(espClient);
delay(10);
// We start by connecting to a WiFi network
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
WiFi.macAddress(mac);
strMacAddress = "";
Serial.print("MAC: ");
Serial.print(mac[5], HEX);
// strMacAddress = strMacAddress + mac[5];
Serial.print(":");
Serial.print(mac[4], HEX);
//strMacAddress = strMacAddress + mac[4];
Serial.print(":");
Serial.print(mac[3], HEX);
//strMacAddress = strMacAddress + mac[3];
Serial.print(":");
Serial.print(mac[2], HEX);
//strMacAddress = strMacAddress + mac[2];
Serial.print(":");
Serial.print(mac[1], HEX);
// strMacAddress = strMacAddress + mac[1];
Serial.print(":");
Serial.println(mac[0], HEX);
// strMacAddress = strMacAddress + mac[0];
}
void(* resetFunc) (void) = 0; //declare reset function @ address 0
void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
d[i] = (char)payload[i];
Serial.print(d[i]);
newv = newv + d[i];
//Serial.println();
//Serial.println("Firmware Version " + newv + " Recieved");
// a = d[4] + d[5] + d[6];
//Serial.print (a);
}
Serial.println();
}
void setup() {
// setup serial port
Serial.begin(115200);
// setup WiFi
setup_wifi();
client.setServer(mqtt_server, 1883);
client.setCallback(callback);
client.connect("ESP8266Client");
}
void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP8266Client")) {
Serial.println("connected");
//client.subscribe(mqtt_topic);
} else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" try again in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
client.subscribe(mqtt_topic);
client.setCallback(callback);
if (a > versions) {
Serial.println("Dowlnloading new version "+newv);
t_httpUpdate_return ret = ESPhttpUpdate.update(espClient, "http://<IP_ADDRESS>/Blink.bin");
 switch (ret) {
   case HTTP_UPDATE_FAILED:
     Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", ESPhttpUpdate.getLastError(), >ESPhttpUpdate.getLastErrorString().c_str());
     break;

   case HTTP_UPDATE_NO_UPDATES:
     Serial.println("HTTP_UPDATE_NO_UPDATES");
     break;

   case HTTP_UPDATE_OK:
     Serial.println("HTTP_UPDATE_OK");
     break;
 }

}
}

This code already contains the following in the 'reconnect' function:
if (client.connect("ESP8266Client")) {
   Serial.println("connected");
   //client.subscribe(mqtt_topic);
} else {

For some reason, you have commented out the subscribe call.
In your loop function you have:
void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
client.subscribe(mqtt_topic);
client.setCallback(callback);
...

You should not be calling subscribe or setCallback in the main loop. You already call setCallback correctly in the setup function, you don't need to keep calling it.
Finally, the last line of your setup() function is a call to client.connect... you should replace that with a call to the reconnect() function so that the subscribe is done from there.


mup setup does nothing.

when I execute mup setup in command line. It pauses for a few seconds and does nothing. I am using Windows.
Here is my mup.js:
'use strict';

module.exports = {
  servers: {
    one: {
      host: 'xxx.xxx.xxx.xxx',
      username: 'root',
      pem: '~/.ssh/id_rsa'
      // password:
      // or leave blank for authenticate from ssh-agent
    }
  },

  meteor: {
    name: 'app',
    path: '../app',
    servers: {
      one: {}
    },
    env: {
      ROOT_URL: 'http://<IP_ADDRESS>/',
      //MONGO_URL: ''
    },

    //dockerImage: 'kadirahq/meteord'
    deployCheckWaitTime: 10
  },

  mongo: {
    oplog: true,
    port: 27017,
    servers: {
      one: {}
    }
  }
};

I was having the same issue. I was able to get 'mup setup' to work by providing a password for the remote server instead of the keyfile:
module.exports = {
  servers: {
    one: {
      host: 'xxx.xxx.xxx.xxx',
      username: 'root',
      password:"*********" <-------- this works
      //pem: '~/.ssh/id_rsa' <----- this did not work
      // password:

I haven't had the time to figure out why this is the case, but I'm thinking it may have something to do with the tilde in the path. I'm going to try an absolute path next and see if that fixes anything.

I had this issue as well and solved it by changing ~ (tilde) in the pem path to the explicit home directory /home/my-user/.

I can confirm as well. Using an absolute path has fixed this issue for me. On OSX, I changed
//...
servers: {
        one: {
           //...
            "pem": "~/.ssh/id_rsa",
            //...
        }
    },
//...

to:
//...
servers: {
        one: {
           //...
           "pem": "/Users/<MY_USER_NAME>/.ssh/id_rsa",
            //...
        }
    },
//...

and everything runs smoothly. Not ideal, but also a pretty simple fix.

+1, Yes gantrim's solution is finely works.


JSX support

Are there anyway i can get emmet to expand to className instead of class, when in jsx files?

You can write your own snippets in ~/emmet/snippets.json, you can see the built-in snippets as an example. You can find more about them in the docs.

There’s pending PR with JSX support: https://github.com/emmetio/emmet/pull/368
Need to test it first in add into editor plugins

hi- when is this PR going to be merged? https://github.com/emmetio/emmet/pull/368
i'm liking atom, but i work with react every day and having .jsx support is becoming more important.

JSX support added in v2.3.6

I am also seeing the same behavior as @kebot

Works for me with latest Emmet. What package you’re using for JSX?

I am using https://github.com/orktes/atom-react, v0.12.3

Same thing as @ryanjuve-porch here.

I'm having the same issues as well. Same versions as @ryanjuve-porch

Works for me with language-babel instead of the react package, except the dot sign translates to class and not className.

I'm having the same issue. Used to resolve to "className" but I've updated, and resolves to only "class".
Other packages I have installed (if that matters):

@sergeche — which JSX package do you recommend?

@zippoxer @mikefisher84 @chiefGui @ryanjuve-porch — Until this is fixed, I just [reverted to v2.3.10] (https://github.com/emmetio/emmet-atom/releases)

@mattwoodnyc $ apm install<EMAIL_ADDRESS>fixed it for me too thank's.

I can confirm that reverting back to 2.3.10 solves the problem

Downgraded but no joy. Pressing tab just inserts indentation. Atom 1.0.15.

@OKNoah — regular shortcut does the trick (⌘ + ⇧ + E). So,
ul.foo
becomes

Atom 1.0.19, had to downgrade from 2.3.13 to 2.3.10.

@Blackrush It's working form me with 2.3.13.

@chodorowicz Nope, it doesn't work for me with "Babel ES6 JavaScript" file type, Atom 1.1.0-beta1-63e8e81 and<EMAIL_ADDRESS>

Atom 1.1.0, Emmet 2.3.14 and ul.foo expands to

 and not className

Doesn't work with emmet 2.3.14 , same config and error as @Restuta

Had the same issue although resolved it by opening Atom > Open Your Keymap and adding the following
'atom-text-editor[data-grammar="source js jsx"]:not([mini])':
'tab': 'emmet:expand-abbreviation-with-tab'
Seems to be working good for me with Emmet 2.3.14, Atom 1.2

@kylepritchard it now expands on tab ,which is cool, but it's still class instead of className

@sergeche could you please consider reopening this?

I can also confirm that the current version of Emmet does not properly use className, which is definitely a reversion of existing functionality.

@sergeche @natecox I have no issues with the class. Comes up with className when expanding. Using the latest emmet and latest atom-react

On OSX with current atom, atom-react, and emmet; inside of a Babel ES6 JavaScript file.
Downgrading emmet with absolutely no other changes causes className to be used properly.

@natecox to which version should I downgrade?

@Restuta apm install<EMAIL_ADDRESS>

confirming that it works for me also with 2.3.10

2.3.11 doesn't work and the only change I see is this https://github.com/emmetio/emmet-atom/commit/75a050c998eb6805bc2c2eaba2e1c2ba17386ae5
This gives confidence that it introduced it. I don't feel like I know enough to fix it, but I hope it would help maintainers.

That commit change actually makes sense. It's probably not finding the babel stuff in the syntaxes and is forcefully setting the syntax back to HTML in the unless block.

As @natecox pointed out, this was because the syntax returned by Babel ES6 JavaScript is babel instead of javascript or jsx, thus emmet assumed it was html and used html pattern.
Adding 'babel' to the test pattern will fix this issue (see my pull request).

@jdinhlife thank you for that, now let's see when maintainers will react and merge + release it

Confirming: After adding the keybinding fix I am able to get tab to work with execution and className is working as expected in<EMAIL_ADDRESS>Thanks everyone! This was driving me crazy and giving me carpal tunnel :)

Adding the below to keymap.cson works well with Atom (1.3.2), Emmet (2.3.17)
'atom-text-editor[data-grammar="source js jsx"]:not([mini])':
  'tab': 'emmet:expand-abbreviation-with-tab'

Also div.foo tab expands to

2.3.17 works for me too

@dv2 thanks, it's works for me too

@dv2 thanks a lot, worked like a charm.

A better solution is to use slightly different selector:
'atom-text-editor[data-grammar~="jsx"]:not([mini])':
  'tab': 'emmet:expand-abbreviation-with-tab'

I can enable this by default, but most likely it will conflict with default autocomplete snippets. Does anyone use any autocomplete-plus provider with JSX? What syntax/extension you’re using for JSX development?

Excellent @sergeche thanks! :100:

@dv2 or @sergeche's solution works very well, no down or upgrading needed. thanks.

@sergeche tab works outside jsx tree with your keybinding. How I can fix this?

Thanks guys! Full solution as stated above:

Use Atom 1.6.1

goto terminal and enter: apm install<EMAIL_ADDRESS>

goto Atom, then find language at bottom right hand side of screen and change to JSX

goto Atom and edit keymap to include the following:
'atom-text-editor[data-grammar="source js jsx"]:not([mini])':
'tab': 'emmet:expand-abbreviation-with-tab'

Enjoy :-)

Case: Atom 1.6.1, emmet 2.4.3
Solution without downgrade:
1) Open Atom ->  Keymap...
2) Edit keymap 'atom-text-editor[data-grammar="source js jsx"]:not([mini])':     'tab': 'emmet:expand-abbreviation-with-tab'

If you have followed the above advice, and if you are using webpack,  you can use the suffix .jsx on your react files and then in your webpack.config.js file make your module loaders accept jsx files using this-->  test: /.jsx$/
this regex will include both js and jsx files in the loader because of the way regex works.
Why? Because this way you do not need to change the language settings in Atom editor, and therefore all files with the suffix .jsx will automatically accept tab completion.

work only with tab, but if i want another keys combination, like ctrl+shift+e?

@AStaroverov
'atom-text-editor[data-grammar~="jsx"]:not([mini])':
  'cmd+shift+e': 'emmet:expand-abbreviation'

@trent-boyd it doesn't work

@sergeche is there a way to do the tab auto complete only inside render function ?

@tbarash There is a way, but it requires additional programming

@sergeche i'll look into that , and try to maybe open a PR if thats cool with you guys?

repeating @max-mykhailenko question..
Can someone tell me if there is way to trigger emmet only inside the render function part, rather than in the entire jsx?

Instead of [data-grammar~="jsx"], using [data-grammar~="js"] which can support both js and jsx files
'atom-text-editor[data-grammar~="js"]:not([mini])':   'ctrl+shift+e': 'emmet:expand-abbreviation'

Instead of [data-grammar~="jsx"], using [data-grammar~="js"] which can support both js and jsx files
 'atom-text-editor[data-grammar~="js"]:not([mini])': 'tab': 'emmet:expand-abbreviation-with-tab'

This gist, "Enable tab completion for JSX with Emmet in Atom", got my Emmet working as expected within jsx.
I found the gist from this other similar gist.