First Class FSMs

Idea
Here's the initial discussion on Zulip, which touches on the motivation for this.
The idea is to be able to instantiate FSMs as a group-like construct in the wires section of a Calyx program. In particular, these FSM constructs might let users specify:

Their own symbolic states
(conditional) Transitions from one state to another (in a case-like block)
(conditional) Assignments that are guarded by the current state
Groups that trigger at the current state

Motivation
As the original post mentions, this was largely motivated by the idea that the Vivado synthesis tool is able to specifically infer a finite state machine construct in Verilog when FSMs are compiled to a Verilog case block. Here's what our FSM transition assignment looked like before:
assign fsm2_in =_guard485 ? 3'd6 : _guard490 ? 3'd5 : _guard497 ? 3'd2 : ... 3'd0

and here's roughly what Vivado expects in order to infer an FSM;
always @ (*) begin
    case (fsm2_out)
       ...
       3'd2 : begin
            if (raise_err_done_out) begin
                fsm2_in = 3'd6;
            end
            else begin
                fsm2_in = 3'd2;
            end
        end
        3'd3 : begin
            if (read_payload_from_mem_pop_done_out) begin
                fsm2_in = 3'd4;
            end
            else begin
                fsm2_in = 3'd3;
            end
        end
      ...

We want Vivado to recognize our register as an FSM because it can do some cool optimizations on it to reduce resources and potentially increase frequency. We're guessing the reason Vivado prefers the second construction is because it can list all transitions, guaranteeing a "finite" number of states, which wasn't necessarily clear from the original transitions specified by the assign.
So, if we want to emit these case statements via the Verilog backend, it would be nice to have an IR construct within the compiler to store these transitions. This prompted the idea to let users define FSMs themselves.
Example
Here's a small example I was working with to map out what a hand-designed FSM in Calyx might look like. It's a silly multiplier for positive integers that adds an input a_in to a result accumulator b_in times, but it importantly contains a loop and conditional transitions. The control looks like this:
control {
    loada;
    loadb;
    while gt.out with cond {
      resincr;
      bdecr;
    }
  }

which currently compiles to 7 states. One for each of loada ( s0 ) and loadb ( s1 ), one for the generated group that updates the comb_reg containing whether b.out > 0 ( s2 ), one for each of updating the accumulator ( s3 ) and b registers ( s4 ), another to once again update comb_reg ( s5 ), and finally the done state ( s6 ).
Here's how I was imagining the equivalent with a new fsm construct, stealing inspiration from what @sampsyo posted on the Zulip thread. For this example, I decided to have only one state in which comb_reg is updated. Also, while I've listed states as integers, I'm thinking of them as symbolic states.
Within the wires section, we might have something like:
fsm myfsm {
    0: { loada; } => <*> 1,
    1: { loadb; } => 2,
    2: { comb_cond_update; } => {
            comb_reg.out: 3, <**>
           !comb_reg.out: 5,
        },
    3: { resincr; } => 4,
    4: { bdecr; } => 2,
    5: => 0
}
* implicit loada.done guard here, plus implicit self-loop if any transition condition not met
** again, implicit guard of comb_cond_update.done, with self-loop

Note that user-defined transition conditions should be exclusive, so the FSM remains deterministic. At least in this example, it seems that we can more specifically expect cond1 xor cond2 xor ... xor cond3 to be true, meaning the only condition on a self-loop will be if the group invoked in that state has a low done signal.
When we compile these fsm constructs to Verilog, we can indeed have a case statement that matches on the current state of the fsm register. Then, within each branch of this RTL case block, we would have an if/else if statement for each condition for a transition (e.g.comb_reg.out from above), which we would guard with the done signal of the group invoked in that state. Then, the else statement (whose purpose is to encode the implicit self-loop mentioned above) would only occur if the done signal of the invoked group is not high.
We would let the assignments within a group be compiled like normal, with one key difference: the only compiler-generated condition on whether an assignment within a group can activate is the state of the FSM from which the group is invoked. Here's why that's different from how we currently do it, and why we think this modification maintains correctness:
Currently, a group with a control block has its assignments is guarded by 1) the state of the FSM fsm.out and 2) tdcc.go. Further, the number of guards increases with nested control that offloads FSMs because you'd instantiate a tdcc group for each new FSM. Based on some discussions during the calyx-opt meeting, the checks of tdcc{x}.go seem unnecessary: if an FSM is currently in the middle of execution, there is never a point at which tdcc.go becomes low. It might offload to another FSM, but the tdcc.go signal remains high. Based on this idea, with nested control and offloaded FSMs, we need only check the "leaf" FSM state, and none of the "node" (i.e. parent) FSM states and none of the tdcc{x}.go signals.
Now, our new fsm construct idea doesn't use tdcc-like generated groups. But, we are thinking of allowing one FSM to call another FSM, similar to a group invoke. We think we can take advantage of this to dramatically reduce assignment guards, and thereby reduce LUT usage!
Brief Static FSM Interlude
I agree with the idea that static fsm constructs should be distinct from dynamic ones. Given that assumption, we could create a new fsm construct for static FSMs, where implicit group.done guards are not added to transition conditions. Since each group will have a latency attached to it, we could compile our static FSMs the same way, where each state is a cycle.
For example,
staticfsm my_fsm {
    s0 : { < 3 cycle group > } => s1,
    s1 : { < 1 cycle group > } => s2,
    s2 : ... => ...
}

would compile to:
always @ (*) begin
    case (fsm_out)
       4'd0 : fsm_in = 4'd1; // s0 begins here
       4'd1 : fsm_in = 4'd2;
       4'd2 : fsm_in = 4'd3;
       4'd3 : fsm_in = 4'd4 // s1 begins here
       4'd4 : fsm_in = 3'd5 // s2 begins here
       ...

Static repeats are the interesting bit. We surely wouldn't want to make users create distinct states for the same loop body, so maybe there can be a repeat construct within the staticfsm block somehow?
Open Questions

Do we allow arbitrary assignments instead of just group/fsm invokes?

One issue might be semantics around implicit group.done guards in the transitions, since, now, it may not be necessary to invoke a group at a state. Static FSMs might get around this by having a different fsm construct specific to static FSMs. Should we do the same and have an "assignment FSM" construct?

How do we allow specification for symbolic representations of state?
Right now, my idea of a user-defined FSM is just "execute this group at this state". What if users have a different idea of what an FSM means?

I'm thinking of an IDLE, CALC, DONE state situation, where control signals are accepted and emitted at every cycle. This probably goes back to allowing arbitrary assignments.

Static repeats are the interesting bit. We surely wouldn't want to make users create distinct states for the same loop body, so maybe there can be a repeat construct within the staticfsm block somehow?

For static repeat we can use a common trick to avoid state explosion, create a separate binary counter register that is incremented every cycle when in a given FSM state. We then transition to the next fsm state when the counter reaches a certain value. This could look some like this (in super fake syntax, just showing the general idea):
reg counter<32>;
cmp = counter == repeat_len;
staticfsm my_fsm {
...
s0 : {incr(counter)} => s1 // start of static repeat
s1 : { loop body state 1 } => s2 // repeat body with static latency of 2
s2 : { loop body state 2 } => { cmp : s3, !cmp : s0}
s3 : { ... } => s4 // after loop body
...
}

So, if we want to emit these case statements via the Verilog backend, it would be nice to have an IR construct within the compiler to store these transitions

I think we have a broader goal here than just emitting FSMs for Vivado to recognize. Broadly, we want to be able to optimize decisions about how a control program is implemented using the FSM representation.

implicit loada.done guard here, plus implicit self-loop if any transition condition not met

I don't like the implicit nature of these guards and transitions. I think we should be very explicit with what the guards and transitions are:
0: { loada; } => loada[done] ? 1; default: 0

The default case must be specified for each transition and should explicitly point out what the state of the FSM is going to be if no transition is valid.

Based on this idea, with nested control and offloaded FSMs, we need only check the "leaf" FSM state

I like this idea! I can't find the issue but @paili0628 was working on changing the go-done interface so we don't have to assert the go signal for the duration of the execution.
Another a possible strawperson proposal for combining static and dynamic FSMs:
staticfsm my_fsm {
    s0 : { mult } => %3 ? s1,

The %3 guard is the usual timing guard that asserts it will be true on the third cycle.

I don't like the implicit nature of these guards and transitions. I think we should be very explicit with what the guards and transitions are:

My thought process here was, if we have a construct for specifying dynamic FSMs in which groups are invoke-able (and, say, arbitrary assignments are not allowed), then some redundancy may exist if we expect users to list all conditions of transitioning to a new state, including the done condition of the group(s) that they just invoked.
For example, if we wanted to invoke A, B, and C in state s_i and transitition to s_j exactly after these groups are completed, then the following syntactic sugar:
fsm my_fsm {
    ...
    s_i: { A, B, C } => s_j,
    ...

may be nicer than:
fsm my_fsm {
    ...
    s_i: { A, B, C } => {A.done && B.done && C.done ? s_j : s_i} ,
    ...
}

I do think that allowing arbitrary assignments, and not just groups, to be executed in a given state is a good reason to make sure all transition conditions are defined by the user. But, if only groups are allowed to be invoked, then would this be redundant?

Welp, you've managed to demonstrate why I think provide arbitrary assignments is good idea. For:
fsm my_fsm {
    ...
    s_i: { A, B, C } => s_j,

There is no guarantee that A, B, and C all end at the same time so the guard A.done & B.done & C.done is not correct. For example, if A takes two cycles and the others take one cycle, that expression above will never be true (remember: done signals are asserted for exactly one cycle).
What you wanted to express there is that A, B, and C all execute in parallel which can only be implemented using distinct FSMs for each of them when we're dealing with dynamic control programs. Now, the two choices are either those invoke FSMs for each thread or you carefully generate all the right assignments.

I guess I can't think of a case where an FSM might want to transition without the guarantee of having completed a group call

I think the intuition is reasonable but here is one case: if you have multiple static groups executing in parallel and some of them complete before others, you want the relevant states to only have particular assignments in those states.

One other possibility is allowing for some kind of fork and join primitive. It might be particularly interesting if you were allowed to fork an FSM's execution and then join in a different state. Not sure if this is a good idea but this will allow hack that I was talking about where a par thread gets to execute on the parent thread:
a;
par {
  seq { t0_a; t0_b }
  seq { t1_a; t1_b  }
}
d

Turns into:
fsm parent {
  0: { a } => { a.done ? 0 : 1 }
  1: {
      b;
      t <= fork thread; // `fork` is immediate. When `thread` is done, it writes its done signal to `t`
  } => { b.done ? 2 : 1 }
  2: { d } => {
    d.done && t.out ? end :
    d.done ? 3 : 2
  } action {
    t.out ? reset t : {};  // set t to 0 if thread finished computing
  }
  3: {} => { thread.done ? end : 3 } action {
    t.out ? reset t : {};
  }
}

fsm thread {
  0: { c } => { c.done ? 1 : end }
}

This is a pretty maximal design with some trade-offs:

Each FSM has it's own end state which finishes execution and asserts the done signal. The done signal is asserted for one cycle and de-asserted afterwards.

This means that the parent needs to have some mechanism to write the child's done signal when it is asserted. The t <= fork thread syntax states that some extra logic is created that stores the done value into register t.

An alternate decision would be to require the FSM to explicitly specify when the FSM resets to the start state. This would get rid of this extra register that needs to be kept around in the parent FSM but would require the parent to manually reset the internals of child FSMs
Another challenge here is that we would have to validate that every forked FSM get blocked on at some point to ensure that we correctly implement par semantics.

Actually, maybe this does not matter? Maybe we want users to be able to define computations that just happen on the side and don't really get checked up on

This is a strawperson design but it highlights some trade-offs in the space. This whole business with resetting shows the the challenges of hand writing FSMs. We need to strike a balance in explicitness and how easy these will be to generate correctly.

I just realized that the above won't work because:
  1: {
      b;
      t <= fork thread;
  } => { b.done ? 2 : 1 }

Would repeatedly attempt to fork thread. We need some other way to continue executing b without forking thread. The problem is that something like this also doesn't work:
  1: {
      b;
      t <= fork thread;
  } => { b.done ? 2 : 3 }
  2: {
    b
  } => { b.done ? 2 : 3 }

Because b might take exactly one cycle to occur in which case we'll execute b two times...

Hey Rachit, thanks for this cool fork/ join primitive idea! In your final code snippet in the first case, I was wondering what the purpose was of transitioning to state 2 if b.done. Wouldn't you want to transition to 3 in this case? If so, then b can take one cycle and not run twice.
Does this achieve what you were thinking?
  1: {
      b;
      t <= fork thread;
  } => { b.done ? 3 : 2 }
  2: {
    b
  } => { b.done ? 3 : 2 }

Huh, I think that would love my problem! Wonderful catch!

Cool idea to write out the “maximal” version with a built-in pair of fork & join operators (category 3 above). It might also be interesting to write out a more minimal alternative, where instead of fork & join, FSMs just have a basic way to trigger other FSMs and a way to access their done signals (category 1 above)? It would look different, surely, but I’m not yet sure how different…

Hm, I'm pretty sure this is a minimal version because any implementation will require a mechanism to trigger several FSMs in parallel (fork) and check their done signal (join).

Sure, maybe it's the same thing—as long as those two constructs are literally nothing more than a control-port access.


Delete ParameterDistributionType

There is no need for this distinction in the EF_Distributions or at least do it in a way that doesn't disturb.

Done


Added word cost feature to user dictionary

I added the word cost (not the connection cost) feature for the user dictionary. The last column represents the cost of its word.
If you have any comments or concerns, I would be very happy to hear them and fix them.
I have tested using the command mvn -Ddownload=true package and it successfully went through all the tests, including newly added tests.
Please do not merge yet (should preferably be after a future quick meeting).

Let's implement this feature as we have discussed in the next major version.


[Chore] Repo name change; Project name change

Description
Maesh changed it's name to Traefik Mesh. This repo and name of the Meshery adapter needs updated to reflect this change.

@layer5io/adapter-maintainers FYI. Repo name changing now.

@leecalcote can we close this issue now?

@leecalcote can we close this issue now?

Sounds good

Sounds good


Crashed when console log was too loong!

When I got thousands output log, try to resize the window, crashed !!!

/feedback


gce-staging-1: enable buildsh tracing, terraform-managed s3 creds

refs https://github.com/travis-ci/reliability/issues/154

Seems to be working! 👍

Since it's already applied, I'll go ahead and merge this. Happy to amend if there is further feedback later.


Rename --uri to --concept or add alias

Concept URI template is specified with concept in vocabularies.yaml and with --uri on command line. The latter should be renamed to --concept to match names.

Yeah, agree. We can add --concept and deprecate --uri. Feel free to add a PR if you have time.


Live data update support?

Quick question does this library support live-data-update from Firestore (eg in Kotlin it's a Stateflow) where the variable is updated if the value in Firestore updates.
If no then ignore the rest of the issue.
if yes how exactly is this being done, I'm making a firebase-sdk for Rust and I'm not sure how to support live-data-updates

I think you're asking about some features that you can find in more generic and high-level libraries.
This library focuses on Firestore integration, so probably it is not a big issue to integrate some reactive/rx library with this if you want to.

Ok 👌 thanks

Hm, listening changes based on gRPC streaming.
Have you seen docs:
https://github.com/abdolence/firestore-rs#listening-the-document-changes-on-firestore
And example?

I'm sorry I wasn't specific enough but I mean like how did you implement as in the source firestore API documentation that you used to make this useful lib not your implementation documentation

I'm probably still a bit confused with the question.
I just read the official Firestore gRPC docs here:
https://firebase.google.com/docs/firestore/reference/rpc
And then it depends on which library you use for gRPC, and in my case it is tonic and they support gRPC streaming.
The source code is not that huge to analyse if you want to implement something yourself:
https://github.com/abdolence/firestore-rs/blob/master/src/db/listen_changes.rs

Ok thank you. Just to clarify, I was wondering how exactly did you implement the listening to changes on Firestore, as even after attempting to reverse engineer the Firebase-android-sdk for Firestore, I did not understand how it was done so that's why i was asking the question


Update Arithmetic.hpp to fix bug in return units of cross function.

Description
Previously, wrong units were generated for cross function for any combination apart from both Cartesians. That bug has been fixed.
References

Link to the issue : https://github.com/BoostGSoC19/astronomy/issues/79

This PR is replaced with https://github.com/BoostGSoC19/astronomy/pull/111


NixOS 21.05: buildPythonApplication import conflict when two applications are built using different interpreter versions and same named dependencies

Describe the bug
I have created an MWE with two dummy applications (they only import numpy and print a message when run) that I build with the following inputs:
  new_app = pkgs.callPackage ./new_app/default.nix {
    inherit (pkgs);
    inherit (pkgs.python3Packages) buildPythonApplication numpy;
  };

  old_app = pkgs.callPackage ./old_app/default.nix {
    inherit (pkgs);
    inherit (pkgs.python2Packages) buildPythonApplication numpy setuptools;
  };

new_app/default.nix:
{ buildPythonApplication, numpy }:

buildPythonApplication {
  pname   = "new_app";
  version = "1.0.0";
  src     = ./.;

  doCheck = false;

  propagatedBuildInputs = [numpy];
}

old_app/default.nix:
{ buildPythonApplication, numpy, setuptools }:

buildPythonApplication {
  pname   = "old_app";
  version = "1.0.0";
  src     = ./.;

  nativeBuildInputs = [setuptools];

  doCheck = false;

  propagatedBuildInputs = [numpy];
}

When I supply these two applications as buildInputs to a package that I create a nix-shell from and run them, I get the following error due to new_app trying to use numpy from python2.7:
$ old_app.py
>> Old app main

$ new_app.py
Traceback (most recent call last):
  File "/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages/numpy/core/multiarray.py", line 13, in <module>
    from . import overrides
  File "/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dynamic module does not define module export function (PyInit__multiarray_umath)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/sg9hssj0mqb9q7fssfdjj4sp9b4v35kf-new_app-1.0.0/bin/.new_app.py-wrapped", line 4, in <module>
    import numpy
  File "/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)

I get the same error with and without the --pure for nix-shell
I get the same error when using the applications in a phase of my downstream package that is built using nix-build

Steps To Reproduce

Download the zip-file containing the MWE
Extract the zip-file
Run nix-build default.nix
See the error unfold

Expected behavior
When I run these apps I should get the following output:
$ old_app.py
>> Old app main

$ new_app.py
>> New app main

Additional context
old_app.py contents:
$ cat $(which old_app.py)
#! /nix/store/xvvgw9sb8wk6d2c0j3ybn7sll67s3s4z-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/4prbf2c5ya287bp8gyi1n7bi4r413bzs-python-2.7.18/bin:/nix/store/i3yi7b9xmc7jp06afcc5fdq8hxf9b02c-old_app-1.0.0/bin:/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/bin'${PATH:+':'}$PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/i3yi7b9xmc7jp06afcc5fdq8hxf9b02c-old_app-1.0.0/bin/.old_app.py-wrapped"  "$@"

.old_app.py-wrapped contents:
$ cat $(dirname $(which old_app.py))/.old_app.py-wrapped
#!/nix/store/4prbf2c5ya287bp8gyi1n7bi4r413bzs-python-2.7.18/bin/python2.7

import sys;import site;import functools;sys.argv[0] = '/nix/store/i3yi7b9xmc7jp06afcc5fdq8hxf9b02c-old_app-1.0.0/bin/old_app.py';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/i3yi7b9xmc7jp06afcc5fdq8hxf9b02c-old_app-1.0.0/lib/python2.7/site-packages','/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages'], site._init_pathinfo());
import numpy

#-------------------------------------------------------------------------------

if __name__ == '__main__':
    print('>> Old app main')

new_app.py contents:
$ cat $(which new_app.py)
#! /nix/store/xvvgw9sb8wk6d2c0j3ybn7sll67s3s4z-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/zv167zw7gwr9h0gpkfcikdfqnscrd0q6-python3-3.8.9/bin:/nix/store/sg9hssj0mqb9q7fssfdjj4sp9b4v35kf-new_app-1.0.0/bin:/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2/bin'${PATH:+':'}$PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/sg9hssj0mqb9q7fssfdjj4sp9b4v35kf-new_app-1.0.0/bin/.new_app.py-wrapped"  "$@"

.new_app.py-wrapped contents:
$ cat $(dirname $(which new_app.py))/.new_app.py-wrapped
#!/nix/store/zv167zw7gwr9h0gpkfcikdfqnscrd0q6-python3-3.8.9/bin/python3.8

import sys;import site;import functools;sys.argv[0] = '/nix/store/sg9hssj0mqb9q7fssfdjj4sp9b4v35kf-new_app-1.0.0/bin/new_app.py';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/sg9hssj0mqb9q7fssfdjj4sp9b4v35kf-new_app-1.0.0/lib/python3.8/site-packages','/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2/lib/python3.8/site-packages'], site._init_pathinfo());
import numpy

#-------------------------------------------------------------------------------

if __name__ == '__main__':
    print('>> New app main')

PYTHONPATH contents:
$ echo $PYTHONPATH | tr ':' '\n'
/nix/store/i3yi7b9xmc7jp06afcc5fdq8hxf9b02c-old_app-1.0.0/lib/python2.7/site-packages
/nix/store/ab2w199fgr4zaa668knd81zwl6bm2yjd-python2.7-numpy-1.16.6/lib/python2.7/site-packages
/nix/store/4prbf2c5ya287bp8gyi1n7bi4r413bzs-python-2.7.18/lib/python2.7/site-packages

It seems that the outputs are wrapped correctly and that the problem comes from PYTHONPATH containing paths to the python2.7 packages that are used in old_app.
Notify maintainers
@FRidh
Metadata
$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.52, NixOS, 21.05.1987.16bf3980bfa (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210707_02dd6bb`
 - channels(root): `"nixos-21.05.1987.16bf3980bfa"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

There is no separate PYTHONPATH for the different interpreters and buildPythonPackage propagates inputs. Propagation of inputs is what we need to get rid off. https://github.com/NixOS/nixpkgs/pull/102613

Thanks for the quick response! :+1:
What is your advice for me to solve the problem I'm having? Wait for that PR to land or can I employ some workaround in the meantime?
The only solution/workaround I can come up on spontaneously is to overwrite PYTHONPATH in the executable wrapper to only include paths to the packages that were given inside propagatedBuildInputs.

In the derivation where you use these two applications as buildInputs and which you nix-shell on, you can unset PYTHONPATH in e.g. the preBuild.

Works like a charm! Thanks a bunch for the help :1st_place_medal:


Environment in env file not loading properly

Hi Roots team,

[+] I've read the guidelines for Contributing to Roots Projects
[+] This request isn't a duplicate of an existing issue
[+] I've read the docs and followed them (if applicable)
[+] This is not a personal support request that should be posted on the Roots Discourse community

Description
Environments added in env file not loaded correctly.
Versions
1.12.6: 2019-05-21
OSX MAC 10.14.3
Additional information
My environment wasn't loading properly so I looked at what was going wrong and was able to fix it by making the following change in the application.php file.
From:
$env_config = DIR . '/environments/' . WP_ENV . '.php';
if (file_exists($env_config)) {
require_once $env_config;
}
To:
$env_config = DIR . '/environments/' . env('WP_ENV')  . '.php';
if (file_exists($env_config)) {
require_once $env_config;
}

I see the mistake I have made. Please disregard this issue.


Why are all my sample's *.GCcontent_RefVsReconstructed.png files the same?

I analyzed 15 samples. The original line and the reconstructed line overlap.
This is my configuration:

Hi chenzemin.
It seems like you are analyzing shallow sequenced sample(s) because the estimated number of processed fragments is below 7 M although 50 M was set as target (preset 2).
Therefore, the entire list of predefined genomic intervals will be used during the GC bias computation. The combined fragment GC content distribution of these genomic intervals (linear combination with region weights between 0.1 and 10, each determined individually) will always be the same if these regions are used consistently for all of your samples.
The printed graph should give an idea if the element-wise optimization is bettter than the naive combination of predefined genomic intervals. In your case, not. Therefore, the original and reconstructed lines are identical. A deviation of ~0.25 GC% of the mean remains.
Sidenote: something looks odd about the height diffference between the reconstructed curve vs. the reference GC% curve. I have to look into that. I think this just affects the visualisation, not the computation though.
Hope that helped.

Hi chenzemin. It seems like you are analyzing shallow sequenced sample(s) because the estimated number of processed fragments is below 7 M although 50 M was set as target (preset 2). Therefore, the entire list of predefined genomic intervals will be used during the GC bias computation. The combined fragment GC content distribution of these genomic intervals (linear combination with region weights between 0.1 and 10, each determined individually) will always be the same if these regions are used consistently for all of your samples.
The printed graph should give an idea if the element-wise optimization is bettter than the naive combination of predefined genomic intervals. In your case, not. Therefore, the original and reconstructed lines are identical. A deviation of ~0.25 GC% of the mean remains. Sidenote: something looks odd about the height diffference between the reconstructed curve vs. the reference GC% curve. I have to look into that. I think this just affects the visualisation, not the computation though.
Hope that helped.

Thank you very much for your response. I indeed only used 1X depth data for analysis and utilized the preset genomic intervals provided by you in the hg38 version. So, are my GC correction results incorrect now? Is it mainly due to the influence of sequencing depth or the preset genomic intervals?

Why would your results be wrong? The preselected genomic intervals only determine which regions are used to compute GC bias. These regions are valid. Their combination closely resembles the expected reference GC distribution.
Only the number of fragments on which the correction is based is limited in case of processing sWGS samples. The computed bias should still be valid even if it will be more of a rough estimate.
I don't see an issue here per se.
Do you have any other reason to suspect an error in the correction? Can you probably share an image of the output weights matrix? (outliers limited and smoothed)

Note: I think I understand the confusion now. The graph you are looking at does not visualize the achieved correction for each sample!
GCparagon tires to combine correction results from various genomic intervals a bit more clever than just averaging them.
GCparagon combines results from genomic intervals using a weighted average, one factor for each region. This factor is computed using the reference sequence of the respective genomic interval and a reference fragment length distribution (-> "fragment GC content distribution"). This has nothing to do with the input BAM file. It is solely defined by the reference sequence and the coordinates of the preselected genomic interval. The computed FGCD (= distribution of GC content of fragments in percent) is computed for each interval using the reference genome sequence, not the alignments form the BAM file. A reference GC content distribution has also been pre-computed using the entire reference genome (this is the red curve in the plot) and the reference fragment length distribution.
This is what is meant with the plot title "Reconstructed from Genomic Intervals vs. Genome".
If you want to validate the correction, you would need to compute the FGCD from your corrected fragments (not the reference genome) and compare against a valid reference GC content distribution (i.e., the red distribution red in the plot; I linked the reference FGCD file for the healthy sample H01) analogous to what was done in the publication.
If you are interested in validating the correction result, have a look at the validation script 03-1_create_corrected_fragment_gc_distributions.py. You will find plotting code for the result in (03-3_plot_per_fragment_GC_content_correction_fidelity.py)[https://github.com/BGSpiegl/GCparagon/blob/including_EGAS00001006963_results/validation/03-3_plot_per_fragment_GC_content_correction_fidelity.py]. beware that these scripts try to compare the GCparagon correction of the 4 samples from the publication to the Griffin correction results. You would need to remove the parts of the code dealing with Griffin if you want to just compare your sample(s) against the reference GC content distribution. And of course change the paths to hte corercted sample BAM files.
I can assure you - your results are not invalid just because no improvement for combining interval results was found over naive averaging (as stated before, this happens when the preselected genomic intervals already closely mimic the reference genome GC content distribution).


"Enable extension on this domain" doesn't show whether it's enabled

Proposed Solution
I think by default the extension should be enabled for github.com domain and the Enable Refined Github on this domain option on right clicking should appear on other domains only. Also, if a particular domain has enabled the extension then the option should say Disable Refined GitHub on this domain and clicking on that should disable extension on that domain.

RGH is obviously already enabled on GitHub.com
What you're suggesting however is good. The contextMenu API lets us add a checkmark next to the item, so that would be a good way to show where it's enabled.
I haven't made this change because it means that the contextMenu needs to be instantiated for each tab (or something like that) instead of once.
Feel free to add the checkmark and thus the ability to disable the domain as well to the original module: https://github.com/bfred-it/webext-domain-permission-toggle
I opened an issue there: https://github.com/bfred-it/webext-domain-permission-toggle/issues/4

This was fixed in https://github.com/fregante/webext-domain-permission-toggle/pull/8


Use path helpers in the redirects

Superseded by: https://github.com/alphagov/govuk-coronavirus-vulnerable-people-form/pull/199


Alternative CreateTemporalActivityDefinition

What Changed
Allow the usage of AsyncServiceProvider in the ServiceProviderExtensions for NET6+ during CreateTemporalActivityDefinitions.

Backwards compatible.
Choose the package to run as NET6.0. Preprocess directives are only looking for NET6_0_OR_GREATER making it very easy to change to NET6+.

Compiler recommend converting delegate variable...
Func<object?[], Task<object?>> invoker = async args =>

...to a local and  async function.
async Task<object?> Invoker(object?[] args)

and I fully agree. After feedback, this went through the rounds as being compatible.
Why?
Uses an AsyncServiceScope which allows the IAsyncDisposable interface to be invoked when created by CreateAsyncScope from the IServiceProvider.
This is a more modern way to handle IAsyncDisposable classes that are used in Dependency Injection. Without it, you can trigger these types of exceptions.
Unhandled exception. System.InvalidOperationException: 'MyClass' type only implements IAsyncDisposable. Use DisposeAsync to dispose the container.

Also handles IDisposable the same as it did before.
Checklist
1. Closes
Should close issue #204
2. Testing
Ran all UnitTests locally to ensure backwards compatibility. Future testing needed when the main Temporalio is brought up to be compatible with NET6 or NET8.
224/224 Passing.

3. Any docs updates needed?
I do not believe so. This code follows typical usage as presented by Microsoft internally to AspNetCore/BackgroundServices or through the dotnet/runtime.

Thanks again!


Long QEMs go off the screen

On StraboSpot version 2.0.0 on iPad and Android, long QEM (e.g., Geologic unit, Measurement, Sedimentary rock types, metamorphic rock, add interval) go off the screen in horizontal orientation.  Users are then unable to close the QEM or save/add interval in some cases.

fixed with commits bca73b2f28081986fbfcda1422163b31bc0c7b21, 608f143fb1c317bd17c3aa163f27e6080a9b1d6c, aecfa620e180c59a23ae1e70acb7de1fd3a99f4b, cc9d04a3e6dbd0c7a01c6337dee12f4fe34728d9


proposal: sub-cell topology configuration to improve replica query performance

Background and Motivation
Currently vitess components in a topology are allocated into one or more cells, which typically correspond to a data center in a physical deployment. For a given shard, it is possible to have replica tablets in multiple cells, while the master tablet is in any one of the cells. The vtgate query routing logic is also aware of this cell topology and will route replica queries to a tablet in the same cell as the vtgate, and by design will only cross cells when routing to a master.
However in many real world scenarios there are topological considerations below the level of a data center. For example, in AWS EC2, hosts are provisioned in a "region" (i.e. data center) and an "availability zone" (i.e. a building/rack within the given data center). As such, there are demonstrable differences in performance, latencies, cost, and fault tolerance between connections that stay inside the same availability zone versus those that cross the availability zone boundary. Other cloud deployments and physical topologies likely share the same characteristics in which the network interconnect between hosts in the same rack / building / cluster can be more performant than between other pairs of hosts.
Proposal
The proposal is to augment the vttablet and vtgate configurations to optionally include a "sub-cell" identifier alongside the existing cell identifier. The sub-cell of each vttablet would be communicated to the vtgates via the normal healthcheck protocol. Then, when selecting a replica for routing queries, the vtgate would prefer a vttablet in its same sub-cell as opposed to round robin among all the tablets in the cell. This way the vitess topology would be made aware of the performance considerations in the real world deployment and could more predictably exploit the connections inside the same AZ.
Because it may not be practical to always have a replica tablet in a given sub-cell, it is important that this logic be implemented as a routing preference and not a constraint. In other words, if there is no replica in the same sub-cell as a vtgate, the vtgate should route it to any other available replica in the cell as it does today.

In case of EC2 you should treat the "availability zone" as Vitess cell and not the "region". (This is also how we would use Vitess cells internally i.e. we would run multiple Vitess cells in the same region.)
Would this avoid introducing this "sub-cell" layer?

@michael-berlin We discussed that idea internally -- because replica queries do not cross cells (by design), it requires us to have an available replica in every availability zone. That's why the sub-cell would be a preference and not a constraint.

We could also go the other way, and add the concept of 'region', group of
cells, and if a cell doesn't have any good replica, we go to the region's
replicas.
Note this would help us internally a lot more to look at it this way, and
would be easier for us to roll out internally...
On Mon, Nov 6, 2017 at 9:23 AM, Michael Demmer<EMAIL_ADDRESS>wrote:

@michael-berlin https://github.com/michael-berlin We discussed that
idea internally -- because replica queries do not cross cells (by design),
it requires us to have an available replica in every availability zone.
That's why the sub-cell would be a preference and not a constraint.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/youtube/vitess/issues/3355#issuecomment-342221478,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEaVMkT7EDJO2O6BZ-o35ikPkwUPViQOks5sz0B6gaJpZM4QTk6A
.

@alainjobart That could work as well and I can see some logic to that. but at first glance that seems like a more intrusive and potentially error prone approach to me for our deployment.
From what I can tell, to do this with the region approach would require something like:

Add the new Region configuration option to all vtgates / vttablets and restart everything with the new setting. Since all components are still in the same cell this should be no change to the behavior.
Create N new cells, one for each AWS availability zone (or rack or whatever).
Reconfigure each of the vtgates with the cell now set to the current AZ. Update the -cells_to_watch setting to include the original cell that still has all the tablets as well as the N new cells. At this point there are no tablets in the same cell as the vtgate, so it would always bubble up to the region.
Reprovision each tablet to put it into the right cell. Ideally this would be just a configuration change and a restart, but looking through the code I don't have 100% confidence that things will "just work" if the same tablet comes up again in a new cell.

In contrast, if we went with the sub-cell approach, then I believe all that's required is to restart all the vtgate / vttablet components one by one with the added sub-cell configuration. Since the point of the new option is just to optimize the replica reads, there is much less risk of breaking something since even if something is misconfigured during the rollout, the worst that can happen is that we'd have sub-optimal read performance, while there is more orchestration needed (including a full redeployment of all tablets) in the region approach.
Can you confirm my understanding is correct here?

Let me add a few things here:

I think we want to find the best target architecture, the migration strategy should not be the deciding factor here (it's important, and if there is no possible migration, maybe we're doing something wrong! but I don't think that's the case)

The 'cell' concept is also used for topology server: we support one topology server per cell now (along with the global one). What makes sense in this cell/sub-cell or region/cell world? Inside Google, we'd need it at the lowest level. So if we went with region/cell, and kept the topology service with the cell, that would work best for us. I also think it's the most flexible, as multiple locations topology services can co-exist in one topo server, it's possible to put all sub-cell topo services in one topo service.
So all in all, I think for this, keeping the topology at the lowest level works best, since then we can support both a topo service per cell, or per region.

do we want to keep one of region or cell optional? I don't think so. If distinction is unused, they should both be set to the same value, probably.

(using region/cell concepts): should the cell names be unique across all regions? Right now the cell name is used in tablet alias, so it has to be. Also used for topology service configuration, again has to be unique. So I think so.

So my vote would be for region (higher level than cell). I agree with your assessment on the transition, it's complicated, because your tables are now per-region, they would need to move to per-cell. but your plan sounds good though.
What do you think?
@sougou @michael-berlin any opinion?

I think HubSpot would also be interested in this. cc @lumost @acharis
We also run our cells cross-AZ currently, which is non-ideal for read replicas. I don't think having a cell per AZ makes sense for us either, similarly to @demmer. I also agree it would be more work to do the region change, but it does sound cleaner TBH.

@alainjobart Thanks for the thorough comments.

You make a very good point -- we should err towards the side of the best final architecture as long as there is some viable migration path.

Speaking on behalf of Slack, we would not want to run a separate topology service at the lowest level. We in fact currently run a single consul topology for both the vitess global and local spheres (though obviously with different paths in consul). I didn't fully consider this in the migration plan above -- it seems like this would also require setting up a topo service in each local cell (or sharing the installation as we do).

My original thought was that cell would be required and sub-cell would be optional. This was partially driven off easy of migration for existing deployments, since no-one would be forced to change their configurations when the feature was introduced. If we go with your proposal for Region and Cell then I agree they should be required but of course allow them to be set to the same thing.

My instinct is yes that all names should be unique. In fact it seems like the names would have to be unique in order to allow deployments (like ours) to share the same topo service in the various cells. I see no clear benefit to allow for non-unique cell names and I think it would just make things more confusing.

With all this in mind, my vote is still for Cell / Sub-Cell as originally proposed.
In closing, I think that Sub-Cell is a terrible name, and something like Zone would be better.

One comment first (and using region/cell naming): if the topo server is per cell, it's trivial to share one in all the cells of a region (and in fact, the paths used in the topo service usually already include the cell). But if the topo server is per region, it's impossible to have a per-cell one if needed. So to keep the flexibility, we'd want the topo service per cell, I think.
Sugu and I talked a bit more about this, and explored a possible solution: add an optional 'Region' string field in the CellInfo record. Then keep vtgate the way it is, with 'cells to watch'. Then when vtgate receives a health check from a tablet (and it's not from the same cell as vtgate), it looks up the CellInfo record, and sees which region it's in (with a cache). If in the same region as the vtgate cell, it uses the tablet as a backup if no local tablets are found. We think it's fairly straightforward to configure (at first, it's not configured, but when you want to use the feature, you just add the 'region' settings, and it works).
@demmer with that proposed solution, since we change the cell definition under you, you would need the migration you mention. :(

But if the topo server is per region, it's impossible to have a per-cell one if needed. So to keep the flexibility, we'd want the topo service per cell, I think
That's certainly true, and as you allude it probably tips the overall best solution more towards Region/Cell as opposed to Cell/Zone.

Sugu and I talked a bit more about this, and explored a possible solution: add an optional 'Region' string field in the CellInfo record.
That seems like a good plan to me.

@demmer with that proposed solution, since we change the cell definition under you, you would need the migration you mention. :(
Well, as they say... no good deed goes unpunished.

When all is said and done I agree that Region/Cell is probably the better approach for the project as a whole, even though Cell/Zone is more immediately convenient for Slack's deployment.
Another idea to throw out there would be to add both the Region and the Zone concepts to the system. Admittedly that is a lot of layers of topology configuration, but besides the immediately applicable convenience benefits for Slack's deployment, I would imagine that there might be topological organization inside the same data center (e.g. the physical rack) where it would make sense to have information exposed to the vitess components, but where you wouldn't want to go through the trouble of setting up a topo service.

👍  for Region + Zone. I realize that the original choice of "Cell" was probably to keep things agnostic, but the reality is that most users of Vitess are going to be running in the cloud. Having familiar terms is going to set up users to succeed without having to drill down into the meaning of "Cell", especially if Vitess can set the right defaults for choosing the best read replicas to hit.

@derekperkins For the record I agree with you that Cell is confusing concept, but my last comment was actually that vitess would have all three constructs:

Region (encompasses one or more Cells)
Cell (what it is today, one or more Zones)
Zone (the smallest unit)

If Vitess was going to use all three, I would flip the last two. Region and Zone have specific meanings, so that third designation that could refer to a datacenter rack would make more sense IMO as a cell. Having a cell be some designation between a region and a zone would be confusing.
I'm not trying to overly complicate things, but in the spirit of @alainjobart's "I think we want to find the best target architecture," making Vitess concepts easier to comprehend will go a long way towards driving adoption.

Given how widespread the Cell terminology and concept is in the code, it seems quite complicated to remove it from the lexicon (and codebase) altogether.
So, in the spirit of moving this forward, I retract my earlier opinion and agree to drop the Zone / Sub-Cell idea altogether and instead add Region as a construct that represents one or more cells, as suggested by @alainjobart.
This would mean that:

Cell remains as the smallest unit of organization.
Region is an optional additional layer of grouping for related cells.
Each cell has either a fully distinct local topo service or a unique path hierarchy in a shared topo service.
vtgate would be configured with -cells_to_watch to include all cells in the topology (as it would in a multi-region setup)
vtgate's routing logic is modified so that replica reads prefer the local cell but are allowed to cross to other cells in the same region.

Open questions / design decisions:

How should the region be configured / stored / communicated?

My original thought was that we would configure each vtgate / vttablet with a region as part of the command line flags. Currently vtgate is configured with a -cell so I presume we would add a required -region parameter for this change. From what I can tell, vttablet gets its cell configuration via the -tablet_path argument which is required to be of the form cell-id. For consistency if we go this route, I'd say we should a required -region flag instead of overloading the tablet path argument. This would also require including the region in the health check protocol so the vtgates would know the region of each tablet.
The proposal that @alainjobart mentioned above would be to add it to be an optional element of the CellInfo, which would require enhancing the AddCellInfo / UpdateCellInfo vtctl methods accordingly to enable the region to be set. One challenge here is that there doesn't seem to be any support for watching the cell info and there is a comment that explicitly states the cell info isn't designed to be changed at runtime. I think that means that if a change is made, all the vtgates would need to be restarted to pick it up.
Of the two I think putting it in the CellInfo makes more sense and is less disruptive.

Does the routing change need to be controlled by a flag (e.g. something like -allow_cross_cell_same_region_replica_reads`. My intuition is no -- we make this the new default behavior and users can control whether or not they want this based on how they lay out the components into cells / regions.

Just a comment from my side. I currently have "pools" of db servers to allow me to isolate traffic between apps and their backend database servers. Normally a backend database is in a single pool but in some cases it may be in more than one pool. I also use "data centres" or "zones" too so am not quite sure if the naming is confusing me or you are looking for something different.
I guess the ideal situation is that apps talk via vtgate to a backend server "as close as possible" but in practice I also have situations where the backend servers in some systems are not located even in the same datacentre and as suggested by @demmer I also don't want to always have to deploy backend database servers just to suit the app.
Latency between one area and another may not be the issue sometimes and inevitably when talking to a master in a multi "zone" db setup the latency to the master is given by it's location: it can't be close to all app servers.
So I would favour vtgate (as this should be close to the app) being able to "route" to some keyspaces which may not be in the same zone/cell etc, and possibly the appropriate weighting to determine that preference need to be configurable as they may change over time or for maintenance you want to move to a backend different location while leaving the apps running happily.
Even in a single cell I currently have different "pools" and this allows me to optionally isolate traffic from one app from traffic from another one. Sougou has said that this concern over isolating apps is less necessary in Vitess due to the way it works and the protection it provides but I do see an advantage to having this isolation. (even if it may not be necessary).
I'm currently using the MySQL interface so probably to support multiple pools I'd need to have more than one vtgate running or maybe allow vtgate to be able to listen for connections on different ports so that it can "identify" the intended target based on the keyspace etc.  Having each vtgate configured separately may be easier even if it requires more proceses to be configured.
So I'm not sure if they comments help in any way: clearly there seems to be a desire to support a more flexible approach which has worked well for YouTube for some time if that's possible.  I'm not far enough in the evaulation to be sure if this is needed yet but will follow this issue carefully.

@alainjobart / @sougou We've started working on this implementation as part of Slack's fork.
@inexplicable will be taking care of the implementation using the approach of adding region to CellInfo -- I tried "assigning" this issue to him but the autocomplete doesn't work. Perhaps that's because he's not a collaborator on the github project?
Also when you get a chance, please comment on my open question 2 above -- whether or not we need a flag to control the routing behavior.

A quick update on this issue based on discussions:

No additional flag should be needed to preserve old behavior. The absence of a specified region for cells should result in old behavior.
This feature interacts with the algorithm that chooses non-lagged replicas. A hard-coded behavior should be chosen. for now In the future, all this must be factored into a user-specifiable policy file that will allow finer control on this behavior.

in progress now: https://github.com/youtube/vitess/pull/3460

@inexplicable pointed out some corner case behavior: if the current vtgate cell has no tablet for a keyspace/shard, there will be no SrvKeyspace for that cell, and we will fail even before we try finding the tablets.
Which got me thinking of a different error case: if the destination cell has a different SrvKeyspace that the vtgate cell (because of an on-going horizontal resharding), we might not only use the wrong shards, but also hit unavailable tablets.
So all in all I'm thinking we're missing one piece of the puzzle here, is that we need to re-resolve keyspace_id to shard if we need to hit a different cell. That resolving happens at a higher level in vtgate. That's where we use the SvrKeyspace to see the shard map.
One way to solve this would be to, instead of returning a tablet in a different cell, returning a specific error. Then have the higher level resolver in vtgate find another cell in the same region, and use that to resolve (reading that SrvKeyspace) and try to route to it. This probably requires a bit more detailed design than this, but I think this could work.
@sougou what do you think?

In more details, the code here:
https://github.com/youtube/vitess/blob/master/go/vt/vtgate/resolver.go#L135
already does a retry, if the error returned by the underlying level is retryable. We could leverage that, and also keep track of which cells work or don't work. So when we re-resolve, we could re-resolve using a blacklist of cells that don't have tablets. The flow would look like:

try the regular cell
no tablets -> return a specific error
higher level catches that error, backlists the current cell
try again with a blacklist of cells that contains the current cell. Need to re-resolve in the possible destination cell.
try to find tablets again.

I had a chat with @alainjobart about this. We agree that the short term fix will be to retry at a higher level.
In reality, this feature request has stretched out the current design beyond its limit. So, we need a new design/model. What we brainstormed was along these lines:

Mapping keyspace ids to shards is not a meaningful function any more. This function should be evolved to map keyspace ids to cell+shards, and this function should take into account the health of the vttablets underneath.
The design has to work correctly for l2 vtgates. Although this is a currently unused server, it's an important architectural component to guarantee our ability to scale indefinitely to "thousands" of shards.
The design has to allow us to define and implement meaningful load balancing policies.

@alainjobart has agreed to write up a design doc for this.

This PR has the design doc:
https://github.com/youtube/vitess/pull/3496
Note it's more of a description of what we do now. It also happens to have a couple proposed changes that would make things so much cooler.

This is done now. Going to close this issue.


User links

Add user links like https://tunnelgram.com/u/hperrin to directly start a new conversation.

I bought tgram.app to make short user links. Like: tgram.app/hperrin.

Actually, I bought tngm.me instead. http://tngm.me/hperrin looks better.


Introduce a new repository for district heating system models

I have been talking to several oemof developers and have learned that a couple of us are working on district heating networks. We already discussed the topic at earlier dev meetings, recently at a conference in Copenhagen and on github.
On github, there is the issue https://github.com/oemof/oemof/issues/618 and the PR https://github.com/oemof/oemof/pull/619 of @joroeder , which proposes and implements a new component for district heating pipes. Also, recently @joroeder  showed this example of a district heating optimisation, which builds a district heating system from input files: https://github.com/quarree100/dhs_optimization
Modeling district heating systems goes beyond a single component. There is a lot of detail in describing the topology and properties of the district heating network as well as in the specific problem setting one aims to answer with the model. These details arise wether you try to build the optimisation models using solph, simulation models using tespy or any other approach. A structured package could help building such models and also coupling them.
Therefore, I would like to introduce a new repository for district heating systems which we can develop together. I imagine the package to provide these functionalities:

Defined data format,
Importers/exporters from/to

geodataframe
graphml
shapefile
...

Import from osm,
Defined problem definition of district heating optimisation and simulation,
Model builders that create a solph/tespy model,
Consistency checks,
Postprocessing functions, derived metrics,
Possibly other nice functions like network aggregation,
Plots/Maps,
Examples,
Tests.

As I mentioned, there are already a couple of efforts to address district heating.  To combine these, it makes sense to join them in a meaningful way. Therefore, it makes sense to have a common repository under oemof. Given the sustained interest in the topic of district heating,  I assume that such a package would be worth to start and also there would be some interest to maintain it.
One could argue that oemof-thermal would be the right place to put it. However, the functionality described above is a bit different to the other thermal energy components. With the advice in mind that repos should represent entities with a well-defined scope and purpose, I would rather do this on a separate one.
I have set up a repo and developed a first structure that we could use to build the package described above. At the moment, it is under the oemof-heat organisation: https://github.com/oemof-heat/DHNx.

module structure
a central network object
importer structure
a structure to build different models (simulation/operational optimisation/investment optimisation)
classes for plots
tests
CI
examples

osm import
plots

If you like the idea and agree, we could move it to oemof and develop it together.
In addition to this, we got some validation data we will be able to publish under an open license, which could serve as a benchmark.
@c-moeller, @FranziPl, @jakob-wo would support the initiative.
I reserved a session on district heating system models on the next dev meeting (Wed, 15.30), where we can discuss and work on the topic together.
What do you think?
@oemof/oemof-heat
@oemof-developer

Thanks for this issue, I think that's a good idea. Let's discuss at the dev meeting how to continue ...

I think there was nothing against such a repo. Hey, how shall we proceed? Put everythink we have in a new repo and than start to structure?
We need a name, how about simply "oemof-dhs", dhs for district heating systems

Hi @joroeder, thanks for asking! Let's proceed as we concluded at the dev-meeting in December. I promised I would clean up of the basic structure of the existing repo in our project (https://github.com/oemof-heat/DHNx) according to the sketch that I showed. When I am ready with this, I would like to move the repo to oemof, so we can start to integrate the different parts.
Today and this week I am working on the cleaner structure, so I hope it will be ready by next week. If you want to have a look in advance to get an impression, check out the repo and especially this PR: https://github.com/oemof-heat/DHNx/pull/21
If you like, you can give some feedback already, but I promise to improve accessibility in the next days.
I started calling it DHNx, alluding to the networkx package. But I realized that a name that is easily pronounced is a good feature. So let's discuss about a good name (now or later). To avoid getting stopped by that, we can take anything as a working name and make a final decision before the first release.

Update: I did not finish all steps for the move last week. This week will work out.

I moved the repo to oemof. I assume that the further discussion will take place in issues on the repo, so I close this one here.


oem/ami: update tests for etcd2

The v1 API has been removed, so use v2 instead. The 10-second sleep was
added because the fleet tests were failing without it. My guess is that
etcd needed some time to warm up before we flooded it with requests.

Hah! I knew we were going to forget about the v1 API in there.
Also ugg on the 10 second sleep... I'm sure whatever is going on there is going to show up in all sorts of places for lots of people.
LGTM

Yeah, that sleep needs to be revisited...


Backup notifications

Have the add-on running for almost a month without problems.
Since a few days getting notifications concerning backups.
Did not change any settings.
Backup folder is set to /backup/dsmrreader

The problem seems to be with conflicting versions of Postgres:
(OSError) dsmr_backup.services.backup.run errored: b'pg_dump: server version: 12.4; pg_dump version: 11.10 (Debian 11.10-0+deb10u1)\npg_dump: aborting because of server version mismatch\n'
Off course the TimescaleDB addon wil get backed up if you use Google Drive Backup anyway...

It's related to this issue. There is a fix on the way, so expect an update in the coming days.

@Toolhead , @ewoutdenheijer just released DSMR Reader 0.1.2. Can you check if it's fixed now?

Seems to be working now. At least, i can make a backup using:
. /app/bash.sh
./manage.py dsmr_backup_create --compact
Thanks!


Convert Kunaki from REXML/Builder to Nokogiri

Remote tests are working too.
@Shopify/shipping for review.

:+1:

:boat:


Making relief mapping faster

Relief mapping is slow, here are some framerate recorded on my system with all of our presets and for each of them I enabled and disabled relief mapping. This is recorded for a 4K screen resolution on an AMD Radeon PRO W7600.

preset
RM off
RM on

lowest
1250 fps
780fps

low
1100 fps
650 fps

medium
600 fps
330fps

high
580 fps
280 fps

ultra
580 fps
290 fps

This single feature removes 500 fps from the framerate of lowest and low presets, and 300 fps from the framerate of medium, high and ultra presets.
Actually don't ask me why with ultra preset and relief mapping enabled I get 10 fps more than with high and same feature enabled, I tested multiple time and reproduced it. 🤔
Anyway, the root problem addressed by this issue is the slowness of relief mapping.
The big performance loss between low and lowest presets compared to medium, high and ultra is the enablement of multitexturing with normal mapping, specular mapping, etc. which adds a lot of code and binds many more textures. We may also lose performance in some way we switch between shaders (something the material branch attempts to fix).
But, the relief mapping single handledly consumes as much as performance as all the features enabled in medium presets.
Among possible improvements we may investigate, @SomaZ said this in developper chat channel:

texture2d is usually slow in loops. using textureGrad can improve performance as it doesnt need to determine mips again and again

Here is some Khronos documentation about textureGrad:

https://registry.khronos.org/OpenGL-Refpages/gl4/html/textureGrad.xhtml

I forgot to say the scene I used for the benchmark was plat23 default spectator scene.
When running the game at 4K with relief mapping enabled, the game CPU usage is only 32%, so the game is waiting on the GPU.

Looking at the current implementation, there's probably also a lot of lane divergence and texture cache thrashing with the current implementation. If I understood it correctly, it's looping over the heightmap 16 times in the direction of view origin to find some sort of depth value, then 6 more times around that point to find the closest value in the heightmap or something like that. The first loop is also missing a break;. Shader compilers might be optimising out the other iterations that do nothing, but they also might not.

Surprisingly, replacing texture2D with textureGrad with constant (for the fragment) derivatives obtained once from dFdx and dFdy actually made performance a lot worse, though I could've made an error somewhere. Maybe this results in a lot of texels being accessed at a lower mip-level?

Is this fixed by 83244ed8442530e2776d4297b66dbb7a1ceacda6? It was claimed to be worth "hundreds of FPS" in the 0.55 release post.

In some way, yes, but the initial issue was not about a bug hurting performance, but about the possibility to rewrite the code using faster functions, though @VReaperV tried and said:

Surprisingly, replacing texture2D with textureGrad with constant (for the fragment) derivatives obtained once from dFdx and dFdy actually made performance a lot worse, though I could've made an error somewhere. Maybe this results in a lot of texels being accessed at a lower mip-level?

I wonder if there was an error or not…


update may/may not be happening when tablet is idle/suspended/closed/standby

Steps to reproduce
(I am not sure the correct term to use for this, and am not sure if this is happening or if this is a my tablet kind of problem)
close the tablet when it is updating library,
open tablet an hour or so later and see it has made 0 progress
Expected behavior
update continues when the table is in the 'idle/suspended/closed/standby' condition
Actual behavior
it seems to just stop waiting for the tablet to be open/on again
Crash logs
No response
Neko version
2.12.5
Android version
android 13
Device
galaxy tab s7+
Other details
Its possible that this is a table specific problem and not a software one, I have no idea which, this could also be in my head but I notice that when I open neko now I don't see a wall of new things like I use to on the kindle, I notice the updating never really stopping, and it always looks about where it was before.
if you happen to know what the problem is and its tablet related, I would appreciate the help.
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] I have tried the troubleshooting guide.
[X] I have updated the app to the newest version Latest.
[X] I have filled out all of the requested information in this form.

I would look through this
https://dontkillmyapp.com/samsung

that did it, thanks


Problems not showing up when making "Zabbix Problems" panel.

I get the following Error when creating a Zabbix Problems panel on my dashboard. I know I have problems on my node, but it says No Problems Found.
This is the error:
Invalid params. Invalid parameter "/output/2": value must be one of "proxyid", "name", "operating_mode", "description", "allowed_addresses", "address", "port", "tls_connect", "tls_accept", "tls_issuer", "tls_subject", "custom_timeouts", "timeout_zabbix_agent", "timeout_simple_check", "timeout_snmp_agent", "timeout_external_check", "timeout_db_monitor", "timeout_http_agent", "timeout_ssh_agent", "timeout_telnet_agent", "timeout_script", "lastaccess", "version", "compatibility".

Software
ZABBIX 7.0
Grafana 4.4.3

duplicate of #1714

@zoltanbedi when will this issue be fixed?

When we add support for zabbix 7. I can't say an exact date sorry.


AxesTimeseries component

Composed component of AxisBottom and AxisLeft. It's specific to timeseries, since the x axis explicitly uses a date scale.
Storybook

I would recommend using the default tickValues and tickFormat until we implement better versions of both functions (we did it in a rush at the time, when we had more than a year of data). Currently, the component doesn't render well if we have shorter or longer time spans (try starting on 2020-01-01 or 2022-06-01).
One option for now is to set numTicks to 8 or so, and to remove the custom tickValues and tickFormat so we can write and test better implementations of those functions.


Validation - inconsistent behavior

Hello
Inconsistent behavior on validation.
It will not throw error when password confirm does not match unless another error was thrown before.
In another case, it will not throw an error when there is more than one constraint.
Also, it will sometimes enable submit even if fields are still invalid.
Specifically on the sandbox below, click on the 'REGISTER' button and attempt to fill in the form.
Input correctly all fields with the exception of the password confirm field. Make it so that it DOES NOT match the password field. You should receive no error.
Then, reload and make an error in any other field before the password confirm. Then make again so that the passwords do not match. Now you should receive the error.
This behavior repeats itself on all forms. If the first field does not throw an error, the following will not throw either.
Sandbox link:
https://codesandbox.io/s/m3nqpqommp
Additionally, and I could not replicate this on the sandbox, on the route below the form will not validate whenever there is more than one constraint.

          <Route

            path="/resetting/reset/:token"
            component={PasswordResetHandlerPage}

On /pages/password_reset_handler/components/password_reset_form.js, it will accept the validation if only one constraint is placed, like so:

validate={[required]}

but not with multiple constraints, like so:

validate={[required, minLength(8), match("fos_user_resetting_form[plainPassword][first]")]}

Environment:
node.js version v8.9.0
    "dependencies": {         "axios": "^0.18.0",         "babel-runtime": "^6.26.0",         "react": "^16.2.0",         "react-dom": "^16.2.0",         "react-redux": "^5.0.7",         "react-router-dom": "^4.2.2",         "react-scripts": "1.1.1",         "redux": "^3.7.2",         "redux-form": "^7.3.0",         "redux-promise": "^0.5.3"     },

@BernardA, I'm just moving this discussion back to here to keep the other issue focused. I recommend using a function like get from lodash when selecting paths from a tree structure.
In your validate function isMatch:
export const isMatch = matchName => (value, allValues, props) =>
  value !== allValues[matchName]
    ? //? `This field must match with ${matchName} field`
      "Passwords do not match."
    : undefined;

You should do instead
export const isMatch = matchName => (value, allValues, props) =>
  value !== get(allValues, matchName)
    ? //? `This field must match with ${matchName} field`
      "Passwords do not match."
    : undefined;

And this will likely work. At the moment you are essentially checking if a key exists on the object allValues like "a[b][c]".

Sorry it took me a while. It works fine as you indicated. Thank you very much.


Introduction of [va_codegen_success_]

This PR introduces a notion of successful code generation. This is to be used purely for debugging purposes, and has no effect on the va_code/va_lemma/etc.
In order to introduce the ability to have arbitrary debug output capabilities, we rely upon the PossiblyMonad's pbool, which is always either ttrue (equivalent to true) or is ffalse reason (equivalent to false but also must provide a string reason).
The expected use case for this is for the .ml extracted files to look at the relevant va_codegen_success and when false, show the message to the user.

The current handling for recursion is quite a hack, and fails under certain cases. The fix for this would be to properly handle if/else constructs, and handle recursion via that.
Handling if/else constructs correctly is also necessary for constructs like KeyExpansionRoundUnrolledRecursive256 in Vale.AES.X64.AES256.fst.

The if/else constructs are now handled better; this both ensures that recursion is handled correctly, as well as ensures that constructs like KeyExpansionRoundUnrolledRecursive256 in Vale.AES.X64.AES256.fst are handled correctly.


Fix release workflow

See: https://github.com/olegtarasov/get-tag/issues/14

LGTM, shouldn't that get covered by dependabot?

Yeah I was thinking the same. For some reason it is missing this one...


[$1000] Chat- When reacting to a message, ask the other user for money, your reaction does not appear

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

First open two accounts with different users, in this case of user A and user B
Start chat and click in the plus sign, the user A
Click Request money the user B
Now with user B reacts to the money request message

Expected Result:
That the user A view the reaction of user B at the exact moment
Actual Result:
User A, to see user B's reaction, have to update the page
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

Version Number: 1.3.43-3
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/93399543/8aa76606-c58b-43bf-871e-db5b3b1701f7
https://github.com/Expensify/App/assets/93399543/b8908fea-65ba-4ca6-8c35-2cbf45d932f6
https://github.com/Expensify/App/assets/93399543/f4fab24f-10d7-44a9-94e8-5d65d6705110
https://github.com/Expensify/App/assets/93399543/be67ff98-f10d-455f-b048-21f22beb809c
Expensify/Expensify Issue URL:
Issue reported by: @Mauri-salazar
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1689862826572919
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~014b7dc3e4e47914f3
Upwork Job ID:<PHONE_NUMBER>590973952
Last Price Increase: 2023-07-25

Looks like backend is not sending the socket event

📣 @ygshbht! 📣 Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:

Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.

 Restating again nonetheless
Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: Link

Had some OOO followed by some unexpected OOO today. Just getting to this now.

Not reproducible. Please clearly describe the reproduction steps

@s-alves10 Can you please ask for better reproduction steps in the corresponding slack thread https://expensify.slack.com/archives/C049HHMV9SM/p1689862826572919

Not overdue. Still looking for proposals (and possibly reproduction confirmation)

Issue not reproducible during KI retests. (First week)

Closing this: https://expensify.slack.com/archives/C049HHMV9SM/p1691097564955689?thread_ts=1689862826.572919&cid=C049HHMV9SM


Ccd

Looks good. Is there a reason why we shouldn't keep the simple test case in as well as the GP one though? I think we need to remove the %matplotlib inline line from ccd_tests.py I believe this is our bug though.

Thanks for the comments Alan, its a bit cleaner now, also I remember you had made some changes to the log marginal objective function, which you can merge in here.

@adhaka Hi - bit late here, so hope this still makes sense.
Alan and I had a go at the CCD stuff - I've tried to summarise our confusion in a jupyter notebook.
https://github.com/lionfish0/ccd/blob/master/Test CCD complete.ipynb

@alansaul @lionfish0 any reason this is still hanging?

I wasn't confident I understood what the correct answers were - so I wasn't sure if I was writing correct test code...I've had a chat with Aki in the meantime and so I think I could probably finish it.
I actually might want this functionality soon, so I'll look at getting it sorted when I get back (November).

Closing due to inactivity, incomplete PR description & incomplete PR name.


[HUDI-8680][DNM] Enable partition stats by default1

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

Closing in favor of https://github.com/apache/hudi/pull/12511


TFW: Archive and upload test coverage reports on PR/Push

Work completed, currently shelved as waiting for an update to the Gradle wrapper included with Yoko.


Stained glass, panes, and concrete powder are broken

1.20.30 introduced separate item names for each variant.
So all stained glass blocks, stained glass panes and concrete powder are not showing in the pack.

Was this resolved by your pr?

Fixed


Trouble with Smart Trades

Thank you for putting this together!
I'm having trouble creating Smart Trades in a paper account. I know I have my API and secret properly entered because I can get an account list from both paper and real.
When I try to create a Smart Trade, I get the error:
{
    "error": "signature_invalid",
    "error_description": "Provided signature is invalid"
}

If I send only account_id and pair, then I get:
{
    "error": "record_invalid",
    "error_description": "Invalid parameters",
    "error_attributes": {
        "position": [
            "is missing"
        ],
        "position[type]": [
            "is missing"
        ],
        "position[order_type]": [
            "is missing"
        ],
        "position[units]": [
            "is missing"
        ],
        "position[units][value]": [
            "is missing"
        ],
        "take_profit": [
            "is missing"
        ],
        "take_profit[enabled]": [
            "is missing"
        ],
        "stop_loss": [
            "is missing"
        ],
        "stop_loss[enabled]": [
            "is missing"
        ]
    }
}

But add any one of those fields and I get the signature error.
Any ideas? Or is this a 3commas problem?

Do you have any spaces in your url? if yes, that could be the issue

No.
It seems to be a problem with how the authentication script is handling an object or array.

Tbh, I think the problem is somewhere on the side of 3commas.
I cant send a Smart trade too.
Because the auth script does not change your object. It does not delete any fields.

Were you guys able to eventually create smart trades, or still borked on their side?

I gave up and used a different method to send the data.


Câmera with gallery

Feature Request
I would like to acess 'gallery' with the camera, so the user can choose his last pictures, or to put an option 'gallery' in the display, so the user that is acessing the camera can type on 'gallery' to acess it (directly from plugin's camera).
Motivation Behind Feature

Feature Description

Alternatives or Workarounds

Thanks a lot for your issue, however the issue template exists for a reason. 😊.
It helps us to debug the issue further and to provide a solution much faster. With important information missing, we unfortunately can't help you.
Therefore, please edit this issue accordingly or close and create a new one and make sure to provide all the required information.


Tooltips rendering behind the game

https://i.imgur.com/5RHLfK2.png

it is working for me

https://i.imgur.com/NojTnWO.gif
The shadow not showing on a the tooltip is probably because of my dying graphics card and not an issue with the client

Fix (at the cost of losing shadow on tooltips):
ToolTipManager.sharedInstance().setLightWeightPopupEnabled(false);
System.setProperty("jgoodies.popupDropShadowEnabled", "false");


Installation process

Can someone please explain how to install this from the start? I really need someone to explain the process.

I am struggling with this as well. From the README, I followed the setup found here in the /app folder. Attempted the npm install but got an error. Opened up the log file that was spat out in the terminal and it seems like there is a complaint with package.json ? 🤔
It's my first time working with ReactNative. Were you able to get this sorted out? I forked the repo and will see what I can find out on my end.

Unscribe me please
On Fri, 10 Dec 2021, 20:21 Gerald S, @.***> wrote:

I am struggling with this as well. From the README, I followed the setup
found here in the /app
https://github.com/sorianog/ar-cutpaste/tree/master/app folder.
Attempted the npm install but got an error. Opened up the log file that
was spat out in the terminal and it seems like there is a complaint with
package.json ? 🤔
It's my first time working with ReactNative. Were you able to get this
sorted out? I forked the repo and will see what I can find out on my end.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cyrildiagne/ar-cutpaste/issues/77#issuecomment-991268431,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOFECDRZGET6CH5LMFU46LTUQJOLPANCNFSM4TFBJIKA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TIL: You should run the commands in the correct directory....
I feel that this issue can be closed if others are able to run the app after seeing the comments here. 🤓


Add ‘On body entered’ entry block for Area2D and RigidBody2D

The block ‘compiles’ to a callback function, which needs to be connected
to the signal in question. To do this, add an additional code-generation
step that generates an _init() function to connect signals.
A more idiomatic approach would be to connect to the signals in the
_ready() function. This would involve merging additional code with the
user-supplied _ready() function, so is deferred for now. _init() works
fine.
https://phabricator.endlessm.com/T35510

In my working copy I deleted most of the blocks in the test game and recreated them to test this functionality; I've left that out here.


More informative warn logs

Describe the feature request
In poll_for_updates, errors are consumed and a fixed log message is printed instead. For example, if the HTTP request fails it simply prints warn!("poll: failed to retrieve features"). This is unhelpful for debugging. It'd be helpful to print the particulars of the error.
Background
I am currently attempting to use unleash-api-client in a cloud-hosted project. It has started sporadically failing, but without information on the particulars I can't easily debug it.
Solution suggestions
It'd be a case-by-case basis, but generally would just need to include a debug-formatted version of the originating error.

Hey, @skairunner 👋🏼  Thanks for the request and thanks for the proposed changes too ☺  That sounds like a great suggestion to me, so I'm all for it! But I'll need to align with the rest of the team internally. We should be able to get back to with some more info some time tomorrow at latest, I think. Is that alright? 😄

Sounds good to me!


"Cocora" is listed in the drop-down menu

Description:
When user writes ”Cocora”, the location appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Type "Cocora" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Cocora, IALOMIȚA".


Skip connector tests

Temporarily skip connector tests until connector APIs are re-added.

:hourglass_flowing_sand: Build in-progress, with failures

Buildkite Build
Commit: 7a525d628cb3d1fb0e98f345a45c15e2ea99a406

Failed CI Steps

:elasticsearch: :javascript: Elasticsearch Serverless Node.js v22 integration tests


[FEATURE]: More settings for YOLO8 face detector

Description
I use Yolo8 as detector backend for DeepFace.
Looking to sources I see that Yolo initialization lacks of parameters:
 return YOLO(weight_file)

    def detect_faces(self, img: np.ndarray) -> List[FacialAreaRegion]:
        """
        Detect and align face with yolo

        Args:
            img (np.ndarray): pre-loaded image as numpy array

        Returns:
            results (List[FacialAreaRegion]): A list of FacialAreaRegion objects
        """
        resp = []

        # Detect faces
        results = self.model.predict(img, verbose=False, show=False, conf=0.25)[0]

I'd like to fine tune Yolo8 params without overriding classes, for example, I'd want to run Yolo detector on GPU instead of CPU, so I need device parameter.
Also it would be good to setup  confidence, IoU and augment.
Probably it should be some dictionary with parameters forwarding from DeepFace constructor to  detector backend's.
Additional Info
No response

We may consider to manage them with env vars
Most probably, we will not able to add these features soon. Still, requests are fair enough and worth to stay opened status.

I confirmed that yolo is using gpu with the current implementation if you have a gpu. So, you don't have to do anything for this task.

will handle confidence score with another environment variable


Latest version appears vulnerable to CVE-2021-30246

We've recently seen an NPM advisory for a critical dependency issue relating to jsrsasign, https://www.npmjs.com/advisories/1672.
We use release 1.5.4 of oidc-client-js, and the advisory suggests we update to release 1.11.5 (latest). However, while trying to determine what the impact of this vulnerability was to us, we stubmled across this file https://github.com/IdentityModel/oidc-client-js/blob/dev/jsrsasign/header.js, which suggest that sometime in the past the jsrsasign dependency was brought "in-house", and at version 8.0.12 (There are commits that suggest this also).
Which would mean that, while the latest version of oidc-client-js does not have jsrsasign in its list of dependencies (And so does not flag dependabot or npm audit), it does still depend on it, and on an old version, so it is still vulnerable to this security issue.
Would you agree that is a fair assessment? We are trying to work out what I we need to to stop being vulnerable to this issue.

Hello there, we're in the same position.
Any updates on this one?
Thanks!

Isn't this a dup of an older issue?

Not that I could see @brockallen , I've just had a read of the titles of the 79 open issues and none jump out at me, and i'm not sure I can read all 1000 closed issues 😅

Any further thoughts on this @brockallen ?

I believe this was the issue I was speaking of: https://github.com/IdentityModel/oidc-client-js/issues/743

No, that is not the same issue (Or I am really misunderstanding). My issue is more of a question (As you have now labeled it), forget the NPM audit stuff for a moment.
First, a CVE exists which states that any version of jsrsasign before 10.2.0 is vulnerable.
Looking at the oidc-client-js source code, I find this file which appears to import jsrsasign from a local file, not from an NPM package.
That local file states at the top of it that it is version 8.0.12. Which would make the built version of oidc-client-js vulnerable to the CVE
Looking at the package.json shows that jsrsasign is now in the devDependencies (Which is why the latest version of oidc-client-js does not trigger the NPM audit warning), but that version is still vulnerable and, more importantly, only ever required in a sample
So yes, I guess my question is "NPM audit thinks everything is fine, but I don't think that is true, looking at what i've found do you agree with me, or am I wrong?" And if I am right, then, my issue is can we update the jsrsasign in ./oidc-client-js/jsrsasign/dist to the latest version to plug the CVE?

I'm happy to accept a working PR for this. The reason we have a copy is due to the lack of molecularity of that package and how heavy weight it is. So how we pull it in is to break it apart. So no idea if what's vulnerable from it is actually included in what we use.

jsrsasign removed the vulnerabilities a few years ago according to their changelog and this file
But please correct me if I'm wrong, cause my team is also looking into this :)


x/exp/cmd/txtar: expansion of environment variables and writing outside of the current directory seems risky

The x/exp/cmd/txtar command does not limit the scope of files it might overwrite, which seems risky (e.g., see https://snyk.io/research/zip-slip-vulnerability). Additionally, the expansion of environment variables could make it even easier for attackers to construct malicious files:
$ go install golang.org/x/exp/cmd/txtar@latest
$ cat hmm.txt
-- $HOME/.ssh/not_authorized_keys --
hmm
$ txtar -x < hmm.txt
$ cat ~/.ssh/not_authorized_keys
hmm

(Being plain text gives a small advantage that it's a bit harder to hide the malicious path, but for sufficiently large txtar files, a user is unlikely to review all of the file paths.)
I'd recommend that x/exp/cmd/txtar default to only writing to files within the current directory. If any resolve outside of the current directory, txtar should exit with an error. (I think exiting without writing any files would be safest, but it seems fine to still write in-scope files if that happens to be easier with txtar's implementation.) A flag can be provided to allow expansion to arbitrary file paths.
I'd also recommend not expanding environment variables by default, instead requiring a flag for that functionality.
/cc @bcmills @rsc @FiloSottile

A flag can be provided to allow expansion to arbitrary file paths.

Or — perhaps even better! — to provide an explicit list of allowed variables.

So I see two related risks:

txtar can write outside the current directory. This is a concern even without variable expansion; e.g., file paths starting with / or ../.
txtar's behavior can be sensitive to environment variables. This makes the above issue riskier (e.g., easy to point to files in $HOME, rather than needing to guess where that might be), but I think it also has concerns of its own (e.g., create a file named $USER within a directory, and then rely on this file being processed by a glob later, revealing the user that ran txtar).

I think providing flexibility for an allow list of safe environment variables would be okay too. I just didn't want to request too many knobs or for too much complexity to be added. I just want that by default when I run txtar on untrusted input, that I don't need to worry about my SSH files being clobbered. :)

I just want that by default when I run txtar on untrusted input

To be clear: you should be careful about untrusted txtar inputs either way. txtar archives can include dotfiles (independent of variable expansion), and there are an awful lot of ways folks can inject malicious behavior via various dotfiles in the working directory (such as .git/config).

I sent two CLs for this:

allow only extraction within the current dir and add an --unsafe flag for extraction outside the current dir
add an --env flag for allowed specific environment variables to be expanded

These flags currently affect only the archive extraction, i.e. they have no effect when writing the archive.
I did experiment with applying the flags also when writing archives, but it seems to me that that would add more complexity than needed.


Service issue

First commit moves the peerAdd/peerDelete operation behind a channel
Second commit deals with a possible deadlock on the pMap Lock. Because of the lazy instantiation logic of the overlay driver, the initSandbox has in all cases to fetch peers from the DB. Moreover this operation has to be atomic on the whole table to avoid a race with subsequent peerAdd/peerDelete that can asynchronously come from the networkDB.

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "service-issue" git@github.com:fcrisciani/libnetwork.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842353899344
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

For better tracking : This PR brings back https://github.com/docker/libnetwork/pull/1861 with the additional changes to avoid the deadlock.


added documentation

improved our haddock values

What went wrong on appveyor?
Doesn't seem related to the changes...
Seems more linked to move to stack 1.6.1 (and back)?

Looks like stack 1.6.1 is less stable than 1.5.1; it couldn't complete compilation even for LTS 9.13.
I added retry mechanism on LTS 10.0 branch. Let's rebase this after that one is merged.

It looks like it's in integersimple variant of ghc; sqatt can be compiled on first try with normal ghc but gives same errors and requires retries with ghc-integersimple.


Add Semantic Release

:tada: This PR is included in version 0.2.9 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Setup turbopack to improve DX

Blocker: sentry does not support turbopack (WIP):
https://github.com/getsentry/sentry-javascript/issues/8105

The DX with webpack is terrible, would be great if we could do this at some point.


assertion.undeclared is unreferenced

The error code assertion.undeclared is described as

An assertion was found in the ingredient’s manifest that was not explicitly declared in the ingredient’s claim.

But this requirement - which, when combined with assertion.missing effectively means the list of assertions in a manifest must be identical to the list of assertions in the manifest's claim - is not listed in the validation steps; assertion.undeclared is not referenced elsewhere in the spec.
Is this an actual current validation requirement, or is this error code a legacy from an now-dropped validation step?

You are correct: this is not a validation requirement. We never required it, although the presence of the status code shows we considered it, and recommend all assertions be referenced. But we couldn't identify any security risk from there being extraneous assertions in the assertion store; it's just wasted space. Validating this requirement would require a separate pass over the assertion store and matching of its contents to the assertions array in the claim. This seemed like a lot of extra work for no benefit, so we did not require it.