conformance: Add test for HTTPS Listener/Route with empty hostnames

What would you like to be added:
An explicit test for TLS/HTTPS requests to a Gateway/Route combination that does not specify any hostnames
Why this is needed:
Inadvertently while adding this test we have added this test, but we should have an explicit one for documentation's sake and so it is not conflated with other features.
We could maybe modify the Routes used in this test to specify a hostname as well so we don't conflate the functionality
Seems like same-namespace-with-https-listener is one of the few Gateways in the conformance suite that does not specify any hostnames

Hi, I wanted to pick this issue, seems a good first issue.  When we say that we want test that does not specify any hostnames, where exactly are we expecting hostname to be undefined, as I checked it is defined for redirect requests.

\assign

/assign

So, after digging around a bit I understand that we want a gateway where a hostname isn't defined for the listeners? Then we want to add tests where we specifically add hostname within the routes. Is my understanding of the issue correct?

/help

/remove-help

So, after digging around a bit I understand that we want a gateway where a hostname isn't defined for the listeners? Then we want to add tests where we specifically add hostname within the routes. Is my understanding of the issue correct?

We want to define a test that has no hostname matches on Gateway Listeners or Routes to make sure that implementations can successfully route requests to arbitrary hosts

@sunjayBhatia, I can't remember, do we specify that this should act as a fallback, that is, that Listeners that do match a hostname should take precedence over ones that don't? (I hope we do!)

@sunjayBhatia Sure, I'll create two tests then. One for the original requirement and the other for the wildcard vs perfect match of hostname.

Just checking in: how are we doing on this one @JSee98, @sunjayBhatia?

@JSee98 are you still working on this? If not we can find someone else who is interested or I can take it

@JSee98 that would be great 👍🏽 Let us know if you need any other support, and thank you for contributing!

Hi @sunjayBhatia, I'm having a little issue to setup the environment to run my tests. I followed the developer setup mentioned here https://github.com/kubernetes/community/blob/master/contributors/devel/README.md#setting-up-your-dev-environment-coding-and-debugging
Using this doc I tried to get a docker container running with the necessary services to run the conformance test, however, wasn't able to.
This is what I've tried:

I explored the make file in the gateway-api repo and found the make command for CRD. When I tried to run it the error was for missing KUBECONFIG env var. So tried setting that to the crd's yaml files but then got the error can't connect to localhost:8080
I also came across this page https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#running-conformance-tests-with-kind This seemed to suggest a needed that specific kubeconfig but had a similar issue there

I'd really appreciate if you can help me out here

Well overall it seems reasonable to do this:
/triage accepted
However it doesn't seem obvious that this needs to be done in a hurry, if someone has a different perspective please share it.
/priority backlog


graphs must be unrolled to align with POA (was: flipping breaks inverting edges)

This fails. The alignment gets corrupted by the node flipping process that's used to prep the graph for partial order alignment. (To be pushed soon as a test:)
vg align -s AGTCCTTGAAAGAGGGCAAAATAAACTGTTAGTAGAGCCAGGTCTGAAAACAACACTTTCTTGC inverting/m.vg >/dev/null

It exposed my own confusion. The inversions are being broken. This isn't quite what I'd expected (or hoped) it does.
For example.
Here's what I'd like:
Loops and inversions get unrolled into new nodes so the graph becomes partially ordered and compatible with gssw, but still contains the same sequence space as the input up to a given length of cycle expansions --- probably on the order of a read length. The translation is recorded. Then the alignment is modified so the path points at the original nodes in their correct orientation.
:tada: VG::unroll

Rendering with arrowheads helps:

Algorithm. Unroll and un-invert the graph into a DAG to prepare for application of partial order alignment (POA) as implemented in gssw.
We start with any node and begin to write a new graph where we un-invert inverting edges and unroll cycles. We need to record how far we go from the start when doing this and break when we reach some base pair limit.
There are four edge types.
For the first two we do nothing, we just copy the structure into the new graph.
from+ -> to+
from- -> to-

For the inverting edges:
from+ -> to-
from- -> to+

We convert from+ -> to- to from+ -> to+ in the new graph, step across and start inverting everything. If we are working in the other direction, and walk through a from- -> to+ link, we change it into a from+ -> to+ link and on the other side start inverting the graph we can reach from it.
Things get nasty if things loop, because we will be able to generate impossible flip combinations, such as by trying to unflip a flipped region of the graph. Unrolling the loops is also helpful for preparing ourselves to apply POA, so it makes sense to do here.
Loop unrolling means we will duplicate some nodes in the base graph in our new graph. To make the resulting graph useful, we have to record a translation from the new nodes into the old ones. This mapping also needs to record if the new node is a reverse complement of the first.
The traversal of the graph needs to be tracked. We want to maintain a list of steppers that traverse the graph in a kind of breadth-first search. Each bifurcation will generate a new pair of steppers from the input stepper. Rather than explode into a tree, in which each bifurcation leads to a new branch (and is inefficient to work with in POA), we want to ensure that we can merge things up again where there are non-looping bubbles. This could be accomplished by detecting the regions of the graph that will be stable under the transformation--- basically where there are no inversions and no cycles. We then aim to generate a tree based on these structures, rooted at our starting node.
What we want to do is first detect these components. We can project these components into a graph that describes their relationships with neighboring components in terms of inversions. Now, we build them into a tree that is long enough to simulate the mapping of our read to any position in the original graph (the unrolling distance). Then, we apply the appropriate reverse complementing, fix up the edges, and emit a ~tree/DAG that is the composition of all these copies of the various non-inverting, non-looping components of the graph.

I dug into this further yesterday with @jltsiren and @edawson.
I think I can now clarify the algorithm for unrolling the graph.
Our objective is to convert a cyclic, bidirectional graph into a unidirectional k-DAG, with some k on the order of 2x the length of the sequence we'd like to align. This allows the application of partial order alignment to the graph.
First, we find the strongly connected components of the graph using an algorithm like Tarjan's strongly connected components algorithm. By definition, strongly connected components must have internal cycles, so we can use these to guide the unrolling process. We start at the entry points of the strongly connected components and enumerate all k-paths that go into the component (a k-path is a path whose concatenated label length is less than k). Now, for each k-path we generate a new path in which the nodes are indexed by the number of times we have traversed them previously in the path. For instance, if we had path ABBCDABC, we would write it A1B1B2C1D1A2B3C2. We now take the paths converted as such and generate a new graph from them, which is a DAG in which each uniquely indexed node becomes a new node. Each Ni is, by definition, equivalent to other Ni's, in that any path crossing it has only seen Ni-1 or lower. This guarantees we generate an acyclic structure in which data dependencies are partially ordered. We extend this model to handle inverting edges by recording if we pass through a node in the forward or reverse direction, so the previous example might become A1B1B'1C'1D1A2B2C1, if N is the forward direction of the original node and N' is the reverse complement. In this way we handle both cyclic paths and also inversions in the unrolling process.
The result is a DAG to which we can apply partial order alignment. A mapping between the new nodes and nodes and orientations in the previous graph may be used to trivially project the mapping against the new unrolled graph back into the old one. In effect, this process enables the direct mapping of reads into cyclic, bidirectional graphs.

Here is the current sketch of the algorithm:

Find the strongly connected components in the graph.
Anything outside the strongly connected components can be copied directly into the DAG. We can also reuse the entry nodes to the strongly connected components.
Use backtracking search starting from each entry node of each strongly connected component, keeping track of the nodes on the path from the entry node to the current node:
If the current node is outside the component, terminate this branch and return to the previous branching point.
Create a new copy of the current node in the DAG and use that copy for this branch.
If we have found the first cycle in the current branch (the current node is the first one occurring twice in the path), initialize path length to 1 for this branch. (We need to find a k-path starting from the last offset of the previous node.)
If we have found a cycle in the current branch, increment path length by the length of the label of the current node. If path length >= k, terminate this branch and return to the previous branching point.

The algorithm creates a lot of redundant nodes, but we don't know how to merge them efficiently without creating cycles.

The algorithm creates a lot of redundant nodes, but we don't know how to merge them efficiently without creating cycles.

We can use partial order alignment to merge them without generating cycles.
We'd want to use partial order to partial order alignment to minimize runtime. I don't have an implemenation of this, and I'm not sure how easy it will be to adapt something like the original POA. Also, it might not help much in terms of removing redundancy, and further it might be very expensive even if completed in O(log(N)) steps for N sequences. It could still be useful if it were cached for the entire graph during the indexing step.

In examining @jltsiren's sketch above, I think there may be a problem with the restart condition:

If we have found the first cycle in the current branch (the current node is the first one occurring twice in the path), initialize path length to 1 for this branch. (We need to find a k-path starting from the last offset of the previous node.)

It seems to me that this means we won't find cycles that are less than k away from our entry points. However, we want to capture these even if they are embedded inside the component more than k steps from the entry points. To resolve, I think it's sufficient to obtain all k-paths for all nodes in the component, and then build them into a DAG by avoiding the introduction of any edges that would create cycles--- and instead adding new nodes. If the original node is N, then we'll call the next node N', and the one after that N'' and so on.
So the updated sketch would be:

find the strongly connected components of the graph
if the component has a single node, add it to the unrolled graph
for each k-path of each node in the strongly connected components:
make the node labels in the k-path unique, for instance ABCBC would become ABCB'C'.
for each node N in the modified k-path:
find the node in the unrolled graph which we are building, adding it if it does not exist
2. connect the node to its predecessor in the path, unless doing so would introduce a cycle
3. if connecting to the previous node in the path would introduce a cycle, add a new node N' and connect it to the previous node in the path

Working this out for the graph drawn at top left unrolls it into the graph at top right. The asterisk "*" marks indicate which nodes would get new identities in the graph due to the potential introduction of loops.

The problem with this technique is that naively checking for loops using strong component detection will cost us O(|N|+|E|) operations per path element. This doesn't need to be done for every element that's added, as unique ones will not need to be checked, but it is still painful.

Note that the example shows the unrolling for the entire graph, not just the strongly connected component.

Here is a simulation of my algorithm sketch in your example graph:
B (entry point, do not duplicate)
    C1 (not seen before)
        B1 (second visit, length = 2)
            E (outside the component, terminate)
            C2 (length = 3, terminate)
        D1 (not seen before)
            C3 (second visit, length = 2)
                B2 (length = 3, terminate)
                D2 (length = 3, terminate)
            D3 (second visit, length = 2)
                C4 (length = 3, terminate)
                D4 (length = 3, terminate)

In addition to the original path A -> B -> E, the acyclic graph would have two additional copies of B, four copies of C, and four copies of D. The graph would not contain the original nodes C and D.
The intuition is the following: For each entry point to each strongly connected component, we create at tree of paths starting from the entry point. To ensure that we end up with a tree, we create a new copy of each node we encounter. The paths can be of any length, but we terminate them once we get out of the component or when the current branch has continued long enough after closing its first cycle.
In the above example, B -> C1 -> D1 is a simple path, but any extension of it closes a cycle. To guarantee that the acyclic graph contains all paths of length 3 in the component, we have to find all paths of length 3 starting from the last node of the simple path (D1) and continuing with a successor that closes a cycle. Hence the length of the path is initialized as 1 + length(C3) = 2 and 1 + length(D3) in the successor nodes C3 and D3.
In the same way, B -> C1 is a simple path. Continuing to B1 closes a cycle, so we need to find all paths of length 3 starting with C1 -> B1.

Thanks @jltsiren, that clarifies things. I'm going to implement this today.

I have an initial implementation of this. Here is the result for the strongly connected component in the graph example I gave earlier.
The graph:

And the unrolled component with k=4.

Adding in the connections to the acyclic components:

For fun ad later reference, here's the input graph in json:
{"node": [{"sequence": "A", "id": 1}, {"sequence": "B", "id": 2}, {"sequence": "C", "id": 3}, {"sequence": "D", "id": 4}, {"sequence": "E", "id": 5}], "edge": [{"from": 1, "to": 2}, {"from": 3, "to": 2}, {"from": 2, "to": 5}, {"from": 2, "to": 3}, {"from": 4, "to": 3}, {"from": 3, "to": 4}, {"from": 4, "to": 4}]}

I'm working now on carrying forward the node translation.
A big missing piece here is flipping inversions, which was part of the objective of unrolling. I have written a draft which does not handle them. There is already functionality in vg to do this, but I'm not sure it's appropriate. Perhaps we can unroll first and then unflip the reversing edges. I need to think about this.

Here is a sketch of the process to reduce the complexity of the trees that are generated by the unrolling process. We want to compress them into DAGs so as to minimize the alignment cost, which is proportional to the length of the labels of the graph.
Say we start with this tree, as obtained earlier:

We then label all the nodes with their identity-relative height from the root. In other words, we label them with the number of times that we see a node with the same original identity before we reach them when walking from the root (here 1:B) This should explain:

The next step is to merge nodes that have the same original identity and identity-relative height. Because our objective is to reduce the complexity of the graph as much as possible, we merge the largest group of these first. In this case, we start with C2. We merge all of these into node 4:C, like so.

Now, continuing, we re-label the graph with the identity-relative height. We would then merge the next-largest group, in this case 5:B, 9:B, and 16:B, which comprise the group B3, resulting in this graph:

Now, we'd merge 6:D, 11:D, and 17:D, which comprise the group D4.

Finally, we merge 10:C and 12:C, which form the group C3.

The resulting graph has no more groups of nodes at the same height with the same original identity, so we stop.
The starting graph is of length 20, while we finish with one of length 11. This is a substantial savings, and will be more substantial with larger k.

Here's an outline of the algorithm to determine the original identity rank of each node in the DAG simplification process.
We will store a vector R with length equal to the number of original nodes in the strongly connected component for each node we reach. For each node n in topological order: Compute Rn as the element-wise max(Rm for all node m in parents of n) or [ 0 0 0 ... ] where n has no parents and increment the vector element corresponding to the original identity of the node in the input graph.
At the end, we traverse all the nodes and label them with the value  the corresponding Rn vector. This establishes the original identity ranks. We then use these as described in the previous post when collapsing the graph. In this way we avoid generating cycles and appropriately simplify the graph without modifying its sequence space.

I've coded up the tree simplification algorithm. Here goes for k=6 on the test graph we have here.
The size reduction is clear from looking at these. The first tree is length 65, the last is 18.
The algorithm does not seem very efficient to me. We could cache these unrolled regions. Computing them on the fly may be fast enough, but I don't have a sense of it.

Outstanding issues.

Inversions should be handled correctly in the unrolling process by flipping the node when building up the initial tree and also recording the relative orientations in the translation. This will require a lot of small tweaks to what I've written in the unroll branch, but seems straightforward provided obvious extensions to the data structures that record the translation.

Where a strong component has multiple input nodes, we should check if the input nodes have partial order relative to each other and choose the top-ordered one when unrolling the component. Otherwise, we end up with extra redundancy in the graph. However, we can't do much if the entry points have no obvious order relative to each other.

Tests of unrolling should use kmer enumeration to verify that input and output graphs have the same sequence space. This is the objective and is easy to check.

I can now handle inversions, at least where these occur within strongly connected components.
Taking this graph:

If we walk forward from B, we get to the reverse complement of C. Then we stay on the reverse strand until we get out by coming back from C reverse into B forward. It is the same as the previous test graph but the cyclic component is inverted.
Adding some special characters (instead of reverse complementing things) we can see the progression of the algorithm easily for k=4.
The original:

And the inverted:

I am going to test a few more cases of this, then push my commits to the pull request.
The next step will be to consider how to handle non-cyclic components with inversions. I don't know if the flipping process is handling these properly. The acyclic components could be handled using the same code so as to resolve inversions.

To test the process of linking the unrolled strongly connected component back into the graph, I modified the graph to include a loop that maintains the inversion, resulting in exit nodes that are inverted.
{"node": [{"id": 1, "sequence": "A"}, {"id": 2, "sequence": "B"}, {"id": 3, "sequence": "C"}, {"id": 4, "sequence": "D"}, {"id": 5, "sequence": "E"}], "edge": [{"from": 1, "to": 2}, {"from": 1, "to": 5}, {"from": 3, "to": 2, "from_start": true}, {"from": 2, "to": 5}, {"to_end": true, "from": 2, "to": 3}, {"to_end": true, "from": 3, "to": 2, "from_start": true}, {"from": 4, "to": 3}, {"from": 3, "to": 4}, {"from": 4, "to": 4}]}

It appears that unrolling works on this and correctly links things back. (k=4)

This is cool. The next step is to target inversions that lie outside of strongly connected components. I need to think about how to do this.

If it's not in a strongly connected component, then you can just flip any nodes you come across where you hit the end of the node when you were expecting the start, right? The ones outside strongly connected components seem like the easy ones.

@adamnovak let me try to explain what's confusing me.
I would like a function which maps an input graph to one that contains no cycles, and represents all sequences of length k in the original graph on the forward strand. It may contain some sequences of length > k that are found in the original graph, but not all those in strongly connected components.
To do this, I can first unroll strongly connected components with the approach described here so that the graph is acyclic and all sequences from strongly connected components up to length k are represented. I have adapted this process to handle inversions and it seems stable. It does not handle inversions in the acyclic components of the graph.
Approach 1: We are representing the reverse complement of the graph implicitly. We could reverse complement the graph to obtain it. Inversions then become crossovers between the forward and reverse graph.
This will work, but it is costly. From the perspective of an aligner, aligning against the full reverse complement of the graph may duplicate effort. We may be certain of our orientation when we initiate the alignment. In vg we collect subgraphs in the same orientation as our read, so this will be the typical state of affairs. If I am reasonably sure of my orientation, I would still at least want to be sure that I can align up to k characters on the other side of the inversion.
Approach 2: I copy the reverse complement of the graph up to k characters on the other side of an inversion into the graph and link it in so that it appears in the natural orientation of the graph.
This can do better than k in some cases. If I have two subgraphs that are connected by a single inverting edge, there is no problem--- I can just choose the orientation of one graph as "forward", then flip the other and I generate a graph that shows me the inversion from the perspective of a sequence that passes through it.
However, if I have edges connecting to both the forward and reverse of the node in question, I can't do this because it introduces contradictions. The node will have to be duplicated as in approach 1 into a forward and reverse complement.
Here is what happens now.
There are no cycles in this graph:

(For clarity: There is an edge from 1:A to 9:B- that inverts, and then another from 9:B- to 5:E. It is easier to note than to try to get it to plot right.)
If I run this graph through orient_nodes_forward, I get a fragmented result:

These cases can't be handled by orient_nodes_forward, so it breaks the graph. We can still align against this graph, but the alignment is greedy and so we don't get a complete alignment, only the alignment across the best-matching subgraph.

Hi Erik,
Somehow I am confused about something - and this might be just me.  You mention:

(For clarity: There is an edge from 1:A to 9:B- that inverts, and then another from 9:B- to 5:E. It is easier to note than to try to get it to plot right.)

But I see 9:C- and 11:B- in the top graph, with 9:C- and 11:-B in the bottom graph.  I am not seeing a 9:B-.  Does the order of the minus sign make a difference?
Am I misreading something?
Thanks,
Paul

Hi Erik and Adam,
I was playing with this with some other code I put together, and was wondering about something.  If a node has both a forward and reverse direction, why not just use the same graph where for each alignment instance one of those edges is disabled (or temporarily removed).  For example, if 11:B- has a direction coming from both 7:C- and 1:A.  Then you can run your alignment with the edge from 1:A (with the one from 7:C- disabled), and another alignment with the edge from 7:C- (with the one from 1:A disabled) on the same graph, without using any expensive space of multiple graphs.  This way the greedy alignment will work for the best-matching subgraph as intended on one connected graph, and should provide the optimal result.
Please let me know if I misinterpreted something.
Thanks,
Paul

In any case, below is the code I put together to test this, and hope it is okay if it is in Java.  Sorry if it looks a little hurriedly put together.  This file would be called ekgDigraph.java, in case you would like to run it:
import java.io.*;
import java.util.*;

public class ekgDigraph {

  Map<String, ArrayList<String>> adj = new HashMap<String, ArrayList<String>>();
  Set<String> dfsMarked = new HashSet<String>();
  int ConnectedComponentCount;
  Map<String, Integer> associatedComponent = new HashMap<String, Integer>();

  public void addEdge( String v, String w ) {
    ArrayList<String> currentAdjacentEdges = new ArrayList<String>();
    if ( adj.get(v) != null ) {
      currentAdjacentEdges = adj.get(v);
    }
    currentAdjacentEdges.add(w);
    adj.put( v, currentAdjacentEdges );
  }

  public ArrayList<String> adj( String v ) {
    return adj.get(v);
  }

  public ekgDigraph( String fileName ) throws Exception {

    FileReader graphFile = new FileReader(new File(fileName));
    BufferedReader bufferedReader = new BufferedReader(graphFile);
    String oneEdge;

    int edgeCount = 0;
    int vertexCount = 0;
    String [] edge;
    while((oneEdge = bufferedReader.readLine()) != null) {

      edge = oneEdge.split(" ");
      if ( !edge[0].startsWith("#") ) {  // Skip edge starting with # sign,
                                         // in order to have only a forward or reverse edge
                                         // into a node.

        int edge_count = 0;

        for( int idx = 0 ; idx < edge.length ; idx++ ) {
          if( edge[idx].trim().length() != 0 ) {
            edge[edge_count] = edge[idx].trim();
            edge_count++; // increment the count
          }
        }

        String v = edge[0];
        String w = edge[1];
        addEdge(v, w);
      }
    }

    bufferedReader.close();

  }

  public void findConnectedComponents() {

    ekgDigraph G = this;

    for ( String vertex : G.adj.keySet() ) {
      if ( !dfsMarked.contains( vertex ) ) {
        dfs( G, vertex );
        ConnectedComponentCount++;
      }
    }
  }

  public void dfs( ekgDigraph G, String vertex ) {

    dfsMarked.add( vertex );
    associatedComponent.put( vertex , ConnectedComponentCount);
    if ( adj.get(vertex) != null ) {
      for ( String adjVertex : adj.get(vertex) ) {
        if ( !dfsMarked.contains( adjVertex ) ) {
          dfs( G, adjVertex );
        }
      }
    }
  }

  public int count() {
    return ConnectedComponentCount;
  }

  public void printComponents() {

    ekgDigraph G = this;

    System.out.println();
    System.out.println( "The graph has " + G.count() + " components.");
    System.out.println();

    ArrayList<String>[] components = (ArrayList<String>[]) new ArrayList[ G.count() ];

    for ( int i = 0 ; i < components.length ; i++ ) {
      components[i] = new ArrayList<String>();
    }

    for ( String vertex : associatedComponent.keySet() ) {
      components[ associatedComponent.get(vertex) ].add(vertex);
    }

    for ( int i = 0 ; i < components.length ; i++ ) {
      System.out.print( "Component " + i + " => " );
      for ( String v : components[i] ) {
        System.out.print( v +  " " );
      }
      System.out.println();
    }
  }

  public void joinComponents() {

    ekgDigraph G = this;

    ArrayList<String>[] components = (ArrayList<String>[]) new ArrayList[ G.count() ];

    for ( int i = 0 ; i < components.length ; i++ ) {
      components[i] = new ArrayList<String>();
    }

    for ( String vertex : associatedComponent.keySet() ) {
      components[ associatedComponent.get(vertex) ].add(vertex);
    }

    boolean [] isComponentConnected = new boolean[components.length];

    for ( int i = 0 ; i < components.length ; i++ ) {
      for ( String v : components[i] ) {
        if ( !isComponentConnected[i] && adj.get(v) != null ) {
          for ( String adjVertex : adj.get(v) ) {
            for ( int rest = 0 ; rest < components.length ; rest++ ) {
              if ( i != rest && components[rest].contains( adjVertex ) && !isComponentConnected[i] ) {
                System.out.println( "Connect component " + i + " with " + rest + " from node " + v + " -> " + adjVertex );
                isComponentConnected[i] = true;
                isComponentConnected[rest] = true;
                break;
              }
            }
          }
        }
      }
    }
  }

  public static void main( String [] args ) throws Exception {

    ekgDigraph G = new ekgDigraph( args[0] );

    G.findConnectedComponents();

    G.printComponents();

    // Print a separator line
    System.out.println();

    G.joinComponents();

  }

}

The graph file has the following contents, which I called ekg-graph.txt.  It is based on the above graph, with only edges 1A 11B and 1A 26B remarked out using #, so that I simulate turning off one edge on a node that has both a forward and a reverse direction:
1A  6B
#1A 11B
1A 8B
1A 25A
#1A 26B
1A 5E
6B 5E
6B  7C
7C 8B
7C 11B
7C 23D
11B 12C
11B 5E
12C 13D
12C 8B
13D 17D
13D 14C
17D 14C
17D 15D
14C 15D
14C 8B
8B 9C
8B 5E
23D 28D
23D 9C
28D 9C
28D 27D
9C 27D
9C 25B
9C 26B
25B 5E
26B 5E

Below are the run results, which show a complete graph in that all the components connect to each other:
> java ekgDigraph ekg-graph.txt

The graph has 5 components.

Component 0 => 14C 15D 8B 9C 5E 26B 27D 25B
Component 1 => 11B 12C 13D 17D
Component 2 => 23D 28D
Component 3 => 6B 7C
Component 4 => 1A 25A

Connect component 1 with 0 from node 11B -> 5E
Connect component 2 with 0 from node 23D -> 9C
Connect component 3 with 0 from node 6B -> 5E
Connect component 4 with 3 from node 1A -> 6B

>

Hope it helps,
~p

I'm calling the "flattening" operation "unfold", with the metaphor being that the inversions are the folds of the graph.
@pgrosu I think this is roughly the same thing that you implemented? I haven't had a chance to run your code.
I've written it up and have what looks like a stable prototype, modulo all the debugging code I've added and weird way that I'm handling the nodes and lengths on the graph (I'm counting 1 for each node as length and adding + and - to represent what operations are done).
Taking our friendly input graph:
which resulted from unrolling:

Which we've unrolled to length 3:

We can unfold to length 50 (in this case, everything) and connect up with the accessible parts of the reverse complement of the graph. We then remove all inverting edges.

The added - behind the node names means they've been flipped in this step.
So it looks to be working and I'm going to cleaning up the code and testing it more seriously. The unrolling algorithm needs to be improved to ensure we don't end up in runaway cases that generate huge initial trees prior to compaction.

I now need to test this as a setup for alignment. Then I will close the issue.

@ekg Awesome, glad to hear you got the problem solved :smile:  Basically what I was doing is letting the program decide the components, and performing edge contraction implicitly to get around the issue of unrolling outside a component.  Afterwards I was just stitching them back together, to ensure a complete graph.  Again, one can use those vertices that have both a forward and backward direction as a bifurcation without unrolling.  So whatever path approaches that node, then the alignment would be topological ordered by proceeding along the 1...n edge(s) that continue in the same direction of the approaching edge.  The alignment would thus be recursive with a radius of k rounds in all satisfiable directions without the need to unroll.

@adamnovak so you were right, and the pathological cases occurred very quickly. I'm stubborn though so I followed through up to the point where things break. I have some hacks that prevent unrolling when there is too much branching (as branching makes the backtracking search blow up) but this isn't ideal and I'm following suggestions from @richarddurbin.
Basically, we unroll the graph "vertically" by copying it and connecting nodes in the first copy to nodes in the second copy where the original graph had such an edge. We then continue this process from the "newest" nodes. The result is a DAG, and it may be possible to run a phase of compression to remove redundancy and reduce the alignment cost. I'll detail the process here as soon as I've got the code to demonstrate them visually. The problem is that it may be expensive to determine when we've unrolled far enough to support queries of a given length. This could be tackled using dynamic programming to find the shortest length path through the unrolled graph, but I haven't worked this out yet.
Here's to being stubborn and trying the impossible :grin: The good news is a lot of the code that I wrote in support of the backtracking solution that @jltsiren and I worked out will be applicable to the O(NM) unroll.

@ekg Is it an issue of keeping it in memory or storing it, and afterwards performing the joins?  How big is the tree - 100s of GBs, several terabytes, petabytes? Those workloads can be easily offset with the tree being indexed on disk.  I'm not seeing why this bottleneck cannot be easily solved.  Sure the shortest path algorithms (i.e. Dijkstra's, A* search, etc.) are fairly common approaches.
Thanks,
~p

@pgrosu The trees grow exponentially with the number of branching edges in the graph, and could quickly approach scales that are simply unthinkable. @jltsiren expressed concern about this and I went ahead and tried it anyway to see what would happen on real data. In real data the trees get very big very quickly. So I am going back and doing the sensible thing with an algorithm that will increase the time and space costs of aligning against strongly connected components only linearly with the length of sequence we want to align against them.

@ekg What would be your longest aligning sequence?  Yes, I know that trees can grow exponentially :)  If very large, why not generate the tree and unroll locally for small sections of the alignment, and the iterate only on paths of subtrees that show stronger alignment and then ignore the other unrolls.  What I am trying to say, is that some paths on the tree are more important than others, and the initial alignment could dictate which paths to keep unrolling on - basically alignments are not random.
~p

I still think there is a much more efficient way to do this dynamically on the graph.
Outer loop along the sequence to be aligned.  Inner loop across possible nodes in the subgraph.  The complication is deletions from the graph,
i.e. the sequence is missing something in the graph.  One way to handle this is to loop multiple times over the inner loop per outer loop, each
addititional one being a single base deletion.  It is possible to make this more efficient by creating new subgraphs that skip multiple steps - I
think there are several ways to do this.
Richard

On 15 Feb 2016, at 05:21, Erik Garrison<EMAIL_ADDRESS>wrote:
@pgrosu https://github.com/pgrosu The trees grow exponentially with the number of branching edges in the graph, and could quickly approach scales that are simply unthinkable. @jltsiren https://github.com/jltsiren expressed concern about this and I went ahead and tried it anyway to see what would happen on real data. In real data the trees get very big very quickly. So I am going back and doing the sensible thing with an algorithm that will increase the time and space costs of aligning against strongly connected components only linearly with the length of sequence we want to align against them.
—
Reply to this email directly or view it on GitHub https://github.com/ekg/vg/issues/182#issuecomment-184068777.

--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

@richarddurbin Or one can perform it the other way around for deletions, by taking paths along the graph and aligning them to the sequence.  Then the gap(s) will show up naturally :)
~p

@richarddurbin

I think there are several ways to do this.

There definitely are.
I am doing the DAGification up front in such a way as to maintain the sequence space of the original graph in the DAGified version up to a certain length. If this length is >= the length of our read, then we can be guaranteed the correct alignment against the first graph.
I think this approach makes sense, because I'd like to maintaining compatibility with POA, for which there are a number of SIMD-based implementations that provide large improvements in performance over non-SIMD implementations. GSSW generalizes Farrar's striped smith waterman to work on graphs. At the hackathon in Sendi I've met Hajime Suzuki, who has (as I understand from our conversations) has generalized a variant of Myers' bit-vector algorithm for approximate string matching to handle gaps. He is currently working on a manuscript describing that and further extending his method to work on graphs (see branch v3). I would like to be able to leverage his work once it is ready, and doing so would be simplest to implement via DAGification of the graph.
There are other ways to do it, and it seems to me there may be some regimes in which they will be preferable, but for the time being I am interested in minimizing effort and using available components as much as possible.
I have resolved alignment against cyclic and inverting graphs in the following way. I will illustrate with the example graph from before. In the test directory, vg view -d graphs/atgc.vg:

We rebuild the graph, first copying over all acyclic components of the graph, and then adding in a dagified version of each strongly connected component.
For each strongly connected component (SCC), we make a copy of the nodes without any edges. To "unroll", we add a new copy of the SCC, and for each edge in the SCC, we add an edge from the node in the first copy to that in the second. At each step, we track the minimum distance from a given node to a node in the first copy of the SCC. When the minimum of these distances is greater than the length of the sequence we want to query against the graph, we stop. Alternatively, we may unroll as many steps as desired.
The first step:

After two steps:

After 3 steps:

After 4 steps:

If we first apply the unfold process to the graph, we can ensure that there are no inverting edges and the reverse complement of the graph is available to reached out to our query length k. Then this dagify unrolling process will not yield any incidental cycles and will be generate a graph in which a read of length <= k can be aligned to the forward complement of the graph.
Note that we maintain a node translation so that alignments against this graph can be projected back into the source graph. The translation of these is trivial.
In summary, this procedure provides a linear time unrolling process, which does not blow up in space and time. I've implemented this and pulled it into master, and am testing it on the MHC.

Can anyone please help me with the vg commands to visualize the DAG of local vg which is fed as an input to gssw module ? There are so many flags associated with vg mod command which deals with one step at a time for unfolding, breaking..etc. What command should be used to visualize the complete DAG of a local vg which is ready to be given as an input to gssw algorithm ?

I think it would be something like vg mod --unfold <length of read> --dagify-to <length of read>. If you really want to see the graph, I think there might be code to dump the actual graph being fed into gssw, behind an #ifdef debug that you could remove.

Thanks. Also can you tell how to run gssw_aligner.cpp module on a separate DAG which is outputted by    vg mod --unfold <length of read> --dagify-to <length of read> with a separate read sequence ?

Also, I am trying to load a subgraph using the vg find command. I am using like this
  ./vg find -r 1:6 -x tiny.xg | ./vg view -d - > viz.dot
But the edges are not shown in the this subgraph. How can I visualize the nodes and edges with orientations of a subgraph ?

I don't think we have code to let you dump an unmodified graph directly
into GSSW.
To get some edges out of "vg find" I think you need to set a context number
of hops to go out from the selected nodes. Try adding -c 1 to the find
command.
On Fri, Nov 18, 2016 at 2:45 AM, vns kavya<EMAIL_ADDRESS>wrote:

Also, I am trying to load a subgraph using the vg find command. I am using
like this
./vg find -r 1:6 -x tiny.xg | ./vg view -d - > viz.dot
But the edges are not shown in the this subgraph. How can I visualize the
nodes and edges with orientations of a subgraph ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vgteam/vg/issues/182#issuecomment-261502577, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE0_X9BznDRP2gfN72SyDsED__iE_SyHks5q_YHJgaJpZM4HF2AS
.

Hi, I tried adding -c 1 but still the complete sub-graph(containing nodes and edges) is not extracted. Please let me know how can we extract a complete sub graph (containing both nodes and edges ) between either range of two nodes or by some other method.
Also, in the above query, I understand we cannot dump unmodified graph directly into gssw. My question is how to dump the DAG output of any local vg using the
vg mod --unfold <length of read> --dagify-to <length of read>
and run the gssw_aligner.cpp module on it independently and check the result of gssw_align on it.

Hm, I think -c 1 is supposed to not leave any dangling edges. Can you
open another bug for that issue?
As for how to get the output of vg mod into gssw_aligner.cpp, we
currently have no command line interface to do it. You would need to write
some C++ code that reads in a graph (probably just as a plain Protobuf
vg::Graph and not as a fully cross-indexed vg::VG), creates a
vg::Aligner, and runs the align() method on the aligner to align
something against the graph. If you want to write a new subcommand that
does that, you would put your main cpp file in the subcommands directory,
make sure it registers itself with the subcommand system like the other
subcommands in there, and then modify the Makefile to build it and link the
resulting object file into the main vg binary.
If you just want to add your own binary to the build that just holds your
C++ code, you would link against libvg.a, which the build process dumps
all the important non-CLI stuff into.
If you are encountering a specific problem, it might make the most sense to
add a C++ unit test under the unittests directory, which (if you put it in
an existing file or set up a new file right in the Makefile) would get
linked into vg and be runnable with vg test.
On Fri, Nov 18, 2016 at 11:29 PM, vns kavya<EMAIL_ADDRESS>wrote:

Hi, I tried adding -c 1 but still the complete sub-graph(containing nodes
and edges) is not extracted. Please let me know how can we extract a
complete sub graph (containing both nodes and edges ) between either range
of two nodes or by some other method.
Also, in the above query, I understand we cannot dump unmodified graph
directly into gssw. My question is how to dump the DAG output of any local
vg using the
vg mod --unfold  --dagify-to
and run the gssw_aligner.cpp module on it independently and check the
result of gssw_align on it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vgteam/vg/issues/182#issuecomment-261699024, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE0_XzKekQXy-o75bSR853hDmBNnF5pdks5q_qVVgaJpZM4HF2AS
.

Okay. Thanks @adamnovak ! I have one more question regarding the GFA file format of vg. The Path line 'P' is different from any other standard GFA file formats. Could you please explain the following Path line "P' in the context of vg GFA file. Here is the sample line from tiny.gfa file of vg
P       6       x       4       +       3M How to interpret this?

I think @ekg might have to explain that one. I know that each P line
represents a visit of a path to a node. The "x" is the path name, the "+"
is the orientation in which the path visits the node, and the "3M" is a
CIGAR giving the alignment between the path and the node. Of the "6" and
"4", one is node ID being visited, and one is the index of the visit along
the path, but I don't remember which is which.
It's not really proper GFA; the mainline GFA people decided to go with
single-line representations of paths, which don't necessarily work for our
use case where a path may visit millions of nodes.
On Tue, Nov 22, 2016 at 3:04 AM, vns kavya<EMAIL_ADDRESS>wrote:

Okay. Thanks @adamnovak https://github.com/adamnovak ! I have one more
question regarding the GFA file format of vg. The Path line 'P' is
different from any other standard GFA file formats. Could you please
explain the following Path line "P' in the context of vg GFA file. Here is
the sample line from tiny.gfa file of vg
P 6 x 4 + 3M How to interpret this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vgteam/vg/issues/182#issuecomment-262211804, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE0_Xw_nxoqPBBYIUW0xs3OPObAHaK0yks5rAswngaJpZM4HF2AS
.

@adamnovak I have one doubt regarding GFA representation of variation graph. In "Link line (L line)" of GFA  if there is path A +  B -  , how should we interpret it ?

According to standard GFA file, this L line is explained as path from A forward strand to reverse complement of B

While according to VG format understanding of mine it is path from B forward strand to A forward strand  as well as path from A reverse complement to B reverse complement since VG graph is bidirectional.

Can you please clarify as to which interpretation is correct  ?

Hey @vnsk ,
I'm going to try to tackle your GFA questions. You're correct that the GFA path syntax has moved to one path per P line, with all path objects list on that line. We're frozen on a version before they implemented this change while we wait for the syntax to firm up.

Here is the sample line from tiny.gfa file of vg
P 6 x 4 + 3M How to interpret this?

We use P as an annotation on a node,where for each segment on the path there is a corresponding P line. This is in contrast to GFA's treatment of paths as first-class objects. So the above example would translate to "Node 6 is on path x; it is the fourth item (rank) in the path; it is in the forward direction; all three bases in 6 are on the path." In some cases, A single path line P in GFA syntax could be generated by collapsing all the vg P lines to one and modifying the cigar as needed, and I plan to support this in the future.
The reason we did this is that our graphs (as @adamnovak said) may have millions of items in a path, and cramming them all on one line makes the GFA format completely illegible to human beings. We've submitted a pull request to implement a W or walk line, but this has not yet been approved.

I have one doubt regarding GFA representation of variation graph. In "Link line (L line)" of GFA if there is path A + B - , how should we interpret it ?

This is actually in the style of proper GFA (assuming I got the flags right). VG does allow directionality in graphs (though it doesn't require it), so in this case you're correct in interpreting it as:

this L line is explained as path from A forward strand to reverse complement of B

i.e. an edge from the "tail" of node A to the "tail" of node B.
Hope this helps clarify things!

By the way, the vg align command does nearly what's wanted with respect to
feeding the graph directly into gssw. We could try to make the input more
"raw" but I am a little confused as to why this is needed. If the graph is
acyclic then it is passed into gssw unmodified.
On Thu, Nov 24, 2016, 12:08 Eric T. Dawson<EMAIL_ADDRESS>wrote:

Hey @vnsk https://github.com/vnsk ,
I'm going to try to tackle your GFA questions. You're correct that the GFA
path syntax has moved to one path per P line, with all path objects list
on that line. We're frozen on a version before they implemented this change
while we wait for the syntax to firm up.
Here is the sample line from tiny.gfa file of vg
P 6 x 4 + 3M How to interpret this?
We use P as an annotation on a node,where for each segment on the path
there is a corresponding P line. This is in contrast to GFA's treatment
of paths as first-class objects. So the above example would translate to
"Node 6 is on path x; it is the fourth item (rank) in the path; it is in
the forward direction; all three bases in 6 are on the path." In some
cases, A single path line P in GFA syntax could be generated by
collapsing all the vg P lines to one and modifying the cigar as needed,
and I plan to support this in the future.
The reason we did this is that our graphs (as @adamnovak
https://github.com/adamnovak said) may have millions of items in a
path, and cramming them all on one line makes the GFA format completely
illegible to human beings. We've submitted a pull request
https://github.com/GFA-spec/GFA-spec/pull/47 to implement a W or walk
line, but this has not yet been approved.
I have one doubt regarding GFA representation of variation graph. In "Link
line (L line)" of GFA if there is path A + B - , how should we interpret it
?
This is actually in the style of proper GFA (assuming I got the flags
right). VG does allow directionality in graphs (though it doesn't require
it), so in this case you're correct in interpreting it as:
this L line is explained as path from A forward strand to reverse
complement of B
i.e. an edge from the "tail" of node A to the "tail" of node B.
Hope this helps clarify things!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vgteam/vg/issues/182#issuecomment-262750933, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAI4EdnOR5ztRft3NExRk-5DdL50J1tqks5rBXASgaJpZM4HF2AS
.

Hi @ekg, @adamnovak, @edawson
..I have on doubt regarding the GFA conversion of VG project. I considered the following sample fasta and vcf file.
sample.fasta
>x
CAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTGCAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTG

And the VCF file as follows
sample.vcf file
##fileformat=VCFv4.1
##fileDate=20141110
##source=mutatrix population genome simulator
##seed=1415643582
##reference=x.fa
##phasing=true
##commandline=mutatrix --dry-run -s 0.05 -i 0.01 -p 2 x.fa
##INFO=<ID=AC,Number=A,Type=Integer,Description="Alternate allele count">
##INFO=<ID=TYPE,Number=A,Type=String,Description="Type of each allele (snp, ins, del, mnp, complex)">
##INFO=<ID=SVTYPE,Number=A,Type=String,Description="Type of SV">
##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of samples at the site">
##INFO=<ID=NA,Number=1,Type=Integer,Description="Number of alternate alleles">
##INFO=<ID=LEN,Number=A,Type=Integer,Description="Length of each alternate allele">
##INFO=<ID=SVLEN,Number=A,Type=Integer,Description="Length of SV">
##INFO=<ID=MICROSAT,Number=0,Type=Flag,Description="Generated at a sequence repeat loci">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	X
x	3	.	A	<DUP>	99	.	SVTYPE=DUP;SVLEN=10	GT	1|1
x	5	.	T	<INV>	99	.	SVTYPE=INV;SVLEN=2	GT	1|1
x	9	.	G	<DEL>	99	.	AC=1;LEN=1;NA=1;NS=1;SVTYPE=DEL;SVLEN=2	GT	1|0
x	20	.	T	<DEL>	99	.	AC=1;LEN=1;NA=1;NS=1;SVTYPE=DEL;SVLEN=7	GT	1|0
x	29	.	T	C,G	99	.	AC=1;LEN=1;NA=1;NS=1;	GT	1|0
x	32	1	A	ATTTTAAAA	99	.	SVTYPE=INS;SVLEN=0	GT	1|1
x	35	.	A	C	99	.	AC=1;LEN=1;NA=1;NS=1;	GT	1|0

To construct and view corresponding GFA file using vg project. I Did the following:
./vg construct -r sample.fa -v sample.vcf.gz > sample.vg

./vg view -g sample.vg > sample.gfa

The output sample.gfa file is as follows:

sample.gfa
H	VN:Z:1.0
S	1	CAAATAAGGCTTGGAAATTTTCTGGAGT
P	1	x	1	+	28M
L	1	+	2	+	0M
L	1	+	3	+	0M
L	1	+	4	+	0M
S	2	T
P	2	x	2	+	1M
L	2	+	5	+	0M
S	3	C
L	3	+	5	+	0M
S	4	G
L	4	+	5	+	0M
S	5	CTA
P	5	x	3	+	3M
L	5	+	6	+	0M
L	5	+	7	+	0M
S	6	TTTTAAAA
L	6	+	7	+	0M
S	7	TT
P	7	x	4	+	2M
L	7	+	8	+	0M
L	7	+	9	+	0M
S	8	A
P	8	x	5	+	1M
L	8	+	10	+	0M
S	9	C
L	9	+	10	+	0M
S	10	TATTCCAACTCTCTGCAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTG
P	10	x	6	+	65M

Could anyone please help why the output GFA file failed to capture the first four lines of VCF file of ,, ,  svtypes ?

That's because vg construct doesn't actually know how to read symbolic
alleles like , , or  yet (at least by default). @edawson has
been working on it.
I thought I was going to need this functionality too, but I managed to get
someone to articulate the VCFs I need using fully specified allele base
strings.
On Mon, Feb 6, 2017 at 10:56 PM, vns kavya<EMAIL_ADDRESS>wrote:

Hi @ekg https://github.com/ekg, @adamnovak
https://github.com/adamnovak, @edawson https://github.com/edawson
..I have on doubt regarding the GFA conversion of VG project. I considered
the following sample fasta and vcf file.
sample.fasta

x
CAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTGCAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTG

And the VCF file as follows
sample.vcf file
##fileformat=VCFv4.1
##fileDate=20141110
##source=mutatrix population genome simulator
##seed=1415643582
##reference=x.fa
##phasing=true
##commandline=mutatrix --dry-run -s 0.05 -i 0.01 -p 2 x.fa
##INFO=<ID=AC,Number=A,Type=Integer,Description="Alternate allele count">
##INFO=<ID=TYPE,Number=A,Type=String,Description="Type of each allele (snp, ins, del, mnp, complex)">
##INFO=<ID=SVTYPE,Number=A,Type=String,Description="Type of SV">
##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of samples at the site">
##INFO=<ID=NA,Number=1,Type=Integer,Description="Number of alternate alleles">
##INFO=<ID=LEN,Number=A,Type=Integer,Description="Length of each alternate allele">
##INFO=<ID=SVLEN,Number=A,Type=Integer,Description="Length of SV">
##INFO=<ID=MICROSAT,Number=0,Type=Flag,Description="Generated at a sequence repeat loci">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	X
x	3	.	A		99	.	SVTYPE=DUP;SVLEN=10	GT	1|1
x	5	.	T		99	.	SVTYPE=INV;SVLEN=2	GT	1|1
x	9	.	G		99	.	AC=1;LEN=1;NA=1;NS=1;SVTYPE=DEL;SVLEN=2	GT	1|0
x	20	.	T		99	.	AC=1;LEN=1;NA=1;NS=1;SVTYPE=DEL;SVLEN=7	GT	1|0
x	29	.	T	C,G	99	.	AC=1;LEN=1;NA=1;NS=1;	GT	1|0
x	32	1	A	ATTTTAAAA	99	.	SVTYPE=INS;SVLEN=0	GT	1|1
x	35	.	A	C	99	.	AC=1;LEN=1;NA=1;NS=1;	GT	1|0
To construct and view corresponding GFA file using vg project. I Did the
following:
./vg construct -r sample.fa -v sample.vcf.gz > sample.vg
./vg view -g sample.vg > sample.gfa
The output sample.gfa file is as follows:
sample.gfa
H	VN:Z:1.0
S	1	CAAATAAGGCTTGGAAATTTTCTGGAGT
P	1	x	1	+	28M
L	1	+	2	+	0M
L	1	+	3	+	0M
L	1	+	4	+	0M
S	2	T
P	2	x	2	+	1M
L	2	+	5	+	0M
S	3	C
L	3	+	5	+	0M
S	4	G
L	4	+	5	+	0M
S	5	CTA
P	5	x	3	+	3M
L	5	+	6	+	0M
L	5	+	7	+	0M
S	6	TTTTAAAA
L	6	+	7	+	0M
S	7	TT
P	7	x	4	+	2M
L	7	+	8	+	0M
L	7	+	9	+	0M
S	8	A
P	8	x	5	+	1M
L	8	+	10	+	0M
S	9	C
L	9	+	10	+	0M
S	10	TATTCCAACTCTCTGCAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTG
P	10	x	6	+	65M
Could anyone please help why the output GFA file failed to capture the
first four lines of VCF file of ,, , svtypes ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vgteam/vg/issues/182#issuecomment-277917081, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE0_X_LUaPabfmyRcXzc79zsW7Lnm_umks5raBWlgaJpZM4HF2AS
.

VG can parse  and  tags if you pass the -S flag to construct and your VCF provides the SVLEN and SVTYPE in the INFO columns. It will parse  if you pass -S and -I insertions.fasta, where insertions.fasta is a fasta file containing the insertion sequence of your VCF variants, and the tag in the alt field is the insertion id in "<>" brackets.
Check out the first part of this wiki page: https://github.com/vgteam/vg/wiki/SV-calling-with-vg
Please let me know if you hit any issues!

Should we consider making -S the default?

Probably yes. I've avoided it so far because it invalidates almost every test and has a few limitations (requires flat alternates and that the variant be at a position >1bp beyond the sequence start, for example).

Hi @edawson ,
I tried to generate GFA using the command you suggested.
USING VG:
./vg construct -f -S -a -r sample.fa -v sample.vcf.gz > sample.vg
./vg view -g sample.vg

The output gfa file is not capturing DUP (Duplicates). It seems that VCF line at POS 3 is ignored in the output gfa file.
Also INV (inversion) is not captured according to the edge convention. Since '-' edge sign was not there. Without the '-' sign the inversion cannot be understood from the GFA file ? Correct me if I am wrong .

Following is the output GFA from vg for sample.fa and vcf shared above in the discussion thread.
sample.gfa
H	VN:Z:1.0
S	1	CAAA
P	1	x	1	+	4M
L	1	+	2	+	0M
L	1	+	3	+	0M
S	2	AT
P	2	_alt_ae86adfba5b06f8b4ddc1cda4a4650aedc380635_1	14	+	2M
L	2	+	4	+	0M
S	3	TA
P	3	_alt_ae86adfba5b06f8b4ddc1cda4a4650aedc380635_0	13	+	2M
P	3	x	2	+	2M
L	3	+	4	+	0M
S	4	AG
P	4	x	3	+	2M
L	4	+	5	+	0M
L	4	+	6	+	0M
S	5	G
P	5	_alt_9415f505f043a840c914dd0ab2bb0a5fcd8ec1b1_1	16	+	1M
L	5	+	7	+	0M
S	6	GCT
P	6	_alt_9415f505f043a840c914dd0ab2bb0a5fcd8ec1b1_0	15	+	3M
P	6	x	4	+	3M
L	6	+	7	+	0M
S	7	TGGAAATT
P	7	x	5	+	8M
L	7	+	8	+	0M
L	7	+	9	+	0M
S	8	T
P	8	_alt_e7ebb7fe1a868b189725dbd95eada2ac65e747f2_1	18	+	1M
L	8	+	10	+	0M
S	9	TTCTGGAG
P	9	_alt_e7ebb7fe1a868b189725dbd95eada2ac65e747f2_0	17	+	8M
P	9	x	6	+	8M
L	9	+	10	+	0M
S	10	T
P	10	x	7	+	1M
L	10	+	11	+	0M
L	10	+	12	+	0M
L	10	+	13	+	0M
S	11	C
P	11	_alt_3a321436cc62676f7a1d8ec28395c00ebe646587_1	20	+	1M
L	11	+	14	+	0M
S	12	G
P	12	_alt_3a321436cc62676f7a1d8ec28395c00ebe646587_2	21	+	1M
L	12	+	14	+	0M
S	13	T
P	13	_alt_3a321436cc62676f7a1d8ec28395c00ebe646587_0	19	+	1M
P	13	x	8	+	1M
L	13	+	14	+	0M
S	14	CT
P	14	x	9	+	2M
L	14	+	15	+	0M
L	14	+	16	+	0M
S	15	ATTTTAAAA
P	15	_alt_6337c431f778e028e6c96110e02be12547f8fa0b_1	23	+	9M
L	15	+	17	+	0M
S	16	A
P	16	_alt_6337c431f778e028e6c96110e02be12547f8fa0b_0	22	+	1M
P	16	x	10	+	1M
L	16	+	17	+	0M
S	17	TT
P	17	x	11	+	2M
L	17	+	18	+	0M
L	17	+	19	+	0M
S	18	C
P	18	_alt_705880aaca0ac6461baf914b1166a4023ebccd36_1	25	+	1M
L	18	+	20	+	0M
S	19	A
P	19	_alt_705880aaca0ac6461baf914b1166a4023ebccd36_0	24	+	1M
P	19	x	12	+	1M
L	19	+	20	+	0M
S	20	TATTCCAACTCTCTGCAAATAAGGCTTGGAAATTTTCTGGAGTTCTATTATATTCCAACTCTCTG
P	20	x	26	+	65M

These two warning messages were generated while using the construct command
warning:[vg::Constructor] Unsupported variant allele "<CN0>"; Skipping variant(s)!
warning:[vg::Constructor] Unsupported variant allele "<DUP:TANDEM>"; Skipping variant(s)!

Also I tried generating GFA file for some other examples apart from the one mentioned. But as we could see this example (sample.vcf) is not actually containing any of warned alle.

Hey @vnsk ,
Thanks for trying this out. I haven't added handling for DUP type tags yet, because I haven't figured out how I want them to go in the graph (i.e. as cycles, or as unrolled repeats). I'm open to suggestions and could also just implement something if you want to give it a try.
INV tags are captured, and should be in the graph. Instead of using reversing edges to represent INV I'm duplicating the ref sequence, reversing it, then adding it as a node at the relevant position. Here's the source code:
      this->ref = fasta_reference.getSubSequence(this->sequenceName, this->position, sv_len);
      string alt_str(fasta_reference.getSubSequence(this->sequenceName, this->position, sv_len));
      reverse(alt_str.begin(), alt_str.end());
      this->alt[alt_pos] = alt_str;

If we look at this graph using dot (save GFA to x.gfa; vg view -Fd x.gfa | dot -T pdf -o o.pdf) , it looks like nodes [2,3] define your inversion:
o.pdf

Hi @edawson
It would be great if DUP can be represented as cycles. Since vg mod command already converts duplications into unrolled repeats to a desired length of 'k', to better interpret the variants, it's beneficial to visualize DUP as cycles in the graph instead of unrolled repeats.

Hi @edawson, @ekg , @adamnovak
Is there any way to visualize the Short Tandem Repeats (Variation Type = STR) as loops using vg commads.
I considered chromosome1.fasta file and variant calls of STR on chromosome1 from sgdp.calls.vcf.gz.  I tried the following command to  visualize the corresponding variation graph.
./vg construct -f -S -a -r chromosome1.fasta  -v sgdp.calls.chr1.vcf.gz > graph.vg
Surprisingly, the output graph is a complete Directed Acyclic Graph (DAG (without any cycles)) which should not be the case if tandem repeats are involved naturally.
But how can one visualize the Short Tandem Repeats (STRs) as effectively as loops or cycles using vg?

Are these variants with an "STR" as the "SVTYPE"? Can you maybe post one (on a new issue?).
I think if you want it to work right now you might have to go out and write the "STR" case handling yourself in the code we use to parse variants. The code for construct is here in vcflib.
It might not quite fit the model we're using, because that code is in our VCF parser, vcflib, which just treats each variant as a set of aligned string alternatives, and doesn't really support cycles. So to make it really work properly you might have to add some special-case code in vg construct, to interpret theses as loops and not just plain alternation between specific numbers of copies.
The other alternative is to not use vg construct, and instead write your own code to generate a JSON or GFA-format graph describing your repeat, and then


How to allow for images and scripts from external domains?

I tried installing & setting up the module but it blocks all external images and scripts, how do I allow external images & scripts or allow specific domains?
I tried this to test in nuxt.config.ts:
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    headers: {
      contentSecurityPolicy: {
        'default-src': ['*'],
      }
    }
  },
})

But it didn't seem to have any effect.

Hey,
Could you please set up a stackblitz link with the issue you are experiencing? I will be glad to show you on the example how to make it work :)

@Baroshem Did that link work or did I make it wrong?

Hey,
Yes, the link is good. It does not work on Stackblitz but I was able to reproduce it on the local environment.
Basically, the nuxt-security module sets some security directives that blocks fetching resources from differen host (in your case wikimedia).
To make it work, you have to add this domain to your nuxt.config.ts file and more specifically the security object for global configuration or in routeRules if you use the per-route approach like following:
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    headers: {
      contentSecurityPolicy: {
        'img-src': ['https://upload.wikimedia.org'], // <--- add the domain you want to fetch the image from here
      }
    }
  }
})

And also, you will need to add a crossorigin attribute to your image tag to instruct the browser that this resource is being fetched from another origin like following:
<template>
  <div>
    <img
      src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/Cat_August_2010-4.jpg/272px-Cat_August_2010-4.jpg"
      alt="Cat Image Here"
      crossorigin  // <-- Here
    />
  </div>
</template>

You can read more about Content Security Policy (img-src) here and about Cross-Origin Embedder Policy here
I hope that it makes sense.
Let me know if that works for you :)

To make stackblitz repro work you need to have this configuration enabled:
  security: {
    headers: {
      crossOriginResourcePolicy: 'cross-origin',
      contentSecurityPolicy: false,
      xFrameOptions: false,
    },
  },

Otherwise it will fail. This is some problem on Stackblitz side with following directives.

Hey @JordashTalon
Did my answer helped you resolve your issue?

Hey @JordashTalon
Adding a crossorigin attribute to image is a good pattern for having content security policy. If you cannot use it that way (although I recommend using it that way as it is a native browser approach) you can always disable this directive like:
      contentSecurityPolicy: {
        'img-src': false
      },

Or you can enable it globally for all https pages like:
      contentSecurityPolicy: {
        'img-src': ['https:']
      },

script src is not working yet as it requires some work on the generated nuxt scripts. For details check out

@JordashTalon
I think that right now you can add any script you like because the script src directive is not enabled yet as I need to configure Nuxt to use it.
If it does not work, could you provide more details like error, nuxt config file, etc?

Also looks like the image fix stopped working too, I have this config:
modules: [
    '@vueuse/nuxt',
    'nuxt-security',
  ],

  security: {
    headers: {
      contentSecurityPolicy: {
        'img-src': false,
      }
    }
  },

And looks like external images are still blocked with the same reason.

That is a very good explanation @alexbidenko1998 , thank you for that!

@JordashTalon
Does that answer your question?

Yeah good for now


Add NuGet package for System.CommandLine

@KrzysztofCwalina, please take a look.
@ahsonkhan, I've pushed System.CommandLine.0.1.0-d101215-1 to: https://www.myget.org/F/netcore-package-prototyping/api/v2/package.

Yeah, I need to update the other specs too, but I did add some dependencies to System.Text.Formatting.nuspec

@terrajobst, are you merging it?

Closed by mistake. reopening.


ENH Handle release branches

Issue https://github.com/silverstripeltd/product-issues/issues/635

Looks good. I will tag this after merge.


(Wind) Job creation output

This pull request adds the feature of generating an approximate number of jobs coming from the wind industry, as a type of post-processing, to the workflow.

Update 14/10:

We will investigate if GEM data can be used to approximate the average/median wind farm size (important assumption for the results).
I will update the script so that it can take this as an argument.

The PR have had a few smaller updates:

The specific wind turbines used in rule build_vre_cf_grid have been moved to the config file and is used both in that rule and in the job creation rules.
The capacity of the turbines is looked up through atlite and the previously hard-coded turbine sizes are now automated.

oh and I would squash and merge as you would most likely not cherry pick individual changes.

To me the feature is finished and we should be ready to pull it into WIMBY


reduce: do not panic on null

also switch to require over assert in testing.

for sure better than panic! however ignoring nils for min/max/etc may also be reasonable.
Obviously depends on where the data came from though -- with group by queries it is easy to end up with a bunch of meaningless nils. We could also consider using nullValueMode this was from the UI settings

I agree. Just going with this for now, as dropping/ignoring nils is a bit more complicated (e.g., what do you do when everything is nil).
What I eventually want to have, is dropdowns with some of these options that control various things. So for reduction, stuff like this. For math, eg $A + $B there may be drop downs how to handle various join scenerios between the items in A and the items in B.
So this is just a quick fix, because that thing I made for fun with Azure and showed on Friday panicked, and I had to fix that ;-)


Implement installation using trust

Closes #115

Regarding the removal of cargo-install-update: It seemed to only serve to update mdbook (see context here), and trust will install the newest version always anyway.

This reduces CI time significantly.

New Build
Old Build

Platform
Old Time
New Time

Total
73min 41 sec
28min 24 sec

Windows
32m 50s
14m 24s

Mac
19m 35s
7m 42s

Linux
21m 16s
6m 18s


Write unit tests for pkg/cache/lrucache.go

can you tell me where should I put the unit tests? @yp969803


Performance tuning discussions

This is just a PR to make sure that random performance experiments don't kill everything, but yeah, I'm glad to know about this.

NP... it's a concerning lock nevertheless :)

I'll be glad to pick up that discussion with @jboeuf and we can figure this one out.

@vjpai, do you want to setup a quick meeting with @ctiller, @yang-g and me on this? I will need to refresh my cache on this one...

Idle a long time and likely obsolete


Usage of internal cross references in Asciidoc

🆕🐥 First Timers Only
This issue is reserved for people who never contributed to Open Source before. We know that the process of creating a pull request is the biggest barrier for new contributors. This issue is for you 💝
👾 Description of the issue
In the migration documentation (https://github.com/AdoptOpenJDK/website-adoptium-documentation/blob/main/migration/index.adoc) several links direct to the migration page of AdoptOpenJDK. Instead of doing so such links should be defined as internal cross references in Asciidoc since all this paragraphs are part of the same documentation file.
📋 Step by Step
To solve this issue and contribute a fix you should check the following step-by-step list. A more detailed documentation of the workflow can be found here.

[ ] Claim this issue: Comment below.
[ ]  Fork the repository in github by simply clicking the 'fork' button.
[ ]  Check out the forked repository
[ ]  Create a feature branch for the issue. We do not have any naming definition for branches.
[ ]  Commit your changes.
[ ]  Start a Pull Request.
[ ]  Done 👍 Ask in comments for a review :)
[ ]  If the reviewer find some missing peaces or a problem he will start a discussion with you and describe the next steps how the problem can be solved.
[ ]  You did it 🎉 We will merge the fix in the master branch.
[ ]  Thanks, thanks, thanks for being part of this project as an open source contributor ❤

🎉 Contribute to Hacktoberfest
Solve this issue as part of the Hacktoberfest event and get a change to receive cool goodies like a T-Shirt. 🎽
🤔❓ Questions
If you have any questions just ask us directly in this issue by adding a comment. You can join our community chat at Slack. Next to this you can find a general manual about open source contributions here.

Hi I am interested in solving this issue.


Error encountered when attempting to convert postman collection to yaml

Describe the bug
openman convert repairq-api.postman_collection.json repairq-api.yaml
Traceback (most recent call last):
File “/usr/local/bin/openman”, line 8, in
sys.exit(cli())
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1137, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1062, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 763, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/openman/cli.py”, line 36, in convert
apispec = spec.get_spec(ignorespec=ignore_file)
File “/usr/local/lib/python3.9/site-packages/openman/spec/spec.py”, line 52, in get_spec
for requestitem in self.postman_collection.get_requestitems():
File “/usr/local/lib/python3.9/site-packages/openman/parser/collection.py”, line 54, in get_requestitems
items.extend(folder.get_requestitems())
File “/usr/local/lib/python3.9/site-packages/openman/parser/folder.py”, line 40, in get_requestitems
request_items.append(RequestItem(item))
File “/usr/local/lib/python3.9/site-packages/openman/parser/requestitem.py”, line 11, in init
raise RequestItemParserException(‘Not a valid request’)
openman.errors.RequestItemParserException: Not a valid request
Expected behavior
Expected the file to be converted from postman collection json to yaml
To Reproduce
Run the convert tool on a collection file downloaded from postman.
My Platform
macOS Big Sur, python3.9, postman collection version v2.1.0
Additional context
Thank you for any assistance you can provide.  I'm happy to provide additional details as would be helpful.

Hi, this issue tracker is for issues with the codebase behind https://pypi.org itself, not the projects hosted on PyPI.
You should report this issue to the tracker for the project in question instead.


Doesn't work on Raspbian JESSIE?

I try to use this gem but I don't manage to control any pin.
I have no error message, but nothing happens on my board.
The LED on pin 17 is on, since the boot of the raspberry. But:
pin = PiPiper::Pin.new(:pin => 17, :direction => :out)
=> #<PiPiper::Pin:0x1ea4448 @pin=17, @direction=:out, @invert=false, @trigger=:both, @pull=0, @last_value=nil, @value=0>

pin.on
=> 1

pin.off
=> 1

pin.on?
=> false

pin.off?
=> true

Does anyone succeed to make it work on Raspbian Jessie + Ruby 2?
root@raspberrypi:~# ruby -v
ruby 2.1.5p273 (2014-11-13) [arm-linux-gnueabihf]

Haven't gotten a chance to upgrade to jessie yet so I am not sure. It seems like the changes in jessie mean you don't need sudo access for GPIO anymore. But to be safe, did you try running your script or irb session using rvmsudo?
I can take a look but AFAIK pin.on and pin.off will always return 1 if it was successful. So the "state" is correct here, when you do pin.off?. Just to clarify, is this your workflow:

Led @ 17 is off
Instantiate pin with PiPiper and do pin.on.
Led @ 17 is now on
pin.off
Led @ 17 is still on?

tell us also which version of PiPiper you are using.

Here is what I notice:

connected to 3.3V pins (pin #1 or #17), the LED if on when the board is powered on.
connected to another pin (GPIO 17 on pin #11), the LED is off on startup.

Then:
# same as rvmsudo but for rbenv
rbenv sudo irb

irb(main):001:0> require 'pi_piper'
=> true
irb(main):002:0> include PiPiper
=> Object

irb(main):005:0* pin = PiPiper::Pin.new(:pin => 13, :direction => :out)
=> #<PiPiper::Pin:0x5665da18 @pin=13, @direction=:out, @invert=false, @trigger=:both, @pull=0, @last_value=nil, @value=0>
# LED is off

irb(main):006:0> pin.on?
=> false

irb(main):007:0> pin.on
=> 1
# LED is still off

irb(main):008:0> pin.on?
=> false

If I try to use pi_piper without sudo:
irb(main):004:0* pin = PiPiper::Pin.new(:pin => 11, :direction => :out)

bcm2835_init: Unable to open /dev/mem: Permission denied
Errno::EACCES: Permission denied @ rb_sysopen - /sys/class/gpio/gpio11/direction
	from /home/pi/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/pi_piper-2.0.0/lib/pi_piper/bcm2835.rb:61:in `write'
	from /home/pi/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/pi_piper-2.0.0/lib/pi_piper/bcm2835.rb:61:in `pin_direction'
	from /home/pi/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/pi_piper-2.0.0/lib/pi_piper/bcm2835.rb:45:in `pin_output'
	from /home/pi/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/pi_piper-2.0.0/lib/pi_piper/pin.rb:56:in `initialize'
	from (irb):4:in `new'
	from (irb):4
	from /home/pi/.rbenv/versions/2.3.0/bin/irb:11:in `<main>'
irb(main):005:0>

It works!
My bad, I was misunderstanding the pin number: I though the pin number was the number in the order of the GPIO (from 1 to 40), but in fact I understand there is no relation between the position of the pin, and it's number.
The GPIO #17 pin if on the place 11, but it's still called "17".
Conclusion:

still works on Jessie
still requires sudo

Thanks for your help! :)

Ah yes, I was thinking that might be it. I will update the documentation to make this clearer. I noticed that others have had that same misunderstanding. I am glad that it works for you though!


Issue with HMR (Fast Refresh) in NextJS

Hi,
I just tried the hooks on a very simple project with NextJS. In development, I get an error on HMR (Fast Refresh), because (I think) the connection is closed and open too fast.

Did you already had this issue? Do you know a workaround?
If not, I will be able to provide you a code sandbox 😊

I also have this issue did you find a solution?

I found a potential solution to this issue. The hook useChannel should not be used in components which are supposed to be re-rendered multiple times because of state updates in parent components (this was what was causing my code to raise this error). So what I did was to declare useChannel in the closest parent component to the one where it was before. In this way the hook wasn't run multiple times because of "parent state" updates. Moreover, pay attention to the operation you do on a channel object because with asynchronous code you risk to make multiple operations on the same channel overlap and this can be another reason why this error is raised.

@NickTaykey I didn't found a solution and your solution is OK for a quick fix I think. I currently not using Ably on a production application so I will not investigate more but I think that the Ably team should if they want more Next users to use Ably 😊

I have just merged a PR which I believe will fix this issue. This was related to us detatching unmounted components, that has been exacerbated by react 18's new debug mode, where it runs useEffect twice.
Please try out the latest version and verify if you can.
Thank you!


If a user is logged in to a server and then tries to connect to the same server nothing happens from their point of view

My Setup

My OS name and version: openSUSE/Fedora
My current app version: 2.4/2.6.1

[x] I have tested with the latest application version
[x] I can simulate the issue easily

Description

One of our users (running on openSUSE with 2.4) just reported an issue where he told me "I've been kicked out of rocket chat and can't get back in". When I went up the rocket chat electron app was sat on the "Add new server" page. The server list on the left was hidden (we use one server, we advise people to hide it).
When the user typed in the URL for our server to connect, it just showed the same page again with the server name removed (basically reloaded the page and did nothing)
The reason the user had this issue, was that he was logged in to our rocketchat server, so when he tried to connect to the server the app must have decided no action was necessary as he was already connected. I don't know he got in to this state, maybe there is a keyboard shortcut to get to the "Add new server" dialogue that I don't know about and he hit accidentally.
I opened up the server list and clicked on our server and everything was fine.
When I got back to my desk I reproduced the issue on my machine (Fedora/2.6.1)
Current Behavior

The app doesn't tell you anything if you try to connect to a server you are already connected to.
Expected Behavior

The app should either take you to that rocketchat server, or pop up a message telling you you are already connected, to avoid confusion.

@Rumbles I've made a change so that if you type the server name again it will load it, rather than ignoring. Will that resolve your issue?
There is a shortcut Ctrl+N to add a new server, so that may be how it happened. It is also possible to switch servers using Ctrl+1 etc and you can manually select them from the Window menu, without having to open the server list.


New version: squidowl.halloy version 2024.13

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

I got
`winget validate --manifest .\manifests\s\squidowl\halloy\2024.13
Manifest has the following dependencies that were not validated; ensure that they are valid:

Packages
Microsoft.VCRedist.2015+.x64
Manifest validation succeeded.`

locally. So what exactly is wrong?


replace yarn with pnpm

replace yarn with pnpm

in the past, i used tauri/action to build, so i am not very clear about what happened
checked and found the following errors
aarch64-unknown-linux-gnu error:
gui_target: aarch64-unknown-linux-gnu

# cmd
pnpm tauri build -- --target $GUI_TARGET --verbose --bundles deb

# error message
error: unexpected argument '--bundles' found

archive artifact error:
target: mips-unknown-linux-musl

other error is can not find the release file:
gui_target: aarch64-unknown-linux-gnu | x86_64-unknown-linux-gnu | x86_64-apple-darwin | aarch64-apple-darwin | x86_64-pc-windows-msvc

# cmd
pnpm tauri build -- --target $GUI_TARGET --verbose

# error message
No such file or directory (os error 2)

if available, please provide assistance to resolve these errors

okay, I'll take a look at night.

i seem to have solved this problem by using tauri/action
https://github.com/m1m1sha/EasyTier/actions/runs/8969651756/job/24631460890


is it possible to long only the requests that returned an error?

so that I can prevent my DB from being filled with normal APIs requests with message 200... it goes fast...
thanks for the amazing work

Closing due to inactivity.


operation error SSO OIDC: RegisterClient, failed to resolve service endpoint, an AWS region is required, but was not found

After upgrading from version 0.17.1 to 0.19.0, when using assume I get the following error:
operation error SSO OIDC: RegisterClient, failed to resolve service endpoint, an AWS region is required, but was not found
Nothing has changed in my settings, and I see no mention of any breaking change in the release notes of the versions after 0.17.1.
The region is present in the profile, and I've done an aws sso login --profile <profilename> before doing the assume.

Not sure, but this looks like the culprit: https://github.com/common-fate/granted/commit/e53483843490a7f289be34af66a5c785003ce0dc

Thanks, @IskanderNovena, I am able to replicate the error. Currently woking on a fix, as a temporary solution please manually downgrade your Granted version: https://docs.commonfate.io/granted/getting-started#reverting-to-an-older-version-of-granted

@IskanderNovena We just released v0.19.1 which should fix this issue. Please let me know if it works for you

Hi @shwethaumashanker
It looks like the new release fixed the issue
Thanks for the quick action!

@shwethaumashanker I can confirm that version 0.19.2 is working for me (didn't have a chance to test 0.19.1).
Thanks!

@danielhanold, I tried replicating the issue with the following configuration that I used last time to replicate it, but it seems to be working for me. Can you please share the AWS profile that is causing this to fail for you? (Feel free to redact any account IDs).
[profile sso-profile]
sso_start_url = https://d-12345678.awsapps.com/start#/
sso_region = eu-west-1
sso_account_id = 123456789
sso_role_name = sso_role
region = eu-central-1
output = text

[profile demo]
role_arn = arn:aws:iam::123456789:role/Administrator
source_profile = sso-profile
region = us-west-2
output = json

@shwethaumashanker Thank you for the quick response.
For the setup of my AWS config, I followed this tutorial from AWS: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html. This resulted in the following configuration (which worked in version 0.19.2):
[sso-session xyz-sso]
sso_start_url = https://abc.awsapps.com/start
sso_region = us-west-2
sso_registration_scopes = sso:account:access

[profile Test_AWSAdministratorAccess]
sso_session = xyz-sso
sso_account_id = 12345678
sso_role_name = AWSAdministratorAccess
region = us-west-2

After you told me that this worked in your testing, I modified my configuration and found the following to work:
[sso-session xyz-sso] # unused in this example
sso_start_url = https://abc.awsapps.com/start
sso_region = us-west-2
sso_registration_scopes = sso:account:access

[profile Test_AWSAdministratorAccess]
sso_start_url = https://abc.awsapps.com/start
sso_region = us-west-2
sso_account_id = 12345678
sso_role_name = AWSAdministratorAccess
region = us-west-2

It looks like version 0.20.0 is not compatible with the sso-session setup.

@JoshuaWilkes @shwethaumashanker Thanks very much for getting this fixed this quickly. Confirmed fixed in version 0.20.2. You're exceeding any expectations for an already amazing tool 🚀


Tracking date of last visit?

(Hope this is the right repo)
Does the Talk back end track when a user most recently visited Talk?
If so we could add substantial value to the front end by allowing the user to view all new posts/threads-with-new-posts since their last visit. This is something users have been asking for ever since we transitioned from standard forum software, and I admit I also find this a hugely useful feature.
I've suggested in zooniverse/Panoptes-Front-End#911 that allowing users to see the fire hose for a particular project (and maybe for all projects they've ever posted on, but I don't want to get ahead of myself) is a critical Talk feature. That's true at the moment, but it's only critical for everyone if we can't offer them a posts-since-last-visit option instead. If we could, then we could just let admins & mods see the fire hose.
Anyway, I figured I should check before I asked for it on the PFE repo.

A user technically "visits" talk every time they visit zooniverse.org, so it's not a useful metric for read tracking.
Ultimately read tracking becomes (or is requested to become) a matrix of every topic, every user that has seen the topic, and a flag that marks whether it has been read -- all of which resets on every new comment/edit.
Since it's kind of a nasty data problem, I'd like to punt it until we get the notifications system integrated.  I'm pretty sure that this use case will be solved with that, so I'm going to close the issue for the moment.
After that has some feedback we should definitely reevaluate though.


feature:  双 Agent / Proxy 安装支持 (closed #1592)

Codecov Report

Merging #1638 (1dcfa1f) into v2.3.2-dev (7fc83bc) will decrease coverage by 0.29%.
The diff coverage is 46.84%.

@@              Coverage Diff               @@
##           v2.3.2-dev    #1638      +/-   ##
==============================================
- Coverage       73.45%   73.17%   -0.29%
==============================================
  Files             389      390       +1
  Lines           26198    26447     +249
==============================================
+ Hits            19245    19352     +107
- Misses           6953     7095     +142

Impacted Files
Coverage Δ

...s/backend/components/collections/agent_new/base.py
92.02% <ø> (ø)

apps/backend/subscription/views.py
51.77% <ø> (ø)

...nents/collections/agent_new/install_other_agent.py
22.91% <22.91%> (ø)

apps/backend/subscription/steps/agent.py
48.89% <23.07%> (-1.57%)
:arrow_down:

apps/backend/agent/manager.py
36.05% <37.50%> (+0.08%)
:arrow_up:

apps/node_man/handlers/validator.py
81.00% <50.00%> (-0.36%)
:arrow_down:

apps/node_man/serializers/job.py
73.56% <58.33%> (-3.10%)
:arrow_down:

apps/node_man/models.py
76.56% <70.00%> (-0.02%)
:arrow_down:

apps/core/gray/tools.py
86.95% <75.00%> (-5.15%)
:arrow_down:

apps/backend/agent/tools.py
92.30% <100.00%> (+0.08%)
:arrow_up:

... and 9 more

... and 10 files with indirect coverage changes


Awesome Banners section is not loading [README.md]

hi @genze121
It is loading fine.

I'm closing this issue. thanks!


[GJ-49] Unify JNI interfaces

Use the unified JNI interfaces to operate the different native engines.
Closes #49.

Locally verified. LGTM!


"?" Is rendered rotated with Roboto font

I have the same issue both in lagrange mobile and desktop, on gemini://ploum.net , all question marks are rendered rotated 45°.
This only seems to happen with Roboto font, and if I copy/paste the text elsewhere it renders fine.

Hmm, this is quite curious. The question marks are preceded by U+202F (Narrow No-Break Space). Apparently this NNBSP code point confuses the glyph selection.
I have no idea at which point this causes the glyph to be misselected by HarfBuzz, but I found a simple workaround for v1.17.

Note: in French typography, question marks should always be preceded by a non-breaking space. (that’s why this character is there ;-) )

@ploum Good to know!


Recommending CTR is dangerous, if the same key is used twice

AES-CTR fails catastrophically when one reuses the same key, and the same IV/counter.
Could you please update the README to warn people to never, ever,  ever reuse the IV when using AES-CTR?
When using CTR, the developer has to create a method of ensuring that a new IV/counter is used every time, if the is a chance that the key can be used.
Therefore the default should probably not start counting at 1, but at a random number, unless the developer explicitly specify a counter.

Whats your opinion? What settings should i use with this lib
Generate new random 16-byte counter for every message and then send it unencrypted alongside a message.

The IV is necessary to decrypt the content in the future, so picking a random IV would be unwise (also, it is hard in many environments to generate a random number portably, so I would recommend against any library not equipped to do so, from generating a random number).
Often the source of the IV comes from either a random source, or from the use of a session (or ephemeral) key mixed with a known public key for another’s private key, and the public key of this session key is prepended to the cipher text.
As an aside, due to the dangers of choosing a “random” number (which is difficult and something easy to hijack), a method similar to RFC-6979 might be preferably, where a hash (HMAC, preferably) of the key and content is used to create a deterministic IV; this means the same plaintext produces the same cipher text and that different plaintexts get different IV. However, do NOT do this if multiple messages will match and you do not want to leak the pattern. You also can’t do this if it is planned as a streaming cipher (one of the advantages of CTR node). This is just an example technique, for very specific cases.
The AES block cipher is a cryptographic primitive, so it’s very important to understand and use it properly, based on its application. It’s a powerful tool, and with great power, yadda, yadda, yadda. :)
Does that make sense?

The IV is necessary to decrypt the content in the future,
Agree. Therefore you need to store the IV. It can be stored as plain text. IVs don't need to be secret.

so picking a random IV would be unwise.  (also, it is hard in many environments
Disagree. In JavaScript, use Crypto.getRandomValues() in a browser or in NodeJS.

@WavePlayz

Whats your opinion? What settings should i use with this lib
Sorry, but it depends on what you are encrypting. :-/

Unique secret keys
If you can use different key/passwords for each encryption, you can you CTR, and just count up from 1. (And stop reading the rest of this post. : )
Reusing secret keys
If you have to use the same secret key/password for multiple/all encryptions, you must pay more heed to the IV.
Synthetic IVs
Hashing the secret key/password to create an IV (as @ricmoo is explaining above) is a often good solution. (The concept is often called SIV, Synthetic IV, and used in AES-SIV, AES-GCM-SIV.)
The downside of SIV is that two equal massages will be encrypted identically.  So an eavesdropper, can pick up that you are sending the same message over and over, and when you do so.
If this is not a problem (or if it is a benefit), feel free to go for the SIV approach. And you'll be free to choose AES-CTR, (or CBC, or another.)
Careful design
If none of these apply, you have to carefully design a IV regime that avoids having the same IV for the same secret key/password. Abd in this case, I would be very careful about using AES-CTR. As it fails catastrophically if you get a collision. The security of AES-CBC is weakened in this situation, but doesn't completely collapse.
Malleable
If I catch an encrypted message, I may not be able to decrypt it. But nothing is stopping me from changing a bit or two in the message, before passing it on. And if this changed message, makes sense, I can "edit" the encrypted messages  (turning  amount=00000.01 into amount=10000.01), and the received has no way of knowing that the message has been tampered with. (Malleable)
If this is not an issue for you case, don't worry about it. If it is a worry, you need to use HMAC (as mentioned above), or another AES mode that includes an HMAC, such as AES-GCM.

@vbakke

As it fails catastrophically if you get a collision

How catastrophically is catastrophically, by the way? For example, of 1000 encrypted messages I've got 2 that share same key and IV; and attacker intercepted all 1000 messages, but doesn't know about IV collissions beforehand - how bad is this situation for me?

It's hard to find proper sources for this. Most just say it "fails catastrophically".
From what I understand, The attacker can manage to decode the two messages with the colliding IVs, but not retrieve the secret key, and therefor not decode the other messages.
One source I've found is this Stack Exchange post: https://crypto.stackexchange.com/questions/2991/why-must-iv-key-pairs-not-be-reused-in-ctr-mode
Which links to another answer: https://crypto.stackexchange.com/questions/2249/how-does-one-attack-a-two-time-pad-i-e-one-time-pad-with-key-reuse
I presume an attack in your example would be XORing all the messages against each other. If the massages are mostly plain text, any colliding IVs would result in an XOR-result with very few of the bit 7 ever being set. And then you start digging. A bit of work involved, though. :)


cloudfront.Distribution Error 412

After successfully creating and updating an aws.cloudfront.Distribution resource, I start getting the following errors:
Updating stack 'default'
Performing changes:

     Type                            Name                    Status                  Info
 *   pulumi:pulumi:Stack             pulumi-scratch-default  done
 ~   └─ aws:cloudfront:Distribution  test-dist               **updating failed**     changes: ~ aliases, 1 error

Diagnostics:
  aws:cloudfront:Distribution: test-dist
    error: Plan apply failed: updating urn:pulumi:default::pulumi-scratch::aws:cloudfront/distribution:Distribution::test-dist: error updating CloudFront Distribution (EYJF95DXKETKL): PreconditionFailed: The request failed because it didn't meet the preconditions in one or more request-header fields.
    	status code: 412, request id: 57185929-a80b-11e8-a357-ad021cd7bac9

Consequently the distribution resource can then no longer be updated or destroyed. The first few updates go through fine but then it eventually starts erroring. I don't know if it's related to the specific updates I'm doing (changing the aliases and acmCertificateArn fields repeatedly) or a certain amount of time passing.

I believe this is due to https://github.com/pulumi/pulumi/issues/1449.
In a nutshell, many operations in AWS will silently mutate the CloudFront distribution's ETAG, and then subsequent attempts to update it will fail (due to this "precondition").
pulumi refresh will fix this, by re-synchronizing the ETAG in your state file, however @pgavlin just fixed a bug that was blocking its usage for this very scenario. I know we plan to release a new CLI imminently, which is likely required for this -- @pgavlin is this ready to go?
In the meantime, #1449 has some potential manual workarounds that we know will work.

Is this the error on refresh?
Diagnostics:
  global: global
    error: diffing urn:pulumi:default::pulumi-scratch::aws:cloudfront/distribution:Distribution::test-dist: active_trusted_signers: 1 error(s) decoding:

    * '[items]' expected type 'string', got unconvertible type '[]interface {}'

@pgavlin has offered to help load balance, as part of M19 shutdown. Thanks Pat!

I have confirmed that pulumi refresh is working as expected for CloudFront distributions. I am closing this as a duplicate of https://github.com/pulumi/pulumi/issues/1449.


feat: Linear order on upper/lower sets

Match https://github.com/leanprover-community/mathlib/pull/19068

bors merge


A problem about fakeplayer relogin

A player made the fakeplayer relogin as often as 2gt on my server, after a few minutes the server mspt increased to a really high value， killing the fakeplayer still didn't fix the problem unless the server was restarted

This issue relates to memory leak of fake player spawning in fabric-carpet, and could be considered as a compatability issue between fabric-carpet and fabric-api. Refer to
https://github.com/FabricMC/fabric/issues/3974
and
https://github.com/gnembon/fabric-carpet/issues/1957
Here's your possiblt solution:

Remove fabric-api or fabric-carpet
Wait until fabric-api or fabric-carpet to upgrade
Wait for us to make a temporarily fix for it

I am doing development about fixing this bug temporarily from Carpet-Org-Addition.
该问题与fabric-carpet生成假人时的内存泄露有关，是fabric-carpet与fabric-api的兼容性问题。参考
https://github.com/FabricMC/fabric/issues/3974
和
https://github.com/gnembon/fabric-carpet/issues/1957
以下是可能的解决方案：

删除fabric-api或fabric-carpet其中一个
等待fabric-api以及fabric-carpet更新
等待我们对此进行一个临时修复

我正在针对此问题进行临时修复的开发。