[Discuss] Relax Layout Transformation

Motivation & Goals
Tensor data layout describes how the data is laid out in memory. It determines the memory access pattern and it can significantly impact performance and memory efficiency. Global layout planning thus becomes an important optimization to achieve performance on various hardware backends. The goal of this document is to present the design of global layout planning in Relax.
Terminology
To classify operators at layout perspective, TVM has defined the following terminologies. Note that we do not use these terms in this document, but these can be used in discussions.

Layout agnostic: relu, pow etc. These operators are not affected, neither functionality nor performance, by data layouts.
Lightly-layout sensitive: pad, concatenate, reduce ops like sum etc. These operators have some attributes that are functionally affected if we do a layout transformation before them. However, performance-wise, the difference is not significant. For these operators, it is beneficial to just adapt to the previous operator output data layout.
Heavily-layout sensitive: conv2d, conv2d_transpose etc. These operators are heavily affected, both functionally and performance-wise, by data layouts. They also have data layout as the op attribute. The performance benefit of a layout transformation for these operations, outweighs the runtime cost of performing a layout transformation. Thus, it is beneficial to modify the input data layouts for these operators (if its not a performant data layout), while the rest of layout agnostic and lightly-layout sensitive operators adapt to the layout governed by the output of these heavily-layout sensitive operators.

We introduce and clarify the following terms used in this doc.

Layout-critical operations: These correspond to heavily-layout sensitive operations in existing terminology. These operations are highly performance sensitive to operand/result layouts. For example, convolution operations.
Frozen layout: Fixed layout decision for an operator.  Typically, heavily-layout sensitive operators will have frozen layouts.
Flow layout constraints: When layout-agnostic and lightly-layout sensitive operators are connected to operators with frozen layouts, they can take those frozen layouts as constraints and adjust themselves without any noticeable performance impact. We call this process “flowing layout constraints”.
Layout rewrite: Explicit conversion for layout when two adjacent operators have different layout constraints.
Merge/Reduce/Cancel/Fold layout rewrites: When a layout rewrite revert its adjacent layout rewrite (e.g., NCHW->NHWC followed by NHWC->NCHW), we can safely eliminate both.

Prior Art
Relay has the Convert Layout pass which first applies the user specified layout to the operations in the Relay graph. For example, the user may specify that they want all convolution operations to have NHWC layout. The ConvertLayout pass would then transform the IRModule so that user desired layouts are honored and transforms the Relay graph with minimal number of data layout transforms.
It achieves this by requiring each Relay operator to have InferCorrectLayout property. The pass uses this property for layout inference. It looks at the original input layouts and the new input layouts and the InferCorrectLayout **property tells operator needs to be modified to accommodate the new input layouts, and what the new output layouts should be. Layout transforms are inserted where new input layouts differ from incoming layouts. This step is done operator by operator in sequence, where ConvertLayout pass keeps on passing the new layouts to the next operator properties, finally resulting in modifying the whole graph operator-by-operator.
Relay has a long list of InferCorrectLayout methods attached to operators. The logic for a number of operators is quite complex, error-prone and a huge burden to maintain. In some cases, Relay operators cannot be modified to accommodate tiling layout transforms.
Also, Relay performs AlterOpLayout pass to conditionally apply the additional layout transformation (e.g., winograd transformation) given operator, target, etc. As its main usage is more like a secondary optimization that can be applied on the top of global layout planning, we will have a separate design doc to account for it.
Relay Example
Let’s take a look at en example. The Relay graph below preforms a convolution followed by a reduction across H axis.
fn (%x: Tensor[(32, 64, 56, 56), float32], %weight: Tensor[(64, 64, 3, 3), float32]) {
  %0 = nn.conv2d(%x, %weight, padding=[1, 1, 1, 1], channels=64, kernel_size=[3, 3]);
  sum(%0, axis=[2])
}

If we modify the input and output layouts of convolution operation, layout_transform operations are inserted in the graph to maintain correctness. At the same time, the InferCorrectLayout property of sum operation modifies the axis attribute of the operation from axis = [2] to axis = [1] to accommodate changing the layout from NCHW to NHWC.
fn (%x: Tensor[(32, 64, 56, 56), float32] /* ty=Tensor[(32, 64, 56, 56), float32] */, %weight: Tensor[(64, 64, 3, 3), float32] /* ty=Tensor[(64, 64, 3, 3), float32] */) -> Tensor[(32, 64, 56), float32] {
  %0 = layout_transform(%x, src_layout="NCHW", dst_layout="NHWC") /* ty=Tensor[(32, 56, 56, 64), float32] */;
  %1 = layout_transform(%weight, src_layout="OIHW", dst_layout="HWIO") /* ty=Tensor[(3, 3, 64, 64), float32] */;
  %2 = nn.conv2d(%0, %1, padding=[1, 1, 1, 1], channels=64, kernel_size=[3, 3], data_layout="NHWC", kernel_layout="HWIO") /* ty=Tensor[(32, 56, 56, 64), float32] */;
  %3 = sum(%2, axis=[1]) /* ty=Tensor[(32, 56, 64), float32] */;
  layout_transform(%3, src_layout="NWC", dst_layout="NCW") /* ty=Tensor[(32, 64, 56), float32] */
} /* ty=fn (Tensor[(32, 64, 56, 56), float32], Tensor[(64, 64, 3, 3), float32]) -> Tensor[(32, 64, 56), float32] */

Global Layout Planning in Relax
In Relax we perform layout planning in a very different way as compared to Relay. In Relay the layout rewrites happens at Relay operator level (i.e., relay.nn.split, relay.sum etc.). In Relax we apply layout rewrites after lowering Relax operators to TIR. The access to the underlying compute in TIR for an operation makes it much easier to flow layout constraints in Relax.
Here is a reduction operation in NCHW layout.
@T.prim_func
def sum(input: T.Buffer[(32, 64, 56, 56), "float32"], output: T.Buffer[(32, 64), "float32"]) -> None:
    for i0, i1, i2, i3 in T.grid(32, 64, 56, 56):
        with T.block("reduce"):
            ax0, ax1, k2, k3 = T.axis.remap("SSRR", [i0, i1, i2, i3])
            T.reads(input[ax0, ax1, k2, k3])
            T.writes(output[ax0, ax1])
            with T.init():
                output[ax0, ax1] = T.float32(0)
            output[ax0, ax1] = output[ax0, ax1] + input[ax0, ax1, k2, k3]

To transform the above operation from NCHW to NHWC layout, we can simply transform the buffer and block layouts using tir.schedule.transform_layout and tir.schedule.transform_block_layout primitives.
Reduce operation in NHWC layout.
@T.prim_func
def sum(input: T.Buffer[(32, 56, 56, 64), "float32"], output: T.Buffer[(32, 64), "float32"]) -> None:
    for ax0, ax1, ax2, ax3 in T.grid(32, 56, 56, 64):
        with T.block("reduce"):
            v0, v1, v2, v3 = T.axis.remap("SRRS", [ax0, ax1, ax2, ax3])
            T.reads(input[v0, v1, v2, v3])
            T.writes(output[v0, v3])
            with T.init():
                output[v0, v3] = T.float32(0)
            output[v0, v3] = output[v0, v3] + input[v0, v1, v2, v3]

In an ideal scenario, layout planning would figure out the most efficient layout for all operations to minimize the end-to-end cost of graph. However, we are not there yet. For now, we need help either from user or auto tuning system like Metaschedule to identify layout-critical operations in the graph and choose the best layout for them greedily. This would result in a graph where layouts for some operations are frozen (i.e., layouts must not be modified by other passes), and layout rewrite operations added to operands and results edges of these layout-critical operations. Next, an optimization pass would flow these layout rewrites in the graph to reduce the overall cost of such copy operations.
Thus, in Relax we break down the layout planning into two sub problems, which are described separately in the document below.

Tune layout-critical operations and freeze layouts. A pass to identify layout-critical operations and find best schedule for them greedily. Mark these operations as having “frozen layouts” and add layout rewrite operations on operands and results to maintain correctness. Such a pass could work with tuning frameworks like Metaschedule to find the best schedule for layout-critical operations.
Minimize layout rewrite cost in graph. An optimization pass to reduce the cost of layout rewrites in the graph. This would be achieved by flowing layout rewrites in the graph to facilitate fusing/folding them with other layout rewrites, operators, or constants.

Tune Layout-Critical Operations & Freeze Layouts
This can be done in conjunction with tuning frameworks like Metaschedule, or a pass that allows user to annotate layouts of specific operations.

Using Metaschedule: Metaschedule could be used to find the best schedule of layout-critical operations greedily. It would add layout rewrite blocks in the prologue and epilogue of TIR function for the operation. A HoistLayoutRewritePass (discussed in appendix) would be added as part of this proposal to  pull out the layout rewrites into separate TIR functions and mark the layout-critical TIR function with “frozen layout” attribute.
User annotated layout: A pass could take user specification of preferred operand-result layouts for relax operations. The pass would insert layout rewrite operations on operand & result edges of these operations if they conflict. It will also mark these operations as having frozen layouts. Such a pass would be implemented as part of layout planning in Relax, but it is out of scope for this design doc which focusses on the problem of minimizing the cost of layout rewrites in the graph. A separate design document would discuss the API and strategy for such a pass.
Other. To promote other strategies of tuning layout-critical operations, any other strategy which tunes specific operations and freezes their layouts is also valid. It is expected that layout rewrite operations would be added by such a strategy. It can then use the layout planning pass to minimize the cost of layout rewrites in the graph.

Minimize Layout Rewrite Cost in Graph
The minimize layout rewrite cost problem can be formulated as below:
Given a directed acyclic graph G(V, E), where v $\in$ V is an operation and e $\in$ E are data values flowing through G, a subset of operations are marked as having frozen layout i.e., the operand & result layouts of the operation cannot be modified. The graph contains layout rewrite operations to satisfy the layout constraints of operations, constants, graph inputs and outputs. The goal of layout planning is to minimize the cost of layout rewrite operations in the graph.
Layout Representation
The layout rewrites in Relax can be represented as any of the following two ways.

PrimFunc Representation. Layout rewrites will be represented as TIR functions in the IRModule. This has the benefit of easy serialization/deserialization of these constraints without the introduction of any new structure in the IR. For example, following is a layout rewrite in Relax. Note that not all TIR functions that do spatial transformation of a buffer would be candidates that can flow through operations and optimized away. Only  a subset of such PrimFuncs, whose transformation (or inverse) can be represented by a compact IndexMap representation would be candidates which can flow and be optimized. While IndexMap is a nice compact representation for expressing layout rewrites, it also helps to use the same layout representation that is used by tir.schedule.transform_layout API.
@T.prim_func
def layout_transform(arg: T.Buffer[(32, 64, 224, 224), "float32"], out: T.Buffer[(32, 16, 224, 224, 4), "float32"]) -> None:
    for i0, i1, i2, i3, i4 in T.grid(32, 16, 224, 224, 4):
        with T.block("T_layout_trans"):
            ax0, ax1, ax2, ax3, ax4 = T.axis.remap("SSSSS", [i0, i1, i2, i3, i4])
            T.reads(arg[ax0, ax1 * 4 + ax4, ax2, ax3])
            T.writes(out[ax0, ax1, ax2, ax3, ax4])
            out[ax0, ax1, ax2, ax3, ax4] = arg[ax0, ax1 * 4 + ax4, ax2, ax3])

Compact Layout Representation: While the PrimFunc representation is very useful in representing layout rewrites generally, it could be difficult to flow/fuse them in their general form. To flow layout rewrites through an operation, a compact representation of layout rewrite is very useful as it could be applied to the operand/result buffers. The compact representation could (1) make it easier to fuse/fold/cancel layout rewrites, and (2) easily be transformed to tir.schedule primitives to actually modify layouts of TIR buffers. We propose to have the compact form of layout rewrites be represented through Relax operations.

Relax operation representation.  The relax operation representation of layout rewrite would use the following relax operations.

relax.layout_transform(input: Tensor, index_map: lambda): This maps the input to a new iteration space. The index_map defines the mapping function. This is a pure layout transform, i.e., index_map is a bijective function.
relax.pad(input: Tensor, pad_width: Tensor, pad_value: scalar): Inserts pad_value to the given pad_width locations. pad_width is an integer tensor with shape [n, 2], where n is the rank of input.  For each dimension d of input, pad_width[d, 0] indicates how many values to add before the contents of input in that dimension, and pad_width[d, 1]indicates how many values to add after the contents of input in that dimension.
relax.crop(input: Tensor, start_indices, slice_sizes, cropped_value): Crops the tensor specified in start_indices and slice_sizes. The optional cropped_value is a hint to the compiler about the values stored in the input tensor regions that were cropped away. This is useful information for the compiler if it intends to cancel this relax.crop with a following relax.pad.

It should be easy to roundtrip between the PrimFunc representation and compact relax operation representation of layout rewrites. There are some valid questions/concerns that we aim to address on the choice of compact representation of layout rewrites:

Since relax.layout_transform only supports bijective transformation, would general PrimFunc layout rewrites (with implicit padding/cropping) have to be broken into smaller primitive TIR blocks in the input IRModule (i.e., Does the layout planner expect prior passes to break general PrimFunc layout rewrites into smaller TIR blocks?)
Answer: No. The input IRModule to layout planning pass could use general PrimFunc representation. We can break such PrimFunc blocks into primitive pad/crop and bijective layout rewrites upon conversion to compact form. So there is no constraint on having only bijective layout rewrites/pad/crop TIR blocks by previous passes. Furthermore, layout rewrite TIR blocks within a Primfunc could be annotated with compact representations (may be use IndexMap representation). HoistLayoutRewritePass could use these compact representations instead of recovering the compact form from TIR block.

Is it necessary to break the layout rewrites into these primitive relax operations (bijective transformation, pad, and crop) for easier cancellation? Can we fuse two layout rewrites in the PrimFunc representation and prove that it is identity?
Answer: Probably no. If we can have such an analysis in TIR that can prove that a PrimFunc is identity for most of our use cases, that would lower the need to have these primitive representation. A compact representation (relax.layout_transform that supports non-bijective rewrites) would still be needed, but we won’t need to break down general layout rewrite into these primitives.

Example
To minimize the cost of layout rewrites in the graph, a pass can flow them across operations until they they can be fused into a constant, cancel out or fuse with other layout rewrites. For instance, in the following graph (which we will use as a running example), we have conv-->add-->conv graph. The convolution operations have “frozen layouts” (marked in orange). The layout rewrite operations are marked in green.
graph LR
    X((x)) --NCHW--> Transform_x(to NCHWc) --NCHWc--> Conv(conv)
		Conv --NCHWc--> Transform_r(to NCHW) --NCHW--> Add(add)
    F((f)) --OIHW--> Transform_f(to OIHWio) --OIHWio--> Conv
		Bias((bias)) -- NCHW --> Add -- NCHW --> Transform_o(to NCHWc) --NCHWc--> Conv2(conv)
		Conv2 --NCHWc--> Transform_conv(to NCHW) --NCHW--> output((result))
		subgraph _
			Transform_r
			Bias
			Transform_o
			Add
		end
		classDef orange fill:#f96,stroke:#333,stroke-width:4px;
		classDef green fill:#16b522,color:#fff
		classDef green_highlight fill:#16b522,stroke:#f96,stroke-width:4px,color:#fff
		class Conv orange
		class Conv2 orange
		class Transform_x green
		class Transform_r green
		class Transform_f green
		class Transform_o green_highlight
		class Transform_conv green

graph LR
    X((x)) --NCHW--> Transform_x(to NCHWc) --NCHWc--> Conv(conv)
		Conv --NCHWc--> Transform_r(to NCHW) --NCHW--> Transform_rinv(to NCHWc)--NCHWc--> Add(add)
    F((f)) --OIHW--> Transform_f(to OIHWio) --OIHWio--> Conv
		Bias((bias)) -- NCHW --> Transform_o(to NCHWc) --NCHWc--> Add --NCHWc--> Conv2(conv)
		Conv2 --NCHWc--> Transform_conv(to NCHW) --NCHW--> output((result))
		subgraph _
				Transform_r
				Transform_rinv
				Bias
				Transform_o
			Add
		end
		classDef orange fill:#f96,stroke:#333,stroke-width:4px;
		classDef green fill:#16b522,color:#fff
		classDef green_highlight fill:#16b522,stroke:#f96,stroke-width:4px,color:#fff

		class Conv orange
		class Conv2 orange
		class Transform_x green
		class Transform_r green
		class Transform_f green
		class Transform_o green_highlight
		class Transform_conv green
		class Transform_rinv green_highlight

graph LR
    X((x)) --NCHW--> Transform_x(to NCHWc) --NCHWc--> Conv(conv)
		Conv --NCHWc---> Add(add)
    F((f)) --OIHW--> Transform_f(to OIHWio) --OIHWio--> Conv
		Bias((bias*)) --NCHWc--> Add --NCHWc--> Conv2(conv)
		Conv2 --NCHWc--> Transform_conv(to NCHW) --NCHW--> output((result))
		subgraph _
			Bias
			Add
		end
		classDef orange fill:#f96,stroke:#333,stroke-width:4px;
		classDef green fill:#16b522,color:#fff
		class Conv orange
		class Conv2 orange
		class Transform_x green
		class Transform_r green
		class Transform_f green
		class Transform_o green
		class Transform_conv green
		class Transform_rinv green

The number of layout rewrites can be reduced by flowing the to NCHWc layout rewrite across add operation from result to operands and then simplifying the graph by folding adjacent layout rewrites (in this case into an identity), and fusing the layout rewrite into a constant.
Folding/Fusing Layout Rewrites
The layout rewrites can be reduced through the following mechanisms

Folding adjacent layout rewrites: Adjacent layout rewrites can be folded into each other. In terms of relax operation representation, this would mean:

two relax.transform_layout operations being folded by folding their index maps
relax.pad operation being folded into relax.crop operation. Folding these two ops is a bit subtle and must obey the following rules, assuming the indices being padded and cropped in the two operations are the same.

a relax.crop operation can always be folded into a prior relax.pad operation.
a relax.pad operation can only be folded into a prior relax.crop operation if all the cropped values are T.undef or same as the values being padded.

Folding layout rewrite into a constant: A layout rewrite on the output edge of a constant can be folded into the constant itself. For example, to NCHWc layout rewrite operation was folded into the bias constant in the above example.
Folding layout rewrite into an operation: Layout rewrites can also be folded into an operation, i.e., modify the layout on one buffer but leave the rest of the operands/results as is. It is unclear though if this could lead to a more optimized graph. In many cases, it would lead to inefficient access patterns in the loops for operands and results (see appendix). For such cases, we can leave the operation + layout rewrite as is in the graph to be optimized by a tuning pass later. The tuning pass can choose to fold the layout rewrite into the operation or materialize it as it sees fit.

Direction of Flowing Layout Rewrites
In order to facilitate folding layout rewrites into other layout rewrites and constants, we intend to be able to flow layout rewrites across operations. In the example above, the layout rewrite post add operation was flowed through it (result to operands) thus allowing it to be folded into an inverse layout rewrite and constant.
The flow of layout rewrites presents us with a design choice on the direction of flow. The following two options were considered.

F0: Operands to Results (Forward). Flow layout rewrite from an operand to results.

Pros

Operations with single operand and multiple results (for example split) would not mess up the access patterns of output buffers.

Cons

If an operation has multiple operands, then flowing the rewrites through it, could mess up the access patterns of other operands.
We cannot fold layout rewrites into constants in this flow, as constants are operations with zero inputs and one output.

F1: Results to Operands (Backward). Flow layout rewrites from a result to operands.

Pros

In ML domain, many operations would fall into the category of multiple operands and single result. So this approach is unlikely to mess up the access pattern of other buffers.
We can expect that in this flow, some of the layouts could be folded into constants.

Cons

If an operation has multiple results, this direction of flow can mess up the access pattern of other outputs.

Acknowledging that none of the two choices are strictly better than the other, the arguments for F1 seem more useful for a large class of operations. So for that reason, we would prefer F1.
Flowing Layout Rewrite through an Operation
For the generic support for flowing layout rewrites, we propose two flow mechanism for each abstraction-level. With these two approaches, we believe our approach can support any IRModule, which may contain a mixture of Relax operators and TIR functions, within the compilation pipeline.
Flowing Layout Rewrite through an Operation at TIR-level
This section would answer the question, “How to generate layout rewrite on the operand buffers of an operation, when flowing layout rewrite through it?”
An analysis of TIR block(s) where the result buffer is written to, can help us identify the layout rewrites for operands.
Let’s say the TIR PrimFunc has one result (output) and multiple operands (arg0, arg1, …). A layout rewrite lambda N, C, H, W: (N, H, W, C // 4, C % 4) is applied on the output buffer. Our goal is to identify the layout rewrites on all of the operands.

Find the block B that writes to output.
Inspect the [T.read](http://T.read) and T.write signatures on B. For output buffer the access would be output[ax0, ax1, ax2, ax3]. Using the mapping N = ax0, C = ax1, H = ax2, W = ax3 apply the layout rewrite to each of the buffers in T.read accesses.

In the presence of temporary buffer allocation in TIR PrimFunc, we might have to flow the layout rewrites through multiple blocks up to the operands.
The following examples show how this is done for broad classes of operations.
Elementwise Operations.
@T.prim_func
def relu(input: T.Buffer[(32, 3, 224, 224), "float32"], output: T.Buffer[(32, 3, 224, 224), "float32"]) -> None:
    for i0, i1, i2, i3 in T.grid(32, 3, 224, 224):
        with T.block("compute"):
            i0_1, i1_1, i2_1, i3_1 = T.axis.remap("SSSS", [i0, i1, i2, i3])
            T.reads(input[i0_1, i1_1, i2_1, i3_1])
            T.writes(output[i0_1, i1_1, i2_1, i3_1])
            output[i0_1, i1_1, i2_1, i3_1] = T.max(input[i0_1, i1_1, i2_1, i3_1], T.float32(0))

In the above PrimFunc, let’s say the output has a layout rewrite lambda N, C, H, W: (N, C // 4, H, W, C%4). We want to flow this rewrite from output to input.  output buffer is written in block compute. The signature of the block has information on the buffers it reads and writes. Here it writes to buffer output[i0_1, i1_1, i2_1, i3_1] and reads buffer input[i0_1, i1_1, i2_1, i3_1]. Since the access to both these buffers are identical, the layout rewrite would be identical too. Thus, the layout planning would modify both input and output buffers with the same layout rewrite as it flows the rewrite operation through relu op.
Broadcast Operations.
@T.prim_func
def add(input: T.Buffer[(32, 256, 213, 213), "float32"], bias: T.Buffer[(256, 1, 1), "float32"], output: T.Buffer[(32, 256, 213, 213), "float32"]) -> None:
    for i0, i1, i2, i3 in T.grid(32, 256, 213, 213):
        with T.block("T_add"):
            ax0, ax1, ax2, ax3 = T.axis.remap("SSSS", [i0, i1, i2, i3])
            T.reads(input[ax0, ax1, ax2, ax3], bias[ax1, 0, 0])
            T.writes(output[ax0, ax1, ax2, ax3])
            output[ax0, ax1, ax2, ax3] = input[ax0, ax1, ax2, ax3] + bias[ax1, 0, 0]

Let’s say the output has a layout rewrite operation lambda N, C, H, W: (N, C // 4, H, W, C%4). Our goal is to flow the layout rewrite operation from result to operands. We know output buffer is written in block T_add. The signature has the access patterns for output[ax0, ax1, ax2, ax3], input[ax0, ax1, ax2, ax3] and bias[ax1, 0, 0] buffers. Mapping the axes C == ax1 and applying the layout rewrite we know that the layout rewrites for input and bias buffers.

input: lambda N, C, H, W: (N, C // 4, H, W, C%4)
bias: lambda i, j, k: (i // 4, 0, 0, i % 4)

Reduction Operations.
@T.prim_func
def sum(input: T.Buffer[(32, 256, 213, 213), "float32"], output: T.Buffer[(32, 256), "float32"]) -> None:
    for i0, i1, i2, i3 in T.grid(32, 256, 213, 213):
        with T.block("rxplaceholder_red"):
            ax0, ax1, k2, k3 = T.axis.remap("SSRR", [i0, i1, i2, i3])
            T.reads(input[ax0, ax1, k2, k3])
            T.writes(output[ax0, ax1])
            with T.init():
                output[ax0, ax1] = T.float32(0)
            output[ax0, ax1] = output[ax0, ax1] + input[ax0, ax1, k2, k3]

Let’s say the output has a layout rewrite operation lambda N, C: (N, C // 4, C%4). Our goal is to flow the layout rewrite operation from result to operand. We know output buffer is written in block rxplaceholder_red. The signature has the access patterns for output[ax0, ax1] and  input[ax0, ax1, k1, k2] . Mapping the axes C == ax1 and applying the layout rewrite we know that the layout rewrites for input  buffer will be lamda i, j, k, l: (i, j // 4, k, l, j % 4)
Fused Operations.
Fused operations could have the added complication of having a series of blocks. We can use the dependency analysis on buffers using block signatures to propagate the layout rewrite rules from result to operands.
Flowing Layout Rewrite through an Operation at Graph-level
Oftentimes, layout rewrites need to flow at graph-level before lowering to TIR-level. BYOC is a good example - (1) BYOC may have certain layout constraints. (2) BYOC codegen works at graph-level by converting each relax op to BYOC equivalent. Relay solves this problem by introducing InferCorrectLayout to provide the manual guidance. Unfortunately, it has been source of many tricky issues (e.g., https://github.com/apache/tvm/pull/10118, https://github.com/apache/tvm/pull/10156, https://github.com/apache/tvm/pull/12007).
To overcome this problem, we propose the following the graph-level flow mechanism that peeks at PrimFunc implementation while leveraging the powerful TIR-level analysis:

Load the simplest PrimFunc implementation for an operator
Flow the constraints at TIR-level. In other words, transform Primfunc to conform the given layout.
Update the operator based on the transformed PrimFunc

Step 3 might be tricky with the current PrimFunc design since we lose the convenient access to op-level info (e.g., operator name, attributes, etc.) at TIR-level; to be clear, PrimFunc implicitly embeds those information but it may not be easy to extract them. Based on our investigation, it seems possible to extend PrimFunc to maintain the op-level information in an accessible way. It also reveals that we might be able to maintain this information when flowing transpose-like layout rewrites through them. Note that BYOC use case only requires transpose-like layout rewrites (e.g., NCHW to NHWC) which is easier to support than layout rewrites with implicit tiling/padding (e.g., NCHW to NCHWc)
Fallback: Layout Rewrite Transformation Callback
Although the above sections cover most of the scenarios, in some cases the user might want to have explicit control over how a relax operator or PrimFunc should be modified when flowing layout rewrites through it. For example, the PrimFunc could have an opaque computation, making it hard to figure out the layout rewrites on operands from result layout rewrites. For such cases, an easy way to register such a callback function on an operation would be provided. Similar to FTVMConvertLayout property in Relay, it would allow user defined alterations to the operation when flowing layout rewrites. When registered, the callback would be used instead of the analyses mentioned in the previous sections.
Advantage of Relax Layout Planning over Relay
The approach described in this doc has multiple advantages over layout planning in Relay.

Robustness: Unlike Relay, we do not need to write and maintain code to tell the pass how to flow layout rewrites through an operation. InferCorrectLayout property currently has many lines of code with complex logic which is error-prone and huge burden to maintain. In the new layout planning pass, all of this code would not be needed which is huge win. The strategy described above is much more robust and can handle a broad class of PrimFuncs even when they do not correspond to operations in Relax operator system.
Support for more general layout transformations: Relay deals with string representation of layout transformations. There is also implicit meaning attached to dimension symbols - H represents height, W represents width, etc. This results in code that could easily break. Relax layout planning does not use strings for layout representation thus avoiding the pitfalls of the Relay and robust against general layout transformations.
Easier operator registration: In Relay, users have to manually define InferCorrectLayout for their new operators, which is not necessarily easy to figure out. However, as Relax layout planning does not rely on this, operator registration becomes much easier.
Cleaner pipeline, improved debuggability and customizability: In Relay, ConvertLayout and OpStrategy are tightly coupled to each other: starting from user annotation in ConvertLayout and then during lowering, OpStrategy will find the right implementation accordingly by using the layout information. As OpStrategy is a complicated inflexible component that lives outside of pass infra, it makes layout optimization hard to debug and customize in many occasions. On the other hand, in Relax, end-to-end flow can live in the pass infra which can be easier to debug and customize. You can perform layout planning at any stage in the pipeline before codegen, even when you have a partially lowered IRModule.

Appendix
Hoist Layout Rewrites Pass
This pass inspects TIR function and lifts any layout rewrite blocks before and after the computation into separate TIR functions. It could either use some analysis of blocks in the PrimFunc or use explicit block annotations to identify layout rewrite blocks within a PrimFunc.
For example, the layout_rewrite block in the matmul PrimFunc below would be lifted out into a separate PrimFunc after this pass.
@R.function
def main(x: Tensor((16, 16), "float32"), w: Tensor((16, 16), "float32")) -> Tensor((16, 16), "float32"):
    gv0 = R.call_tir(matmul, (x, w), (16, 16), dtype="float32")
    return gv0

@T.prim_func
def matmul(A: T.Buffer[(16, 16), "float32"], B: T.Buffer[(16, 16), "float32"], C: T.Buffer[(16, 16), "float32"]) -> None:
    B_ = T.alloc_buffer([16, 4, 4], dtype="float32")
    for i0_o, i1_o in T.grid(16, 16):
        with T.block("layout_rewrite"):
            i0, i1 = T.axis.remap("SS", [i0_o, i1_o])
						# optional attribute annotation to identify layout rewrite blocks
            T.block_attr({"meta_schedule.layout_rewrite_preproc": True})
            B_[i1, i0 // 4, i0 % 4] = B[i0, i1]
    for i0, j, k0, i1, k1 in T.grid(4, 16, 4, 4, 4):
        with T.block("matmul"):
            vi = T.axis.spatial(16, i0 * 4 + i1)
            vj = T.axis.spatial(16, j)
            vk = T.axis.reduce(16, k0 * 4 + k1)
            with T.init():
                C[vi, vj] = T.float32(0)
            C[vi, vj] = C[vi, vj] + A[vi, vk] * B_[vj, vk // 4, vk % 4]

@R.function
def main(x: Tensor((16, 16), "float32"), w: Tensor((16, 16), "float32")) -> Tensor((16, 16), "float32"):
    gv = relax.call_tir(layout_rewrite, (w,), (16, 4, 4), dtype="float32")
    gv0 = relax.call_tir(matmul, (x, gv), (16, 16), dtype="float32")
    return gv0

@T.prim_func
def matmul(A: T.Buffer[(16, 16), "float32"], B: T.Buffer[(16, 4, 4), "float32"], C: T.Buffer[(16, 16), "float32"]):
    for i0, j, k0, i1, k1 in T.grid(4, 16, 4, 4, 4):
        with T.block("matmul"):
            vi = T.axis.spatial(16, i0 * 4 + i1)
            vj = T.axis.spatial(16, j)
            vk = T.axis.reduce(16, k0 * 4 + k1)
            with T.init():
                C[vi, vj] = T.float32(0)
            C[vi, vj] = C[vi, vj] + A[vi, vk] * B[vj, vk // 4, vk % 4]

@T.prim_func
def layout_rewrite(SRC: T.Buffer[(16, 16), "float32"], TGT: T.Buffer[(16, 4, 4), "float32"]):
    for i, j in T.grid(16, 16):
        with T.block("layout_rewrite"):
            i0, i1 = T.axis.remap("SS", [i, j])
            TGT[i1, i0 // 4, i0 % 4] = SRC[i0, i1]

Thank you everyone for the discussion yesterday and bringing up very important points. I'll summarize them below.
Q1. [@Hzfengsy] Would layout planning interfere with fusion?
For the usecases we have in mind - MetaSchedule and User annotation, we do expect that fusion has already happened before layout planning. Specifically I see the following phase ordering: Fusion --> MS Tuning individual PrimFuncs (freezes layouts) --> HoistLayoutRewritePass --> RelaxLayoutPlanning Pass. So in these scenarios we do not expect that layout planning would prevent fusion in any way.
Another approach to tackle this would be to pick forward and backward flow based on user choice. Technically the infra to flow layout rewrites should work regardless of the chosen direction of flow. But I do not have a phase ordering example where that would be necessary and there. Can you elaborate on the use case you had in mind where layout planning and fusion would interfere?
Q2. [@Hzfengsy] Can we expose an API at Relax level to easily transform layouts of PrimFuncs and operators?
Yes, perhaps we can provide an API to modify the layout of a PrimFunc similar to how tir.schedule.transform_layout and tir.schedule.transform_block_layout are used to transform buffers and blocks. We can do that as part of this work.
Q3. [@slyubomirsky] Can you share an example of how layout rewrites would flow through operators?
@sunggg has expanded on this here https://github.com/tlc-pack/relax/issues/278
Q4. [@jinhongyii ]How will you handle layout rewrite flow through pad operation?
Pad operation is itself  a layout rewrite. So it will follow similar behavior to other layout rewrites. For example, two successive pad operations could possibly be fused into each other. Or may be it can be cancelled out with an adjacent crop operations. I expand on when this would be legal in the original post.
It is possible I misunderstood your question. Can you expand on it?

Given broad interests, let me add clarification about how relax layout planner would work with BYOC. Although there could be better approaches with the future improvement in BYOC, in this post, I will explain with the current BYOC approach to focus on delivering the main idea around the interaction between layout planner and BYOC.
For each external library/codegen,  BYOC interface manages a set of specifications about its supported operators and their constraints. For instance, TensorRT supports conv2d only when data_layout==NCHW && kernel_layout==OIHW.
Largely, offloading to BYOC happens in several steps:
// Step 1: Identify valid operators that we can offload
// `pattern_table()` stores the set of operators and their constraints
// If an operator cannot satisfy the check, it won't be annotated in the next step.
transform.MergeComposite(pattern_table())

// Step 2: Annotate each of valid operators
// This example assumes `tensorrt`
transform.AnnotateTarget("tensorrt")

// Step 3: Try to merge annotations to minimize the number of subgraphs to offload
// This is important for performance as invocation of a BYOC runtime module would impose some overhead
transform.MergeCompilerRegions()

// Step 4: Partition the graph according to annotation
transform.PartitionGraph()

To best use BYOC, Relay expects users to convert layouts preferably before the offloading steps above.
transform.ConvertLayout(
                {
                    "nn.conv1d": ["NCW", "default"],
                    "nn.conv2d": ["NCHW", "default"],
                    "nn.conv3d": ["NCDHW", "default"],
                    "nn.conv2d_transpose": ["NCHW", "default"],
                }
            ),
// Offloading step 1-4
transform.MergeComposite(pattern_table())
transform.AnnotateTarget("tensorrt")
// ....

transform.ConvertLayout essentially freezes the layouts of specified operators with user-provided layouts, flow & cancel layout ops at operator level by using InferCorrectLayout property. Since Relax layout planner proposes its own way to flow & cancel, we expect it would be possible to implement the equivalent of transform.ConvertLayout at least while being open-minded about new ideas to attack this BYOC-layout problem.
In case you want to offload only a part of graph (although the graph is fully offload-able), you can customize the annotation by selectively adding/removing the annotation. Please note that the annotated operator nodes will be ignored by internal pipeline by delegating optimization & codegen to external components. By doing so, you may offload one part of graph to BYOC while still leveraging optimization passes like fusion, metaschedule tuning, etc for another.

Would you be able to use ConvertLayout to fix only specific operators, namely the ones you intend to match in BYOC? Perhaps that could be added as an interface to BYOC (e.g., fix layouts at each pattern match)

@slyubomirsky Yeah, that seems possible. That is one of the potential improvements discussed offline with @YuchenJin and @psrivas2.

@psrivas2 I thought that flowing layout through pad will result in some different behavior, but it proved that I'm wrong. So Q4 is no longer a problem.

What's the status of this?

Given broad interests, let me add clarification about how relax layout planner would work with BYOC

I'm interested in tackling this problem. I've worked on two BYOC backends recently (DNNL and CUTLASS), they both want layouts that are likely different from the input mod (NCHWc or NHWC).
Probably the first step is to add a high level layout_transform op. We also need to make it executable via legalizer.

@masahi that would be awesome! @sunggg also proposed a direction here. Would be great to get alignment on the approach.

UPDATE: just found that the proposal talks about relax.layout_transform op). We also need to make it executable via legalizer.

Yes, I have a pending PR locally that adds the layout_transform op. It does not add legalization support though. Let me send it out today.
Note: Relax already has ops like permute_dim and reshape ops, which could be used to handle layout transformation without any implicit padding and cropping.


Refreshing file while syncing creates extra autocmd

When you enter a buffer b.rsync_init is checked if null and only then is continued. But when starting a sync rsync_init is set to nil which can trigger creating an extra autocmd for the buffer if :e is called while syncing.
Actually while syncing a per project variable should be set instead of the current per buffer approach.

Related to #6

This should not be possible but it would be nice to have per project sync status and not per buffer.


4K Video Playback Sometimes Fails

As stated, 4K playback sometimes fails seemingly only for LTT-specific videos. TQ, TL, MA, and other channels don't seem to be affected.
As has been mentioned in Discord, Floatplane's own Android app uses a combination of React/JS and ExoPlayer, the latter of which we also use for AndroidTV.
If anyone has any thoughts/suggestions, I'm all ears (or eyes, rather). I'm at a loss as of right now.

If when you say "fail" you mean simply doesn't start with no error then I can confirm, I have that issue on an Nvidia shield. Clicking a video does nothing at all.

We're already aware of the issue related to 4K on some particular devices. The reason the playback is failing is because the devices fail to decode the video files because we're out of the specs they support. This is because of the resolution we output for LTT videos and their weird video ratio. We'll likely change the transcoding resolution for new 4k videos soon. This is not something that can actually be fixed by the client, other than have the client fallback to a lower rendition when the player gets a decoding failure.

That's good to hear. I personally stopped my subscription since casting seems to cap out at 1080p and this client wasn't working for 4k, I just ended right back on youtube. Keeping an eye on this since I'd happily subscribe again if 4k started working though.

Ah, I see. It's 4320 x 2160, not 3840 x 2160, thus exceeding the TV's max resolution! Thanks for the insight!

yeah some videos have weird resolutions/aspect and they are encoded at higher profile levels than what some standalone hardware devices can decode

Hey, just to say that newer videos should be properly boxed to a resolution compatible with level 5.1 or 4.0, which should make playback better overall. We might re-transcode older videos to fix their resolution, but for now, only newer LTT 4K videos will be fixed.

Thanks Ajax!

Given the update, going to close this.


Configurable summary threshold

The summary for entry targets shows any pairs where the spread is positive. It would be nice to have it so it would display pairs where the spread is within a difference of the target. For example, if the target is 0.008 you could configure 0.005 and it would show anything 0.003 or higher.
(8 - 5 = 3)
Also the summary output doesn’t need to show the target on every line since it’s the same.

I have some other alternatives in mind for how to present this information. Strongly considering setting up a web UI with authentication and websockets to give a secure, realtime overview of what the bot is doing and what the markets look like to it. It'd be a big project and a lot more work but:

Conveys more information in a more compact, efficient way
Easier to access remotely via browser or mobile
Would be a good place for remote controls (eg. exit positions, shutdown/restart, auto-balance accounts, runtime configuration changes...)


Fixed #28670 -- Added FETCH/OFFSET support on Oracle.

https://code.djangoproject.com/ticket/28670

buildbot, test on oracle.

buildbot, test on oracle.

buildbot, test on oracle.

queries.tests.SubqueryTests.test_distinct_ordered_sliced_subquery should failed, because SQL is invalid on Oracle. I prepared separate PR with fix #9207.

buildbot, test on oracle.

@timgraham Thanks for review.


codemirror

Use CodeMirror/next for the sandbox to assist Json editing with errors and predictions.

Giving up in favour of a LSP driven approach, using the Monaco Editor.


jetbrains: verify download urls

Description of changes
For MPS, JetBrains has changed its download URL scheme:
(The format in nixpkgs)
https://download.jetbrains.com/mps/2023.3.1/MPS-2023.3.1.tar.gz

(The format on JetBrains' website)
https://download.jetbrains.com/mps/2023.3/MPS-2023.3.tar.gz

This PR improves the update script to verify the URLs before committing them. It also supports the old URL scheme.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Just out of curiosity: has anyone reached out to jetbrains to check if this change was intentional or an accident? Because if this is intentional we might want to prepare for it to happen for other products too, if not they might fix the urls on their side?

Two things are worth mentioning: a) They have changed the URLs for all MPS versions, not just for the most recent version. b) The schemes for other products are different from that for MPS. In fact, versionMajorMinor is solely used for MPS in the templates right now.
Besides, this change to the script is generic enough to support both the old and new schemes and does not restrict it to MPS.

@zlepper I had a misunderstanding. It was not because they changed the scheme but rather because this is a new version of MPS with three numbers in it (2023.3.1). It is amazing how this has never happened in the past to trigger an error in the update script.

I have updated the description.

Result of nixpkgs-review pr 320186 run on x86_64-linux 1

2 packages built:

jetbrains.dataspell
jetbrains.rust-rover

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1757

@leona-ya Could you please also take a look at this PR. It fixes the URL issue in #319290.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1832

Please notify me when someone is ready to merge this. I will resolve the conflicts only then in case of there are further needs to resolve future conflicts.

I would merge this, can you rebase please? Sorry for the delay

I would merge this, can you rebase please? Sorry for the delay

Done. Please take a look. Thanks.

Can you remove the merge commit, and rebase your branch on master  instead? We don't use merge commits within PRs here

OK. I will create a new PR to include only the changes to update_bin.py.


Add Git push --set-upstream

Additional option was added to the push wrapper.

@perlun @tarcinil Could you please take a look at this PR, and possibly re-open / merge it?
It would be very useful to do the equivalent of:
git push -u origin my_branch

@LoicAG - Is it possible that you can add a test that would test to ensure that the upstream is set?
Lightly based on https://github.com/ruby-git/ruby-git/blob/master/tests/units/test_remotes.rb#L18

Closing due to no test.  Happy to accept if a test is added.

Are there any update of this pr?


Free video asks for login credentials

Checklist

[X] I'm reporting a new site support request
[X] I've verified that I'm running yt-dlp version 2022.06.29 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that none of provided URLs violate any copyrights or contain any DRM to the best of my knowledge
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and am willing to share it if required

Region
Global
Example URLs
https://www.pokergo.com/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48
Provide a description that is worded well enough to be understood
Video is available to public without any login. But yt-dlp asks for login credentials (still doesn't work even if you login)
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
C:\>yt-dlp https://www.pokergo.com/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48 -vU
[debug] Command-line config: ['https://www.pokergo.com/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.29 [9d339c4] (win32_exe)
[debug] Python 3.7.9 (CPython 32bit) - Windows-10-10.0.19041-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 2022-06-16-git-5242ede48d-full_build-www.gyan.dev (setts), ffprobe 2022-06-16-git-5242ede48d-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.06.29/_update_spec
Latest version: 2022.06.29, Current version: 2022.06.29
yt-dlp is up to date (2022.06.29)
ERROR: [PokerGo] ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48: This video is only available for registered users. Use --username and --password, or --netrc (pokergo) to provide account credentials
  File "yt_dlp\extractor\common.py", line 638, in extract
  File "yt_dlp\extractor\common.py", line 547, in initialize
  File "yt_dlp\extractor\pokergo.py", line 28, in _real_initialize
  File "yt_dlp\extractor\common.py", line 1106, in raise_login_required

the website requires login (with or without subscriptions),
to show anything, even free videos.

if you've got your login info,
try run it with --username and --password for authentication,
and probably with --cookies as well,
for persistent cookies across navigation...

pokergo.com/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48

This URL is opening for me without login

C:>yt-dlp https://www.pokergo.com/collections/dea7a3be-d673-47bf-997f-75b7b61bfac5 -vU

But this does not

Sorry for mixing up URLs in the log. Fixed.
Yes, this link is free and open, no login is required, it is not invalid.

Another free video link (no account needed, video can be viewed anonymously)
Without login:
C:\>yt-dlp https://www.pokergo.com/videos/040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd -F --cookies-from-browser Chrome -vU
[debug] Command-line config: ['https://www.pokergo.com/videos/040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd', '-F', '--cookies-from-browser', 'Chrome', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.29 [9d339c4] (win32_exe)
[debug] Python 3.7.9 (CPython 32bit) - Windows-10-10.0.19041-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 2022-06-16-git-5242ede48d-full_build-www.gyan.dev (setts), ffprobe 2022-06-16-git-5242ede48d-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[Cookies] Extracting cookies from chrome
[debug] Extracting cookies from: "C:\Users\user\AppData\Local\Google\Chrome\User Data\Default\Network\Cookies"
[debug] Found local state file at "C:\Users\user\AppData\Local\Google\Chrome\User Data\Local State"
[Cookies] Extracted 104 cookies from chrome
[debug] cookie version breakdown: {'v10': 104, 'other': 0, 'unencrypted': 0}
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.06.29/_update_spec
Latest version: 2022.06.29, Current version: 2022.06.29
yt-dlp is up to date (2022.06.29)
ERROR: [PokerGo] 040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd: This video is only available for registered users. Use --username and --password, or --netrc (pokergo) to provide account credentials
  File "yt_dlp\extractor\common.py", line 638, in extract
  File "yt_dlp\extractor\common.py", line 547, in initialize
  File "yt_dlp\extractor\pokergo.py", line 28, in _real_initialize
  File "yt_dlp\extractor\common.py", line 1106, in raise_login_required

With login:
C:\>yt-dlp https://www.pokergo.com/videos/040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd -F --cookies-from-browser Chrome --username [edited] --password [edited] -vU
[debug] Command-line config: ['https://www.pokergo.com/videos/040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd', '-F', '--cookies-from-browser', 'Chrome', '--username', 'PRIVATE', '--password', 'PRIVATE', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.29 [9d339c4] (win32_exe)
[debug] Python 3.7.9 (CPython 32bit) - Windows-10-10.0.19041-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 2022-06-16-git-5242ede48d-full_build-www.gyan.dev (setts), ffprobe 2022-06-16-git-5242ede48d-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[Cookies] Extracting cookies from chrome
[debug] Extracting cookies from: "C:\Users\user\AppData\Local\Google\Chrome\User Data\Default\Network\Cookies"
[debug] Found local state file at "C:\Users\user\AppData\Local\Google\Chrome\User Data\Local State"
[Cookies] Extracted 104 cookies from chrome
[debug] cookie version breakdown: {'v10': 104, 'other': 0, 'unencrypted': 0}
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.06.29/_update_spec
Latest version: 2022.06.29, Current version: 2022.06.29
yt-dlp is up to date (2022.06.29)
[PokerGo] Logging in
[PokerGo] Downloading JSON metadata
[debug] [PokerGo] Extracting URL: https://www.pokergo.com/videos/040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd
[PokerGo] 040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd: Downloading JSON metadata
ERROR: 040b0661-fbe8-4bd8-82bf-9cc8aab0d9dd: An extractor error has occurred. (caused by StopIteration()); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 640, in extract
  File "yt_dlp\extractor\pokergo.py", line 67, in _real_extract
StopIteration

fetch
get source
your playlist is https://cdn.jwplayer.com/manifests/XXXXXXXX.m3u8
ffmpeg -i "https://cdn.jwplayer.com/manifests/XXXXXXXX.m3u8" -c copy out.mkv
no authentications no nothing.

several ways:
https://api.pokergo.com/v2/properties/1dfb3940-7d53-4980-b0b0-f28b369a000d/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48
{
  "data": {
    "id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
    , "property_id": "1dfb3940-7d53-4980-b0b0-f28b369a000d"
    , "title": "High Stakes Poker Free Episode "
    , "description": "Enjoy this free High Stakes Poker episode from Season 2. PokerGO has every High Stakes Poker episode featuring your favorite players including Daniel Negreanu, Phil Ivey, Tom Dwan, Doyle Brunson, Phil Laak, Antonio Esfandiari, and Phil Hellmuth."
    , "aired_at": "2022-02-22T17:00:00Z"
    , "published_at": "2022-02-22T17:00:00Z"
    , "created_at": "2022-02-22T22:03:52.52955Z"
    , "updated_at": "2022-07-02T13:40:10.277972Z"
    , "deleted_at": null
    , "source": "N9SJUtDX"
    , "entitlements_rule": "none"
    , "label": "high-stakes-poker-free-episode"
    , "ended_at": "0001-01-01T00:00:00Z"
    , "type": "vod"
    , "subheading": " "
    , "trailer_source": ""
    , "duration": 2616.3
    , "live_end_point": ""
    , "live_media_id": ""
    , "show_tags": [{
      "id": "48b866b6-f896-4073-a434-3883111d79b0"
      , "video_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
      , "show_type": "Episode"
      , "season": 1
      , "year": 2022
      , "tag_id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
      , "tag": {
        "id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
        , "name": "free experience"
        , "tag_type": "show"
      }
      , "video": null
      , "episode_number": 2
    }]
    , "other_tags": [{
      "id": "48b866b6-f896-4073-a434-3883111d79b0"
      , "video_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
      , "show_type": "Episode"
      , "season": 1
      , "year": 2022
      , "tag_id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
      , "tag": {
        "id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
        , "name": "free experience"
        , "tag_type": "show"
      }
      , "video": null
      , "episode_number": 2
    }]
    , "video_entitlement": []
    , "video_blocked_location": []
    , "video_source": []
    , "subscriber_video_progress": {
      "video_id": ""
      , "subscriber_id": ""
      , "progress": 0
      , "last_watched": "0001-01-01T00:00:00Z"
    }
    , "search_exclusions": {
      "id": "503eefb8-3c44-4d97-9fea-a2798765c6dd"
      , "entity_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
    }
    , "images": [{
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-SD.jpg"
      , "label": "SD"
      , "mime_type": "image/jpeg"
      , "width": 720
      , "height": 576
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-HD.jpg"
      , "label": "HD"
      , "mime_type": "image/jpeg"
      , "width": 1280
      , "height": 720
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-FHD.jpg"
      , "label": "FHD"
      , "mime_type": "image/jpeg"
      , "width": 1920
      , "height": 1080
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-UHD.jpg"
      , "label": "UHD"
      , "mime_type": "image/jpeg"
      , "width": 3840
      , "height": 2160
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-4K.jpg"
      , "label": "4K"
      , "mime_type": "image/jpeg"
      , "width": 4096
      , "height": 2160
    }]
    , "pre_air_images": [{
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-SD.jpg"
      , "label": "SD"
      , "mime_type": "image/jpeg"
      , "width": 720
      , "height": 576
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-HD.jpg"
      , "label": "HD"
      , "mime_type": "image/jpeg"
      , "width": 1280
      , "height": 720
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-FHD.jpg"
      , "label": "FHD"
      , "mime_type": "image/jpeg"
      , "width": 1920
      , "height": 1080
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-UHD.jpg"
      , "label": "UHD"
      , "mime_type": "image/jpeg"
      , "width": 3840
      , "height": 2160
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-4K.jpg"
      , "label": "4K"
      , "mime_type": "image/jpeg"
      , "width": 4096
      , "height": 2160
    }]
    , "entitled": true
    , "favorite": false
  }
}

get .source » N9SJUtDX

https://cdn.jwplayer.com/v2/media/N9SJUtDX
{
  "title": "Pg-free-experience-ep03-hsp-s02-ep11"
  , "description": ""
  , "kind": "Single Item"
  , "playlist": [{
    "title": "Pg-free-experience-ep03-hsp-s02-ep11"
    , "mediaid": "N9SJUtDX"
    , "link": "https://cdn.jwplayer.com/previews/N9SJUtDX"
    , "image": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=720"
    , "images": [{
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=320"
      , "width": 320
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=480"
      , "width": 480
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=640"
      , "width": 640
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=720"
      , "width": 720
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=1280"
      , "width": 1280
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=1920"
      , "width": 1920
      , "type": "image/jpeg"
    }]
    , "duration": 2616
    , "pubdate":<PHONE_NUMBER>
    , "description": ""
    , "sources": [{
      "file": "https://cdn.jwplayer.com/manifests/N9SJUtDX.m3u8"
      , "type": "application/vnd.apple.mpegurl"
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-sc8uFhWQ.mp4"
      , "type": "video/mp4"
      , "height": 180
      , "width": 320
      , "label": "180p"
      , "bitrate": 296941
      , "filesize": 97099930
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-OZA8pCe1.mp4"
      , "type": "video/mp4"
      , "height": 270
      , "width": 480
      , "label": "270p"
      , "bitrate": 444146
      , "filesize": 145235840
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-Yst1vCOp.mp4"
      , "type": "video/mp4"
      , "height": 720
      , "width": 1280
      , "label": "720p"
      , "bitrate": 1268193
      , "filesize": 414699112
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-XM7Nnf2V.m4a"
      , "type": "audio/mp4"
      , "label": "AAC Audio"
      , "bitrate": 113394
      , "filesize": 37080080
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-TZwR2iuw.mp4"
      , "type": "video/mp4"
      , "height": 1080
      , "width": 1920
      , "label": "1080p"
      , "bitrate": 2613775
      , "filesize": 854704717
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-8gR6FZg3.mp4"
      , "type": "video/mp4"
      , "height": 360
      , "width": 640
      , "label": "360p"
      , "bitrate": 491063
      , "filesize": 160577846
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-9cqarGOI.mp4"
      , "type": "video/mp4"
      , "height": 540
      , "width": 960
      , "label": "540p"
      , "bitrate": 826248
      , "filesize": 270183387
      , "framerate": 30.0
    }]
    , "tracks": [{
      "file": "https://cdn.jwplayer.com/strips/N9SJUtDX-120.vtt"
      , "kind": "thumbnails"
    }]
    , "variations": {}
  }]
  , "feed_instance_id": "9cf5c876-31bb-4c32-88ca-f1887282bad6"
}

@eladkarako Try not to delete comments if you can

sorry that's an error in script running on my pc that uses github api, also deleted half of my repositories. I'll add it back.

no javascript needed, you can do it all with string manipulation and regular-expression matching.

view-source:https://www.pokergo.com/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48

.......

https://www.pokergo.com/js/app.c95e0058.js

.....
.....
},
    progressVideos: null,
    videoId: null,

    property_id: "1dfb3940-7d53-4980-b0b0-f28b369a000d",

    searchKeyword: ''
  },
  mutations: {
    setLoginText: function setLoginText(state, value) {
      state.loginPageText = value;
    },
....
....

https://api.pokergo.com/v2/properties/ + property_id + /videos/ + URL's last pathname (split by /, pop last one)

https://api.pokergo.com/v2/properties/1dfb3940-7d53-4980-b0b0-f28b369a000d/videos/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48
(note:
can not re-open in tab
{"errors":[{"code":401,"title":"Unauthorized","detail":"auth token incorrect or was't supplied"}]}
xhr + cookie + mock browser (referer/useragent), csrf??
)

    , "source": "N9SJUtDX"

full json, various meta data:
{
  "data": {
    "id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
    , "property_id": "1dfb3940-7d53-4980-b0b0-f28b369a000d"
    , "title": "High Stakes Poker Free Episode "
    , "description": "Enjoy this free High Stakes Poker episode from Season 2. PokerGO has every High Stakes Poker episode featuring your favorite players including Daniel Negreanu, Phil Ivey, Tom Dwan, Doyle Brunson, Phil Laak, Antonio Esfandiari, and Phil Hellmuth."
    , "aired_at": "2022-02-22T17:00:00Z"
    , "published_at": "2022-02-22T17:00:00Z"
    , "created_at": "2022-02-22T22:03:52.52955Z"
    , "updated_at": "2022-07-07T20:00:15.378086Z"
    , "deleted_at": null
    , "source": "N9SJUtDX"
    , "entitlements_rule": "none"
    , "label": "high-stakes-poker-free-episode"
    , "ended_at": "0001-01-01T00:00:00Z"
    , "type": "vod"
    , "subheading": " "
    , "trailer_source": ""
    , "duration": 2616.3
    , "live_end_point": ""
    , "live_media_id": ""
    , "show_tags": [{
      "id": "48b866b6-f896-4073-a434-3883111d79b0"
      , "video_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
      , "show_type": "Episode"
      , "season": 1
      , "year": 2022
      , "tag_id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
      , "tag": {
        "id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
        , "name": "free experience"
        , "tag_type": "show"
      }
      , "video": null
      , "episode_number": 2
    }]
    , "other_tags": [{
      "id": "48b866b6-f896-4073-a434-3883111d79b0"
      , "video_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
      , "show_type": "Episode"
      , "season": 1
      , "year": 2022
      , "tag_id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
      , "tag": {
        "id": "a63f7c67-2248-47ce-8b96-b8afb6a51a14"
        , "name": "free experience"
        , "tag_type": "show"
      }
      , "video": null
      , "episode_number": 2
    }]
    , "video_entitlement": []
    , "video_blocked_location": []
    , "video_source": []
    , "subscriber_video_progress": {
      "video_id": ""
      , "subscriber_id": ""
      , "progress": 0
      , "last_watched": "0001-01-01T00:00:00Z"
    }
    , "search_exclusions": {
      "id": "503eefb8-3c44-4d97-9fea-a2798765c6dd"
      , "entity_id": "ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48"
    }
    , "images": [{
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-SD.jpg"
      , "label": "SD"
      , "mime_type": "image/jpeg"
      , "width": 720
      , "height": 576
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-HD.jpg"
      , "label": "HD"
      , "mime_type": "image/jpeg"
      , "width": 1280
      , "height": 720
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-FHD.jpg"
      , "label": "FHD"
      , "mime_type": "image/jpeg"
      , "width": 1920
      , "height": 1080
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-UHD.jpg"
      , "label": "UHD"
      , "mime_type": "image/jpeg"
      , "width": 3840
      , "height": 2160
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48-4K.jpg"
      , "label": "4K"
      , "mime_type": "image/jpeg"
      , "width": 4096
      , "height": 2160
    }]
    , "pre_air_images": [{
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-FHD.jpg"
      , "label": "FHD"
      , "mime_type": "image/jpeg"
      , "width": 1920
      , "height": 1080
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-UHD.jpg"
      , "label": "UHD"
      , "mime_type": "image/jpeg"
      , "width": 3840
      , "height": 2160
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-4K.jpg"
      , "label": "4K"
      , "mime_type": "image/jpeg"
      , "width": 4096
      , "height": 2160
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-SD.jpg"
      , "label": "SD"
      , "mime_type": "image/jpeg"
      , "width": 720
      , "height": 576
    }, {
      "url": "https://storage.googleapis.com/gxm-video-platform-images/1dfb3940-7d53-4980-b0b0-f28b369a000d/ebb1e54f-a6d4-4a5c-850d-cdb78b93ad48__pre_air-HD.jpg"
      , "label": "HD"
      , "mime_type": "image/jpeg"
      , "width": 1280
      , "height": 720
    }]
    , "entitled": true
    , "favorite": false
  }
}

https://cdn.jwplayer.com/v2/media/ + source

https://cdn.jwplayer.com/v2/media/N9SJUtDX
various download links,
poster (for thumbnail), MP4 (various resolutions), m3u (.ts - mpeg), your choice...
...also subtitles! (look for VTT below).
{
  "title": "Pg-free-experience-ep03-hsp-s02-ep11"
  , "description": ""
  , "kind": "Single Item"
  , "playlist": [{
    "title": "Pg-free-experience-ep03-hsp-s02-ep11"
    , "mediaid": "N9SJUtDX"
    , "link": "https://cdn.jwplayer.com/previews/N9SJUtDX"
    , "image": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=720"
    , "images": [{
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=320"
      , "width": 320
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=480"
      , "width": 480
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=640"
      , "width": 640
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=720"
      , "width": 720
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=1280"
      , "width": 1280
      , "type": "image/jpeg"
    }, {
      "src": "https://cdn.jwplayer.com/v2/media/N9SJUtDX/poster.jpg?width=1920"
      , "width": 1920
      , "type": "image/jpeg"
    }]
    , "duration": 2616
    , "pubdate":<PHONE_NUMBER>
    , "description": ""
    , "sources": [{
      "file": "https://cdn.jwplayer.com/manifests/N9SJUtDX.m3u8"
      , "type": "application/vnd.apple.mpegurl"
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-sc8uFhWQ.mp4"
      , "type": "video/mp4"
      , "height": 180
      , "width": 320
      , "label": "180p"
      , "bitrate": 296941
      , "filesize": 97099930
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-OZA8pCe1.mp4"
      , "type": "video/mp4"
      , "height": 270
      , "width": 480
      , "label": "270p"
      , "bitrate": 444146
      , "filesize": 145235840
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-Yst1vCOp.mp4"
      , "type": "video/mp4"
      , "height": 720
      , "width": 1280
      , "label": "720p"
      , "bitrate": 1268193
      , "filesize": 414699112
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-XM7Nnf2V.m4a"
      , "type": "audio/mp4"
      , "label": "AAC Audio"
      , "bitrate": 113394
      , "filesize": 37080080
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-TZwR2iuw.mp4"
      , "type": "video/mp4"
      , "height": 1080
      , "width": 1920
      , "label": "1080p"
      , "bitrate": 2613775
      , "filesize": 854704717
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-8gR6FZg3.mp4"
      , "type": "video/mp4"
      , "height": 360
      , "width": 640
      , "label": "360p"
      , "bitrate": 491063
      , "filesize": 160577846
      , "framerate": 30.0
    }, {
      "file": "https://cdn.jwplayer.com/videos/N9SJUtDX-9cqarGOI.mp4"
      , "type": "video/mp4"
      , "height": 540
      , "width": 960
      , "label": "540p"
      , "bitrate": 826248
      , "filesize": 270183387
      , "framerate": 30.0
    }]
    , "tracks": [{
      "file": "https://cdn.jwplayer.com/strips/N9SJUtDX-120.vtt"
      , "kind": "thumbnails"
    }]
    , "variations": {}
  }]
  , "feed_instance_id": "b8587c0a-ab43-4cfc-8bca-cd8a48d33a22"
}

you can skip most of those stuff if you just want the video and you've found the 'source':
https://cdn.jwplayer.com/manifests/ + source + .m3u8

https://cdn.jwplayer.com/manifests/N9SJUtDX.m3u8
you don't need anything complex other than just the link..
ffmpeg -i "https://cdn.jwplayer.com/manifests/N9SJUtDX.m3u8" -c copy out.mkv

sorry, no fancy screenshots this time..


[Feature]: Auto Commenting feature for PR Merged

Description
As current workflow files only autocomments on issues created and PR raised,
This feature aims to solve the problem of inconsistent and delayed communication following the merging of PRs. By automating the commenting process, it ensures that contributors receive timely feedback and acknowledgments. It also reduces the administrative burden on maintainers, allowing them to focus more on code reviews and other critical tasks.
Screenshots

Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (optional)

Congratulations, @Asymtode712! 🎉 Thank you for creating your issue. Your contribution is greatly appreciated and we look forward to working with you to resolve the issue. Keep up the great work!We will promptly review your changes and offer feedback. Keep up the excellent work! Kindly remember to check our contributing guidelines

@pooranjoyb please assign me this issue under gssoc'24

Nice one, we can go ahead with this! @Asymtode712


Receiving 6 notification for each alert

Been working on Estalert for the past few days and all seems set but receive 6 notification for each alert (in Slack).
When I run the elastalert-test-rule its shows 1 alert. have the below in the rule (only 1 rule file). Can ome one please tell me what could the problem be.
name: billing-alert
es_host: #host
es_port: #post
index: web-
type: any
query_key: msg
realert: {minutes: 0}
filter:

query:
query_string: {query: 'msg: "SUCCESS Trial signup" OR msg: ERROR OR msg: "Failed login" OR msg: "Payment received" OR msg: "New Ticket"'}

alert: slack
slack_webhook_url: #url
slack_msg_color: "good"

Can you show the output of elastalert-test-rule? You are sure that a single document caused 6 different alerts?

Please see the result below. Yes whenever an event occurs, six messages are received in Slack, same copy of the message.
The below rule was created with a single query, but it still triggered 6 slack notification
====rules/payment.yaml====
name: payment-alert
es_host: XXX
es_port: 80
index: web-
type: any
query_key: msg
realert: {minutes: 0}
filter:

query:
query_string: {query: 'msg: "Payment received"'}

alert: slack
slack_webhook_url: XXX
slack_msg_color: "good"
=================
elastalert-test-rule  rules/payment.yaml
Successfully loaded payment-alert
INFO:urllib3.connectionpool:Starting new HTTP connection (1): XXX
Got 5 hits from the last 1 day
Available terms in first hit:
username
description
@log_level
app
detail.city
detail.firstname
detail.companyname
detail.lastname
detail.lastlogin
detail.country
@timestamp
msg
amount
tag
invoice
fees
action
transid
gateway
INFO:urllib3.connectionpool:Starting new HTTP connection (1): XXX
INFO:elastalert:Queried rule payment-alert from 2016-02-01 19:26 IST to 2016-02-01 20:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 20:11 IST to 2016-02-01 20:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 20:56 IST to 2016-02-01 21:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 21:41 IST to 2016-02-01 22:26 IST: 1 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 22:26 IST to 2016-02-01 23:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 23:11 IST to 2016-02-01 23:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-01 23:56 IST to 2016-02-02 00:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 00:41 IST to 2016-02-02 01:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 01:26 IST to 2016-02-02 02:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 02:11 IST to 2016-02-02 02:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 02:56 IST to 2016-02-02 03:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 03:41 IST to 2016-02-02 04:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 04:26 IST to 2016-02-02 05:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 05:11 IST to 2016-02-02 05:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 05:56 IST to 2016-02-02 06:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 06:41 IST to 2016-02-02 07:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 07:26 IST to 2016-02-02 08:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 08:11 IST to 2016-02-02 08:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 08:56 IST to 2016-02-02 09:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 09:41 IST to 2016-02-02 10:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 10:26 IST to 2016-02-02 11:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 11:11 IST to 2016-02-02 11:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 11:56 IST to 2016-02-02 12:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 12:41 IST to 2016-02-02 13:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 13:26 IST to 2016-02-02 14:11 IST: 2 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 14:11 IST to 2016-02-02 14:56 IST: 2 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 14:56 IST to 2016-02-02 15:41 IST: 1 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 15:41 IST to 2016-02-02 16:26 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 16:26 IST to 2016-02-02 17:11 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 17:11 IST to 2016-02-02 17:56 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 17:56 IST to 2016-02-02 18:41 IST: 0 hits
INFO:elastalert:Queried rule payment-alert from 2016-02-02 18:41 IST to 2016-02-02 19:26 IST: 0 hits
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-01T21:58:51+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-01T21:58:51+05:30
_id: AVKdqykYS2YWIbB8W7Gd
_index: web-
_type: mmin
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 0
gateway: XXX
invoice: 13694
msg: Payment received
tag: XXX
transid:<PHONE_NUMBER>23
username: XXX
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-02T13:40:42+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-02T13:40:42+05:30
_id: AVKhCKyyU4A2KYJD56Ir
_index: web-
_type: mmin
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 0
gateway: XXX
invoice: 13700
msg: Payment received
tag: XXX
transid:<PHONE_NUMBER>64
username: XXX
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-02T13:55:16+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-02T13:55:16+05:30
_id: AVKhFmkTS2YWIbB8Xkvk
_index: web-
_type: mmin
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 0
gateway: XXX
invoice: 13702
msg: Payment received
tag: XXX
transid:<PHONE_NUMBER>65
username: XXX
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-02T14:29:48+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-02T14:29:48+05:30
_id: AVKhNnY3U4A2KYJD585C
_index: web-
_type: mmin
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 0
gateway: XXX
invoice: 13703
msg: Payment received
tag: XXX
transid:<PHONE_NUMBER>82
username: XXX
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-02T14:51:30+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-02T14:51:30+05:30
_id: AVKhSbENS2YWIbB8XoRU
_index: web-
_type: XXX
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 4.41
gateway: XXX
invoice: 2231
msg: Payment received
tag: XXX
transid: 9YS15023K14626149
username: XXX
INFO:elastalert:Alert for payment-alert, Payment received at 2016-02-02T15:37:46+05:30:
INFO:elastalert:payment-alert
@log_level: info
@timestamp: 2016-02-02T15:37:46+05:30
_id: AVKhc9C6U4A2KYJD6BHH
_index: web-
_type: mmin
action: payment
amount: XXX
app: XXX
description: Invoice Payment
fees: 0
gateway: XXX
invoice: 13704
msg: Payment received
tag: XXX
transid:<PHONE_NUMBER>28
username: XXX
Would have written the following documents to elastalert_status:
elastalert_status - {'hits': 6, 'matches': 6, '@timestamp': datetime.datetime(2016, 2, 2, 13, 56, 40, 117344, tzinfo=tzutc()), 'rule_name': 'payment-alert', 'starttime': datetime.datetime(2016, 2, 1, 13, 56, 39, 898498, tzinfo=tzutc()), 'endtime': datetime.datetime(2016, 2, 2, 13, 56, 39, 898498, tzinfo=tzutc()), 'time_taken': 0.21692991256713867}

The issue of multiple alerts is gone now.
Seems multiple instance of supervisor/ elastalert scripts (while setting up) were running in the background so they were initiating multiple notifications. Rebooted the machine and started elastastart via supervisord and now receiving single alert as expected.

Is there a config sample of supervisord for elastalert? Have been trying various configs and init scripts but elastalert would not start with init script.
/etc/supervisord.conf
[program:elastalert]
command=elastalert (have also tried python -m elastalert.elastalert)
process_name=elastalert
numprocs=1
user=root
autostart=true
autorestart=false (also tried true)
startsecs=0 (also tried 20)
startretries=3
exitcodes=0,2
umask=022
2016-02-02 20:50:40,046 INFO spawned: 'elastalert' with pid 5621
2016-02-02 20:50:40,235 INFO success: elastalert entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2016-02-02 20:50:40,247 INFO exited: elastalert (exit status 1; not expected)

Ditched supervisrd and got elastalert work with upstart easily. Hope upstart meets the purpose of supervisrd.

It's probably writing the error somewhere. You could try adding
stderr_logfile=/var/log/elastalert_err.log

It might be because ElastAlert cannot find it's config.yaml if you are calling supervisor from the wrong directory. That's good it works with upstart though.

Thanks for the information, yes seems that was what happening and without the log file could not find the issue.
Even with upstart got the same error of missing config.yaml.
Anyways after struggling for 2 days with supervisord settled down with upstart.
Thanks

You can pass --config /path/to/config.yaml, otherwise it will look in the current directory.


Added delay before reporting and enforced a minimum interval of 10s

Solves #6570
I tried to set min pingInterval to be 10s and introduced a delay before reporting the nodeLatencyStats.
seeking your guidance in resolving this issue and looking forward to your feedback on the proposed changes

@antoninbas tried to decouple by adding 2 tickers. But I didn't add jitters as of now.

@Yushmanth-reddy any update regarding this PR?

Sorry for the delay @antoninbas.  I've decided to close this pull request for now. I feel that I need to take some more time to better understand the project. Once I'm confident, I'll revisit and solve issues.
Thank you for your  time and feedback during the process.


Missing word in sentence

"You can keep development settings (for example, local MySQL password) in Web.config or appsettings.json, but production secrets (for example, Azure MySQL database password) safe in App Service." I think this means "ARE safe in App Service". I.e. "but production secrets are safe in App Service."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1e25899c-1ed8-7cfc-2c37-73bb010a87f0
Version Independent ID: 8b366c42-43f8-d5eb-b619-b356e75a9916
Content: Configure apps in the portal - Azure App Service
Content Source: articles/app-service/configure-common.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

Hi Sam, thanks for bringing this to our attention! We will fix this issue and update the doc shortly @samrueby .

Hi @samrueby, thanks for catching this error in the doc. The intended word we wanted to use was "safely" not "safe". I have submitted a PR to fix the issue, the change is merged internally and will go-live in the public docs shortly.


Uncaught NotFoundError in utils.py line 6432

Aider version: 0.58.2.dev7+g9789668c
Python version: 3.12.5
Platform: macOS-10.16-x86_64-i386-64bit
Python implementation: CPython
Virtual environment: No
OS: Darwin 24.0.0 (64bit)
Git version: git version 2.44.0
An uncaught exception occurred:
openai.PermissionDeniedError: Error code: 403 - {'error': {'message': 'Project `proj_9KoeXdtHe0YQJbsF2FQAKmzl` does not have access to model `gpt-4o-mini`', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

Thank you for filing this issue.
Looks like you did not activate in this model in Vertex AI Model Garden ->

Project proj_9KoeXdtHe0YQJbsF2FQAKmzl does not have access to model gpt-4o-mini


on startup, don't start translation until all resources have been loaded

xref #447
xref #413
xref https://github.com/telepresenceio/watchable/pull/6
On startup/restart, EG should not start translating/sending xDS traffic until all resources have been loaded, to avoid issues with incomplete inputs.

Note that this fix is needed to ensure lastTransitionTime is not updated when EG is restarted.

This issue should be resolved when #413 is fixed.


how to use user function with node package

Hello everyone, I faced similar problem like this one. https://github.com/arangodb/arangodb/issues/1092
but my problem is with moment library (I think it will occur with any node packages).
my registered function is
function date(date) { var moment = require(‘moment’); var dd = moment(date).fromNow(); return dd; }
and I have got this error:
user function runtime error: {“error”:true,“errorNum”:3100,“errorMessage”:“cannot locate module\nFile: moment”}
when I have used AQL like this from FOXX UI:
FOR doc IN api_v1_user  FILTER doc._key == “1804774”  RETURN APPLY(“DATE::RELATIVE”, [doc.userCreateDate])
I have installed moment with package.json. what am I missing?
my environment running ArangoDB
I'm using the latest ArangoDB of the respective release series:

[ ] 2.8
[ ] 3.0
[ ] 3.1
[ ] 3.2
[*] 3.3
[ ] self-compiled devel branch

Mode:

[ ] Cluster
[*] Single-Server

Storage-Engine:

[*] mmfiles
[ ] rocksdb

On this operating system:

[ ] DCOS on

[ ] AWS
[ ] Azure
[ ] own infrastructure

[* ] Linux

[ ] Debian .deb
[ ] Ubuntu .deb
[ ] SUSE   .rpm
[ ] RedHat .rpm
[ ] Fedora .rpm
[ ] Gentoo
[ ] docker - official docker library
[ ] other:

[ ] Windows, version:
[ ] MacOS, version:

I'm issuing AQL via:

[* ] web interface with this browser:     running on this OS:
[ ] arangosh
[ ] this Driver:

Hi,
I've tried to require the moment library shipped with joi like this from arangosh:
<IP_ADDRESS>:8529@_system> moment = require("joi/node_modules/moment")
function utils_hooks__hooks() {
        return hookCallback.apply(null, arguments);
    }

<IP_ADDRESS>:8529@_system>

which should work similar from UDFs.

Hi @nymul-islam,
Did you have a chance to try the solution suggested by dothebart? Do you still have issues with moment library?
Best,

I am closing this issue for now as we have not received any feedback for over one month.
Please feel free to comment and reopen in case you have further questions.


Laravel 9.36.4 Shift

This pull request includes updates for the recent minor version release of Laravel as well as updates to your package dependencies. Feel free to commit any additional changes to the shift-ci-v9.36.4 branch.
Before merging, you need to:

Checkout the shift-ci-v9.36.4 branch
Review all pull request comments for additional changes
Run composer update
Thoroughly test your application (no tests?, no CI?)

:warning: The Shift build servers run PHP 8.1. If you are running a different version of PHP you should run composer update locally to ensure you have the proper versions of your dependencies to reflect your PHP environment.

:information_source: Shift detected your application only has the default tests. You may quickly start adding tests by using the Test Generator to automatically create the model factories, test classes, and sample tests cases for your Laravel application.


Set the OPENTOFU_RUNNER_URL ENV var

This is needed so the app knows the location of the opentofu-runner API

Backported to radjabov in commit 3f598319048e78b8cff2e9bda5a441f56de7c3bf.
commit 3f598319048e78b8cff2e9bda5a441f56de7c3bf
Author: Jason Frey<EMAIL_ADDRESS>Date:   Tue May 28 13:59:18 2024 -0400

    Merge pull request #385 from agrare/set_terraform_runner_url_env_var

    Set the OPENTOFU_RUNNER_URL ENV var

    (cherry picked from commit 6d21b5192ceaa0ad3d3dd06b2929ed47d52440e3)


Pipeline

Address issue #22
Upgrade scikit-learn to ~1.5

Upgrade rdflib library to ~7.0 to make it compatible with current annif  versions

Upgrade python major versions to support 3.12

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.54%. Comparing base (db92a63) to head (2948f35).

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #25   +/-   ##
=======================================
  Coverage   94.54%   94.54%
=======================================
  Files          23       23
  Lines         898      898
=======================================
  Hits          849      849
  Misses         49       49

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Complex custom formatter for x-axis

Is your feature request related to a problem? Please describe.
I'd like to display two series with data related to different periods of time. Example: google analytics shows 2 lines, comparing This week stats with Previous week stats.

For now I'll hide tooltips, it's ok. I understand such feature implies custom SVG elements, positioning etc and I don't expect to have a customSVGFormatter function, it's overkill.
Describe the solution you'd like
I'm thinking of 'categories' in series, not in xaxis. Example:
series: [     {         data: ....,         categories: ['March 1', 'March 2', 'March 3']      },     {         data: ....,         categories: ['March 5', 'March 6', 'March 7']      } ]
and in xaxis to specify which categories to display. The default x tooltip formatting will then handle the rest ;)

Easy!
Codepen example

The tooltip is drawn with HTML elements allowing very flexible customization

Wow!!! Great!
Thanks for the cool example.
I was reading about tooltip.x.formatter but didn't notice the Custom Tooltip chapter!
P.S. That's really powerful! I can add my own custom data to the chart config, and then later in Custom Tooltop function, with the w variable, I can use that data. Having series.categories is a good example.

hello,  thanks for this example. I took this example exercise to update the values by ajax but I don't know why if I update the chart data with chart.updateOptions it updates the chart fine but the tooltip shows the value of w.config.series[0].categories[dataPointIndex] like as undefined.
Does anyone know why this it happen?
I have a graph painted with 5 points, and then I update those values by ajax with chart.updateOptions, if the new graph has 10 values, the first 5 show me the data correctly and the others appear as undefined.
...
success: function( data ) {
...
chart.updateOptions({
            xaxis: {
              type: "datetime",
              categories: data.days,
            },
            tooltip: {
              custom: function({ series, seriesIndex, dataPointIndex, w }) {
                return (
                  '
' +
                  '
' +
                  "
Current
" +
                  w.config.series[0].categories[dataPointIndex] +
                  "
" +
                  '
Nº: ' +
                  series[0][dataPointIndex] +
                  "
" +
                  "
" +
                  '
' +
                  "
Previous
" +
                  w.config.series[1].categories[dataPointIndex] +
                  "
" +
                  '
Nº: ' +
                  series[1][dataPointIndex] +
                  "
" +
                  "
" +
                  "
"
                );
            }
          },
 })
...


Wink Server Name Change

Home Assistant release (hass --version):
0.23.1
Python release (python3 --version):
3.4
Component/platform:
Wink
Description of problem:
SSL Error. Wink changed name of API server. New server name is api.wink.com
Additional info:
Tracked this down in another group using the Wink API. The old server name is winkapi.quirky.com. Not surprisingly they've changed it to api.wink.com. Updating the server name solved the issue on the other project. I assume the same would be true here.

python-wink PR is opened, waiting on it to be pulled in.
https://github.com/python-wink/python-wink/pull/43

It appears that Wink renewed the cert for *.quirky.com.

It would be awesome if we could get this into the next release this Saturday.

I agreed, I have some other changes I would like to submit as well. Waiting on the latest python-wink to get pushed to pypi and we will be good to go. However in my last comment I stated Wink renewed the cert that was causing the issue, so this isn't an issue anymore.


[WIP] Fetch details for GQL queries payment from config file

Part of Fetch GQL queries payment config from rates file

Add nitro.payments config
Read rates config from filepath set in nitro.payments.ratesFile

Closed by https://github.com/cerc-io/ponder/pull/16


Add support for macros

Implementation for https://github.com/cerner/splunk-pickaxe/issues/17

This looks good, thanks!