Create an initcontainer for persistent home setup

What does this PR do?
The problem
Today, the persistent home setup occurs in the entrypoint [1] of the tooling container (assuming it is based on the UBI). Notably, the entrypoint runs stow [2] to make symbolic links for the home directory to be saved on the PVC.
If a poststart event depends on files or edits files located in the home directory, there is a race condition because the poststart event may start running before stow has finished running. Stow takes about 6 seconds to complete.
This race condition causes https://github.com/eclipse-che/che/issues/22914, because the checode poststart event runs before the entrypoint script completes.
Proposed solution
To solve the race condition altogether, the entrypoint script must be complete before the post start events run. To do this, an initcontainer that runs the entrypoint script can be used.
What is the container image for the initcontainer?
To have a minimal effect on workspace startup times, the container image for the initcontainer should be the same image as the tooling container's image to avoid pulling an extra image. Assuming that the image is already pulled, I've observed no significant startup impact (see demo video in the next section) when running an extra container running the tooling image, even if the tooling image is large (quay.io/devfile/universal-developer-image:ubi8-latest has a compressed size of 2.9GB).
When thinking about it in the Docker perspective, running a container is only creating a writable layer on top of the already-pulled image layers, therefore if the image is already pulled the image size would not have a significant impact on container startup. I'm guessing this is what also happens in OpenShift/CRI-O, because adding the initcontainer did not increase workspace startup time significantly.
What issues does this PR fix or reference?
https://github.com/eclipse-che/che/issues/22914
Demo: https://youtu.be/RQeKKJgaogM
In the demo above, the init container named init-persistent-home runs for than 9 seconds at the 0:20 second mark. Stow is only run once (because of this if statement [3]) to set up the new PVC.
So for subsequent workspace starts, the init-persistent-home init container takes less than a second to finish. Here is a second demo showing that:
https://youtu.be/hKnb1mgBSqA?si=yuJ3EKmj_dX0qHj0&t=20
Is it tested? How?

To test this PR with Eclipse Che:
Install DWO on your cluster by adding this catalog source and installing DWO from OperatorHub:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: devworkspace-operator-catalog
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/dkwon17/devworkspace-operator-index:init-container
  publisher: Red Hat
  displayName: DevWorkspace Operator Catalog
  updateStrategy:
    registryPoll:
      interval: 5m

Install Eclipse Che with chectl:
chectl server:deploy -p openshift --skip-devworkspace-operator

Enabled persistent user home by adding the following in the CheCluster CR:
spec:
  devEnvironments:
    persistUserHome:
      enabled: true

Create a workspace with the following repositories and confirm that the podman info command runs without errors after workspace startup (just like in the demo video):

https://github.com/che-samples/web-nodejs-sample
empty sample workspace
https://github.com/dkwon17/repo-with-poststart
https://gist.githubusercontent.com/dkwon17/9ecac5397bef1178f77f308565814996/raw/4801b3c07a2cababd53b3594754c628cde7b8947/child-devfile.yaml

PR Checklist

[ ] E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)

[ ] v8-devworkspace-operator-e2e: DevWorkspace e2e test
[ ] v8-che-happy-path: Happy path for verification integration with Che

Is it tested? How?

PR Checklist

[ ] E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)

[ ] v8-devworkspace-operator-e2e: DevWorkspace e2e test
[ ] v8-che-happy-path: Happy path for verification integration with Che

Some cases to think about in general:

User is using a parent devfile
User's main container image does not have the /entrypoint.sh script
User's main container image has ["tail", "-f", "/dev/null"] in it's entry point, which would cause the init container to never terminate

Amazing work so far David. I think this is a good proposed solution to fixing the race condition between the UDI's entrypoint and any post-start commands.

This is something you mentioned in your earlier draft. I think it makes the most sense for the init container component to be added in DWO:

We want to minimize the transformation of a devfile -> devworkspace, so that DWO's devfile bootstrap feature continues to work. If the Che-Dashboard was adding an extra component to the devworkspace for home persistence, this extra component would be lost when DWO restarts the devworkspace from the cloned devfile (as the bootstrap feature will use components from the devfile, not the devworkspace)
I'm not sure if the Che-Dashboard can currently "read" the Che Cluster CR in order to determine the home persistence settings? The dashboard would have to watch for changes in the Che Cluster CR or get re-deployed (with the configured home persistence settings) when the home persistence configuration changes.
There's already some level of coupling between the UDI and DWO for home persistence: mounting /home/user/ as a volume assumes that $HOME == /home/user/, which is the case for the UDI but not necessarily true for other user-provided images.

Since there's already coupling between DWO and the UDI, we should keep this the same rather than move the coupling to be between the UDI and Dashboard.

Should we call the main/init containers entrypoint? Or just the code relevant to home persistence (stow, cp)?
I don't think we want to call the UDI's entrypoint directly, as there's other code in it that is irrelevant to home persistence. Instead, we'll want to move the stow-related code into its own script and execute that.
This script could potentially reside in the UDI under a pre-determined name (e.g. /home-persistence-init.sh), or it could be injected by DWO as the initContainer's command. In the latter case, we could test whether stow is installed in the container before calling it.
Should we hope that the main/tooling component has stow installed? Or consider shipping a dedicated init container with stow installed.
Instead of depending on the first container component in the devworkspace being the UDI & having stow installed, we could ship a minimal image that has stow installed (similar to how we ship the project clone container).
This would simplify the inferInitContainer() step, but require us to productize a new container downstream that we'd be shipping.
For a first pass of your proposed change, I think it's fine to require users to use a main/tooling image based on the UDI (with stow installed, as well as tooling installed in /home/tooling/ and $HOME set to /home/user/). This is already the case for $HOME persistence to work as expected, anyhow.

Some cases to think about in general:
* User's main container image does not have the /entrypoint.sh script

* User's main container image has `["tail", "-f", "/dev/null"]` in it's entry point, which would cause the init container to never terminate

Both of these can be resolved by having DWO call stow and cp directly (by default), rather than requiring them to be in the entrypoint. We could then let users configure the init container command through the DWOC  if they don't want to use the default we provide.

Sorry, the PR still had rough work, I've still updated the PR recently with some of your feedback

@dkwon17 since we're adding new fields to the DWOC, you'll have to modify mergeConfig() and GetCurrentConfigString() in sync.go to account for the new fields.
Additionally, the CRDs will have to be re-generated by running the following and adding the resulting changes into a separate commit: make generate manifests fmt generate_default_deployment generate_olm_bundle_yaml.
Since we're still working through the details of this PR, these 2 steps can wait until we're set on the new DWOC fields.

@tolusha could you provide a reproducer? Does this happen when enabling persistUserHome and starting an existing workspace?

@dkwon17 your latest changes look good to me :) I still need to do some final manual testing on a cluster but I think this is just about ready to merge.
My 2 last thoughts on this PR:

We need to verify this doesn't break the Podman wrapper in the UDI (i.e. kubedock) since now Stow is being invoked before the podman wrapper is symlinked. However, as we briefly discussed on call, I don't think this is an issue since /home/tooling/ (where the Podman wrapper exists) is on PATH
It'd be nice to squash some of the commits from this PR together to create an easier-to-follow git log. My suggestion is that you could probably just have ~3 commits: one for the feature implementation, one for changing the CRDs/resources and one for adding tests. I know some commits (like this one) change both the implementation and test cases, so don't worry if these remain their own separate commits.

Thanks for the feedback, I was able to verify the kubedock functionality [1] in the UDI for this PR
[1] https://access.redhat.com/documentation/en-us/red_hat_openshift_dev_spaces/3.9/html/release_notes_and_known_issues/new-features#enhancement-crw-3367

Thanks for the feedback, I was able to verify the kubedock functionality [1] in the UDI for this PR
[1] access.redhat.com/documentation/en-us/red_hat_openshift_dev_spaces/3.9/html/release_notes_and_known_issues/new-features#enhancement-crw-3367

@dkwon17 awesome, thank you for verifying this. I'll do some final testing and have this merged.

Sounds good @AObuchow , do you want me to squash the commits and force push?

@dkwon17 yes please squash your commits then force push if you get a chance. I know you're going to be on PTO however, so I can do this for you.

@dkwon17 thanks for squashing your commits. Will do final review & have this merged by Wednesday morning :)

Note to myself: need to create a DWO issue before this is merged for tracking in the 0.28 release miletstones


Is there a way to check if a file exist before to load content?

Hi @kensoh
I wrote a little snippets to check if a entry exist in a file before to append new entries, but I also need before to check if the file exit.
How can I do that to complete the following code?
//email to check/append
<EMAIL_ADDRESS>
//path of file to create/use
path="/tmp/emails.csv";

//*******CHECK HERE IF FILE EXIST*********//
load '+path+' to database_emails
echo 'database_emails: '+database_emails

if (database_emails.indexOf(email)>-1)
{
echo 'email already exist in database'
}
else
{
echo 'email added in database'
write email to '+path+'
}

I didn't found it in tuto or closed issues...
Thanks

Hi @adegard following block of code can be used. As TagUI underlying JS engine is PhantomJS, the file system module that comes with PhantomJS can be used.
fs = require('fs');
target_file = '/tmp/emails.csv';

if (!fs.exists(target_file))
fs.touch(target_file);

Alternatively, you can make use of TagUI user-defined functions. By creating a file tagui_local.js and put in same folder as the script, or creating a file tagui_global.js and putting in tagui\src folder, users can define their own functions to be used. More on this here under helper functions.
contents of tagui_local.js or tagui_global.js
function touch(full_path) {
        fs = require('fs');
        if (!fs.exists(full_path))
        fs.touch(full_path);
}

to use touch() function in your tagui script
touch('/tmp/emails.csv');

Great @kensoh
It works perfectly! I use it as local database (to avoid repeat the same action to the same item)
Good also your user-defined functions creation system!

Also adding on, below egs evaluate to a boolean true or false that can be used in conditions directly -
file_exists = require('fs').exists('/tmp/test.txt')
...
echo require('fs').exists('/tmp/test.txt')


Tweak message spacing.

@keybase/react-hackers, this adjusts the message spacing ever so slightly. There were too many nested paddings/margins causing different kinds of messages to have different levels of spacing. This cleans that up.

For instance, take a look at a message that has reactions.


Database type and version properties in \yii\db\Conection?

In yii2-dynamic-ar we haven't found a clean way to discover a connection's database type and version. And I think anyone building an extension (or other reusable code) that depends on advanced DBMS features from more recent versions might face a similar problem.
For example, yii2-dynamic-ar can use Dynamic Columns if the DBMS is Maria 10.0+. It can use JSON of the DB is MySQL 5.7+. And it can use jsonb if it has PostgreSQL 9.4+. But it can't even see the difference between Maria and MySQL and it can't check versions at all.
Reading the version string from the database, parsing it and caching the result is possible but that's as appealing as branching on a parsed user agent string. (I don't like the idea, is what I'm saying.) I would prefer an application configuration that the developer is responsible for. But I don't know where it should go. \yii\db\Conection?
Any thoughts on this would be welcome.

Connection seems to be natural choice.

Note, that there is already \yii\db\Connection::driverName.


ipv6 ?

I don't have an IPv4 address, but I have a fixed IPv6 public address,
Does the node support IPv6?

Yes, this is supported.


No module named util (Django 1.11.2)

Hi,
I setup a clean install of the defect dojo using the ./setup.bash (master branch) on the following system:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

Django 1.11.2

If i want to delete an exisiting test under  /Home / Active Engagements / 1 / Engagement: This Engagement the application returns an ImportError "No module named util" based on the django.contrib.admin.util import in line 117 in /dojo/test/views.py:
117     from django.contrib.admin.util import NestedObjects
The error based on the django version (from 1.8 to 1.11.2). The package name was change from django.contrib.admin.util to django.contrib.admin.utils. The code below will fix the error in Defect Dojo
117     try:
118         from django.contrib.admin.util import NestedObjects
119     except ImportError:
120         from django.contrib.admin.utils import NestedObjects

Stack:

ImportError at /test/3/delete
No module named util
Request Method: 	GET
Request URL: 	http://host.de/test/3/delete
Django Version: 	1.11.2
Exception Type: 	ImportError
Exception Value:
No module named util
Exception Location: 	/opt/django-DefectDojo/dojo/test/views.py in delete_test, line 118
Python Executable: 	/usr/bin/python
Python Version: 	2.7.12
Python Path:
['/opt/django-DefectDojo',
'/usr/local/lib/python2.7/dist-packages/DefectDojo-1.0.5-py2.7.egg',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/home//.local/lib/python2.7/site-packages',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/opt/django-DefectDojo']

Confirmed, working on this now.


cargo release changes only shows commits since origin/master

I have a repository with the following history:
* cb9f950 [2023-04-17] (HEAD -> master) Increment version
* baa3c3c [2023-04-17] Set up most of cargo-release
* b02a8c8 [2023-04-17] (origin/master) Give the upload-artifacts job a more human-readable display name
* 8cb75ef [2023-04-17] rm debug
* 246ddf0 [2023-04-17] (tag: v0.1.0) NUL-separate xargs input
* 783af46 [2023-04-17] Remove CRs from jq's output on Windows
* 7b911a3 [2023-04-17] Debug
* 76dcaad [2023-04-17] Fix Windows PowerShell Core envvar syntax
* ec20fd2 [2023-04-17] Workflow fix
* 76a0843 [2023-04-17] Init

(In case it's relevant, the v0.1.0 tag is a signed annotated tag.)
Running cargo release changes -vvvv (using cargo-release v0.24.10) in this repository gives the following output:
[2023-04-17T22:24:19Z DEBUG globset] glob converted to regex: Glob { glob: "**/*", re: "(?-u)^(?:/?|.*/)[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true }, tokens: Tokens([RecursivePrefix, ZeroOrMore]) }
[2023-04-17T22:24:19Z DEBUG globset] built glob set; 0 literals, 1 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 1 regexes
[2023-04-17T22:24:19Z TRACE cargo_release::ops::git] origin/master: b02a8c8cca13e915a8a375952e81902350cb21de
[2023-04-17T22:24:19Z TRACE cargo_release::ops::git] merge base: b02a8c8cca13e915a8a375952e81902350cb21de
     Changes for cargo-dist-test from v0.1.0 to 0.1.1
             cb9f950 Increment version
             baa3c3c Set up most of cargo-release

As you can see, only the commits after origin/master are shown; based on the fact that cargo release changes --help says "Print commits since last tag", I would have expected it to also show commits b02a8c8 and 8cb75ef.

All of the code looks correct.  To really be able to tell what is going on, we'd need to know what the content of b02a8c8  and 8cb75ef are.
When cargo release changes runs, it filters out commits that do not touch files reported by cargo package --list

Ah, that explains it; the missing commits only touched a GitHub workflow, and I exclude all dot-files and -directories from my crates.  Is this aspect of cargo release changes's behavior documented anywhere?  I didn't see a mention of this in the reference.

Its showing you the changes on a per-crate basis.  Those changes are not part of the crate, so ergo they won't show up?  I think the current framing makes the most sense when working on a workspace.  Maybe there are ways to improve it but not having a good idea atm that is clear without being too verbose that it is less likely to be relied on


Create an example with next.js and NextAuth.js

PR approved on NextAuth.js, should appear in the next release (>4.2.1)


Plugin not toggling relative linenumbers (Linux, 0.169.0)

The Plugin seems to do nothing when trying to toggle the relative linenumbers.
The absolute numbers are always displayed.
Standard Shortcut, custom shortcut and Button "Relative Lines->Toggle" in Menu Bar were tried, including reinstall of plugin.
Is this just for me?
(New Atom-User, never had it working before.)

I don't think this issue is Linux specific, I'm seeing it on OSX build (0.177.0).

Hi and thanks for reporting this issue, I'm aware of the problem which is caused by that I haven't had the time to keep the plugin up to date with the API changes the Atom team has made. I will try to fix this but I can't promise anything. I'm will happily accept any pull request which solves this issue.

I believe the proper way to do this would be to change the atom GutterComponent instead of modifying the DOM when we receive events... It would be easier if it was an independant package

I made a pull request to atom core: https://github.com/atom/atom/pull/5449


List torch layers: and their arguments.

Listing torch layers is pretty easy by listing the contents of nn for example.
Any recommendations on getting the arguments of the constructor of the layers?

As far as I know there's no easy way beyond checking the docs, but if you have quick questions about Torch it's worth asking on Gitter.

I could just regex each <layername>.lua for contents <layername>: __init(<args>). Probably the only way.


[Fix] boolean-prop-naming: add check for typescript "boolean" type

[x] Fixes #2892
[x] Includes test

Overview of change:
Add a check for typescript boolean type without the need to explicitly define react PropTypes.
type Props = {
  enabled: boolean
}
const Hello = (props: Props) => <div />;

Will throw a linting error for the above code.

(note, the new tests weren't actually running since [].concat wasn't being used; i've fixed that)

(note, the new tests weren't actually running since [].concat wasn't being used; i've fixed that)

that's weird, because it did run locally using this
node node_modules/mocha/bin/_mocha tests/lib/rules/boolean-prop-naming.js

@vedadeepta the tests completed, but weren't actually running, because parsers.TS() returns an array, and the tests are objects.

This is the output i get when i run the tests without concat using the command node node_modules/mocha/bin/_mocha tests/lib/rules/boolean-prop-naming.js. Should it be something else?

parsers.TS returns either its first argument (which is meant to be an array), or an array.
I'll make some changes so it's harder to misuse.

Thanks for looking into it.


what's transaction detail?

i find method of offline_sign(),but i don't know the detail of parameter(transaction),can you help me?
I don't find detail in trx document.In addition,do you know how to mul-sign in trx?It looks like you should be good at eth and ripple,i can mul-sign in eth and ripple.Do they have anything in common?trx can use solidity too.

offline_sign() - Outdated
-- Current
sign() - offline sign
online_sign() - Not recommended! But in the case of attachment of additional parameters, the signature should be carried out online.
multi-sign - It is not yet implemented

No, the development of multi signing in TRON is not finished.


replace non-ascii characters with their "close enough" ascii counterparts (e.g. 'é' and 'e')

you can run translitcodec on your inputs, I would argue that's not in scope of this lib

@vvaezian Would you mind updating this to resolve the merge conflict? I'd be happy to merge once that is done.

I started to modify this to fix merge conflicts, but realize on second thought (and after seeing the failing tests) that I agree with @ddelange that this is out of the scope of the library.


Deeper Json Hierarchy

Here's a toy example of a json file with deep nesting
{
  "1": {
    "2": {
      "3": {
        "4": {
          "5": {
            "6": {
              "7": {
                "8": {
                  "9": {
                    "10": {

                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

If I scroll down to hide the top several lines, I see the hierarchy in the breadcrumbs only up to "4".

Please show more, or possibly even configurable. Thanks!

You can increase the value of the editor.stickyScroll.maxLineCount setting.

You can increase the value of the editor.stickyScroll.maxLineCount setting.

This worked for the example above. Interestingly, it doesn't work for my real file, and now I see that it's a bit glitchy. I pasted the large, test json file from https://github.com/json-iterator/test-data/blob/master/large-file.json and it's more or less nonfunctional. Truncating that file to different lengths, 10000 lines seems fine, 100000 not so much.
Is it OK to move the goal posts on this feature request to supporting stickyScroll for large JSON files?

Hi @blewis2-blue thank you for posting this issue. For very large files, we have disabled sticky scroll because it is computationally expensive. When you open such a file, you should see a notification which mentions that certain features like tokenization and sticky scroll are disabled, to keep VS Code running smoothly. It could also be you are seeing glitches, because sticky scroll is computationally expensive. I will close this feature in favor of https://github.com/microsoft/vscode/issues/191973


Feature request: Trace estimation algorithms

The trace is often used to summarize curvature matrices in second-order methods or for generalization metrics.
I could not find libraries that provide trace estimation methods for scipy.sparse.LinearOperators. The closest library is Nico's matfree which has Hutchinson trace estimation for JAX. pyhessian has Hutchinson trace estimation in PyTorch, but does not use a LinearOperator interface and only considers the Hessian.
So it would be useful to offer trace estimation through a scipy-based linear operator interface through this library.
Possible algorithms are:

[ ] (Basic) Hutchinson trace estimation (see Section 4)
[ ] (Advanced) Hutch++ (paper, matlab implementation)
[ ] (Advanced) NA-Hutch++ (paper)

Brilliant!
I also want to point out that Hutchinson can be extended to compute the trace of  H^2, as follows:
Sketch of classical Hutchinson to estimate tr(H):
result = 0
for _ in range(N):
    x = sample_rademacher()
    x = l2_normalize(x)
    result += x @  hessian_vector_product(H, x)  # x.T @ H @ x
result /= N

Since tr(H) = <H, I> = <H, mean(v @ v.T) where v are random vectors. Also, we don't normalize sample vectors since Rademacher guarantees that the diagonal entries of our "I" all equal N, so we just divide by N.
Now, the only modification we need to compute tr(H^2), is to replace the x.T @ H @ x operation with x.T @ H @ H @ x. This can be achieved at no extra cost in memory or computation by changing a single line, namely:
result += l2_norm_squared(hessian_vector_product(H, x) )

Since  || Hx ||_2^2  equals indeed x.T @ H @ H @ x.

I believe this operation can be useful whenever we care about the magnitude of the trace or the top spectrum, since H^2 is PSD and H is generally not. And it is trivial to implement with zero overhead.

Here is my current implementation, hope it helps:
class HutchinsonSquared:
    """
    """

    @staticmethod
    def rademacher(dims, nonzero_idxs, dtype=np.float32):
        """
        """
        if nonzero_idxs is None:
            sample = (((np.random.rand(dims) < 0.5)) * 2 - 1).astype(dtype)
        else:
            noise = (((np.random.rand(len(nonzero_idxs)) < 0.5)) * 2 - 1)
            sample = np.zeros(dims, dtype=dtype)
            sample[nonzero_idxs] = noise.astype(dtype)
        #
        return sample

    @classmethod
    def hutchinson_squared(cls, hessian, num_iters=1, nonzero_idxs=None):
        """
        :returns: An estimate of ``tr(H^2)`` for the diagonal entries
          corresponding to ``nonzero_idxs``.
        """
        h, w = hessian.shape
        result = 0
        for i in range(num_iters):
            sample = cls.rademacher(w, nonzero_idxs, hessian.dtype)
            sample = hessian @ sample
            result += (sample * sample).sum()
        # we don't normalize sample vectors, since Rademacher guarantees
        # that all diagonal entries of R @ R.T will equal num_iters.
        # So to approximate identity, just divide by num_iters.
        result /= num_iters
        return result

How should an interface for Hutchinson look like? Suggestion:
def hutchinson_trace(
    A: LinearOperator,
    num_samples: int,
    sampling: str = 'rademacher', # could also be 'normal'
    return_samples: bool = False, # allow to access individual samples, e.g. to estimate the variance
) -> Union[float, Tuple[float, List[float]]:
    ...
    # only return the mean unless user requests individual samples

any thoughts?

I think a class-based approach may be better, especially for Hutch++ which has a pre-computation step (computing an orthonormal basis and the trace in its span).
class HutchinsonTrace:

    def __init__(self, A: LinearOperator):
         self.A = A

    def sample(self, distribution='rademacher') -> float:
        ...

Brilliant!
I also want to point out that Hutchinson can be extended to compute the trace of H^2, as follows:
Sketch of classical Hutchinson to estimate tr(H):
result = 0
for _ in range(N):
    x = sample_rademacher()
    x = l2_normalize(x)
    result += x @  hessian_vector_product(H, x)  # x.T @ H @ x
result /= N

Since tr(H) = <H, I> = <H, mean(v @ v.T)> where v are Rademacher iid vectors. Also, we don't normalize sample vectors since Rademacher guarantees that the diagonal entries of our "I" all equal N, so we just divide by N.
Now, the only modification we need to compute tr(H^2), is to replace the x.T @ H @ x operation with x.T @ H @ H @ x. This can be achieved at no extra cost in memory or computation by changing a single line, namely:
result += l2_norm_squared(hessian_vector_product(H, x) )

Since || Hx ||_2^2  equals indeed x.T @ H @ H @ x.
I believe this operation can be useful whenever we care about the magnitude of the trace or the top spectrum, since H^2 is PSD and H is generally not. And it is trivial to implement with zero overhead.

Quick correction: The l2_normalize is not required in Hutchinson's trace estimator.

Sorry for dropping the ball.

Thanks for the correction! You are right that the normalization is not needed, looking at my snipped I also don't find one. Good catch in any case
As for the interfacing, if both functional and class-based provide the same efficiency, and unless there are clear cons, I'd go for consistency with the rest of the library. I don't know much about Hutch++ but if you are coupling functionality with data, OOP is probably the natural choice.


Implement MetalRenderer

Not much to say, it's in src/gpu/metal/!
TODO:

[x] SetGpuBufferName
[x] SetTextureName
[x] SetStringMarker
[x] GetTransferData
[x] CopyBufferToBuffer
[x] GenerateMipmaps
[x] DownloadFromTexture
[x] DownloadFromBuffer
[x] BindVertexStorageTextures
[x] BindVertexStorageBuffers
[x] BindFragmentStorageTextures
[x] BindFragmentStorageBuffers
[x] DrawPrimitivesIndirect
[x] DrawIndexedPrimitivesIndirect
[x] QueryFence
[x] SupportsSwapchainComposition
[ ] CreateOcclusionQuery
[ ] OcclusionQueryBegin
[ ] OcclusionQueryEnd
[ ] OcclusionQueryPixelCount
[ ] ReleaseOcclusionQuery

Cleared up the finished items on the list - all examples should now work, including those that use compute shaders.

Ended up dropping occlusion queries, so Metal's filled in now!


Use labels in AlignedUMAP if provided

Fix issue #801

Looks like all the test failures are due to some tensorflow/keras changes that have broken parametricUMAP; I'll just merge this since it looks like a very simple change.


🚀 Enhance deploy script

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6f6fe29) 80.46% compared to head (a5534e0) 80.46%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #42   +/-   ##
=======================================
  Coverage   80.46%   80.46%
=======================================
  Files          10       10
  Lines         476      476
  Branches      109      109
=======================================
  Hits          383      383
  Misses         76       76
  Partials       17       17

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


comments inside of vector can get printed oddly when they wrap

If a comment is inside of a vector (for which :wrap? true is set, which it is by default), then if it ends up near the right margin when the comment wraps it can end up in a strange place.

Can't reproduce this in 0.5.0.


Use Boost.StaticString

Beast is supposed to use Boost.StaticString now instead of using its own copy.

This is already done


EAX support?

So I know that vmware supports eax because it emulates sb128, but is it possible on virtualbox? I feel this is an important question because a lot of games stand to benefit from eax support.

There is a VirtualBox package providing MPU401, OPL3 and AWE32: https://git.javispedro.com/cgit/vmusic.git/about/
As for EAX - agree it would be a good addition. Potentially useful code:

AudioPCI/SB128 is emulated also by QEMU, PCem, 86Box (which are open source in contrast with VMware)
Linux Live! and Audigy drivers
Illumos Live! and Audigy drivers: code
EAX wrappers, etc. joncampbell123/dosbox-x/issues/3597


jay: 1.2.0 -> 1.3.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/mahkoh/jay/releases.
meta.description for jay is: A Wayland compositor written in Rust
meta.homepage for jay is: https://github.com/mahkoh/jay
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.3.0 with grep in /nix/store/a7dlr1qs564wrbn1nnj8c3wx2h9sidik-jay-1.3.0
found 1.3.0 in filename of file in /nix/store/a7dlr1qs564wrbn1nnj8c3wx2h9sidik-jay-1.3.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

jay

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/a7dlr1qs564wrbn1nnj8c3wx2h9sidik-jay-1.3.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A jay https://github.com/r-ryantm/nixpkgs/archive/6abf602b1cbac7b94ff681b18528a5af014dbd52.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/6abf602b1cbac7b94ff681b18528a5af014dbd52#jay

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/a7dlr1qs564wrbn1nnj8c3wx2h9sidik-jay-1.3.0
ls -la /nix/store/a7dlr1qs564wrbn1nnj8c3wx2h9sidik-jay-1.3.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

jay

Maintainer pings
cc @dit7ya for testing.

Result of nixpkgs-review pr 314587 run on x86_64-linux 1

1 package built:

jay


vertex search for RAG with custom documents

I would like to setup a pipeline with GCP modules to make RAG with vertex search for custom documents in formats of xlsx, pdf, xlsx, etc.
I saw there are some notebooks related to BigQuery + vertex search, but could not find one for document retrieval.
Is there any reference that I can see for this?
Thanks,

See following samples:
https://github.com/guruvittal/codesamples/blob/main/Retrieval_Augmented_Generation_Using_Vertex_Matching_Engine_-_Contract_Analyst.ipynb
https://github.com/guruvittal/generative-ai/blob/main/language/orchestration/langchain/intro_langchain_palm_api.ipynb


Fix filtering dropdowns in soviet

There's something that overlaps when screen is too narrow

TU SEI NARROW!

Accidentally duplicated in #87 but that contains more details so leaving that one open


Bug in Fixed point modulo maths

This operation does not output the correct result.
https://github.com/Bonfida/agnostic-orderbook/blob/0acdcea2ec4a4760be68a076c5a883ce9f7daa20/program/src/processor/new_order.rs#L123-L125
Example:
Tick price: 0.1
As FP32: 429496729
Limit price: 0.9
As FP32:<PHONE_NUMBER>
3865470566 % 429496729 = 5
This is presumably because of the floor operation in the FP32 calculation.
Pre floor values are3865470566.4 and 429496729.6
So these value probably need to be converted back to floats before doing the modulo operation?

Hi @Henry-E!
The underlying ticksize is defined in FP32 terms. This means that your limit price must be a multiple of the tick size in FP32. The easiest way to compute this would be with something like :
floor(0.9 / 0.1) * 429496729.
We think it wouldn't be a good idea to add some float logic to the AOB, because it's too computationally expensive for little benefit. This problem should be solved on the client side. This might be a usability issue so thanks for reporting it, we'll think about adding a helper util to perform this kind of computation.

Yes, it might be useful to add some typescript tests demoing the different basic functionality of placing orders, reading the event queue and the order book. I couldn't find any examples in the dex v4 repo but I might have been looking in the wrong place.


Running the "DeployVisualStudioTestAgent" task reboots the machine; Is it by design?

We have TFS 2017.
As part of TFS build we are deploying Test Agent with this task "Deploy Visual Studio Test Agent".
When it is used for the first time in a machine, it works fine, running Test AGent as User2, while another user is logged in to the machine as User1.
On a consecutive build to the same machine when User1 is logged to the machine, the machine gets rebooted, as the build runs the task "Deploy Visual Studio Test Agent".
Upon checking the logs, I see that there is a reboot after this log
You have chosen to run the test agent process as Domain\User2. Restart the machine for this change to take effect.
IN the second run why is it trying to delete the service and reinstall it?

If the task is configured to run as interactive process for UI Tests it tries its best effort to make sure the test user has a active UI session . If this is Windows Client OS it tries to get user2 (test agent is configured as this user) an active session by rebooting the machine and adding a EnableAutoLogon settings.

Hi I have below queries:

With Win 7 SP1 machine the user with which the Tests are run, is already logged on in the PC. But the reboot is still happening.

After running for the first time, the service is left installed. But the service is stopped. During the next run why does the task try to reinstall the agent?

Can you please set build variable system.debug = true and share those build logs with us at<EMAIL_ADDRESS>In your case, the reboot might be happening if test agent did not find an active user session while trying to run UI test.

Any update?

Dear Tanvi,
Thank you for the follow up and apologies for the delay in getting back.
With Interactive:

I am running the build agent as User1 and running the tests also as User1.
There is an active session for User1 already in the machine.
User2 is also logged in.

Now when the Test Agent is configured there is not a reboot.
However I am yet to try the below option:

Run the agent as User1.
Run the tests as User2.
Have an active session for User1.
I hope there won't be a need to have an active session for User2 as well.
Login as User3 and try the build to see if there is a reboot done.

@BaskiGIT If you already have active session for user 1, then reboot would not happen, as you have mentioned above.
Rebooting of test agent machine is by design and it happens only if there is no active session for the user with which you want to run tests.
You can simply disconnect the machine or log out, and let the build run. It will reboot the machine, if required to establish active session for mentioned user, and run the test. If you log into the machine with the same user during run functional test task, you can see the tests running.

Hope you got unblocked. Closing this thread for now. If you feel the issue is not addressed please re-open.


Removed Useless Symbol - Docs

Found another useless semicolon while reading the docs,
These might be caused by, prettier (docs)

Thanks;


ci: try a different strategy for rabbit health check

WIP

coverage: 73.597%. remained the same
when pulling ae1f927d3baa5f2c1900b50223d3f09471bba098 on massi/rabbit-test
into 3d5674e309f8466cee85b9b653deab21461162f1 on main.


Close issue notes not showing immediately after closing a case

Close issue notes not showing immediately after closing a case, navigating away from the Case Details page entirely and coming back is necessary for it to display.
Expected Behavior: should show immediately on case closure

Can't reproduce


Add order confirmation modal to display after a user checks out, and added marketplace home page that shows before an aisle is selected

Looks good


How to not trigger data stream using watchwhen upserting/updating data which same columns?

Drift's query stream are updated by a heuristic, you can't rely on them only updating when data has changed.
If your intention is to not update streams unnecessarily because you react to new events with side-effects, you may be able to use .distinct() (or perhaps .distinct(const ListEquality().equals) for streams with multiple rows) so that it only updates when data truly has changed.
If your idea is to now update streams too often because running the select statement itself is expensive, you can group multiple inserts in a batch or a transaction. They will only update streams once in that case.


Windows 10 x64 - Fatal Python error: Py_Initialize: unable to load the file system codec

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.08.17. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2016.08.17

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
C:\path\to\youtube-dl>youtube-dl.exe -v
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

Current thread 0x00003a38 (most recent call first):
<end of log>

Description of your issue, suggested solution and other information
Similar issue:
I found #10053, which reports the same problem, but was closed because the user had placed it in C:\Windows\System32. I did not place mine in that location.
Additional notes:

I downloaded the executable file just prior to executing, so it is the latest version
Windows 10 x64 machine
Microsoft Visual C++ 2010 redistributable package is installed
The machine was recently upgraded to Windows 10, and prior to the upgrade, youtube-dl functioned just fine

What is the C:\path\to\youtube-dl?
What is the output of where youtube-dl.exe?

The path to youtube-dl is C:\Users\REDACTED\Downloads\Youtube. I'm using the executable from within the folder that it resides in, so it does not exist in the PATH.

C:\Users\REDACTED\Downloads\Youtube>where youtube-dl.exe
C:\Users\REDACTED\Downloads\Youtube\youtube-dl.exe

Same issue in Windows 10 x64.  Tried in the cmd.exe interpreter and powershell v5.1.  I'm running it out of the current directory.  Path includes "c:\Python34;c:\python34\scripts".
c:\..>  .\youtube-dl.exe -v
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

Current thread 0x00005464 (most recent call first):

@Slaymeister Could you remove the Python 3.4 installation and try again? They may conflict with each other.

What happens if you boot into safe mode and run youtube-dl.exe there?

I removed Python 3.4 and had the same error.
Running in safe mode also produced the same error.

Check with Tasklist (or Procexp) for running processes and services, try closing or deactivating non-necessary stuff and try again..

I got this today. It seems a random error...

In all invocations youtube-dl.exe is the same file. G:\ is a mounted Virtualbox share.

I have such an error only when I try to run the program from any other directory other than the one in which the program resides.
Windows 10 x64

I can reproduce by simply signing youtube-dl.exe using SignTool.
Ex:

Download youtube-dl.exe then run it in many different ways (from same folder, etc.). Always works.
Sign it using signtool.exe sign ...
Run it with in many different ways, always fails with the following error:

Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

I guess signtool.exe changes the content of youtube-dl.exe? If so py2exe
can be broken. What's the need of using signtool? youtube-dl.exe is already
signed and the detached signature can be downloaded from
https://yt-dl.org/downloads/latest/youtube-dl.exe.sig
On Sat, Sep 30, 2017 at 4:56 PM, Romain Cointepas<EMAIL_ADDRESS>wrote:

I can reproduce by simply signing youtube-dl.exe using SignTool.
Ex:

Download youtube-dl.exe then run it in many different ways (from
same folder, etc.). Always works.
Sign it using signtool.exe sign ...
Run it with in many different ways, always fails with the following
error:

Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rg3/youtube-dl/issues/10372#issuecomment-333294793,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2RGUCzGOcqiU-vRzUhsjwOVUBtEK6Eks5sngImgaJpZM4Jm9qT
.

@yan12125 are you sure youtube-dl.exe is signed?
When I run signtool.exe verify youtube-dl.exe, I get SignTool Error: No signature found.

The signature is not included in the executable itself. Instead, it's
stored in an external, detached file, which is available at
https://yt-dl.org/downloads/latest/youtube-dl.exe.sig. You can download
both the executable and the signature and verify it with gpg:
gpg --verify youtube-dl.exe.sig youtube-dl.exe
As a reference, the last paragraph in
http://rg3.github.io/youtube-dl/download.html provides more information and
useful tips.
On Sat, Sep 30, 2017 at 6:55 PM, Romain Cointepas<EMAIL_ADDRESS>wrote:

@yan12125 https://github.com/yan12125 are you sure youtube-dl.exe
https://github.com/rg3/youtube-dl/releases/download/2017.09.24/youtube-dl.exe
is signed?
When I run signtool.exe verify youtube-dl.exe, I get SignTool Error: No
signature found.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rg3/youtube-dl/issues/10372#issuecomment-333300859,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2RGWlJE0pbyibHIEZx3UQiRAD6mVt7ks5snh4cgaJpZM4Jm9qT
.

@yan12125 Oh got it.
I'm shipping youtube-dl as part of an Electron app, and as part of the packaging process I'm signing the main executable and all embedded executables. I've just discovered that in order to sign a py2exe executable, I will need to build youtube-dl myself and detach the library from the exe loader (as explained here: https://medium.com/@bjoernstiel/code-signing-py2exe-executables-a222c0d06d3f).
I'm currently trying to figure out how to build youtube-dl on Windows, but that shouldn't be a problem.
I just thought I would mention this specific cause as it ends up failing in the exact same way as this issue mentions.

Thanks for the link! Maybe we can sign youtube-dl.exe when building it. Seems it requires buying a certificate to get a valid signature? I'm not sure if getting another signature is worth it - we already have PGP signatures.
If there's really a need, could you open a new issue? This issue targets unmodified youtube-dl.exe

@yan12125 New issue created.
Just to clarify here as well, the official youtube-dl release should not be signed (that's really not the problem, and you would encounter the same problem to sign it anyway). The solution simply require an additional build of youtube-dl for windows (one that separate the exe loader and the library, that's it).

Did anyone get any workaround for this?

Getting it outside of system32 folder makes it work for me.
E.g C:\myapps\ for example

Can confirm @yafrica's comment:
Here it was in C:\Windows\System32:
C:\Users\Dell>youtube-dl --version
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Current thread 0x000014d4 (most recent call first):
Move the executable to C:\Windows (also on the PATH by default):
C:\Users\Dell>youtube-dl --version
2018.04.16
Marvellous! (and utterly bonkers)
(yes, my "username" is "Dell" - I'm still commissioning this PC!)


Texture Streaming issues with Vulkan

Fortunately, I was able to get rid of an "AwaitReady" texture hack after xen2 (from the main branch) found issues were related to texture streaming & Vulkan.
He patched it up by initially loading a low resolution texture (so Vulkan had something to chew on) while the higher resolution texture gets streamed in. Reportedly this works on the main, official branch.
When I ported it, textures never got rendered at a higher resolution during testing, even though the code seemed to be getting the higher resolution texture.
Ultimately, I disabled texture streaming for Vulkan. This way, textures are loaded immediately and ready for use. This has its pros and cons... like reducing overhead loading textures a pro, while a con is having to wait for the texture to load (which is probably best loaded in another thread or during an app startup as a workaround). A con is also taking this choice away from the developer.
Anyway, I consider this progress (should result in faster texture loading and less hacks)... two steps forward, one back.

Commit doing the above: https://github.com/phr00t/xenko/commit/cf7528cf667e5ca5ee4eed9464010fb3f3f7c783


Add money comparisson

There are several methods to compare two Money object but there is no Money.Compare(Money) function.
The proposal is to add a Compare function for Money.
// Compare returns an integer comparing two Money objects
// If Money objects are of different currency, an ErrCurrencyMismatch is returned.
// If both objects are for same currency, their amounts are compared.
// The result will be 0 if a == b, -1 if a < b, and +1 if a > b.
func (m *Money) Compare(om *Money) int {
    if err := m.assertSameCurrency(om); err != nil {
        return false, err
    }

    return m.compare(om), nil
}

I'll provide a PR.

Hey @totemcaf
Great idea! Let's implement it


Obtaining TSV output with R10.4

Hi,
I sequenced with R10.4 and now I wanted to use these data in PycoMeth pipeline. They only mention that the input should be taken from the output TSV of nanopolish. Given that I can't call methylation with Nanopolish (because of the R10.4), what could do as an alternative to get the TSV?
Thanks in advance! :)
Pol

Try f5c. https://github.com/hasindu2008/f5c/


Try to add ssl

Almost identical to previous PR. Contains some minor fixes.

Requires new release of scylla-cdc-java with fixes


[release-4.14] OCPBUGS-24269: add CLI oauthclient

manual pick of #3238 and #3249

/jira refresh

/retest-required

/lgtm

/override "Red Hat Trusted App Test / mce-24-klk72 / tektok-runner-enterprise-contract-mce-2.4"


Allow deriving SystemParam on private types

Examples creating a public type to derive SystemParam on were updated
to create a private type where a public one is no longer needed.
Resolves #1869

bors r+


fix(deps): code change should have been done with KMS bump

Description
The DA use of kms-all-inclusive was broken by upgrading the module version without associated (required) code changes.
This pull request adds the required code changes.
Release required?

[ ] No release
[x] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

:tada: This PR is included in version 8.13.4 :tada:
The release is available on:

GitHub release
v8.13.4

Your semantic-release bot :package::rocket:


Surface for 8-bit grayscale image

Hello all,
I recently tried to use Julia/Cairo for an image generation of 8-bits gray scale.
In fact, what I need is an uint8 array with each element represent the gray scale for the further calculation. (It is conventional 8-bit grayscale image, I think)
However, I cannot find the way to make grayscale surface with Cairo.
At first, I assumed that CAIRO_FORMAT_A8 is the grayscale surface and also assumed that the data array has to be an uint8 array. However, there's error like follows:
julia> sf = CairoImageSurface(dt,2);
ERROR: MethodError: `CairoImageSurface` has no method matching CairoImageSurface(::Array{UInt8,2}, ::Int32)
   Closest candidates are:
   CairoImageSurface(::Real, ::Real, ::Integer)
   CairoImageSurface(::Array{UInt32,2}, ::Integer)

dt is Array{UInt8,2}
And then, I convert the dt array to uint32:
julia> sf = CairoImageSurface(dt,2)
ERROR: AssertionError: stride == 4w
 in CairoImageSurface at C:\Users\10189524\.julia\v0.4\Cairo\src\Cairo.jl:175

Of course, I can make the image surface with ARGB or RGB format.
Is this a bug or my short knowledge of Cairo?
Thanks in advance

CAIRO_FORMAT_A8 (see http://cairographics.org/manual/cairo-Image-Surfaces.html#cairo-format-t) is 8 bit, but alpha channel only. This is usually used as a alpha mask for painting. You could e.g. use this to paint black color source to a white background with your image in the _A8 alpha map to paint a grayscale image, but the easier way is to use one of the color formats (_RGB).

Thanks @lobingera.
Maybe I can use RGB surface to draw my figures.
However, I want to use 8-bit surface if possible since my target array is somewhat large one. (There will be a few 20000 * 20000 float64 array and my PC has relatively small memory.)
Also, if possible, binary b/w image will be more useful for me.
In fact, I don't need figure itself, I just want the arrays.
Is there any possible way to do this?

Is there any reason you need to use Cairo? The ColorTypes and Images packages supports 8-bit grayscale natively. In previous benchmarking, we've found julia code, especially when dealing with grayscale, to be faster than libcairo by fairly large factors (3-10 or so).

@timholy Can ColorTypes/Images deal with drawing shapes/lines? What I need is to make arrays whose elements can form many "shapes" (such as rectangles or triangles...)
I'm new to Julia (just a few weeks) and I have not read the package manuals completely.
I think Images package can convert images to integer array. However, I don't know I can "draw" something with them.

@timholy, could you please link to this benchmarking?
@xanadupark, Tim is right, handling large bitmaps/images with Cairo might not be the most efficient way (i'm e.g. not sure if 8bit Alpha format isn't using 32bit with masking...). Usually Cairo is used as the step before painting or displaying and so you might e.g. need only to bring 512x512 pixels clipped to screen, but your image stored is 20000x20000. libcairo has some infrastructure for this (http://cairographics.org/manual/cairo-Raster-Sources.html) but i think it's not available in Cairo.jl right now.

@xanadupark, no, there is no drawing. You could see if Drawing.jl would be interested in a non-Cairo backend, but then someone would have to implement it.
Two benchmarks that come to mind:

https://groups.google.com/d/msg/julia-users/wG1g9C7S6Mg/OFEIbLteAdYJ
https://groups.google.com/forum/#!searchin/julia-users/cairo$20path/julia-users/oNPhJ9iHIDY/wh4L5jaFMwAJ and https://github.com/nolta/Winston.jl/issues/89

@lobingera, you were on both of those threads, if you wanted links you could have provided them yourself :stuck_out_tongue:

@timholy, i'm always hoping for something more recent.

By all means, run the code in those benchmarks and report the results.

Just for the record, after copying the code Tim's code from the first thread and working around the dep warnings in 0.4.1 ->
julia> include("image-resize.jl")
Cairo:
0.046598 seconds (31 allocations: 51.400 MB, 1.75% gc time)
Julia:
0.085892 seconds (9 allocations: 6.047 MB)

Fascinating. On my machine the julia code is still faster, especially if I replace trunc with unsafe_trunc.  (Back when that was written, julia's trunc was unsafe.) But the gap is now only 2x.
The difference is that the Cairo version got faster, not that the julia code (if you make that change) got slower. Nice!

I think there was some work on pixman (the SW that deals with pixel operations and blitting) especially on interpolation in finishing 1.14. But still, Cairo is not the place for image operations, but elegant and efficient image display.

Thanks everybody.
I found out there is no proper Julia package at this time for my use.
I look into Drawing.jl and it seems like use Cairo as backend.
I also look into Compose package but it is somewhat complicated for me.
I think I can use some python packages to draw my arrays and use Julia for the calculation.
Thanks all for your advise.


Create Clip-n-Ship Module

This should be the clip-n-ship class/module that one should instantiate.
Some proposals:
/*
  @param video (required) - blob?? - the video file to convert
*/
const converter = new ClipConverter(video); // additional args?? should return the canvas element for consumer to do whatever with maybe?

ClipConverter API:
/*
  @param scale (required) - number - the scale of the video layer
  @param filter (optional) - string - the filter to apply to the layer. defaults to "none"
*/
converter.addLayer(0.5, "blur(20px")) // adds a new layer to the clip.  should it return an index or id??

converter.getLayers() // returns list of layers (their properties)

/*
  @param index (required) - number - the index of the layer to get
*/
converter.getLayer(0) // returns the layer properties for specified layer

/*
  @param index (required) - number - the index of the layer to update
  @param scale (required) - number - the scale value to change to
*/
converter.updateLayerScale(0, 0.75) // updates the layer's scale property

/*
  @param index (required) - number - the index of the layer to update
  @param filter (required) - string - the filter value to change to
*/
converter.updateLayerFilter(0, "none") // updates the layer's filter property

converter.render() // renders the clip using the specified layers, returns some kind of progress observer (which returns an object URL when complete)??

closed by #4


Support more import/export formats

Which formats should XeTune support?
Some future-gazing questions that might help answer that:

Would users benefit from interoperability with other apps, and preserve metadata that the .scl format loses?
Do we need to store XeTune tunings as a portable file, and is a proprietary format required for that?
Do other formats store essential data that we cannot infer otherwise?
Do we need to persist as a file, data that adds value to tunings, notes, analysis, or configuration profiles. For example, user-provided annotations, generators of tunings, or filtered views?

My instinct is to await user needs (features) that require file persistence, before reaching for proprietary or compatible formats.

once again thanks for this amazing project and tool
my biggest request would be the import of SCL files, KBM import is not as crucial for my needs but I can't speak for others, not even sure if keyboard mapping would even be feasible, however being able to set the root note and its frequency would certainly be something worth considering

my biggest request would be the import of SCL files

Good news: you can already drag-and-drop Scala files into XeTune.

KBM import is not as crucial for my needs but I can't speak for others, not even sure if keyboard mapping would even be feasible, however being able to set the root note and its frequency would certainly be something worth considering

Certainly! The workflow is fragmented for reference frequencies. Discussed a bit more in #23.

Good news: you can already drag-and-drop Scala files into XeTune.

this is great news, I think I vaguely remember you mentioning this on Discord, thanks for the response


entry is null in getTimingEntry callback

Using the basic setup instructions, in my current app I'm getting:
Uncaught TypeError: Cannot read property 'secureConnectionStart' of null
This is odd in that it works perfectly in another app we have, using the exact same backend endpoint to run the latency test against.
Thoughts?

More detail: testing in Chrome, the first ping is sent to the server and responds correctly.
It looks to me, that there are "entries" for 150 requests in my app, everything except the network ping it should be filtering down to with return ~entry.name.indexOf(lastURLToken);. All of these entries are for the local server, whereas my ping request actually points to a remote server. Possibly a config issue?

And it turns out the Performance API specifies a limit of 150 requests. I was able to fix the matter by clearing the entries immediately before invoking network JS like so:
window.performance.clearResourceTimings()
var net = new window.Network( config )


Enhance CSS and add Help section

Issue #5 ,
Add reactstrap to give a better look, and add Modals from reactstrap to create Help section where instructions can be given.

WIP
This the basic theme I can think of. If you like it I can make more changes and bring all the Send options in a drop-down or a Navbar. I have made a few changes in QRCodeScanner changing the position of Scanner and close option also.

We have forked away from this version so I'll close this because of merge conflicts. Thanks for looking into it!

@austintgriffith , should I send new PR ?

Not yet, basically, we need to decide on our favorite UI over in https://github.com/austintgriffith/burner-wallet/issues/5 first and then we can spec out the build to integrate it.


Add env-var-docs.json and PR automation scripts

Description
Followup to https://github.com/civiform/civiform/pull/4225.  Adds the env-var-docs.json documentation file and the scripts that ensure it has all the proper variables and that any provided regex_tests pass.
Checklist
General

[x] Added the correct label: < feature | bug | dependencies | infrastructure | ignore-for-release >
[x] Created tests which fail without the change (if possible)
[x] Extended the README / documentation, if necessary
[x] Add test in check_vars_documented_test.py for imported conf files
[x] Add bin scripts
[x] Add new scripts to pr_env_var_docs action

@bion @jessbumblebee PR is ready for review


Support integration with Zentao

See: https://www.zentao.net/
Zentao is an open-source project management platform written in PHP. There are several ways for integrating Zentao with Casdoor SSO:

Via 3rd-party OIDC module: https://github.com/wingfeng/zentao-oidc
Via Zentao's own non-password login: https://www.zentao.net/book/zentaopmshelp/344.html
Via CAS custom code snippet: https://www.bianchengquan.com/article/323615.html

@leo220yuyaodog

Fixed by:https://github.com/casdoor/casdoor-website/pull/205


Logbook Form Investigation

Background:
Trips and log books represent the last 2 paper forms.
Trips (formerly known as effort) is a record of the day. Who went out, which boat, and where they went. It includes a summary of the encounters & logbook entries and some key metrics for the day (distance covered, conditions etc)
The logbook is currently a handwritten notebook which is taken out onto the boat to record, at 30-minute intervals, the conditions being encountered.
This is burdensome and hard to analyse. Often, the log book will not be transposed into the MS Access database despite it containing valuable data due to the manual pain involved in doing so.
We want to fix this so in this ticket we want to investigate and create a solution for the MVP. For this spike to be marked as complete, we would need a full solution description, pros and potential drawbacks of the suggested solution, and finally to communicate this to Charlotte before implementation is complete. The MVP is detailed below:
MVP (the very thinnest slice that can be built to complete the user journey)

Add essential UI elements to allow the inputting of a trip
Add essential UI elements to allow the inputting of a log book entry
Export script to allow for importing of trips & log book to MS Access DB

The previous team created this mural which will be helpful in this investigation for potential effort form stories:
https://app.mural.co/invitation/mural/thoughtworksclientprojects1205/1658485030752?sender=u3cc89491750dce6cd5621603&key=916e6445-a3db-4b72-8173-a7f8c92fd7e3

Closing this as the logbook has been implemented by the stories: #509 #500


fox.com needs fixing please.

hi, can you please update youtube-dl so it works with the new fox.com urls
thank you
free  https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/
login  https://www.fox.com/watch/30056b295fb57f7452aeeb4920bc3024/
i use a youtube-dl gui so i don"t get any error messages but fox was working perfect a few days ago and could download locked videos that needed a login without logging in with no problems
but now the fox site has changed the urls and site layout

I fed the free url to youtube-dl, here's the output:
casval@casval-X6815 /tmp % youtube-dl -v https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.02
[debug] Python version 2.7.13 - Linux-4.10.0-33-generic-x86_64-with-Ubuntu-17.04-zesty
[debug] exe versions: avconv 3.2.4-1build2, avprobe 3.2.4-1build2, ffmpeg 3.2.4-1build2, ffprobe 3.2.4-1build2, rtmpdump 2.4
[debug] Proxy map: {}
[FOX] 087036: Downloading webpage
ERROR: Unable to extract drupal settings; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 35, in _real_extract
    webpage, 'drupal settings'), video_id)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 797, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract drupal settings; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I have no fox.com password, so someone else do that.

It's also geo-restricted.

@timendum Would it help if I tried to let it run again via Tor Onion with an US exit node?

I'm looking into it. It looks like a site redesign.

still major issues on there end.....I managed to feed m3u8 into youtube-dl on recent show.......720p dimensions where
Width                       : 404 pixels
Height                      : 228 pixels
as was 432p and 1080p(1080 is new btw)
funny looking at 6mb/s 228p video lol
also ads(national ads only) are in video feed too atm........so lot of kinks on there end also

can it be fixed in the next few days?
fox are due to remove alot of episodes soon
thanks.

downlaoder works.........but all there stuff has embedded ads now :(

Perhaps the easiest way to 'fix' the ad issue this would be for youtube-dl to start a new file every time UPLYNK-SEGMENT items occur in the m3u8 file, since UPLYNK-SEGMENT occurs at changes in the stream.  Or alternately, to add an option to process each fragment like an individual video when --keep-video --keep-fragments is used.  As it is right now, the Frag* files aren't playable.  If they were, users could identify/delete the problematic fragments and stitch the items back together themselves.

Perhaps there's a way to filter out the #UPLYNK-SEGMENT: ...,00000000,ad segments. (I'm assuming this would require a modification to common.py)

From what I encountered when dealing with The Orville, technically yes, but it comes with a couple significant caveats or drawbacks.  Some are easy to filter out based on the video properties, but others could only rely on length and could potentially cause the end credits to be skipped too.  I guess one way to know is if multiple streams occur under one UPLYNK-SEGMENT, it's probably part of the main program, but without checks on properties or length of the whole UPLYNK-SEGMENT you can't be sure (and it could always get trickier if all the ads suddenly start having the same properties as the main program*, or if the main program's UPLYNK-SEGMENTS suddenly start occurring mid-act; that's why I suggested having a playable-fragment fallback option so end users could brute force it themselves later).
*especially if this means they stop flagging the ads separately as well.
Comedy Central's extractor (despite probably being served in an entirely different fashion) puts each main segment of the program into a separate file, and always has for as far back as I can remember.  A fringe bonus of that approach is that the end user can append all of the good segments back together with, say, MKVToolNix, and have it automatically insert chapter markers where the commercial breaks had been, since those are the natural places where chapters tend to be inserted anyway.  AFAIK, chapter information is something youtube-dl has absolutely no control over (and would definitely be lost in some sort of smart filtering to a single output file), but outputting by big, main segments would allow the user to reconstruct that information with virtually no overhead or significant hassle.

The initial theplatform request should redirect to a preplay json.
The data contents of the preplay json should have a "playURL" (including pbs code) at the very end of the json data.  That playURL can be parsed down to an ad-free parent hls master url.
For example, for a playURL that looks like this;
https://content-ause2.uplynk.com/preplay2/6c68ec3ffc3c4df7972591ba15629fcf/447ef3b45492a2e10d37d1742781b57f/6luJp5COKhZUvGh2fSdjTB34T0DrGRd0si8T4ohVViFH.m3u8?pbs=b88ec83f195142119194f92cbd3eec47
Parse it to;
https://content-ause2.uplynk.com/6c68ec3ffc3c4df7972591ba15629fcf.m3u8?pbs=b88ec83f195142119194f92cbd3eec47

the method for bypassing Uplynk preplay ads were already been used in viceland extractor, it should be usable as well for fox, the uplynk preplay json url should be delegated to UplynkPreplayIE.

@remitamine  For clarification (and since there has been any update on this), does that mean that all that needs to be done is to add
'add_ie': ['UplynkPreplay'],
to the FOXIE class under _TESTS?

does that mean that all that needs to be done is to add
'add_ie': ['UplynkPreplay'],
to the FOXIE class under _TESTS?

no, the changes that need to be done is:

do no extract the formats in https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/fox.py#L57-L63.
extract the real uplynk preplay link(theplatform redirect to uplynk so it should be possible to get the url with a HEAD request).
pass uplynk preplay url in the returned info dict(and also add 'ie_key': 'UplynkPreplay').

this is one way to do it.
the other possibility is to make theplatform handle uplynk preplay urls and than delegate the extraction to ThePlatformIE.

@dstftw As you implemented the original extraction of the formats, can you please implement the fixes @remitamine suggested?

This commit does not handle videos behind a TV provider (e.g. some shows up to a week after original airing)
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.fox.com/watch/818529c3d72584486365f78ae851fa67/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.15
[debug] Python version 3.6.0 - Darwin-17.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.3.4, ffprobe 3.3.4, rtmpdump 2.4
[debug] Proxy map: {}
[FOX] 818529c3d72584486365f78ae851fa67: Downloading JSON metadata
[FOX] 818529c3d72584486365f78ae851fa67: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 503, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2177, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)```

I concur.  The commit does not handle videos behind a TV provider.  Although, I get a different error message.  Fox requires shows to be verified with a TV provider login the first week when posted online and then after 5 episodes are online.
https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/

The error would be helpful.
Download Video Streams
Enter video source link: https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--get-filename', '--ap-mso', 'DTV', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE',
'--rm-cache-dir', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrom
e/63.0.3218.0 Safari/537.36', '--verbose', '-f', 'mp4[height=720]', '-o', '%(series)s - s%(season_number)02de%(episode_nu
mber)02d.%(ext)s', 'https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 3.4.4 - Windows-10-10.0.15063
[debug] exe versions: ffmpeg N-82092-g89ec4ad, ffprobe N-82092-g89ec4ad, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https:/
/yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl w
ith the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplnz97faa\build\youtube_dl\extractor\common.py", line
 503, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplnz97faa\build\youtube_dl\YoutubeDL.py", line 2195,
in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

i also get the same error
is it possible to fix?

@dstftw or @remitamine what you say?
I can test. Just let me know. :)

jaylweb, have you found a fix yet? the old version oct 1st never gives a issue downloading from fox but it also downloads loads of adds
i use the same dns number so i know the issue is the the new oct 29th version of yt-dl
i am new to downloading streams so i have no idea how to fix problems like this

diff --git a/youtube_dl/extractor/fox.py b/youtube_dl/extractor/fox.py
index 5f98d017b..75322f93e 100644
--- a/youtube_dl/extractor/fox.py
+++ b/youtube_dl/extractor/fox.py
@@ -98,7 +98,7 @@ class FOXIE(AdobePassIE):
         urlh = self._request_webpage(HEADRequest(release_url), video_id)
         video_url = compat_str(urlh.geturl())

-        if UplynkPreplayIE.suitable(video_url):
+        if False and UplynkPreplayIE.suitable(video_url):
             info.update({
                 '_type': 'url_transparent',
                 'url': video_url,

fixes it.

And for Adobe Pass locked stuff. (Yes, I'm aware this patchset is awful, but I'm very new to the ytdl codebase, and I don't have the time to contribute. But I fixed this up and figured I should push back the changes, in case someone more knowledgable and patient can get it merged.)
diff --git a/youtube_dl/extractor/adobepass.py b/youtube_dl/extractor/adobepass.py
index 231b8a57e..0776c0587 100644
--- a/youtube_dl/extractor/adobepass.py
+++ b/youtube_dl/extractor/adobepass.py
@@ -1368,10 +1368,10 @@ class AdobePassIE(InfoExtractor):
                 })

         def raise_mvpd_required():
-            raise ExtractorError(
+            print(
                 'This video is only available for users of participating TV providers. '
                 'Use --ap-mso to specify Adobe Pass Multiple-system operator Identifier '
-                'and --ap-username and --ap-password or --netrc to provide account credentials.', expected=True)
+                'and --ap-username and --ap-password or --netrc to provide account credentials.')

         def extract_redirect_url(html, url=None, fatal=False):
             # TODO: eliminate code duplication with generic extractor and move
@@ -1436,7 +1436,7 @@ class AdobePassIE(InfoExtractor):
                     elif 'automatically signed in with' in provider_redirect_page:
                         # Seems like comcast is rolling up new way of automatically signing customers
                         oauth_redirect_url = self._html_search_regex(
-                            r'continue:\s*"(https://login.comcast.net/oauth/authorize\?.+)"',
+                            r'continue:\s*"(https://oauth.xfinity.com/oauth/authorize\?.+)"',
                             provider_redirect_page, 'oauth redirect (signed)')
                         # Just need to process the request. No useful data comes back
                         self._download_webpage(oauth_redirect_url, video_id, 'Confirming auto login')
diff --git a/youtube_dl/extractor/fox.py b/youtube_dl/extractor/fox.py
index 75322f93e..5a0ba85ea 100644
--- a/youtube_dl/extractor/fox.py
+++ b/youtube_dl/extractor/fox.py
@@ -1,6 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import urllib
+
 from .adobepass import AdobePassIE
 from .uplynk import UplynkPreplayIE
 from ..compat import compat_str
@@ -47,6 +49,11 @@ class FOXIE(AdobePassIE):
     def _real_extract(self, url):
         video_id = self._match_id(url)

+        resource = self._get_mvpd_resource(
+            'fbc-fox', "null", video_id, "null")
+        query_auth = self._extract_mvpd_auth(
+            url, video_id, 'fbc-fox', resource)
+
         video = self._download_json(
             'https://api.fox.com/fbc-content/v1_4/video/%s' % video_id,
             video_id, headers={
@@ -58,6 +65,9 @@ class FOXIE(AdobePassIE):
         title = video['name']
         release_url = video['videoRelease']['url']

+        if query_auth:
+            release_url = release_url + "&auth=" + urllib.quote(query_auth)
+
         description = video.get('description')
         duration = int_or_none(video.get('durationInSeconds')) or int_or_none(
             video.get('duration')) or parse_duration(video.get('duration'))
@@ -76,10 +86,6 @@ class FOXIE(AdobePassIE):
         episode_number = int_or_none(video.get('episodeNumber'))
         release_year = int_or_none(video.get('releaseYear'))

-        if data.get('authRequired'):
-            # TODO: AP
-            pass
-
         info = {
             'id': video_id,
             'title': title,

I also needed https://github.com/rg3/youtube-dl/pull/11258 with mekantor's fix in https://github.com/rg3/youtube-dl/issues/11257

...apparently above patchset is a duplicate of https://github.com/rg3/youtube-dl/pull/14489/files. RIP.


Introduced AbstractStratPers as new supertype

The introduction of the abstract supertype AbstractStratPers allows the user to simplify dispatch on StratPers from TwoLevel and TwoLevelTree.
In general, it can be beneficial to use the same approach for RepPers and OpScens. In this case, we however have to consider how we handle TimeStructInnerIter and TimeStructInnerOuter. It can be seen as an extension to remove unnecessary code.
Additionally, I fixed a minor bug I ran aground regarding eachindex(StratPers).

LGTM. Maybe we should make a new release before starting on RepPers and OpScens improvements?

Yeah, I would say so. The changes there are in fact only related to internal routines. These are by no means a necessity to fix fast.