TabControl Spec

Spec review for TabControl.

@YuliKl @mdtauk @mrlacey @Poopooracoocoo @MikeHillberg @teaP
Thank you all for the great feedback, suggestions, and ideas. I replied to a bunch of the open threads in the PR, but there are still a handful that are outstanding. The ones that are open but haven't been replied to are the ones that I need to think more about -- working on it!
Additionally I've compiled some open questions at the end of the doc for discussion. We'll be discussing internally (and I'll be sure to reply with our thoughts!), but I'd also love to know your thoughts on the following open questions:

Should TabControl support tabs that don't close?
Does TabControl need two areas to the right of the tab pane (as spec'd) or can there just be one content area?
The spec outlines several keyboard behaviors which may be considered "non-standard". Should those be built into the control directly?
Compact mode isn't required for Terminal v1 - can we push this feature out to v2?
Do we need a TabClosed event? It would be useful from a consistency point of view, but is it required for v1?

Tab Control Team Spec Review Summary:
Hi all! Following up on the discussion topics from the Open Questions list. I'll be updating the spec to reflect the changes discussed here and feedback from the PR review.

TODO: Add sample for tab recombination. There is some nuance where tearing out the last tab of Window A and recombining it into an existing Window B should close the window A.

Q: Should TabControl have a global knob for making the whole set of tabs not closeable?

A: No. No global knob is needed, as individual tabs can still mark themselves as not closeable.

Q: Could Ribbon be built with TabControl? Should it even be called TabControl?

A: TabControl is being built specifically to support document style tabs and the first customer is the Terminal. There are a set of things that the proposed control won’t support yet (like positioning, rows of tabs, etc.). Rather than sitting on the “good name” (TabControl) the proposed change was to update the name of the control to “DocumentTabControl” or something to help tip developers that this might not be the tabs they are looking for. To that end, Ribbon is an “out of scope” scenario.

Q: Should we provide a “+” button in the control itself?

A: Yes. We will provide a “+” button in the control. It will have the following capabilities: 1) it can be turned off; 2) it will fire an event when clicked

Q: Should we have two areas to the right of the tab pane (as spec’d) or can there just be one content area?

A: Inconclusive. We discussed that having two areas would be an easier app dev experience, but there are questions about how to properly lay out the two areas. We will do a side-by-side comparison between the two areas and the one area to figure out the delta in the developer experience.

Q: The spec outlines several keyboard behaviors which may be considered "non-standard". Should those be built into the control directly?

A: Inconclusive. We definitely don’t want the tab control to “eat” keyboard presses that the app may use – and the CTRL+# combination is expected to be fairly common. Maybe we could have an API to opt-in to the keyboard behaviors? We will have another followup checkpoint specifically on keyboarding and accessibility.

Q: Compact mode isn't required for Terminal v1 - can we push this feature out to v2?

A: Yes. This isn’t a P0 requirement and can be cut for now.

Q: Do we need a TabClosed event? It would be useful from a consistency point of view, but is it required for v1?

We didn’t discuss this.

Does code for the TabControl spec'd here exist yet or is this just a spec for a control to be written in the near future?  I'd like to take it for a spin, even if I need to compile from a branch to do so.

Does code for the TabControl spec'd here exist yet or is this just a spec for a control to be written in the near future? I'd like to take it for a spin, even if I need to compile from a branch to do so.

The code exists, but not in the publically released package.  The new Windows Terminal repo is using it and "dogfooding" it.

Does code for the TabControl spec'd here exist yet or is this just a spec for a control to be written in the near future? I'd like to take it for a spin, even if I need to compile from a branch to do so.

@zodmanperth TabView should be available as a preview API in the current prerelease NuGet package:
https://www.nuget.org/packages/Microsoft.UI.Xaml/2.2.190611001-prerelease
Although it will be a bit behind source branches, e.g.:
https://github.com/microsoft/microsoft-ui-xaml/tree/user/teaP/TabViewAdditions

Thanks folks, I'm playing with the TabView control now.
I do have a follow-up question; is the TabView control in Microsoft.UI.Xaml the control where this spec will be applied, or can I expect to see a TabControl to emerge in the near future?
For background, we've got "urgent" need for multiview functionality and are evaluating tear-away tabs as a potential solution.  We would like to contribute and/or help guide development driven by our real-world use case to ensure success (for everyone).

@ZodmanPerth - thank you for trying out the preview control and "kicking the tires" 😀 I'd love to hear any feedback you have about TabControl.

I do have a follow-up question; is the TabView control in Microsoft.UI.Xaml the control where this spec will be applied, or can I expect to see a TabControl to emerge in the near future?

The exact name/API is still being finalized. We will continue to develop and iterate on the TabView control (ie. there won't be both a TabView and a TabControl - we'll pick one name and update TabView to that name).

For background, we've got "urgent" need for multiview functionality and are evaluating tear-away tabs as a potential solution. We would like to contribute and/or help guide development driven by our real-world use case to ensure success (for everyone).

We'd love to hear your thoughts and get your feedback. TabView is still in preview as we iterate, so please be aware that functionality and names are subject to change. However, this is also a great time in the control development lifecycle to provide feedback/contributions.

We'd love to hear your thoughts and get your feedback. TabView is still in preview as we iterate, so please be aware that functionality and names are subject to change. However, this is also a great time in the control development lifecycle to provide feedback/contributions.

Thank @stmoy .  I've had a good look at the TabView TearOff Sample and initial impressions are good.  I like how customisable the UI of the control is, and the events for managing tear off seem logical.  Progress so far looks promising.
At present we have two concerns:

We want to provide the same tear-away tab experience as Edge does; being able to drag and position the new window on a secondary monitor.  We're concerned about Known issue #1 which states it's not currently possible.  What are the chances of this being possible before Win UI 2.2 drops?  Or Win UI 3.0?  Or beyond?

AppWindow does not have native support for DirectX (see Remarks from the docs) which may prevent some of our drawing scenarios.  Further investigation needs doing on our side, however development of tear-off windows for the TabControl appears to be moving with AppWindow as a focus.  I do hope support for this edge case is not lost in the rush to complete TabControl; it would be disappointing to see our first concern addressed only to find we cannot use TabControl because of dependence on AppWindow to complete the use case.

I know you're all working hard to get this ready; we really appreciate your effort!
We hope you will take our concerns with you as you make decisions for this exciting control.  Users of our software have been asking for this feature for some time and we would love to be able to provide it for them using the TabControl (or whatever it ends up being named).

@ZodmanPerth - thank you for the feedback! Here are my thoughts on the two concerns raised:

We want to provide the same tear-away tab experience as Edge does; being able to drag and position the new window on a secondary monitor. We're concerned about Known issue #1 which states it's not currently possible. What are the chances of this being possible before Win UI 2.2 drops? Or Win UI 3.0? Or beyond?

Tab tear-away is an important feature for this control and is considered a P0 scenario for v1. We just haven't built it in yet.

AppWindow does not have native support for DirectX (see Remarks from the docs) which may prevent some of our drawing scenarios. Further investigation needs doing on our side, however development of tear-off windows for the TabControl appears to be moving with AppWindow as a focus. I do hope support for this edge case is not lost in the rush to complete TabControl; it would be disappointing to see our first concern addressed only to find we cannot use TabControl because of dependence on AppWindow to complete the use case.

I'll bubble this concern up to the AppWindow team. The control itself doesn't have a coupling to AppWindow - I believe that you could achieve a similar experience using CoreApplicationView. @michael-hawker , do you have an example of this?

Thanks @stmoy.
I've now completed a spike for how we would use AppWindow and it looks like we may be OK.  It would be great to discuss our needs directly with the AppWindow team if anyone from that team wants to reach out.
All our concerns appear to be being addressed.
Best of luck with the development of the TabControl.  I look forward to seeing what you come up with.

Wanted to give my feedback on the name.
Currently XAML Controls don't use Control for names. There's no NavigationControl, FlipControl, ListControl, GridControl, etc. It's all NavigationView, FlipView, ListView, GridView.
So I believe TabView would be more intuitive and more consistent with other controls name.

Wanted to give my feedback on the name.
Currently XAML Controls don't use Control for names. There's no NavigationControl, FlipControl, ListControl, GridControl, etc. It's all NavigationView, FlipView, ListView, GridView.
So I believe TabView would be more intuitive and more consistent with other controls name.

Agreed.  I was about to say that for the cases that do have a name that ends in "Control" are more low-policy controls (i.e. ContentControl, ItemsControl), but then I remembered we had a RatingsControl.  I'll defer to @MikeHillberg on naming conventions.  :)  I suspect that one was a case of aligning the name with what was already the prevailing name used elsewhere.

@vitorgrs @micahl - very good points. This control is expected to take up the entire page, similar to NavigationView, so I agree that TabView is probably the way to go!

The ListView within the TabView, should be given appropriate Add, Remove, Re-arrange animations for TabViewItems being added, removed or dragged.
Edge Canary and Dev has a flag for a new opening animation - which should probably be the basis for the transitions used by the XAML control.
edge://flags/#new-tab-loading-animation

The ListView within the TabView, should be given appropriate Add, Remove, Re-arrange animations for TabViewItems being added, removed or dragged.
Edge Canary and Dev has a flag for a new opening animation - which should probably be the basis for the transitions used by the XAML control.
edge://flags/#new-tab-loading-animation
https://youtu.be/jn_pUsnKwgM

Thank you for calling out the animations and also the video. Naively, the Tab control will end up having the animations from ListView by default, since TabView contains a ListView. That said, the Edge Canary animations look much more appropriate than the ListView animations - we'll need to investigate how we can make these animations happen.

@mdtauk. That's not an Edge exclusive flag. That flag changes the loading spinner on the favicon so you can see it while it's loading. Before there was just a spinner.
(I actually wrote this comment tons better but since I'm using firefox focus and opened your video in a new tab, my comment was cleared)

@mdtauk. That's not an Edge exclusive flag. That flag changes the loading spinner on the favicon so you can see it while it's loading. Before there was just a spinner.
(I actually wrote this comment tons better but since I'm using firefox focus and opened your video in a new tab, my comment was cleared)

@Poopooracoocoo There are slight tweaks to the movement, but also how the Icon animates into view.  The ProgressRing control is changing by WinUI 3.0 - so hopefully it will work a little better in the tab header, compared to the current version.

Hi all! I wanted to provide a super quick update and let you know what's coming next. TL;DR - the feedback provided has helped evolve and solidify the spec to a point where we want to create a "checkpoint of truth" 😉, so I will be completing this PR and submitting the spec to our internal API review board. If you have more comments/feedback, you can still provide them here - we just wanted to give the review board a "clean" PR so they can focus specifically on the spec.
We are actively developing TabView (you can check out its progress on the main WinUI repo). Once the API review is completed, we can start the process of moving TabView from "preview" to "release".
Thank you again for all of your comments/suggestions/feedback! I know there are some features that didn't make it in for v1, so keep an eye out for a v2 tracking issue soon!

Will the updated spec be published on GitHub to view?
At the moment the current Spec uses images from UWP Edge, the current control design, and a design with selected indicator.
Will the proposed actual design be included, or is that beyond the scope of the spec document?

Good questions! This is a pretty new process for the team so we're still trying to flesh a few things out. With that in mind...

Will the updated spec be published on GitHub to view?

The current spec (as it exists in this PR) is the most up-to-date spec for v1 (sans my answer to your next question 😉)

At the moment the current Spec uses images from UWP Edge, the current control design, and a design with selected indicator.
Will the proposed actual design be included, or is that beyond the scope of the spec document?

Yup, you've touched on an area that is still being actively iterated upon. We're working with our design team to come up with better and more representative images for the spec and docs. Since the next step in the process is API review, and the API board isn't as concerned with the visual spec, we didn't want to block on that piece. Before we check into master, we'll have more representative images of what our actual POR is.

@stmoy  Thank you for that.  I think I have seen others "cleaning" up Specs and Proposals for pitching to teams - so wasn't sure if that is what happens with API reviews also.
I am sure the Windows Terminal team are looking forward to getting their hands on a V1 release, and addressing the many design related feedback items they have received.
I look forward to seeing how First Party apps implement it, or even the Windows Shell itself (I'm looking at you Property Dialog) 😉


Move max-messages and max-runtime into subscribers

In the consumer the max-messages and max-runtime configuration is quite ugly IMO. I would add a cycle event to the end of the while, so these checks are:

optional, only done when a subscriber is added
removed from consumer making it much cleaner

What do you think @henrikbjorn?

I have plans on refactoring the Consumer in a way that would allow using the pthreads extension. Also this makes it impossible to configure this stuff from the cli.


kivy.org https certificate expired

Is your feature request related to a problem? Please describe.
The website, docs, and all subdomains (e.g. chat.kivy.org) are not loadable with Firefox (and maybe more browsers), because HSTS is activated and the https certificate is expired. In Chrome you'll get a warning and need to manually confirm that you still want to load the ("unsafe") website.
Describe the solution you'd like
Renew https certificate

Thanks for the heads’ up, this is now resolved.


Water filter still applied

The underwater filter is still applied when your body is in water

Not able to reproduce this one... any ideas?

I got a screenshot of it.

Get a friend(s)
Get mine-shaft
Drown yourself in the pools of water
Switch to the other person(s)

I would recommended checking if the target for spectating is in water before applying the filter

Seems that this can't be fixed using the regular detection method as that depends on a PlayercontrollerB instance, which doesn't exist upon death.
I think an acceptable compromise is using the CauseOfDeath attribute as there are very few cases where a body can be recovered from a drowned player, so most likely it will remain underwater.
Worst case scenario the filters can be disabled entirely upon death, but maybe someone has a better solution?

Should be fixed in latest commit


[WIP] Check Hooks

The Sensu client’s primary function is to execute check commands and publish their results for processing, allowing a Sensu server to take the appropriate actions. The practice of using Sensu for auto-remediation tasks and automated data gathering for incident triage continue to gain traction amongst Sensu users. Although these practices are achievable, Sensu does not currently make it easy, as all actions must be orchestrated by a Sensu server.
This pull request introduce check hooks, commands run by the Sensu client in response to the result of the check command execution. The Sensu client will execute the appropriate configured hook command, depending on the check execution status (e.g. 1). Valid hook names include (in order of precedence): "1"-"255", "ok", "warning", "critical", "unknown", and "non-zero".
Example check hook configuration:
{
  "checks": {
    "my_check": {
      "command": "echo WARNING && exit 1",
      "...": "...",
      "hooks": {
        "ok": "echo ALL IS WELL",
        "warning": "echo DANGER WILL ROBINSON!",
        "critical": "ps aux"
      }
    }
  }
}

Now to get feedback from users 🎉
YARDOC and sensu-settings validation to come tomorrow.

For posterity, I did consider using an array of hook commands, e.g. "warning": [{"command": "..."}, {"command": "..."}], in order to support multiple sequential executions (if the previous exited 0), however, it adds unnecessary complexity as the same can be achieved with existing (and more powerful) shell operators, e.g. && and ||.

@portertech agreed that really we can just rely on shell/ruby operators to accomplish different that.
Regarding different hook types when we discussed this (yes a while ago) I could think of two basic types of hooks: command and http(s). Are you thinking that we would just write a ruby, shell, python, etc script to retrieve data (such as $HOST:PORT/metrics) or should we have a bit more native approach?
If so I was thinking something along these lines:
{
  "checks": {
    "my_check": {
     "hook_type" : "(command|http)"
      "hook_exec": "echo WARNING && exit 1",
      "...": "...",
      "hooks": {
        "ok": "echo ALL IS WELL",
        "warning": "echo DANGER WILL ROBINSON!",
        "critical": "ps aux"
      }
    }
  }
}

Basically just whether I need to either write wrapper scripts, use curl $HOST:$PORT/$PATH, or if we should support an HTTP endpoint natively.

To be clear I am not saying we should support it natively just asking the question. An argument to be made against it is that now we basically have to support every header option under the sun as someone will ask for it.

@majormoses I am indeed thinking that we will write and use shell, ruby, golang, python, etc scripts to retrieve data and do things 👍

"echo DANGER WILL ROBINSON!"

I am not gonna lie I kinda want to add this to all my monitoring now.

@davideauee I have update the PR with hook "stdin" support, an option to write JSON serialized Sensu client and check data to the hook command process via STDIN. The JSON data closely resembles the Sensu event data that goes through the event pipeline on Sensu servers. I avoided any reference to "events", as check hooks are not intended to be client side event handlers.

We have an opportunity to add "stdin" to checks, providing optional client and check data to check plugins.

So this is implemented now and this is the way to define them?
{
"checks": {
"my_check": {
"command": "echo WARNING && exit 1",
"...": "...",
"hooks": {
"ok": {"command": "echo ALL IS WELL"},
"warning": {"command": "echo DANGER WILL ROBINSON!"},
"critical": {"command": "ps aux"}
}
}
}
}

Its not in the example docs (would love a PR to update them:
{
  "checks": {
    "my_check": {
      "command": "echo WARNING && exit 1",
      "...": "...",
      "hooks": {
        "ok": {
          "command": "echo ALL IS WELL",
          "timeout": 1
        },
        "warning": {
          "command": "echo DANGER WILL ROBINSON!",
          "timeout": 5
          },
        "critical": {
          "command": "ps aux",
          "timeout": 10
          }
      }
    }
  }
}```


improv(tests): adopt aws cdk cli lib

Description of your changes

Note
This PR supersedes #1624 and proposes a much reduced scope compared to that PR.

This PR introduces a new testing utility that at the moment contains only a new class called TestStack. This class is a simple wrapper around the @aws-cdk/cli-lib-alpha module and provides only the minimum amount of methods needed to synth, deploy, and destroy a CDK Stack.
This driver behind this change, as detailed in the linked issue, stems from the reliance of our current integration/e2e tests on an internal CDK command line which has changed in a recent release and thus has locked us out of newer and patched versions of CDK.
Since our first implementation of the integration tests over a year ago, CDK has released a new package (@aws-cdk/cli-lib-alpha) that allows to interact with the CLI programmatically, and that can replace our current implementation. Even though the package is in alpha, we consider this still to be an improvement over reliance on internal API, and we hope that through our usage we can help shape the package.
As mentioned above, this is the second PR that tries to apply this change. The previous PR was much larger in scope and in addition to introducing the new CLI wrapper, it also added a ton of new utilities. After discussing internally we have however decided to leave these changes out and work on a RFC before attempting to implement them.
The changes in this PR constitute a stop-gap solution that allows us to move away from the internal CDK API and access current versions of CDK. Once the RFC mentioned will be out and discussed, I am open to decommission the changes introduced this PR in favor of a more solid implementation.
While working on the PR I prioritized keeping changes in the codebase as low as possible, even though there are several areas (i.e. standard resource factories) that could have resulted in a more streamlined implementation.
Detailed list of changes:

Removed of all aws-cdk-related (i.e. http-proxy, vm2, etc.) dependencies from the project
Removed cdk-cli util from commons [1]
Created a new internal testing utility
Added aws-cdk-lib as dependency scoped to the testing module
Added aws-cdk & aws-cdk-lib as scoped dependencies to the layers package [2]
Minimal changes to tracer e2e tests to use the new package + added the new package as devDependency
Minimal changes to logger e2e tests to use the new package + added the new package as devDependency
Minimal changes to metrics e2e tests to use the new package + added the new package as devDependency
Minimal changes to idempotency e2e tests to use the new package + added the new package as devDependency
Minimal changes to parameters e2e tests to use the new package + added the new package as devDependency
Changes to layers e2e tests to use the new package + added the new package as devDependency [3]

Notes:
[1] This was an internal module that was never published to npm, so I don't think it constitutes a breaking change
[2] Since the CLI is used as an actual CLI in the CI for this package, we don't need the additional dependencies and we are not relying on any internal
[3] The diff here is slightly higher compared to the other packages because this e2e test deploys two CDK apps/stacks
Related issues, RFCs

Issue number: #1623
Checklist

[x] My changes meet the tenets criteria
[x] I have performed a self-review of my own code
[x] I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my change is effective and works
[x] The PR title follows the conventional commit semantics

Breaking change checklist
Is it a breaking change?: NO

[ ] I have documented the migration process
[ ] I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Integration test, using the new module, ran successfully: https://github.com/aws-powertools/powertools-lambda-typescript/actions/runs/5682665966

Also, SonarCloud is going to most probably bring up a ton of findings because the PR touches many existing files that have a lot of code.
I will happily address any finding on the newly added code (packages/testing/*), but in order to keep the diff contained & in light of what discussed in the PR body as well as the fact that these are internal test files, I would suggest to not dedicate time to rework other findings.

I will happily address any finding on the newly added code (packages/testing/*), but in order to keep the diff contained & in light of what discussed in the PR body as well as the fact that these are internal test files, I would suggest to not dedicate time to rework other findings.

I agree, let's address the changes in a dedicated PR.


spl-token metadata is missing wrappers for unverify_collection and unverify_sized_collection_item

spl-token metadata spl/src/metadata.rs is missing wrappers for unverify_collection and unverify_sized_collection_item

I created them in my own repo, I can create a PR
https://github.com/coral-xyz/anchor/compare/master...web3omega:anchor:master

Sure, PRs welcome but please add a small test for each if possible

It seems there are no tests for SPL or spl-token-metadata - yet. Only some token-proxy tests.
https://github.com/coral-xyz/anchor/tree/master/tests/spl
I can derive some tests from my own project repo to here as I'm already using these new functionality locally. Will do it later this week.


Allow clip paths to nest in the canvas API.

Closes #372.

@bors-servo: r+

:pushpin: Commit c8a6d8e has been approved by pcwalton

:hourglass: Testing commit c8a6d8e1f7feee3ff4521838ff2b4d1402b058e7 with merge e48b2759dd6aebd7d682fd7ce550dec7bcea7fd4...

:sunny: Test successful - checks-travis
Approved by: pcwalton
Pushing e48b2759dd6aebd7d682fd7ce550dec7bcea7fd4 to master...


The ported jsagent7 gets the wrong gid on first start

When you start the ported jsagent7 with the correct gid, the controller always identifies it as gid = 0
Currently the only way to fix it is:

Remove the node model from osis (on agentcontroller machine)
Restart both the controller and the agent and then it gets assigned the correct gid

This issue only happens when running this agent for the first time.

Fixed:
https://github.com/Jumpscale/jumpscale_core8/commit/0b30504d2dff009b74dfa4c2cedd7be24336eb38


libmodsecurity api issue

Describe the bug
When I was using libmodsecurity v3 c api, I didn't find any documentation on this aspect. May I ask if there is an api to get the detection result synchronously, or documentation on this aspect.
Expected behavior
I hope there is documentation in this regard, or tell me the api function of synchronous or asynchronous callback.
Server (please complete the following information):

ModSecurity version (and connector): ModSecurity v3.0.4
OS (and distro): Centos7

Rule Set (please complete the following information):

Running any public or commercial rule set? OWASP modsecurity
What is the version number? latest

Hi @Magicskys,
The API documentation is built along with the code. You can use Doxygen to transform it into whatever feels more comfortable to you.
Here is an example of a piece of documentation:
https://github.com/SpiderLabs/ModSecurity/blob/a2be19d8e53f4d2d90d435e187086afa2396c373/headers/modsecurity/modsecurity.h#L18-L57
The C API is just a wrapper for the C++, the same documentation is valid for both, the specificities are noted on the C wrapper.
Atop of the documentation, there are some examples on the examples folder:

multiprocess_c
simple_example_using_c
using_bodies_in_chunks

More: examples


Fix wrong title in recommendation's page

The ======= doesn't seems to be in the good place.

Good catch; thanks!


fix: resetWidget doesn't reset child widgets inside a list

Description

Fixes issue with resetWidget platform function not honouring resetChildren argument. Also, takes care of resetting private widgets inside a list widget.
These bug was introduced in #29151, where we moved the resetWidget logic to the worker thread, to support restoration of default values in meta fields (post setter methods invocation).

PR fixes following issue(s)
Fixes #30116
Type of change

Bug fix (non-breaking change which fixes an issue)

Testing

How Has This Been Tested?

[x] Manual

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

Summary by CodeRabbit

Refactor

Streamlined evaluation logic for better performance and maintainability.

New Features

Enhanced widget reset functionality with improved cache handling.
Upgraded library installation process for JavaScript libraries within the app, ensuring more reliable and consistent performance.

/ok-to-test

/ok-to-test<EMAIL_ADDRESS>

/build-deploy-preview

Tested and verified! Changes look good to me.

/ok-to-test<EMAIL_ADDRESS>


Implement keyboard smash

Epic: #14
Description
Migrate keyboard smash function to the playground project
Output
(dataframe original, dataframe result)
dataframe original: dataframe with the data will be processed (string)
dataframe result: dataframe with result of keyboard smash features about dataframe original
Checklist

[ ] Add char_frequency feature
[ ] Add special_character_sequency feature
[ ] Add number_sequency feature
[ ] Add vowels_sequency feature
[ ] Add consoants_sequency feature

Pre processing -

the pre-processed addresses database + keyboard smashing features for STREET_ADDRESS_1 and STREET_ADDRESS_2 concatenated

remove null values

Features extracted:

street_address_sequence_vowels_concat
street_address_sequence_consonants_concat
street_address_sequence_special_characters_concat
ksmash_number_address_concat
char_counter_street_address_concat
Word Embedding

Model

Random forest


Custom message headers not propagated when using scheduling API

As discussed on discord...
It seems as if the recurring scheduling API doesn't propagate custom message headers as it normally would when using Send, Publish or even SchedulePublish.
Consider this example:
var message = new { __Header_Foobar = "some custom header value" };
await _bus.ScheduleRecurringSend<ICleanupExpiredBaskets>(destinationQueue, schedule, message);

The Foobar header is not propagated on the scheduled message.
{
    "messageId": "7c590000-493b-e4e7-dc8e-08d9b999240c",
    "correlationId": "7c590000-493b-e4e7-80d4-08d9b9992396",
    "conversationId": "7c590000-493b-e4e7-98f5-08d9b9992492",
    "sourceAddress": "amazonsqs://eu-west-1/XXX_XXX_bus_xtcoyynj8x1qqyfubdc5ugjnfy?durable=false&autodelete=true",
    "destinationAddress": "amazonsqs://eu-west-1/MassTransit_Scheduling-ScheduleRecurringMessage?type=topic",
    "messageType": [
        "urn:message:MassTransit.Scheduling:ScheduleRecurringMessage"
    ],
    "message": {
        "correlationId": "7c590000-493b-e4e7-80d4-08d9b9992396",
        "schedule": {
            "misfirePolicy": 1,
            "timeZoneId": "Romance Standard Time",
            "startTime": "2021-12-07T16:49:22.6434607+01:00",
            "scheduleId": "Sales.Application.Schedules.CleanupExpiredBasketsScheduler+CleanupExpiredBasketsSchedule",
            "scheduleGroup": "Sales.Application",
            "cronExpression": "*/10 * * * * *"
        },
        "payloadType": [
            "urn:message:Sales.Contract.Commands:ICleanupExpiredBaskets"
        ],
        "destination": "queue:to-cleanup-expired-baskets-handler",
        "payload": {}

    },
    "sentTime": "2021-12-07T15:49:23.5593358Z",
    "headers": {},
    "host": {
        "machineName": "XXX",
        "processName": "XXX",
        "processId": 28740,
        "assembly": "XXX",
        "assemblyVersion": "<IP_ADDRESS>",
        "frameworkVersion": "5.0.12",
        "massTransitVersion": "<IP_ADDRESS>",
        "operatingSystemVersion": "Microsoft Windows NT 10.0.18363.0"
    }
}

This occurs on both regular and recurring schedulers using the endpoint message scheduler.

Also mentioned, there's a work around by adding the header via a custom IPipe.
var message = new {  };
var pipe = new FoobarHeaderPropagator("some custom header value");
await _bus.ScheduleRecurringSend<ICleanupExpiredBaskets>(destinationQueue, schedule, message, pipe);

...

private class FoobarHeaderPropagator : IPipe<SendContext>
{
	private readonly string _foobar;

	public FoobarHeaderPropagator(string foobar)
	{
		_foobar = foobar;
	}

	public async Task Send(SendContext context)
	{
		await Task.CompletedTask;
		context.Headers.Set("Foobar", _foobar);
	}
}

This will be resolved in MassTransit v8 (because I just finished coding it and testing it).
Headers will also work in state machines via Init<T> now as well.


[CS2103-F09-4] ReadyBakey

ReadyBakey assists home bakers with order management. It simplifies all the necessary requirements and information into a central application.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@8d5b66e). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #7   +/-   ##
=========================================
  Coverage          ?   72.15%
  Complexity        ?      399
=========================================
  Files             ?       70
  Lines             ?     1232
  Branches          ?      125
=========================================
  Hits              ?      889
  Misses            ?      311
  Partials          ?       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d5b66e...90a98c1. Read the comment docs.


Sync service complains about multiple connections from the same client

Multiple people have reported this happen for different reasons.
Here's one such report:
When using the Dart client, after the app closes and Electric considers the client disconnected because of not responding to the ping, Electric will complain about multiple connections from the same client.
How is that possible? Maybe the service is not clearing the client_id connection that was closed due to the unresponded ping?
-- App disconnects, without gracefully closing the socket connection --
-- After a while, the server disconnects the client --

electric-1  | 16:00:17.172 pid=<0.4058.0> client_id=8c120869-9cc5-43cb-b8bb-c042e747e07a instance_id=4f15a93c-91b1-4f99-98da-3e916829b330 user_id=6bdd56fa-acd7-4349-8de8-7735859494bf [info] Client is not responding to ping, disconnecting

-- User reopens the app and reconnects --

electric-1  | 16:00:26.029 pid=<0.4063.0> [info] GET /ws
electric-1  | 16:00:26.030 pid=<0.4063.0> instance_id=4f15a93c-91b1-4f99-98da-3e916829b330 [info] Sent 101 in 343µs
electric-1  | 16:00:27.351 pid=<0.4063.0> client_id=8c120869-9cc5-43cb-b8bb-c042e747e07a instance_id=4f15a93c-91b1-4f99-98da-3e916829b330 [info] attempted multiple connections from the same client

Issue report on the Dart client with full client and server logs: https://github.com/SkillDevs/electric_dart/issues/23

👋 we've been working the last month on a rebuild of the Electric server over at a temporary repo https://github.com/electric-sql/electric-next/
You can read more about why we made the decision at https://next.electric-sql.com/about
We're really excited about all the new possibilities the new server brings and we hope you'll check it out soon and give us your feedback.
We're now moving the temporary repo back here. As part of that migration we're closing all the old issues and PRs. We really appreciate you taking the time to investigate and report this issue!


Added guide when tfx missing

Added information on, how to get tfx command installed, in order to generate the extension

@cpoDesign : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@willsmythe Could you review this pull request? Thanks.

@willsmythe @ktoliver Let me know if you need anything else

@willsmythe Will, could you review this PR from October 2019?
@aaronhallberg

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@mijacobs : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@mijacobs: I'm sorry - only the author of this article, @apawast, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

Thanks @mijacobs; merging now.


MWEM laplace noise

in mwem.py:
lap = self._laplace(
                    (2 * self.iterations * len(self.histograms)) / (self.epsilon * len(dimensions))
                )

why is len(dimensions) in the denominator?
I have a 4D histogram (dataset with 4 attributes), where the sensitivity of a single query is still 1 (or 2), then you apply composition over all iterations (self.iterations), and use the exponential mechanism (hence factor 2 in the nominator). But why do we have to divide by the number of dimensions (in my case, 4)?

Hmm, I don't remember why we did this.  It doesn't seem to be necessary per the paper [2].  @lurosenb, do you recall why we reduce the budget by a factor of the number of dimensions in the line at [1]?
[1] https://github.com/opendp/smartnoise-sdk/blob/main/synth/snsynth/mwem.py#L266
[2] https://www.cs.huji.ac.il/~katrina//papers/mwem-nips.pdf

Update on this; I am pretty sure you are correct.  We will be deploying a method to test the empirical privacy of our synthesizers, and will fix this when we can show empirically with that tool that we can safely less noise and stay within the overall privacy budget.

Apologies for the delay on this!
You are correct, I believe this is an artifact of a faulty sensitivity calculation for linear range queries. I'll hot fix this immediately - thanks for pointing it out!

Please approve https://github.com/opendp/smartnoise-sdk/pull/474 @joshua-oss

Thanks for the fix, @lurosenb!  Approved, merged, and released to PyPi


Video Upload and Detail Page

Description
You are required to create the form that will enable creators upload a Video, and a corresponding page where the uploaded video can be viewed.

[ ] Create two model classes for the Video and Comment
[ ] Create the form and views that will handle Video upload
[ ] Make a research on how to restrict video uploads to a max of 15 mins using either client side (Javascript) or backend (Python) techniques.
[ ] Create a detail view for a Video instance
[ ] Enable addition of comments to a Video by users

The Video upload screen for an idea is here: https://www.figma.com/file/HmFzbwjxQiDFUxPYxsp8Q6/Col-films-designs-(Copy)?node-id=2557%3A15720

The Video detail screen is available here: https://www.figma.com/file/HmFzbwjxQiDFUxPYxsp8Q6/Col-films-designs-(Copy)?node-id=2540%3A14223

@Klinteen Please ask any further questions here if you need clarifications or guides on how to work on this feature.


doc: external components: Introduce a section about external tooling

Since tools that are not software that is compiled into the final images may have very different requirements from regular external components, introduce a new section outlining the process and rules to introduce new tools into the project.

End users seeking any certification level may need to know the qualification level achieved by each tool they use. Within the Zephyr ecosystem this should be consistently reported. The guidelines on tool integration should cover how this is done.
Similarly, each tool should have some preliminary documentation within the Zephyr documentation structure. This documentation should exist within the repository of each tool to avoid extraneous content when the tool is not included in the workspace.

I really do not have enough knowledge to document or discuss this to be honest. Would you like to send a PR once this one is merged to add the requirements you mention?

approved by the TSC.


youtube-dl unable to download private vk video do to allegedly invalid credentials

Checklist

[x] I'm asking a question
[x] I've looked through the README and FAQ for similar questions
[x] I've searched the bugtracker for similar questions including closed ones

Question

WRITE QUESTION HERE
I am trying to download a video with my vk login credential but it fails do to invalid credentials. I am able to log in to my account using the same credentials that are returned in the failed youtube-dl pull request
here is the command line
youtube-dl -v -u <email/phone_number> -p <password>  -vg "https://vk.com/video-133441491_456261066"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', '-vg', 'https://vk.com/video-133441491_456261066']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.13.0-1023-aws-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
# I edit file /usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/vk.py with:
#     def _login(self):
#        username, password = self._get_login_info()
#        print("-----------------------")
#        print(username)
#        print(password)
-----------------------
<username> # Used this username (copy/paste) to log in to my account via web UI
<password>  # Used this password (copy/paste) to log in to my account via web UI
ERROR: Unable to login, incorrect username and/or password
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 533, in extract
    self.initialize()
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 437, in initialize
    self._real_initialize()
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/vk.py", line 66, in _real_initialize
    self._login()
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/vk.py", line 62, in _login
    raise ExtractorError(
youtube_dl.utils.ExtractorError: Unable to login, incorrect username and/or password

From what I found is that this should be working, and it is very likely that it is my credentials, but I was able to validate my credentials by login in with it to vk. Maybe my credentials need to wrapped in some chars like ' or " or some other char?

video does not require login:
https://vk.com/video-133441491_456261066

@89z that was the case until a few days ago, then I started to get this error
youtube-dl -v -vg "https://vk.com/video-133441491_456261066"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-vg', 'https://vk.com/video-133441491_456261066']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.13.0-1023-aws-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
-----------------------
None
None
ERROR: This video is only available for registered users. Use --username and --password or --netrc to provide account credentials.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/vk.py", line 328, in _real_extract
    payload = self._download_payload('al_video', video_id, data)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/vk.py", line 75, in _download_payload
    self.raise_login_required()
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 941, in raise_login_required
    raise ExtractorError(
youtube_dl.utils.ExtractorError: This video is only available for registered users. Use --username and --password or --netrc to provide account credentials.

As I just wrote in the previous issue:

As a rule, the login function of any extractor is liable to break because sites change the procedure, by adding captchas or whatever. If the login function doesn't succeed, or even always, omit -u .../--username ... and use --cookies ....

If, as I understand it, you disabled the login function, that applies in Spades.

@dirkf If I did disabled the login function it would be unintentional and unlikely. Where do I check for that? This comes from within a tool I written. That code has not changed. It worked in the beginning of the week. I think vk updated their back end and by doing so it broke youtube-dl (vk portion of it). The problem with this theory (vk changed its backend) is that I could not find any recent activity in changelog.
The original problem was not even a login. It failed to pull public video with an error saying that the video is private.
youtube_dl.utils.ExtractorError: This video is only available for registered users. Use --username and --password or --netrc to provide account credentials.
I tried using my credentials and it failed with them too.
youtube_dl.utils.ExtractorError: Unable to login, incorrect username and/or password
I tried redownloading the video that I was able to download before. It produced the same error.

You wrote:

# I edit file /usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py with:
#     def _login(self):
#        username, password = self._get_login_info()
#        print("-----------------------")
#        print(username)
#        print(password)

If that means you commented out the YoutubeDL._login() method, then anything could happen.
The extractor diagnostics are confusing because it doesn't suggest using the --cookies ... option which is the most likely to succeed when --username ... fails.
Anyhow there are further new site issues that will break VK extraction: see https://github.com/yt-dlp/yt-dlp/issues/4097.

oo no, sorry for the confusion I commented that out for the output showing this is what is happening in the background (and not a part of the stderr) and the credentials that are printed out I used to log in via UI.
Thank you for sharing the link.

Vk change login page. Cookies don't work.

They're using some really tight code to prevent downloads of their livestreams. It seems difficult for for any app or program to generate even a downloadable link or alternative stream link, even the ones that guarantee it.

So not only me getting error about downloading vk video? I created _netrc, created config file, tried to use -u -p/--username --password but yet it getting me this error

No, it's a general issue with VK as other commenters have noted; see the link in https://github.com/ytdl-org/youtube-dl/issues/31035#issuecomment-1159733957.

The authentication algorithm on the site vk.com changed. Now on the start page in the input form you need to specify only the login, after clicking on the button in the next form you need to enter only the password. Please correct the authentication for vk.com in your module vk.py

Does anyone know another way to download videos from vk.com by python?

Vk change login page. Cookies don't work.

How not? (Remember to not use -u .../--username ...)

For me, nor youtube-dl -u ... -p ... , youtube-dl -n or cookies works.

For
please tell us the details how you use cookies for downloading from vk.com? I tried to save cookies by the chrome extension (get cookies.txt), but I got an error:

youtube-dl -v --cookies vk.com_cookies.txt https://vk.com/video-31480508_456243099
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--cookies', u'vk.com_cookies.txt', u'https://vk.com/video-31480508_456243099']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.18 (CPython) - Darwin-21.1.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.0.1, ffprobe 5.0.1, rtmpdump 2.4
[debug] Proxy map: {}
[vk] -31480508_456243099: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

What am I doing wrong?

What am I doing wrong?

You're doing everything right, the VK has changed something...

There are two issues:

The site's login workflow has changed, as above.
The request that the extractor uses to fetch video data, except for embedded videos, no longer works, and has to be re-worked as in https://github.com/yt-dlp/yt-dlp/pull/4128.

Consequently:

only embedded videos like https://vk.com/video_ext.php?oid=-77521&id=162222515&hash=87b046504ccd8bfa can be downloaded with the extractor as-is
if there were an embedded video that needed a login then it would presumably be accessible using --cookies ..., but perhaps embedded and private are mutually exclusive.

This patch fixes the extractor for public videos and should, I hope, enable private videos with --cookies ... as well (it also includes _VALID_URL improvements from yt-dlp).
--- old/youtube_dl/extractor/vk.py
+++ new/youtube_dl/extractor/vk.py
@@ -16,6 +16,7 @@
     orderedSet,
     str_or_none,
     str_to_int,
+    try_get,
     unescapeHTML,
     unified_timestamp,
     url_or_none,
@@ -64,10 +65,14 @@

     def _download_payload(self, path, video_id, data, fatal=True):
         data['al'] = 1
+        url = 'https://vk.com/%s.php' % path
         code, payload = self._download_json(
-            'https://vk.com/%s.php' % path, video_id,
+            url, video_id,
             data=urlencode_postdata(data), fatal=fatal,
-            headers={'X-Requested-With': 'XMLHttpRequest'})['payload']
+            headers={
+                'X-Requested-With': 'XMLHttpRequest',
+                'Referer': url,
+            })['payload']
         if code == '3':
             self.raise_login_required()
         elif code == '8':
@@ -87,22 +92,22 @@
                             )
                             ext\.php\?(?P<embed_query>.*?\boid=(?P<oid>-?\d+).*?\bid=(?P<id>\d+).*)|
                             (?:
-                                (?:(?:m|new)\.)?vk\.com/(?:.+?\?.*?z=)?video|
+                                (?:(?:m|new)\.)?vk\.com/(?:.+?\?.*?z=)?video|clip|
                                 (?:www\.)?daxab.com/embed/
                             )
-                            (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+))?
+                            (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+|ln-[\da-zA-Z]+))?
                         )
                     '''
     _TESTS = [
         {
             'url': 'http://vk.com/videos-77521?z=video-77521_162222515%2Fclub77521',
-            'md5': '7babad3b85ea2e91948005b1b8b0cb84',
+            'md5': '56df4e32658e1ffb77fdf2548327f73a',
             'info_dict': {
                 'id': '-77521_162222515',
                 'ext': 'mp4',
                 'title': 'ProtivoGunz - Хуёвая песня',
                 'uploader': 're:(?:Noize MC|Alexander Ilyashenko).*',
-                'uploader_id': '-77521',
+                'uploader_id': '39545378',
                 'duration': 195,
                 'timestamp':<PHONE_NUMBER>,
                 'upload_date': '20120212',
@@ -124,7 +129,7 @@
         {
             'note': 'Embedded video',
             'url': 'https://vk.com/video_ext.php?oid=-77521&id=162222515&hash=87b046504ccd8bfa',
-            'md5': '7babad3b85ea2e91948005b1b8b0cb84',
+            'md5': '56df4e32658e1ffb77fdf2548327f73a',
             'info_dict': {
                 'id': '-77521_162222515',
                 'ext': 'mp4',
@@ -133,7 +138,7 @@
                 'duration': 195,
                 'upload_date': '20120212',
                 'timestamp':<PHONE_NUMBER>,
-                'uploader_id': '-77521',
+                'uploader_id': '39545378',
             },
         },
         {
@@ -180,6 +185,19 @@
                 'view_count': int,
             },
             'skip': 'Removed',
+        },
+        {
+            'url': 'https://vk.com/video-93049196_456239755?list=ln-cBjJ7S4jYYx3ADnmDT',
+            'info_dict': {
+                'id': '-93049196_456239755',
+                'ext': 'mp4',
+                'title': '8 серия (озвучка)',
+                'duration': 8383,
+                'timestamp':<PHONE_NUMBER>,
+                'upload_date': '20211222',
+                'uploader': 'Dizi2021',
+                'uploader_id': '-93049196',
+            },
         },
         {
             # video (removed?) only available with list id
@@ -227,6 +245,7 @@
             'params': {
                 'skip_download': True,
             },
+            'skip': 'This video has been deleted',
         },
         {
             # video key is extra_data not url\d+
@@ -254,8 +273,6 @@
                 'title': 'ИгроМир 2016 День 1 — Игромания Утром',
                 'uploader': 'Игромания',
                 'duration': 5239,
-                # TODO: use act=show to extract view_count
-                # 'view_count': int,
                 'upload_date': '20160929',
                 'uploader_id': '-387766',
                 'timestamp':<PHONE_NUMBER>,
@@ -297,6 +314,10 @@
         {
             # The video is not available in your region.
             'url': 'https://vk.com/video-51812607_171445436',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://vk.com/clip30014565_456240946',
             'only_matching': True,
         }]

@@ -314,7 +335,7 @@
         mv_data = {}
         if video_id:
             data = {
-                'act': 'show_inline',
+                'act': 'show',
                 'video': video_id,
             }
             # Some videos (removed?) can only be downloaded with list id specified
@@ -323,8 +344,10 @@
                 data['list'] = list_id

             payload = self._download_payload('al_video', video_id, data)
+            opts = try_get(payload, lambda x: x[-1], dict)
+            if not opts:
+                raise ExtractorError('Invalid payload data')
             info_page = payload[1]
-            opts = payload[-1]
             mv_data = opts.get('mvData') or {}
             player = opts.get('player') or {}
         else:

def _download_payload(self, path, video_id, data, fatal=True):

It works! Thank you!

this is great @dirkf, I update the file with the changes but unfortunately I do to see the download happening I get this instead
youtube-dl -vg "https://vk.com/video-133441491_456261193"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vg', 'https://vk.com/video-133441491_456261193']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.13.0-1023-aws-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
https://vkvd262.mycdn.me/expires/1656117961051/srcIp/<IP_ADDRESS>/pr/40/srcAg/CHROME/ms/<IP_ADDRESS>/type/3/sig/eQKgW9u4xMs/ct/8/urls/<IP_ADDRESS>/clientType/13/zs/43/id/2634730506936/video/

@dirkf I think it was my mistake with the command line. Once I ran my tool that uses it as a lib it worked
youtube_dl.YoutubeDL({"outtmpl": file_path}).download(
                    [self.video_files_contents[video]["source_link"]]

thank you for getting this resolved.
I am assuming that at some point this will be a part of the rc build?

-g/--get-url just prints the download link(s).
There will be a PR. If someone manages to fix the login soon, I'll add that before merging. Then, real soon now, a 2022.0n.mm release.

я всё сделал по гайду обновил но. пишет опять тоже самое
[vk] 77496727_456240355: Downloading JSON metadata
ERROR: This video is only available for registered users. Use --username and --password or --netrc to provide account credentials.

вы можете просто файл загрузить куда-то целиком, а не кусками. может я не туда вставил что-то

я всё сделал по гайду обновил но. пишет опять тоже самое
[G Translate] I did everything on the guide, I updated but. writes again the same
[my Translate] I did everything according to the guide, I updated, but still get the same result
...

вы можете просто файл загрузить куда-то целиком, а не кусками. может я не туда вставил что-то
[G Translate]You can just download the file somewhere, not in pieces. maybe I put something in the wrong place

Here I'm not sure what point the commenter is making.
Read https://github-com.translate.goog/ytdl-org/youtube-dl/issues/31035?_x_tr_sl=auto&_x_tr_tl=ru&_x_tr_hl=en#issuecomment-1161719063.
Don't expect the released version to work at the moment without patching.
Even then, don't expect the login or netrc functions to work: use cookies instead.
Post your verbose log (use -v/--verbose option) if it's significantly different from that already posted.

[G Translate]
Читайте https://github.com/ytdl-org/youtube-dl/issues/31035#issuecomment-1161719063`.
Не ожидайте, что выпущенная версия будет работать сейчас без исправления.
Даже тогда не ожидайте, что функции входа в систему или NETRC будут работать: вместо этого используйте файлы cookie.
Разместите свой журнал словеса (используйте опцию -v/-verbose), если он значительно отличается от того, что уже опубликовано.

Спасибо, а есть какие-то гайды как использовать файлы cookie я не совсем понимаю как их подключить

def _download_payload(self, path, video_id, data, fatal=True):

It works! Thank you!

а что это? надо вставить куда-то?

I can't login , the username and password are correct

Спасибо, а есть какие-то гайды как использовать файлы cookie я не совсем понимаю как их подключить

Нужно узнать, какой у браузера User-Agent, и указать параметр --user-agent такой же. Потом если один раз запустить с параметром --cookie vk.txt, то пока yt-dl потыкается в сайт, он создаст vk.txt с кукисами в правильном формате. Открываем в браузере список куки, в Firefox при открытом залогиненном ВКонтакте Ctrl+Shift+K, вкладка Хранилище, раздел Куки, и по одной чик-чик-чик Ctrl+C, Ctrl+V имя и значение в vk.txt, через табуляцию, как требует формат файла. С таким содержимым vk.txt можно будет качать видео. У меня получилось скачать generic движком из мобильной (m.vk.com) версии сайта, но для этого ещё в эту мобильную версию зайти надо, у мобильной свои куки, и только потом снимать куки в vk.txt
youtube-dl --cookies vk.txt --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" --force-generic-extractor https://m.vk.com/video-12345_67890
I had success with cookies method, generic extractor and a MOBILE (m.vk.com/video) version. You need to be sure saved cookies match the mobile version, so login to mobile version first in real browser

I can't login , the username and password are correct

https://github.com/ytdl-org/youtube-dl/issues/31035#issuecomment-1179766308 applies, as well as the comment above.

FWIW, the video URI posted in the opening post,
https://vk.com/video-133441491_456261066
can be "handled" by latest yt-dlp, at least from my location (SE-Europe), without any cookies or credentials:
youtube-dl -F "https://vk.com/video-133441491_456261066"

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://vk.com/video-133441491_456261066']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2<IP_ADDRESS>0 [47a60be] (win32_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] Checking exe version: "..\ffmpeg" -bsfs
[debug] Checking exe version: "..\ffprobe" -bsfs
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] [vk] Extracting URL: https://vk.com/video-133441491_456261066
[vk] -133441491_456261066: Downloading JSON metadata
[vk] -133441491_456261066: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for -133441491_456261066:
ID       EXT           RESOLUTION FPS |   FILESIZE   TBR PROTO | VCODEC    VBR ACODEC  ABR
------------------------------------------------------------------------------------------
hls-193  mp4           256x144     25 | ~  4.46MiB  194k m3u8  | unknown  194k unknown  0k
url240   unknown_video 240p           |                  https | unknown       unknown
hls-369  mp4           426x240     25 | ~  8.52MiB  369k m3u8  | unknown  369k unknown  0k
url360   unknown_video 360p           |                  https | unknown       unknown
hls-725  mp4           640x360     25 | ~ 16.74MiB  725k m3u8  | unknown  725k unknown  0k
url480   unknown_video 480p           |                  https | unknown       unknown
hls-1185 mp4           852x480     25 | ~ 27.35MiB 1186k m3u8  | unknown 1186k unknown  0k
url720   unknown_video 720p           |                  https | unknown       unknown
hls-2460 mp4           1280x720    25 | ~ 56.77MiB 2461k m3u8  | unknown 2461k unknown  0k
url1080  unknown_video 1080p          |                  https | unknown       unknown
hls-5058 mp4           1920x1080   25 | ~116.70MiB 5058k m3u8  | unknown 5058k unknown  0k

So, again, some backporting might be needed 😉 ...
Be that as it may, I'm not sure whether yt-dlp can handle "true" private VK videos...

@gsw17 : Which version of yt-dlp is that on?
Both the stable yt-dlp release (2022.09.01) as well as latest "daily" build (which you can get from:
https://github.com/ytdl-patched/yt-dlp/tags )
work fine here with that sample VK URI:
yt-dlp -v -F "https://vk.com/video-133441491_456261066" --ie vk

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://vk.com/video-133441491_456261066', '--ie', 'vk']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (win32_exe)
[debug] Python 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] Checking exe version: "..\ffmpeg" -bsfs
[debug] Checking exe version: "..\ffprobe" -bsfs
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1 extractors
[debug] [vk] Extracting URL: https://vk.com/video-133441491_456261066
[vk] -133441491_456261066: Downloading JSON metadata
[vk] -133441491_456261066: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for -133441491_456261066:
ID       EXT           RESOLUTION FPS |   FILESIZE   TBR PROTO | VCODEC    VBR ACODEC  ABR
------------------------------------------------------------------------------------------
hls-193  mp4           256x144     25 | ~  4.46MiB  194k m3u8  | unknown  194k unknown  0k
url240   unknown_video 240p           |                  https | unknown       unknown
hls-369  mp4           426x240     25 | ~  8.52MiB  369k m3u8  | unknown  369k unknown  0k
url360   unknown_video 360p           |                  https | unknown       unknown
hls-725  mp4           640x360     25 | ~ 16.74MiB  725k m3u8  | unknown  725k unknown  0k
url480   unknown_video 480p           |                  https | unknown       unknown
hls-1185 mp4           852x480     25 | ~ 27.35MiB 1186k m3u8  | unknown 1186k unknown  0k
url720   unknown_video 720p           |                  https | unknown       unknown
hls-2460 mp4           1280x720    25 | ~ 56.77MiB 2461k m3u8  | unknown 2461k unknown  0k
url1080  unknown_video 1080p          |                  https | unknown       unknown
hls-5058 mp4           1920x1080   25 | ~116.70MiB 5058k m3u8  | unknown 5058k unknown  0k

Are you able to watch that video inside your browser?
If VK geo-blocks certain European regions, then the error issued by yt-dlp should be more appropriate for such a case ...

Hi! I still have the same issue. Here's what have:
I've verified that I'm running yt-dlp version 2023.1.6 (update instructions) or later (specify commit)
I've checked that all provided URLs are playable in a browser with the same IP and same login details
I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
I've read the guidelines for opening an issue
Provide a description that is worded well enough to be understood
I have an error "Access to this video has been restricted by its creator" on my video that only I can watch and error "Access to this video has been restricted by the community's managers" which is available for donors.
Cookies are passed from browser and I tried pass them with a file exported via Get cookies.txt. But it's suspicious there is no an error like "Login failed" when cookies is not passed.
Here is my code:
 import yt_dlp, pathlib

def download_video (URL):
     path = pathlib.Path(r"C:\users\Иван\Documents\Python_programs\DownloadKV\vk.com.txt")

     ydl_opts = { "cookies-from-browser": "chrome", "verbose": True, "ie": "vk" }
     with yt_dlp.YoutubeDL (ydl_opts) as ydl:
          ydl.download (URL)

download_video("https://vk.com/video/subscriptions?z=video-168725999_456240334%2Fpl_cat_subscribes")

Complete Verbose Output
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.06 [6becd2508] API
[debug] params: {'verbose': True, 'ie': 'vk', 'compat_opts': set()}
[debug] Lazy loading extractors is disabled
[debug] Python 3.11.1 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1q  5 Jul 2022)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1767 extractors

What I have tried:

The first that I tried is fix vk video extractor for youtube_dl and it helped but a week later the errors occured.
last release of yt_dlp and with "Fix vk video extractor".
yt_dlp_patched

I’d appreciate your help!

Please discuss this in the yt-dlp tracker to avoid confusion.

Please discuss this in the yt-dlp tracker to avoid confusion.

Yep, my mistake

Back-port of the newly updated yt-dlp extractor handles https://vk.com/video-133441491_456261066.


Fix model.plot() to work with new Numpy update.

This fixes both the issues
https://github.com/SheffieldML/GPy/issues/723
and
https://github.com/SheffieldML/GPy/issues/728
I am reasonably sure this doesn't break any backward compatibility with earlier Numpy versions. As per my previous comments in the first link:

My reason for thinking this is probably ok, is that we are already inside an if block stating that len(free_dims) == 1. Furthermore, if Xnew cannot be squeezed to a 1D array then the original statement would have failed anyway.

I think the issue must be in a Numpy update that has changed the rules for assignment.

Looks like it does break backwards compatibility via CI test failure - I'm going to go ahead and close this due to inactivity.


Previews are not working with Xcode12-beta 1 (latest main)

The app builds and runs fine using the simulator, but when trying to view a preview, it fails building the "Backend" target:
ld: warning: linking against a dylib which is not safe for use in application extensions: /Users/matthias/Library/Developer/Xcode/DerivedData/ACHNBrowserUI-cjtwdnwejuzvkbgipxuioxpcebsl/Build/Intermediates.noindex/Previews/ACHNBrowserUI/Products/Debug-iphonesimulator/PackageFrameworks/Purchases.framework/Purchases
Undefined symbols for architecture x86_64:
  "_OBJC_CLASS_$_SDWebImageDownloader", referenced from:
      objc-class-ref in Backend.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

This is a bug of Xcode 12. Should be fixed in on of the next beta releases.


Fix custom function resolving in insert with select and upsert queries

Adds custom function resolution for Select and Upsert subnodes of an Insert node. I think we want this behavior, right?
This is the issue I was talking about btw @sanderpick where I thought BLOCK_NUM() wasn't working in local-tableland, I finally looked into why today and realized that it wasn't local-tableland that was the issue but that we are using it in an insert with a select-query

nice catch 💯

Do you want me to merge this to main or should I retarget staging?

Do you want me to merge this to main or should I retarget staging?

you can merge it into main


Create Phoenix and Reva  tutorial

This is a tutorial how to run Reva together with Phoenix locally.

Some syntax error:
https://reva.link/docs/tutorials/phoenix-tutorial/
@LovisaLugnegard


Ui-radio with object not checked

Hello and thank you for this project.
I'm trying to use the ui-radioand setting the value and current to objects, it does the binding well, but it doesn't check the one that is already pre-selected.

{{#each shippingMethod in shippingMethods}}
  <div class="field">
    {{ui-radio name="shippingMethod" label=shippingMethod.name value=shippingMethod current=model.shippingMethod}}
  </div>
{{/each}}

We use radios in multiple places and as long as value and current are the same object it does select them. Are you sure shippingMethod and model.shippingMethod are both the same object? If they are two different objects or shippingMethod was a string, it wouldn't see it as being equal.

Didn't see any further response, closing.


Code blocks not rendering variables

I realize this is probably by design, but I actually have a use case where it would be useful to render variables within code blocks.  Can you provide a way to override a code block so that it actually renders the variable in the markdown preview?
Perhaps overriding it by using some sort of syntax language value , like note_vars or something could be a solution?
Example:

variable
value

var_name
erik

Code view:
```note_vars
var_name
```

Rendered as:
erik

@etho201 I didn't implement it on code blocks because of how MD works parsing different tokens. I currently don't know if this is possible, but I'm open to suggestions.

I'd be really interested in this also but without restriction to a specific type of fence (as suggested by using 'note_vars') as I'd like to build mermaid graphs with variables.
I managed to prototype it localy by reusing the markdownIt rule for standard text in markdownItPlugin.ts:
// Save default fence renderer, as for text
const defaultFenceRender =
  markdownIt.renderer.rules.fence ||
  function (tokens, idx, options, env, self) {
    return self.renderToken(tokens, idx, options, env, self);
  };

// ...

// Replace fence renderer, as for text
markdownIt.renderer.rules.fence = function (...) {

  /**
   * Copy content from the existing markdowniIt.renderer.rules.text
   */

  // Replace return statement with:
  tokens[idx].content = newText;
  return defaultFenceRender(tokens, idx, options, env, self);
};

Nota: to implement the rule for some specific fences only (as suggested), I guess the easiest would be to check the token.info value. Ref: Markdown-it 14.0.0 API documentation