SubjectGroup selection bug - incorrect URLs

API bug
The URLs returned by the SubjectGroup API are in an incorrect format.

example of an URL from API: https://panoptes-uploads.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png
CORRECT URL: https://panoptes-uploads-staging.zooniverse.org/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png

Related: #3530 - the initial addition of the SubjectGroup API end points
Testing
Setup

Project: Survos Survey 2020 (staging)
Workflow: 3412 - workflow.configuration set up to use the SubjectGroupViewer and the Survos API. Whitelisted to use the Survos API.
Subject Set: Subject Set 4780, which has 6 Subjects, each having 25 images

Action:

pinged https://panoptes-staging.zooniverse.org/api/subjects/grouped?workflow_id=3412&num_rows=5&num_columns=5 via Postman

Expected:

A pseudo-Subject that consists of 25 images. Resource looks the same as any other Subject, as the Front End is concerned.

Result:

A valid pseudo-Subject is received, but the URLs are all invalid.

Example:
{
    "subjects": [
        {
            "id": "134898",
            "metadata": {
                "#subject_group_id": 6,
                "#group_subject_ids": "85110-85111-85112-85114-85113-85109"
            },
            "locations": [
                {
                    "image/png": "https://panoptes-uploads.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png"
                },
                {
                    "image/png": "https://panoptes-uploads.zooniverse.org/staging/subject_location/a25dc6fa-e056-4e82-a480-d541ebb3ca8e.png"
                },
                {
                    "image/png": "https://panoptes-uploads.zooniverse.org/staging/subject_location/8ed9d95f-ac06-4868-8323-c9c29658bfb5.png"
                },
                ...total of 150 URLs...
            ],
            "zooniverse_id": null,
            "created_at": "2021-01-11T15:58:14.786Z",
            "updated_at": "2021-01-11T15:58:14.786Z",
            "href": "/subjects/134898",
            "selected_at": "2021-01-11T15:58:15.206Z",
            "retired": false,
            "already_seen": false,
            "finished_workflow": false,
            "user_has_finished_workflow": false,
            "favorite": false,
            "selection_state": "normal"
        }
    ],
    "meta": ...
}

Side note (irrelevant to the issue, but noting for posterity):

I receive 150 image URls instead of 25... which sorta makes sense, because the API finds only 6 Subjects out of 25, pulls them all anyway, and it turns out those 6 Subs have 25 images each (6*25 = 150). This is fine, because Subject Set 4780 isn't using the "correct" setup of 500 Subjects with one image each.

Status
Low priority bug, because the SubjectGroup feature is used only for Survos, and the front end can create an adaptor/transformer to make that URL work.
Tagging @camallen

I've added a new redirect rule on the panoptes-uploads.zooniverse.org frontdoor CDN to redirect any /staging/* path urls to the panoptes-uploads-staging.zooniverse.org domain. These images should now load correctly in the UI.
$ curl -I https://panoptes-uploads.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png
HTTP/2 302
content-length: 0
location: https://panoptes-uploads-staging.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png
server: Microsoft-IIS/10.0
x-azure-ref: 0dF4AYAAAAACquDHif9PaQpqqDROIItpKTE9OMjFFREdFMDEwOAAyMDgxYjI0OS00MTYyLTQ5YmUtOWMwYy1mMTk1YzBkODljODY=
date: Thu, 14 Jan 2021 15:08:36 GMT

I've added a new redirect rule on the panoptes-uploads.zooniverse.org frontdoor CDN to redirect any /staging/* path urls to the panoptes-uploads-staging.zooniverse.org domain. These images should now load correctly in the UI.
$ curl -I https://panoptes-uploads.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png
HTTP/2 302
content-length: 0
location: https://panoptes-uploads-staging.zooniverse.org/staging/subject_location/b71027f2-2416-4d8f-aa42-316791ca0d8a.png
server: Microsoft-IIS/10.0
x-azure-ref: 0dF4AYAAAAACquDHif9PaQpqqDROIItpKTE9OMjFFREdFMDEwOAAyMDgxYjI0OS00MTYyLTQ5YmUtOWMwYy1mMTk1YzBkODljODY=
date: Thu, 14 Jan 2021 15:08:36 GMT

Side note (irrelevant to the issue, but noting for posterity):

I receive 150 image URls instead of 25... which sorta makes sense, because the API finds only 6 Subjects out of 25, pulls them all anyway, and it turns out those 6 Subs have 25 images each (6*25 = 150). This is fine, because Subject Set 4780 isn't using the "correct" setup of 500 Subjects with one image each.

This is a good point - i think i'll add a check on the grouped subject creator to ensure that there is only 1 image media resource for each 'group' subject, otherwise, the resulting media resources can explode and possibly break the UI.

Side note (irrelevant to the issue, but noting for posterity):

I receive 150 image URls instead of 25... which sorta makes sense, because the API finds only 6 Subjects out of 25, pulls them all anyway, and it turns out those 6 Subs have 25 images each (6*25 = 150). This is fine, because Subject Set 4780 isn't using the "correct" setup of 500 Subjects with one image each.

This is a good point - i think i'll add a check on the grouped subject creator to ensure that there is only 1 image media resource for each 'group' subject, otherwise, the resulting media resources can explode and possibly break the UI.

I think this is closed by the CDN changes and the PR fixes in #3554 so i'm going to close for now. Feel free to re-open / comment on this issue if it's not fixed and i'll re-open.

I think this is closed by the CDN changes and the PR fixes in #3554 so i'm going to close for now. Feel free to re-open / comment on this issue if it's not fixed and i'll re-open.

Bug Un-Squashed
I'm reopening this issue, sorry! Following #3554 I've done some tests I've found the Subject Group API endpoint introducing a slightly different variation of the erroneous URL.
Action:

pinged a GET request to https://panoptes-staging.zooniverse.org/api/subjects/grouped?workflow_id=3412&num_rows=5&num_columns=5 via Postman

Result:
{
    "subjects": [
        {
            "id": "134955",
            "metadata": {
                "#subject_group_id": 63,
                "#group_subject_ids": "134711-134822-134706-134861-134855-134603-134845-134691-134820-134617-134707-134630-134866-134769-134893-134658-134612-134754-134869-134885-134743-134745-134598-134749-134686"
            },
            "locations": [
                {
                    "image/png": "https://subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"
                },
                {
                    "image/png": "https://subject_location/0736d6b9-b3d1-41d5-a192-c7af6657aaed.png"
                },
                {
                    "image/png": "https://subject_location/f734e36b-56cc-4d0c-9fab-d68aa0462c29.png"
                },
                {
                    "image/png": "https://subject_location/857a8e3f-5aae-4f87-9987-c2cc140e28ba.png"
                },
                {
                    "image/png": "https://subject_location/8dcd6d18-6c16-4dde-8aea-ac278bab3863.png"
                },
                {
                    "image/png": "https://subject_location/f6fe6a5a-c850-49a6-8c9d-65a86f1d5fb9.png"
                },
                {
                    "image/png": "https://subject_location/3630fc62-e96e-4af5-bfd8-3a5fcd3d3229.png"
                },
                {
                    "image/png": "https://subject_location/32304289-3e1a-402e-8de0-35467d61acc2.png"
                },
                {
                    "image/png": "https://subject_location/9ec81da0-4511-480e-90c9-e79396c14be3.png"
                },
                {
                    "image/png": "https://subject_location/61f6abb8-f1cd-4455-b1c5-520f228e17ed.png"
                },
                {
                    "image/png": "https://subject_location/67f315ab-8219-42a7-98cc-694156d585ca.png"
                },
                {
                    "image/png": "https://subject_location/e4597fba-d3bf-4484-af53-6ced7f7ebef5.png"
                },
                {
                    "image/png": "https://subject_location/440ee50a-4365-4d5e-9182-38c0a6bb9613.png"
                },
                {
                    "image/png": "https://subject_location/f3c0353a-ac86-4bc7-992f-861ca7812c31.png"
                },
                {
                    "image/png": "https://subject_location/e2c44338-7bce-45c0-8a64-c33f3682f7e8.png"
                },
                {
                    "image/png": "https://subject_location/03829109-cd1d-44a5-96d8-19a964434440.png"
                },
                {
                    "image/png": "https://subject_location/2fb859de-024b-4146-a0ef-888f8235e543.png"
                },
                {
                    "image/png": "https://subject_location/421f6d5e-eedb-4c86-87e1-05b97e9fc109.png"
                },
                {
                    "image/png": "https://subject_location/b90e3324-902a-456c-baf9-80284c4d7e0f.png"
                },
                {
                    "image/png": "https://subject_location/d708f076-db21-478f-b0bd-335590941b2b.png"
                },
                {
                    "image/png": "https://subject_location/7f61eed7-93e5-448f-b352-6e2ab0acf922.png"
                },
                {
                    "image/png": "https://subject_location/aa5d1029-1bc8-4f9f-96aa-7cad1e8da018.png"
                },
                {
                    "image/png": "https://subject_location/50ed0cbc-8a4d-45dd-bb59-9b651e8e5c2c.png"
                },
                {
                    "image/png": "https://subject_location/d6ebdf83-60c5-4605-8fe2-cc5d00e8cae7.png"
                },
                {
                    "image/png": "https://subject_location/46ad27cb-128a-4c41-957e-ac474f19d219.png"
                }
            ],
            "zooniverse_id": null,
            "created_at": "2021-01-25T18:08:34.516Z",
            "updated_at": "2021-01-25T18:08:34.516Z",
            "href": "/subjects/134955",
            "selected_at": "2021-01-25T18:08:35.101Z",
            "retired": false,
            "already_seen": false,
            "finished_workflow": false,
            "user_has_finished_workflow": false,
            "favorite": false,
            "selection_state": "normal"
        },
        ...subject 64...,
        ...subject 65...
    ],
    "meta": {
        ...
    }
}

Analysis:

as you can see, the URLs from the API have gone rather bonkers: "https://subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"
correct URL should of course be "https://panoptes-uploads-staging.zooniverse.org/subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"

Status
I'm genuinely baffled as to what's going on here. I've taken a look at 3554's code and I don't see what could possible cause the panoptes-uploads-staging.zooniverse.org/ to be stripped out of the image URLs. Is there a missing ENV variable?
Good news is, while this is still an important bug to fix, it remains a low urgency issue at the moment since the front-end has a temporary (simple yet effective) URL replacement workaround for continued Survos development.

Bug Un-Squashed
I'm reopening this issue, sorry! Following #3554 I've done some tests I've found the Subject Group API endpoint introducing a slightly different variation of the erroneous URL.
Action:

pinged a GET request to https://panoptes-staging.zooniverse.org/api/subjects/grouped?workflow_id=3412&num_rows=5&num_columns=5 via Postman

Result:
{
    "subjects": [
        {
            "id": "134955",
            "metadata": {
                "#subject_group_id": 63,
                "#group_subject_ids": "134711-134822-134706-134861-134855-134603-134845-134691-134820-134617-134707-134630-134866-134769-134893-134658-134612-134754-134869-134885-134743-134745-134598-134749-134686"
            },
            "locations": [
                {
                    "image/png": "https://subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"
                },
                {
                    "image/png": "https://subject_location/0736d6b9-b3d1-41d5-a192-c7af6657aaed.png"
                },
                {
                    "image/png": "https://subject_location/f734e36b-56cc-4d0c-9fab-d68aa0462c29.png"
                },
                {
                    "image/png": "https://subject_location/857a8e3f-5aae-4f87-9987-c2cc140e28ba.png"
                },
                {
                    "image/png": "https://subject_location/8dcd6d18-6c16-4dde-8aea-ac278bab3863.png"
                },
                {
                    "image/png": "https://subject_location/f6fe6a5a-c850-49a6-8c9d-65a86f1d5fb9.png"
                },
                {
                    "image/png": "https://subject_location/3630fc62-e96e-4af5-bfd8-3a5fcd3d3229.png"
                },
                {
                    "image/png": "https://subject_location/32304289-3e1a-402e-8de0-35467d61acc2.png"
                },
                {
                    "image/png": "https://subject_location/9ec81da0-4511-480e-90c9-e79396c14be3.png"
                },
                {
                    "image/png": "https://subject_location/61f6abb8-f1cd-4455-b1c5-520f228e17ed.png"
                },
                {
                    "image/png": "https://subject_location/67f315ab-8219-42a7-98cc-694156d585ca.png"
                },
                {
                    "image/png": "https://subject_location/e4597fba-d3bf-4484-af53-6ced7f7ebef5.png"
                },
                {
                    "image/png": "https://subject_location/440ee50a-4365-4d5e-9182-38c0a6bb9613.png"
                },
                {
                    "image/png": "https://subject_location/f3c0353a-ac86-4bc7-992f-861ca7812c31.png"
                },
                {
                    "image/png": "https://subject_location/e2c44338-7bce-45c0-8a64-c33f3682f7e8.png"
                },
                {
                    "image/png": "https://subject_location/03829109-cd1d-44a5-96d8-19a964434440.png"
                },
                {
                    "image/png": "https://subject_location/2fb859de-024b-4146-a0ef-888f8235e543.png"
                },
                {
                    "image/png": "https://subject_location/421f6d5e-eedb-4c86-87e1-05b97e9fc109.png"
                },
                {
                    "image/png": "https://subject_location/b90e3324-902a-456c-baf9-80284c4d7e0f.png"
                },
                {
                    "image/png": "https://subject_location/d708f076-db21-478f-b0bd-335590941b2b.png"
                },
                {
                    "image/png": "https://subject_location/7f61eed7-93e5-448f-b352-6e2ab0acf922.png"
                },
                {
                    "image/png": "https://subject_location/aa5d1029-1bc8-4f9f-96aa-7cad1e8da018.png"
                },
                {
                    "image/png": "https://subject_location/50ed0cbc-8a4d-45dd-bb59-9b651e8e5c2c.png"
                },
                {
                    "image/png": "https://subject_location/d6ebdf83-60c5-4605-8fe2-cc5d00e8cae7.png"
                },
                {
                    "image/png": "https://subject_location/46ad27cb-128a-4c41-957e-ac474f19d219.png"
                }
            ],
            "zooniverse_id": null,
            "created_at": "2021-01-25T18:08:34.516Z",
            "updated_at": "2021-01-25T18:08:34.516Z",
            "href": "/subjects/134955",
            "selected_at": "2021-01-25T18:08:35.101Z",
            "retired": false,
            "already_seen": false,
            "finished_workflow": false,
            "user_has_finished_workflow": false,
            "favorite": false,
            "selection_state": "normal"
        },
        ...subject 64...,
        ...subject 65...
    ],
    "meta": {
        ...
    }
}

Analysis:

as you can see, the URLs from the API have gone rather bonkers: "https://subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"
correct URL should of course be "https://panoptes-uploads-staging.zooniverse.org/subject_location/7d5b2efb-31c7-456d-86cd-e27a0d77621d.png"

Status
I'm genuinely baffled as to what's going on here. I've taken a look at 3554's code and I don't see what could possible cause the panoptes-uploads-staging.zooniverse.org/ to be stripped out of the image URLs. Is there a missing ENV variable?
Good news is, while this is still an important bug to fix, it remains a low urgency issue at the moment since the front-end has a temporary (simple yet effective) URL replacement workaround for continued Survos development.

this was fixed by #3554


Microsoft.MixedReality.Input package causes a headlocked display with Holographic Remoting

Describe the bug
Tracking an external issue where the presence of the Microsoft.MixedReality.Input package causes the display to lock to the head when doing holographic remoting. This package is used for deploying to the HP Reverb Headset, so this may impact users working across both WMR and Hololens platforms.
To reproduce

Add the Microsoft.MixedReality.Input package
https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/unity-reverb-g2-controllers#installing-microsoftmixedrealityinput-with-the-mixed-reality-feature-tool

Try to start Holographic Remoting
https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/features/tools/holographic-remoting?view=mrtkunity-2021-01

Expected behavior
Holographic remoting works without the headlocking behavior
Your setup (please complete the following information)

Unity Version 2019.4.x
MRTK Version 2.6+

Target platform (please complete the following information)

HoloLens
HoloLens 2
WMR immersive

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!


'See all theme modules" button is not responsive

On some devices, the button is not responsive.

Fixed by https://github.com/PrestaShop/ps_themecusto/pull/8

Thank you @apacios


Return if is new lock file

I think we can probably determine if it's new, based on the lack of Features for our particular target.
Having a boolean value only set when it's new feels like we're breaking the purity of this a little.

Yep closing in favour of pre features


[EventRequest] Report 202 "Sales Document - Test" - existing event "OnAfterCreateDimTableIDs" - change reference-type

Please change the reference-type of the parameters "TableID" and "No" at event "OnAfterCreateDimTableIDs" of Report 202 "Sales Document - Test".
OLD CODE:
[IntegrationEvent(false, false)]
local procedure OnAfterCreateDimTableIDs(var SalesLine: Record "Sales Line"; TableID: array[10] of Integer; No: array[10] of Code[20])
begin
end;

NEW CODE:
[IntegrationEvent(false, false)]
local procedure OnAfterCreateDimTableIDs(var SalesLine: Record "Sales Line"; var TableID: array[10] of Integer; var No: array[10] of Code[20])
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


GROOVY-8135: SecureASTCustomizer whitelist does not work

For arrays we should get componentType instead of type

Thanks!


feat: allow overriding the power select components used in the templates when extending

Similar in scope to https://github.com/kaliber5/ember-bootstrap/pull/685
Basically allows overriding which power-select components  the ember-bootstrap-power-select uses when extending ember-bootstrap-power-select compoennts.
This is useful when using some custom extended power-select components - for example if there's some internal my-power-select variant in the consuming app which we'd like to be used in ember-bootstrap.

@simonihmig all tests now passing - good catch on that incorrect property

@simonihmig any chance of a release with this PR included? 😃

@lougreenwood oh, sorry, missed this. Will do that later today. Feel free to ping me again, if not! :wink:

Published as 1.1.0! :tada:


Highlight end users and industries

It would be great to highlight the adopters by moving that to the nav bar and maybe creating pages for specific industries that Cilium is helpful in

Verticals I am thinking right now:

[ ] Media/Telecommunications
[ ] Financial Services
[ ] Consulting
[ ] SaaS/Software
[ ] Cloud Provider
[ ] Retail

We could make pages similar to https://www.hashicorp.com/industries/financial-services

I imagine this would look like "learn" navbar, with verticals for each industry and corresponding pages?

I think we could actually do it as a drop down under Use Cases now that it is there

Hey, @xmulligan do we need to design the pages before starting the development?

Hey @kirtanchandak sorry @paularah is already working on this, I just forgot to assign him until now

I've discussed with @paularah and we've come up with these categories
Financial services
Media and Entertainment
Telcos and Data center providers
Cloud providers (emphasis on hyper-scale clouds)
Consulting
Cyber security - (emphasis on tetragon and cilium security features)
SaaS/Software/DBaaS
E-commerce

closed with #343


vitest extension does not pick any tests in vscode

I have a regular TS-based sveltekit project with vitest added.
When I run "npm run test" from the command line, everything works as expected with standard cli-based vitest running.
I was trying to have it in the vscode test explorer. I installed the extention above. Alas, it need not find any tests and nothing happened.

I'm also in same/similar trouble
'vitest' is gray out

I'm also dealing with the same problem.
It also does not work with the debugger.
I'm using OS Linux Ubuntu.
I'm using vitest version ^0.34.4
I'm also using the latest version of this extension as well.
Here is my vitest config
import { defineConfig } from 'vitest/config'

export default defineConfig({
  build: {
    lib: {
      entry: './lib/main.ts',
      name: 'Class Name Filter Sorter',
      fileName: 'index',
      formats: ["es", "iife"]
    }
  },
  test: {
    globals: true,
    cache: {
      dir: 'node_modules/.vitest',
    },
    environment: 'node',
    include: ['lib/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}'],
  }
})

Here is the repo.
https://github.com/louiss0/class-names-filter-sorter


Getting artifacts on screen when displaying the shortcut guide

ℹ Computer information

Windows build number: [run "winver"]
PowerToys version:
PowerToy module:

📝 Provide detailed reproduction steps (if any)

…
…
…

✔ Expected result
What is the expected result of the above steps?
❌ Actual result
What is the actual result of the above steps?
📷 Screenshots
Are there any useful screenshots? WinKey+Shift+S and then just paste them directly into the form

Also getting an overlay of something that looks like the task view

How about some specifics? Give us something to work with.

Hi,
Found out the artifacts were due to an issue with my graphic card driver.


[SPARK-5714][Mllib] Refactor initial step of LDA to remove redundant operations

The initialState of LDA performs several RDD operations that looks redundant. This pr tries to simplify these operations.

Test build #27193 has started for   PR 4501 at commit 9af1487.

This patch merges cleanly.

Test build #27193 has finished for   PR 4501 at commit 9af1487.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27193/
Test FAILed.

an unrelated failure...

cc @jkbradley.

Test build #27268 has started for   PR 4501 at commit 4870fe4.

This patch merges cleanly.

Test build #27268 has finished for   PR 4501 at commit 4870fe4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27268/
Test PASSed.

LGTM. Merged into master and branch-1.3. Thanks!


"exec format error" when building on hypriot

Error:
$ docker run -it alpine sh
standard_init_linux.go:175: exec user process caused "exec format error"

Docker info:
$ docker info
Containers: 24
 Running: 0
 Paused: 0
 Stopped: 24
Images: 33
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.4.15-hypriotos-v7+
Operating System: Raspbian GNU/Linux 8 (jessie)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 862 MiB
Name: black-pearl
ID: QLG7:HLFA:YNZ2:SNNF:5XC3:ORPG:WUM6:NC3K:JK45:RXPO:SGCL:NI6J
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No cpuset support
Insecure Registries:
 <IP_ADDRESS>/8

Machine info:
$ cat /etc/*-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
HYPRIOT_OS="HypriotOS/armhf"
HYPRIOT_OS_VERSION="v1.0.0"
HYPRIOT_DEVICE="Raspberry Pi"
HYPRIOT_IMAGE_VERSION="v1.0.0"

This image is amd64 and your system appears to be armv7l, which is likely
why it won't run. :smile:

i had the same error when i tried something like
docker-compose -f docker-compose-shovel.yml run --entrypoint "./import" importer

but when i tried something like:
docker-compose -f docker-compose-shovel.yml run --entrypoint "sh ./import" importer

now it works well. My base image is a FROM gliderlabs/alpine:3.3 so i think it lacks bash.
I started reading this http://stackoverflow.com/questions/5725296/difference-between-sh-and-bash too.

Is ./import a script? I think what is going on here is that the --entrypoint "./import" form is the exec format which won't process a script under a shell (it would need to be a binary). It works in your latter example because sh is a binary. Feel free to reopen if this isn't the case.


gentle restructuring

patching humpty together again after the repo cutover
gentle restructuring to create slimmer modules
post review changes from the hackpad

Are we good to go on this, EarthmanT?


[BUG]Controllers of longhorn managers will fight for the ownership of the objects when the related node is disconnected

Describe the bug
There are tons of logs related to the ownership change of the objects on the disconnected nodes. This issue is found during testing #1545
To Reproduce
Steps to reproduce the behavior:

Create and attach a volume
Disconnect the node that the volume is attached to
Check the longhorn manager log

Expected behavior
For each object, the ownership won't change once it is picked by a running node.
Log
[longhorn-manager-k5mmp] time="2020-07-07T10:28:52Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-e-30c65e73"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:52Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-r-abc2bf5a"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:52Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-238dd3b5"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:52Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol3-r-4fa6eccf"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:53Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol2-r-765e4c92"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:53Z" level=debug msg="Engine controller shuo-longhorn-cluster-1-worker-1 picked up vol1-e-9ea4903d"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:53Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-4bcf8a47"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:53Z" level=debug msg="Requeue longhorn-system/instance-manager-e-30c65e73 due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-e-30c65e73\": the object has been modified; please apply your changes to the latest version and try again"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:54Z" level=debug msg="Requeue longhorn-system/instance-manager-r-abc2bf5a due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-r-abc2bf5a\": the object has been modified; please apply your changes to the latest version and try again"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:54Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-238dd3b5"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:54Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol3-r-4fa6eccf"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:54Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol2-r-765e4c92"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:54Z" level=debug msg="Engine controller shuo-longhorn-cluster-1-worker-1 picked up vol1-e-9ea4903d"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:55Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-4bcf8a47"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:55Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-e-30c65e73"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:55Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-r-abc2bf5a"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-238dd3b5"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol3-r-4fa6eccf"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol2-r-765e4c92"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Engine controller shuo-longhorn-cluster-1-worker-1 picked up vol1-e-9ea4903d"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-4bcf8a47"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:56Z" level=debug msg="Requeue longhorn-system/instance-manager-e-30c65e73 due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-e-30c65e73\": the object has been modified; please apply your changes to the latest version and try again"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:57Z" level=debug msg="Requeue longhorn-system/instance-manager-r-abc2bf5a due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-r-abc2bf5a\": the object has been modified; please apply your changes to the latest version and try again"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:57Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-238dd3b5"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:57Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol3-r-4fa6eccf"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:57Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol2-r-765e4c92"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:58Z" level=debug msg="Engine controller shuo-longhorn-cluster-1-worker-1 picked up vol1-e-9ea4903d"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:58Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-4bcf8a47"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:58Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-e-30c65e73"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:58Z" level=debug msg="Instance Manager Controller shuo-longhorn-cluster-1-worker-1 picked up instance-manager-r-abc2bf5a"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:59Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-238dd3b5"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:59Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol3-r-4fa6eccf"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:59Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol2-r-765e4c92"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:59Z" level=debug msg="Engine controller shuo-longhorn-cluster-1-worker-1 picked up vol1-e-9ea4903d"
[longhorn-manager-k5mmp] time="2020-07-07T10:28:59Z" level=debug msg="Replica controller shuo-longhorn-cluster-1-worker-1 picked up vol1-r-4bcf8a47"
[longhorn-manager-k5mmp] time="2020-07-07T10:29:00Z" level=debug msg="Requeue longhorn-system/instance-manager-e-30c65e73 due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-e-30c65e73\": the object has been modified; please apply your changes to the latest version and try again"
[longhorn-manager-k5mmp] time="2020-07-07T10:29:00Z" level=debug msg="Requeue longhorn-system/instance-manager-r-abc2bf5a due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-r-abc2bf5a\": the object has been modified; please apply your changes to the latest version and try again"

Environment:

Longhorn version: 1.0.0
Kubernetes version: 1.17
Node OS type and version: Ubuntu 18.04

It seems that the manager pod on the disconnected node is still able to update the object.

It seems that the manager pod on the disconnected node is still able to update the object.
This is probably due to some iptables setup issue when blocking the traffic. Move out from the release unless we can reproduce it again.

remove stale due to the incorrect exempt label

Same issue with 1.4.1, 1.24 EKS


Intel Gen 3 Platinum 8360Y

Hi All,
`

Operating system: Ubuntu 20.04.2 LTS
Kernel: 5.4.0-80-generic
Intel® Crypto Multi-buffer library from the ipp-crypto release version IPP Crypto 2021.3
Intel® Multi-Buffer crypto for IPsec Library release version v1.0
OpenSSL* 1.1.1k

`
My test platform is Intel 8360Y x 4 with AVX512 and cpuid all enabled.  I am using type 1 host passthrough for dedicated CPU and network cards.
I ran the step by step here https://software.intel.com/content/www/us/en/develop/articles/building-software-acceleration-features-in-the-intel-qat-engine-for-openssl.html
Adapting for newer versions, when I get to step
./autogen.sh LDFLAGS="-L/opt/ipsec-mb/v1.0/lib -L/opt/crypto_mb/2021.3/lib" CPPFLAGS="-I/opt/ipsec-mb/v1.0/include -I/opt/crypto_mb/2021.3/include" ./configure --prefix=/opt/openssl/1.1.1k --with-openssl_install_dir=/opt/openssl/1.1.1k --with-openssl_dir=$HOME/openssl-1.1.1k --enable-qat_sw
I get an error on the compile in bold below
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:12: installing './compile'
configure.ac:9: installing './missing'
Makefile.am: installing './depcomp'
autoreconf: Leaving directory `.'
I have searched across the forums and as this is compiling for the new Gen3 CPU, is there something else that requires to be installed?
any assistance is greatly appreciated.

@jimthedj65  The Logs highlighted in bold are expected from autogen.sh which does autoreconf to generate autconf files like configure . Or did you see any errors from ./configure step ? Please share config.log and we should be able to figure out any issues with setup.

@Yogaraj-Alamenda  thanks for the quick response,  ./configure steps seemed to go without any issues and error-free.  I have searched in my /var/log and various source directories and couldn't find a config.log where would this be normally located?
thanks

If configure.ac runs fine then after make install the qatengine.so should be copied to /opt/openssl/1.1.1k/lib/engines-1.1/ .  Please check if the library is copied into that path. config.log is present in same directory where you run autogen.sh and configure.
If Libraries are copied you can run the tests mentioned here to confirm build is fine

Just to confirm the previous ./configure steps were for the other libraries.
It does appear to be an issue with the QAT_Engine ./autogen.sh step?
I looked in QAT_Engine dir where I run the ./autogen.sh step and there is no config.log, do I need to explicitly switch the log file?
I ran autogen separately and got the error and then ran configure as a second step and it appeared to complete ok but wouldn't complete the make step.
giving me the error
In file included from e_qat.c:86: e_qat.h:61:12: fatal error: cpa.h: No such file or directory    61 | #  include "cpa.h"       |            ^~~~~~~ compilation terminated.
hope it helps

I refreshed my bash session and ran autogen.sh and ./configure.sh seperately and it foinally compiled and now I have a working QAT_Engine
(qatengine) Reference implementation of QAT crypto engine(qat_sw) v0.6.7
[RSA, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, X25519]
[ available ]
ENABLE_EXTERNAL_POLLING, POLL, ENABLE_HEURISTIC_POLLING,
GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE
Weird error but seems ok now.


[LX-2552]: Fix annotated_video and audio blocks in pathways

This merge request fixes two XBlocks that don't work properly on pathways:

lx_audio: removed code cruft and dependencies on transcript files (not used anymore).
annotated_video: fixed code that locates child video block to work in a generic way.

Jira tickets

FAL-2925
LX-2552

Testing instructions

Install this branch on your devstack.
Check that lx_audio and annotated_videos continue working from the library (http://localhost:4556/library/items/lb:LabXchange:1109ec43:lx_annotated_video:1 and http://localhost:4556/library/items/lb:LabXchange:1f29b8d5:lx_audio:1).
Check that the modified blocks retain their behavior when logged out.
Create a pathways adding those two items and check that they work.

Author's notes and concerns

I also took the opportunity to fix the tests and remove unused bits of code!

Release notes
Release as usual - this will only fix the currently broken blocks.
Reviewers

[ ] @pomegranited

I've got one nit above, but 👍 conditional on resolving that.
Thank you for sorting this out @giovannicimolin !

[x] I tested this on my devstack using the PR instructions. I also ensured that:

Video blocks and their transcripts work as expected, when standalone, in Annotated Videos, and when in Pathways.
Audio transcripts work as expected, standalone and in Pathways.

[x] I read through the code
[x] I checked for accessibility issues N/A
[x] Includes documentation N/A
[x] I made sure any change in configuration variables is reflected in the corresponding
client's configuration-secure repository. N/A


add example to compute the determinant of an array (python-numpy)

fixes #1236

Thanks man!
Deployed: https://onelinerhub.com/python-numpy/compute-the-determinant-of-an-array


Replaced requests-based API call with use of yfinance library.

Fixes #1

Note that I couldn't compare against the previously working version so some of the tag and value names may have changed


DDC-1025 - Absolute Namespace Annotations

Request pull into master using proper procedure this time:
https://github.com/doctrine/doctrine2/pull/215

A regex replace over phpdoc annotations to use absolute (i.e. \Doctrine) namespace references, described in issue DDC-1025 - http://www.doctrine-project.org/jira/browse/DDC-1025
This should fix code completion in multiple IDEs including PhpStorm.

Finally! Thanks a lot for this change! IDEA seems to be fixing it's behaviour forever!


Fix Global EF Registration (if possible)

@brettsam Need clarification

Closing as not a commitment, especially as we move towards .NET Core


Typehint string on loadOrCreateTag

... as TagManager itself does put name in an array.
Right now PHPStorm triggers a warning on loadOrCreateTag because we send it a string. It actually should be typehinted as string, because the method wraps it in an array itself.

No response, assume this will not get merged.


Italian translation contains inappropriate word

The Italian translation of this page contains an inappropriate word as the translation of CLI.
Screenshot below.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e5c4674-9e05-1381-8202-5225afd95e33
Version Independent ID: 62b343f9-9b2a-0be0-2582-e3795e9cdef5
Content: Azure CLI sample for Azure Pipelines and variable groups - Azure Pipelines
Content Source: docs/pipelines/scripts/cli/pipeline-variable-group-secret-nonsecret-variables.md
Product: devops
Technology: devops-cicd-scripts
GitHub Login: @steved0x
Microsoft Alias: jukullam

Hi @matteote - Thanks for finding this. It looks like the translation is not working right. Only happens for that one instance here, but there must be other pages where it goes wrong.
This board addresses technical content. Please open at the docs board.


Documenation: How do I use this?

This package already contains documentation on how to build and start it and how to write own nodes, but it doesn't contain any instructions for newcomers.

How do I create a new tree?
Where do I start / What kind of options for a root node do I have?
How do I use the web GUI? Where do I add inputs, etc

I think a tutorial on building an example tree and explaining all the steps along the way would be very handy.

Are we missing something here after #118 ?

@Oberacda


get_transcript not working

Hi, I'm in a Linux environment and have verified the following before raising this issue:

The version in use is youtube-transcript-api==0.4.1
From my local Ubuntu 18.04.5 LTS, I'm able to get desired response for https://www.youtube.com/watch?v=FStcqEIH9G0
Trying to hit API from Ubuntu servers setup for dev and prod environments leads to not getting any transcripts. (https://github.com/jdepoix/youtube-transcript-api/issues/74))
I tried ping youtube.com from my dev and prod envs and both of them seem to default to an IPv6 address as can be seen in the screenshot:

@jdepoix Any idea what could be the reason?
Thanks!

Hi @salonygupta76, could maybe try and run curl 'https://www.youtube.com/watch?v=FStcqEIH9G0' from your server and upload the resulting HTML somewhere and post the link here?
Also, since you posted that specific video, is the error only on that video or every video you try?

No, I'm getting issue with any video with available transcript that I try. Here's the result on running above command: link

@salonygupta76 are you sure you pulled that html from the host the module was failing on? The HTML you uploaded seems just fine and I have no problem extracting transcripts from it.

I've exposed this service as an API in dev environment and I'm trying to query the same using Postman from my local system. The response html is shared after ssh'ing into dev and running curl command that was shared by you.

Just so that I am understanding you correctly: curling YouTube from your local machine returns the same html as curling YouTube from your server, yet this module works on your machine, but not on your server, right? That seems really odd! What python version are you using btw? And could you please post the exact error message which is returned by this module.

@jdepoix
Hello, I don't know if it's the right place to post, if no, I am very sorry, maybe I should have created a separate issue.  But I have problems with your tool using it from the EU zone countries when I do it from my command line. When I do with EU countries VPN on the sites like replit or pythonanywhere, it works fine (it sends requests from their IPs I suppose). When I use VPN for out-of-EU country, it works fine. When I do from the EU (and my friends), it doesn't work. Maybe they've applied some law about it because I have troubles with YouTube tool downloading live chats as well.

@vanyamlb could you please explain your infrastructure a bit more, I am not sure if I am understanding you correctly. Also, what do you mean by live chats? This module only supports transcripts. And what version of this module are you using?

@vanyamlb could you please explain your infrastructure a bit more, I am not sure if I am understanding you correctly. Also, what do you mean by live chats? This module only supports transcripts. And what version of this module are you using?

Windows 10, 21H1, 64 bit. Python 3.9.1.
I meant that I have another YouTube tool for live chats from another developer (not connected with this topic), but it also stops working in the same conditions.
I live in Ukraine (so out of the EU sadly).  Your tool (last version) and another tool for live chats from another developer work fine together, but once I turn EU country's VPN on - they stop working. When I change VPN to, for example, Israel - tools start working again.
The same happens with my friends who actually live in that countries (for example, Germany and the UK).
Thus, I came to the conclusion that there is some law about personal data maybe in the EU  or something like that which changed the tools' behavior.

Date when the tools stopped working: the beginning of this year's April.

@jdepoix
Just checked everything again.

1 - under Italy's VPN or any other EU
2 - under Israel's VPN or just my own Ukrainian's IP (or any other out-of-EU)

@jdepoix
Just checked everything again.

1 - under Italy's VPN or any other EU
2 - under Israel's VPN or just my own Ukrainian's IP (or any other out-of-EU)

Well, I live in Germany and I don't have a problem using this, so I'm pretty sure it's not a problem with EU law 😄
Sound like it's a problem with your VPN to me. What happens if you just open youtube.com from one of those VPNs were it's not working?

@jdepoix the problem is that I am not alone, my friend from Germany has this issue too and friends from other EU countries :/ (without a VPN)  If it was only with VPN, then sure I would think about that... Maybe it's IPS blocking it or youtube...? I don't know the explanation...  I checked YouTube with my VPN and it worked fine...  But once more, when I use the same VPN app for non-EU country - i get the tool working! :/
just wondering... could you please check one thing for me? my friend developed his own tool based on yours (it's called yxd and is installed through pip as well). with it, you can scan an entire channel to get transcripts using YouTube API key for video listing + your tool (by the way there was an issue about scanning the entire channel, you can tell that person she can use it).  you just enter yxd, then enter your API, then enter yxd -c linktothechannel --first=10 and it starts downloading. Just interesting if it works for you living in Germany. Thanks in advance!

I'm sorry @vanyamlb but I can't provide support for other modules. However, I might be able to help me if you upload the HTML you receive when accessing any given video (with subtitles) on youtube.com through curl or a browser.

@jdepoix with curl just got too many requests errors (429)  with VPN :/  wondering why did they block the IP and how to avoid that..

@vanyamlb probably you're sharing an IP address with other users of that VPN and that IP has been blocked because of too many requests. Is there any way to change the IP address?

@jdepoix ok I realize that with VPN that's possible... but people who used this for themselves with their IPS  and got the block (while I used it so so much too and didn't get it :/)... Maybe they have static IPs while I have dynamic...
do you know any way to get your IP unlocked without changing IP address?
yes, when I changed, it started working, but idk how to change it within the country to check if that will help (but looks like it should)

Unfortunately, there is no way I know of to get around the block without changing the IP or simply waiting until the block gets removed. So there's not really anything you can do here.
I guess this issue lost track a bit. @salonygupta76 any news on your end?

@salonygupta76 any news? Otherwise I will close this issue.

Hey @jdepoix Sorry about the delay in getting back. Unfortunately, at this point of time, I'm unsure what the root cause could be. For a video like this: url (where a transcript exists), sometimes the API simply throws this error instead of retrieving it:

Could not retrieve a transcript for the video https://www.youtube.com/watch?v=-em-_gFlDfQ! This is most likely caused by:
The video is no longer available
If you are sure that the described cause is not responsible for this error and that a transcript should be retrievable, please create an issue at https://github.com/jdepoix/youtube-transcript-api/issues. Please add which version of youtube_transcript_api you are using and provide the information needed to replicate the error. Also make sure that there are no open issues which already describe your problem! - -em-_gFlDfQ

Note that this happens only in case of trying to access the code available on one of dev/prod environments (through an API) and not while testing on my local environment.
Also of imp maybe, sometimes simply rebuilding the project from Jenkins resolves the problem.

@salonygupta76 where did you deploy your application? Maybe you are running into a problem similar to what @vanyamlb is describing?

I'm deploying the code in Linux environments as a Flask application. I've been capturing the error tracebacks for a while now and some of them do look like the one @vanyamlb shared above. I'm even using proxies.

@salonygupta76 but what infrastructure are you hosting your application on? If it is a cloud provider like GCP, AWS etc. it is likely that you are sharing a public IP with other users and therefore are being blocked by YouTube. What happens when you curl https://www.youtube.com/watch?v=-em-_gFlDfQ from that environment as it is throwing that error? Do you an 429 as well? Otherwise could you maybe upload the returned html so that I can have a look at it?

@jdepoix Infra is AWS. Right now, the block has been lifted and I'm able to get results. Can share Curl response when it reverts to throwing errors.

Hey, facing the issue yet again and error is mostly "Video is not available..." one when trying your code.
When I hit curl -L https://www.youtube.com/watch?v=-em-_gFlDfQ, I get the following as response:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.youtube.com/watch?v=-em-_gFlDfQ</title>

<form id="captcha-form" action="index" method="post">

<input type='hidden' name='q' value='EgQNOG6qGLnCsIcGIhBk7JGbIK-s219AkHLO2dTFMgFy'><input type="hidden" name="continue" value="https://www.youtube.com/watch?v=-em-_gFlDfQ">
</form>

About this page

Our systems have detected unusual traffic from your computer network.  This page checks to see if it's really you sending the requests, and not a robot.  Why did this happen?

This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.  In the meantime, solving the above CAPTCHA will let you continue to use our services.

This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  Learn more

Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.

IP address: DEV_IP_ADDRESS}
Time: 2021-07-12T11:02:18Z
URL: https://www.youtube.com/watch?v=-em-_gFlDfQ

Is this the curl response you're looking for? There seems to be some request limit, which when surpassed throws this error.

Hi @salonygupta76, thank you very much for the detailed information. That is exactly what I was looking for. Unfortunately, this confirms my assumption that you are being blocked by YouTube. The only way to work around this is to

Manually solve the captcha in a browser, then export the cookie and use it for future requests
Use a different IP address
Wait until the ban on your IP has been lifted

I am aware that non of these solutions are great, but it's all we can do unfortunately (at least afaik).
While this doesn't directly solve your problem, I could at least use this html to make sure a more suitable error is raised in this case. Maybe you could catch this error and implement a sleep, while hoping that the ban will be lifted in the meantime (unfortunately I haven't been able to figure out how long you'll have to wait until bans get lifted and I feel like it's not really consistent). If you want to implement a more sophisticated solution you could catch the error and trigger a change of IP address, which definitely will be the most reliable solution, but also the most expensive one.

@jdepoix Yes, I've thought of these solutions, IP rotation in particular.
Inducing a sleep could work if we know which quota limit is specifically in
play here, that is, requests per min or requests per day or any other. Do
you have any idea which of these leads to our issue?
I tried making requests via a Proxy (in the methods exposed where proxies
is None by default) but in vain. My assumption is that they're not even
being used as they're supposed to be because my specific IP is being
captured and blocked.
Is there any way I could see what request payload is being sent to YouTube
using your package? (Like we do with requests by printing out their request
headers and data attributes)
Confidentiality Note: The contents of this email and any
attachments thereto are intended only for the use and consumption of the
individual or entity to which it is addressed and may contain information
that is privileged, confidential and exempt from disclosure. If the reader
of this email / message is not the intended recipient or a duly authorized
employee or agent responsible for delivering the message to the intended
recipient, you are hereby notified that any use, misuse, dissemination,
distribution, or copying of this communication and any attachments thereto
is strictly prohibited. It is not our intention to waive confidentiality,
work-product doctrine nor any proprietary rights in the said
information. If you have received this communication in error, please
notify the sender immediately by e-mail and delete the original message
along with attachments, and any copies thereof that may have been
downloaded by you on your system / server. Thank you.
On Mon, Jul 12, 2021, 5:38 PM jdepoix @.***> wrote:

Hi @salonygupta76 https://github.com/salonygupta76, thank you very much
for the detailed information. That is exactly what I was looking for.
Unfortunately, this confirms my assumption that you are being blocked by
YouTube. The only way to work around this is to

Manually solve the captcha in a browser, then export the cookie and use
it for future requests
https://github.com/jdepoix/youtube-transcript-api#cookies
Use a different IP address
Wait until the ban on your IP has been lifted

I am aware that non of these solutions are great, but it's all we can do
unfortunately (at least afaik).
While this doesn't directly solve your problem, I could at least use this
html to make sure a more suitable error is raised in this case. Maybe you
could catch this error and implement a sleep, while hoping that the ban
will be lifted in the meantime (unfortunately I haven't been able to figure
out how long you'll have to wait until bans get lifted and I feel like it's
not really consistent). If you want to implement a more sophisticated
solution you could catch the error and trigger a change of IP address,
which definitely will be the most reliable solution, but also the most
expensive one.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jdepoix/youtube-transcript-api/issues/117#issuecomment-878224876,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASRA7BFMGCNZBZ4VNRCUDPDTXLLKFANCNFSM4524W2IQ
.

@salonygupta76 Unfortunately, I don't know how long the "sleeping interval" would have to be to be sufficient. You'd have to play around with that. But if you do so, I would greatly appreciate if you could share your findings!
If you want to look deeper into the requests which are being sent, you'll have checkout the code and add some logs or run it in a debugger. More specifically youtube_transcript_api._transcripts.TranscriptListFetcher._fetch_html is the method which does the actual request, so if you want to log the response, that's were you'll have to look.

I am using the statement transcripts=YouTubeTranscriptApi.get_transcripts((video_id)); in a for loop, however, there are a few videos which have their id disabled. Is there a success or failure call for YouTubeTranscriptApi.get_transcripts((video_id))

@AaditBhatia what do you mean by success/failure call? You can simply wrap your call in a try/except and ignore the exception.

I will close this issue now, as there isn't really much we can do here and the discussion went off rails a bit.


a little problem with understanding the code in ContainerPool

Hello, I wonder what actor it refers to in the line below. It really confuses me a lot.
https://github.com/apache/openwhisk/blob/f9e469e50aac5a345a010e2bf1f71596f1b101fc/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala#L199

It's bounded here:
https://github.com/apache/openwhisk/blob/f9e469e50aac5a345a010e2bf1f71596f1b101fc/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala#L167-L168
this is the actor which represents the proxy to the actual container for running an action.

Thank you very much! It is such a complicated project.


self trained model param mismatch

Dear all, I tried to use train_mono.py on nyu to do the training, and I chose DPT_BEiT_B_384 as my base. After training done, I wanted to test my model by using evaluate.py, but got param mismatch error, just like that below.
I guess in the training process, the original zoeD_N model was overwrite, right? What should I do prevent this mismatch mistake?

i think you are using checkpoint wrong. you should use zoedepth_m12_n only other wise it will not work. as far as i know


Make honeypot field invisible via inline style

This should simplify the usage of this nice little library. :)

Thanks for this.
There are some reason why not including the css inline:

It's more discoverable by bots that this input is not visible. It's a feeling of mine, without any objective data.
I've seen some cases using javascript to hide the input instead css.

So, although I'm agree with you that this change simplify the usage, I'd like to keep it the more agnostic possible. What do you thing about make it optional? Some ideas:

Use a boolean second argument to insert the display:none:echo Honeypot::getField('name', true);

Use an array as second argument to insert any attribute (this allows to add classes, ids, etc):echo Honeypot::getField('name', ['style' => 'display: none']);

Adding other helper that works exactly as getField but adding the display:none style:echo Honeypot::getHiddenField('name');

Okay, I went with your third suggestion. Let me know what you think.

It's perfect. Thank you!


About the performance of COCO14?

Have anyone trained co-deformable DETR + R50 with 12epochs on COCO2014 dataset?
I wonder what is the performance of it when testing on COCO2014val.

We have not tried Co-DETR on the COCO2014

We have not tried Co-DETR on the COCO2014

Thank you,  and i will try it myself.


python3Packages.openwrt-ubus-rpc: init at 0.0.3

Motivation for this change
Python API for OpenWrt ubus RPC
https://github.com/Noltari/python-ubus-rpc
This is a Home Assistant dependency
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 115140 run on x86_64-linux 1

3 packages built:

python37Packages.openwrt-ubus-rpc
python38Packages.openwrt-ubus-rpc
python39Packages.openwrt-ubus-rpc

Result of nixpkgs-review pr 115140 at eb54158e run on aarch64-linux 1

3 packages built:

python37Packages.openwrt-ubus-rpc
python38Packages.openwrt-ubus-rpc
python39Packages.openwrt-ubus-rpc


加载poc数量很少

root@VM-8-11-ubuntu:~/MyWork/scan4all# ./scan4all -l 14ip.txt.txt -p 1-65535 -ceyeapi xxxxxxxxxxxxxxxxxx xxxxxx.ceye.io -ec -o 14all.csv
2023/07/26 18:40:48 wait for init config files ...
2023/07/26 18:40:48 init config files is over .
2023/07/26 18:40:48 MyDebug
Can't to build ./logs/errror.log
2023/07/26 18:40:48 DbName  config/scan4all_db
2023/07/26 18:40:49 wait for all threads to end
[INF] Running SYN scan with CAP_NET_RAW privileges
[INF] Port scan starting....
[INF] Port scan over,web scan starting
[WRN] Found 6260 templates with syntax error (use -validate flag for further examination)
[INF] Using Nuclei Engine 2.7.7
[INF] Using Nuclei Templates
[INF] Templates added in last update: 0
[INF] Templates loaded for scan: 46
[INF] No results found. Better luck next time!
看起来我只加载了46个模板,有人遇到过么!

The nuclei template is not Backward compatibility. Please use the template of the corresponding version of the program, or it will fail to load
nuclei/v2 v2.7.7


[WIP] Use convenient functions to compute raw._first_time, raw._last_time

What does this implement/fix?
Several places in the codebase we recompute raw._first_time despite having a helper method for that. I propose to replace them by the helper method. i,e:
- offset = self.first_samp / self.info['sfreq']
- self.annotations.onset -= offset
+ self.annotations.onset -= self._first_time

More over, I would also add self._last_time and do the same wherever we use raw.last_samp / raw.info['sfreq'].
Additional information
I would like RCF in unifying self.first_samp and self._first_time, maybe instead of two properties, this can be a function that takes a parameter to chose whether to return the first sample in latency or time.

to make it MRG, I would like to solve this https://github.com/massich/mne-python/pull/5 first. But we can get it in as it is, and migrate RFC to an issue (or anywhere)

the problem in https://github.com/massich/mne-python/pull/5 is that the object is not BaseRaw but _RawShell where we monkeypatch raw.first_samp to spit out a number rather than calling the BaseRaw.first_samp.

but first_samp and sfreq is not defined for _RawShell so here you replaced cases where it was used so I don't get it why _RawShell becomes the pb


Blur image generation in synthetic dataset

Hello, I have a question about the blurry image in synthetic dataset Ev-REDS. In the paper, you had written that "For each sequence, we generate high frame-rate videos by interpolating7 images between consecutive frames using RIFE [11], and then synthesize blurry frames by averaging 49 sharp images of the high frame-rate videos." Are sharp images from the .npz file in the Ev-REDS dataset used to generate blurry images?

Yes, these sharp images are used to synthesize the blurry frames. Specifically, we take 7 sharp images from the original REDS dataset, and then interpolate 7 new frames between consecutive sharp images with RIFE. The interpolated sequence contains 49 frames in total, and we average them to generate a blurry frame. Thus, each blurry frame (normal blur ones) corresponds to 7 sharp images from the original REDS dataset. You can find the blurry-sharp correspondence by comparing timestamps in the npz files. For example, you can find the sharp images used to generate the first blurry frame by conducting data['exp_start1']<=data['sharp_timestamps']<=data['exp_end1'] (specific data format is explained here).

Thank you for your reply, I have understand the whole process of synthetic blurry image generation.


add support for AMQP over websockets for Azure IoT Hub

Here's a first stab at separating the transport layer from connection.js (meaning extracting the dependencies to net and tls in specific files) and adding websocket support to enable amqp over websockets for Azure IoT Hub
the websocket library used has the benefit of being entirely written in javascript, unlike other popular ones. the good thing about this change is that it makes it easier to change or add transport layers by just creating another client in the socket folder, and then figuring out in connection.js how to decide whether we should be using plain sockets, secure sockets (tls) or websockets.
I have run unit tests but not integration tests - I might need a little help doing this :)

@pierreca very cool work, thanks for the contribution! I think we can and should definitely get this in, but I have a few comments (and I'm sure @noodlefrenzy has thoughts as well):
amqp10 was originally designed to be a pure javascript implementation, avoiding building native components at all costs (there was an initial module that built against apache qpid's proton that was the impetus for this module iirc). Having said that, your changes here would bring in a hard requirement to build a native module (I believe ws is in there somewhere, below nodejs-websocket), so we'd be breaking one of our core rules there. That doesn't mean that we shouldn't think of a way to get this in, so this is a great first step.
My proposal would be to make a new module with this work (amqp10-ws, amqp10-transport-ws? etc), and to "plug it in" to amqp10, so people who don't need websockets don't need to pay the penalty of downloading and building ws. At this point I think the conversation drives towards what a proposal for such a plugin system would look like, should it just be for transport? it looks like it will also affect the url parsing (support for wss in the protocol there). I imagine something like:
var amqp = require('amqp10');
amqp.use(require('amqp10-ws'));

var client = amqp.Client();
client.connect('wss://<something>')
  .then(function() { ... });

Thoughts?

Huh, crap, didn't realize this depended on native code. I know Node has, what, 7 websocket libraries? Any chance there's one that doesn't have native code?
Another option, similar to what @mbroadst is suggesting, is a protocol->provider map, with an ability to register protocol providers in amqp_client. So e.g. the code might look like:
var amqp = require('amqp10');
require('amqp10-ws').register(amqp);

var client = amqp.Client();
client.connect('wss://<something>')
  .then(function() { ... });

With register implemented as:
register = function(client) {
  client.registerProtocolHandler('wss', this);
};

The only reason I'm thinking in that direction is that use just seems a bit too generic.

Mike and I had this conversation about using library with no native dependency, and from what I could understand (I'm kinda new to this) nodejs-websocket (this one: https://github.com/sitegui/nodejs-websocket) had no dependencies at all (cf package.json). I did try to avoid ws because of that... What did I miss that makes you say it has a dependency on ws?
This being said, I think it's a good idea to implement a better decoupling anyway. it'd be nice if we could have a provider system that allow users to choose what websocket library to use anyway.

@pierreca ah you're totally right my apologies, things using websockets generally use the ws package for its speed. It looks like nodejs-websocket indeed uses pure javascript to get the job done (perhaps at a performance penalty?).
That satisfies my complaints from earlier, but I'm still marginally interested in the idea of "transport plugins." I think it would be nice to encapsulate the testing in a separate module, and you could theoretically use ws if building a binary protocol wasn't an issue for you. I'll try to squeeze in a talk with @noodlefrenzy when he's up :smile:

Whew, good, I thought he'd said it didn't have a native dependency.
So here's where I stand on the plugin idea:

I like mine better than yours, short-term, because it seems focused and easy to do.
I like yours better than mine, long-term, because it would potentially allow plugins for a variety of features, and could replace some of the more awkward parts of the policy concept. I just don't understand yet how we would get it to work in a manner that made sense to both clients and implementors, without building a "framework" and that's the last thing I want to do.
I would be willing to do the code changes required to move from mine to yours if and when we ever figure that shit out.

@pierreca do you want to take a first stab at the protocol plugin? If not, I can probably get to it tomorrow.

@noodlefrenzy sounds good to me. On that note I have a few review-related questions/observations:

should we call them "tranports" rather than "clients" (as we might want to support something crazy, like zeromq or whatever)
IMHO each "transport" should inherit from an abstract base (AbstractTransport or whatever), so we can get a feel for expected behaviors (this would also serve as the interface to code against for transport plugins)
Internal variable for Client would in turn be renamed this._client to this._transport (or socket, whatever you guys want to name these "sparkle ponies")
As you mentioned, we'd just have Client.prototype.registerTransport

I think that makes the work pretty manageably split. If someone wanted to tackle the refactorings in-place (as in inside the codebase as the PR is currently split up), I would be totally willing to do the work of making the external modularity work. I suppose it's just based on everyone's schedules at this point :smile:

@mbroadst yes, transports makes sense, as does the abstract base. I'm in a conference all day today and part of tomorrow, so earliest I would be able to help is tomorrow afternoon. If no-one else tackles it before then, I'll take it on.

Happy to try and implement this tomorrow, @noodlefrenzy I can start and we can meet later to review/push. One clarification though, I assume you would keep net and tls in the amqp10 library to provide basic transport functionality built-in, right?

@pierreca - exactly (re: net/tls), they would just implement whatever AbstractTransport we set up.

ok, there's a start here: #195

no need for this PR anymore, replaced by #195


Allow logging above the progress bar

This allows logging some information without interfering with the progress bar.
extern crate indicatif;

use std::thread;
use std::time::Duration;

use indicatif::ProgressBar;

fn main() {
    let pb = ProgressBar::new(100);
    for i in 0..100 {
        thread::sleep(Duration::from_millis(250));
        pb.println(format!("[+] finished #{}", i));
        pb.inc(1);
    }
    pb.finish_with_message("done");
}

Resolves #27

Does this work if a line in the log is wrapped?

@mitsuhiko yes, works nicely. If we print 2 lines, one is a very long log line and the 2nd one is the progress bar, the terminal treats it as 3 lines (assuming one wrap), while indicatif has 2 lines recorded. Without the orphan logic it would try to clear the last two lines. This would remove the log line after the wrap and the progress bar. Since we set orphan_lines to 1, it would only clear one line, which is the progress bar.

Thanks. Merged. Will need to fix the appveyor stuff before I release this.

This is an awesome feature. Is this going to be released at some point?

I integrated this with my application. It does not seem to work with MultiProgress.
Is that something that can be added in an easy way? I might try to contribute here but I need some pointers.


changing source of cereal

What is the purpose of this PR? You generally open a PR against https://github.com/commaai/cereal for cereal changes, not update the openpilot submodule


Please submit your extension to the web stores

Because its not installed from the official store I keep getting prompted to disable the extension!
Edge:
https://learn.microsoft.com/en-us/microsoft-edge/extensions-chromium/publish/publish-extension
Chrome:
https://chrome.google.com/webstore/devconsole/register
Firefox:
https://addons.mozilla.org/en-US/firefox/extensions/

Hi @achard ,
There is no short term plan to publish the extension on the stores, but this is the first time I'm getting this feedback.
Can you let me know which os and which browser version you are using, and if there is any company policy that may generate this behavior?
thanks

Hi @tmunzer,
This is built in behaviour of the Edge browser and cannot be modified.
https://answers.microsoft.com/en-us/microsoftedge/forum/all/msedge-1220236580-turn-off-turn-off-extensions-in/b482fad6-9cce-4b99-a860-1ba4f5688a6b
Regards,
Alex

Hi @achard ,
Ok, I'm also seeing this on Windows + Edge (I'm not using Windows daily). All the other combinations are not showing this popup (Edge on other OS and other web browsers on Windows).
I'll see what I can do.
Regards,
Thomas


Added name_regexp to tailscale_devices data source.

This can be used to identify/match devices based on a regular expression RE2.
What this PR does / why we need it:
Adds option to search for devices using regular expression RE2.
Which issue this PR fixes:
Fixes #202
Special notes for your reviewer:

Thank you for proposing this change, and apologies that it took so long to review it.
I am not sure I understand why this is necessary. If filtering is happening on the client anyway, can't you use a for expression to select the devices you need as part of your Terraform configs? For example:
[for d in data.tailscale_devices.sample_devices.devices : d if length(regexall("-(mobile|laptop)$", d.name)) > 0]


Doesn't seem to be working in IE10

Hey, we've implemented this awesome scroll library on our project, works perfect across all browser except IE10 for some reason (did not test below, do not support)

I had the same issue as well. Seems like it breaks on Line 32 in /lib/mixins/scroll-spy.js
I fixed it by replacing the line:
isMounted(scrollSpyContainer) {
with:
isMounted: function (scrollSpyContainer) {

isMounted issue should be fixed now.


Allow Sticky session per service basis

https://www.haproxy.com/blog/load-balancing-affinity-persistence-sticky-sessions-what-you-need-to-know/

https://github.com/kubernetes/ingress/blob/master/examples/affinity/cookie/nginx/README.md


Recipe: Chisel 2 chisel

The chisel 2 mod's chisel seems to have reverted back to old recipe which conflicts with Furniture mods knife.

But not on single player?

same in singleplayer

Well in the new -1 beta the chisel simply do not have a recipe

Well chisel just updated 11 minutes ago so a id probably changed. We'll kind of be in a beta mode for a few days.


Jackson Afterburner and the servlets

Commit the local branch with the Jackson Afterburner module enabled for the servlet test.
Edit: also adding implementation with the Jackson streaming API for comparison will be good. I'll do new issue.

The branch is ready.


Trouble importing on Mac OSX

I've been using Macports to get this running as far as I can but I'm stuck with these errors right now:
shiny:~ ovi$ beet import /Users/ovi/Music/iTunes/iTunes\ Media/\!DelaDap
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.15', 'console_scripts', 'beet')()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/ui/__init__.py", line 1163, in main
    _raw_main(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/ui/__init__.py", line 1149, in _raw_main
    subcommands, plugins, lib = _setup(options, lib)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/ui/__init__.py", line 1028, in _setup
    plugins = _load_plugins(config)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/ui/__init__.py", line 1014, in _load_plugins
    plugins.send("pluginload")
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/plugins.py", line 457, in send
    for handler in event_handlers()[event]:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/plugins.py", line 440, in event_handlers
    for plugin in find_plugins():
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beets/plugins.py", line 288, in find_plugins
    _instances[cls] = cls()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beetsplug/replaygain.py", line 813, in __init__
    self.config, self._log
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beetsplug/replaygain.py", line 378, in __init__
    self._import_gst()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/beetsplug/replaygain.py", line 445, in _import_gst
    gi.require_version('Gst', '1.0')
AttributeError: 'module' object has no attribute 'require_version'

Any hints are much appreciated.

sudo su
then it works. is this really needed?
h-3.2# beet import /Users/ovi/Music/iTunes/iTunes\ Media/\!DelaDap

/Users/ovi/Music/iTunes/iTunes Media/!DelaDap/Cigani Ruzsa + Angelo (12 items)
Tagging:
    !DelaDap - Cigani Ruzsa + Angelo
URL:
    http://musicbrainz.org/release/9de98b44-f503-4e83-90dc-4b975f621f7c
(Similarity: 100.0%) (CD, 2004, AT, ecco.chamber)

/Users/ovi/Music/iTunes/iTunes Media/!DelaDap/Dela Paji (14 items)
Tagging:
    !DelaDap - Dela Paji
URL:
    http://musicbrainz.org/release/1091cb58-4a8e-4274-b455-fe9a2688e2aa
(Similarity: 98.0%) (missing tracks) (CD, 2006, AT, Chat Chapeau)
Missing tracks (1/15 - 6.7%):
 ! Spielurh (#15) (1:00)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort?

The exception is coming from the replaygain plugin, which is for some reason not getting what it needs from its GStreamer-related dependencies. I'm not really sure why—but I would recommend either disabling the plugin or choosing a different backend.
Sudo should not be required.
Maybe there's a different Python module installed called gi that's getting confused here?

aacgain seems to work from Macports. no more errors even without sudo :-)
Thanks!


11662<=gnu(5952)<=11663

Constructed a list of length 11662
It contains 11661 groups and 1 sublists
Length of sublists are [ 2 ]
11662<=gnu(5952)<=11663
Runtime of this construction: 11741501 ms (3.26 hrs)
Now checking non-isomorphism of groups in sublists ...
System : Windows Vista

Thank you for submitting new gnu(n). We will recalculate it and add to the database if we get the same result.
Checklist for the reviewer:

[ ] checked and assigned labels "new (n<50000)" or "recomputation", if needed
[ ] set "replication needed" label (unless it starts immediately)
[ ] changed it to "replication in progress" label when the replication has been started
[ ] changed it to "replicated" after successful replication review
[ ] reproducible value of gnu(n)
[ ] no essential runtime differences
[ ] sufficient level of technical details

Checklist for the maintainer:

[ ] correct arguments of command to store data
[ ] correct location of new line in the data file
[ ] mention submitter and reviewer in the commit message
[ ] Close corresponding issue from the commit message

@gnufinder thanks. I only suggest to not to include the line
Now checking non-isomorphism of groups in sublists ...

if you're not going to run the non-isomorphism check, to make it clear.
We're trying to improve and automate a better check to avoid time-consuming calls of IsomorphismGroups.

gnu(5952)=11663
GAP      4.8.3
GrpConst 2.5
Runtime of ConstructAllGroups: 6784164 ms (1.88 hrs)
Returned a list of length 11662 containing 11661 groups and 1 sublists
Length of sublists : [ 2 ]
Runtime to check non-isomorphism of groups in pairs: 22821778 ms (6.34 hrs)
Replicated by @alex-konovalov on Ubuntu 16.04 on Azure cloud standard DS13 v2 instance (8 cores, 56 GB memory)
To maintainers: for new results, add to data/gnudata.g this line:
GNU_SAVE( 5952, 11663, WITH_GC );


Use pyarrow S3 file system at read time for arrow parquet engine

As discussed in https://github.com/dask/dask/issues/9619, there seem to be some performance benefits of using pyarrow.fs.S3FileSystem to open s3-based Parquet files for data ingest. This PR proposes a simple change to the defaults open_file_options contents for the "arrow" read_parquet engine.

[ ] Closes #xxxx
[ ] Tests added / passed
[ ] Passes pre-commit run --all-files

It also appears that pyarrow has utilities for converting between pyarrow filesystem objects and fsspec-compatible filesystem objects (see https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems-with-arrow and https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec). Do these help us here?

It also appears that pyarrow has utilities for converting between pyarrow filesystem objects and fsspec-compatible filesystem objects ... Do these help us here?

Unfortunately not :/
The first link is already what we are doing to create pyarrow Dataset objects (the backend filesystem is still python-based fsspec). The second link (ArrowFSWrapper) explains that we can wrap a pyarrow-based filesystem object in fsspec to expose fsspec-compatible APIs for ops like ls, mkdir, etc. The problem is that the wrapper does not seem work for opening/reading files. Even if we could use ArrowFSWrapper for data ingest, we still need to create the pyarrow-based filesystem somewhere.

Thanks for taking this on @rjzamora

@martindurant - Do you think we should be able to use ArrowFSWrapper (https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec) to expose an fsspec API for a pyarrow file-system? It would be nice to use this functionality to support temporary/specific optimizations without diverging from an fsspec-based design.

https://github.com/fsspec/filesystem_spec/issues/1113 is what I was referring to. It ought to be fine to pass various connection strings to arrow, but I don't really know what their API expects. Assuming this is fine, the rest of the interface should work as normal. Note the options in _open (https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/arrow.py#L149), where presumably we always need seekable.

things that appear to work on the client will maybe not work on workers

Hmm I don't follow this point. Could you give an example fail case?

The user does pd.read_parquet() on part of the dataset in the client. This uses s3fs only, and succeeds. Then they do dd.read_parquet() on the full dataset, where the arrow FS doesn't know how to establish auth. Strange errors will result,. OR we fall back to s3fs and it works; but it's hard for the user to reason about what's happening.

if storage_options has anything other than anon/anonymous use s3fs

There are many ways to configure botocore sessions, not just kwargs. (same as last point above). It is super common in some circumstances, e.g., each user in a jhub, and probably any situation where the user does not own the compute instance they are on.
Note that for non-local read, e.g., from my laptop, I already showed that s3fs is just as fast as pyarrow, saturating the bandwidth, so no need for the change in that case. This is probably the biggest user base (but not in enterprise).

Here's a data point.
Did this after seeing mixed results in some of my tests.  The following is 10 runs of Query 1 on a 50GB datasets for H2O benchmarks with this branch vs dask==2022.11.1 against s3fs==2022.11.0.  This data suggests users won't see a significant impact, and any difference we're seeing is likely noise.
Details:  These are 10 runs of Query1 for each environment.  We compare 10 runs on the same cluster, calling client.restart() between each run (left two boxplots) vs 10 runs with each run on a fresh cluster (right 2 boxplots).

@hayesgb can I ask you to dig into this and see what's actually happening?  I would look at profiling, task streams, etc..  I'm also curious about the difference in the benchmarks that we've seen above, which seem inconsistent with what you're presenting here.

@rjzamora I'd be curious about your thoughts here

I am having qualms about merging this PR for two reasons:

It feels "hacky" to construct an fsspec filesystem on the client, only to replace it with an additional pyarrow filesystem at data-ingest time. This feels even worse now that @martindurant has pointed out that storage_options is not guaranteed to include all necessary configuration options (and so an empty storage_options argument does not mean we will successfully construct a S3FileSystem object).
It is not clear to me (yet) that switching from fsspec-pyarrow provides a consistent performance bump. I have seen mixed results locally. I am only "comfortable" with (1) if it is clear that we are improving performance in most (real-world) cases.

One possible "middle-ground" solution is to make it easier for the user to define/pass-in the desired filesystem themselves:
import dask.dataframe as dd
import pyarrow.filesystem as pa_fs

fs = pa_fs.S3FileSystem(...)
# OR MAYBE allow {"arrow", "fsspec", <fs-object>}
ddf = dd.read_parquet("s3://...", filesystem=fs)

The "arrow" engine could simply add non-default logic to wrap a Pyarrow-based filesystem with ArrowFSWrapper (and the default logic would raise an error for anything other than an fsspec filesystem implementation).
The obvious drawback to this approach is that it doesn't address the original goal of automatically using a Pyarrow-based filesystem for s3. With that said, it does provide a more-intuitive mechanism for overriding the default behavior.
Also , if we do find that S3FileSystem consistently outperforms s3fs for GIL-related reasons, we could use the clearly-defined filesystem= argument to make default behavior engine-dependent. For example, engine="arrow" could be designed to use from_uri to infer the filesystem, while the default engine would use fsspec's url_to_fs.

I would like for us to find a solution where the default behavior of reading Parquet from S3 was close to hardware speeds.  I don't think that this should require special configuration.

I would like for us to find a solution where the default behavior of reading Parquet from S3 was close to hardware speeds. I don't think that this should require special configuration.

Yeah, I agree that we need good default behavior here.  However, I'm just pointing out that this PR feels like a footgun to me. I think it makes sense to enable the use of an optimized filesystem object, but my gut tells me that we should be very transparent about it. For example, https://github.com/dask/dask/pull/9699 is still very rough, but it proposes a mechanism to use a Pyarrow-based filesystem for metadata parsing and IO (in a way that may be easier to document).

Footguns sound bad.  Do you have suggestions for how to make the default behavior fast?  Does #9699 bring us towards that outcome?

Additional work on this topic, which makes a pretty solid case for some version of this PR being implemented.  Shoutout to @gjoseph92 for his help with profiling, and some digging to find thatpyarrow.S3Filesystem starts 6 threads by default.
Summary
Evidence below indicates that, in this use case, the pa.S3Filesystem will yield similar to faster end-to-end wall clock times when running workloads that originate from S3 as Parquet Datasets.  Differences are more substantial as the number of cores per worker increases.
All of these runs are with "Query1" from a 50GB H2O-benchmark dataset.  We run on 3 different types of EC2 instances, as follows:
m6i.large -- 10 workers, 2 cores/worker
m6i.xlarge -- 5 workers, 4 cores/workers
m6i.2xlarge -- 3 workers, 8 cores/worker
All of these machines a maximum network bandwidth of "up to 12.5Gbps".  Networking performance of these instances is burstable.  We have between 10 and 30 repeats, depending on the run.
We're comparing s3fs + dask==2022.11.1 vs this PR.

pyspy profile from a single worker of the 10 worker cluster:

A few interesting things about this.

pyspy shows 6 threads (52-57) that are not visible in the pyarrow.S3Filesystem run (below).  Pyarrow starts 6 threads, but in the pa.S3Filesystem case, there's no interaction with Python, so they are not visible.
in the above plot, we see pthread_cond_wait@@GLIBC_2.3.2, which is time spent waiting on Arrow Futures, takes 34 seconds, or 54% of the total time in this thread.
The open call takes about 11 seconds (17% of total time).

For comparison, the below is the pyarrow.S3Filesystem profile.

Note the difference in the number of threads.
Time spent waiting on Arrow Futures is 13 seconds (37% of total time)

This plot is from the run with s3fs, and shows one of the pyarrow threads running fsspec code.  These are the threads that are not visible in the pyarrow.S3Filesystem run.

Run to run variance -- When we run multiple runs on the same cluster with restarts between, wall clock times get less jittery.  Presumably S3 is doing some caching of data, but the first 1-2 runs can be pretty variable in their run times.

@martindurant, @jrbourbeau and I had an offline discussion about this work. Rough summary of this discussion:

It is probably a bad idea to (automatically) use a different filesystem backend on the client and workers. For this reason, we should probably go in the direction of https://github.com/dask/dask/pull/9699.

This seems both dangerous and confusing to explain/debug

It makes sense to support a user-specified filesystem in read_parquet, since both fastparquet and pyarrow already allow this on their own.
It may take significant time before the performance gap between S3FileSystem and s3fs can be closed for the specific case of a multi-threaded Dask cluster that is co-located with storage (if ever). Therefore:

Users should definitely be able to pass in their own S3FileSystem filesystem in cases like this
The “arrow” read_parquet engine should probably try creating it’s own S3FileSystem on the client by default

Remaining Uncertainty:

How do we go about changing default behavior without doing more harm than good?

Tentative Plan:

Add filesystem argument to read_parquet

This could be a top-level argument or a dataset-level argument
Will need to add an optional class method to set the filesystem object to be used throughout the engine. Default will use ffspec

Always try to initialize an S3FileSystem in the “arrow” engine for paths with the “s3” protocol

Fall back to fsspec/s3fs in the case that S3FileSystem initialization fails
If default S3FileSystem-initialization does not fail, throw a meaningful error message if/when file-opening fails. The message should explain the recent change in behavior for s3 data, and tell the user how to use the filesystem argument.

(James and Martin: Feel free to make corrections)

Thanks for writing up the summary

Closing this in favor of https://github.com/dask/dask/pull/9699 (but will re-open if necessary).


[kong] initContainer for job #390

What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #390

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] PR is based off the current tip of the next branch and targets next, not main
[x] Title of the PR and commit headers start with chart name (e.g. [kong])

for tpl related, i will removed it soon. aside of this, we can ask for vote on the community about tpl feature which supported in helm. as it is very handy and bundled function as part of helm instead of rely of another different tools.

aside of this, we can ask for vote on the community about tpl feature

Yeah, feel free to create an issue outlining the proposed change, and we'll await input from other users looking for something similar and/or note down when it'd be useful for handling some other need.


Add support for llava-1-5-phi3-mini

llava-phi-3-mini uses the Phi-3-instruct chat template. I think is similar with current llava-1-5, but with Phi3 instruct template instead of llama 2.
format:
<|user|>\nQuestion <|end|>\n<|assistant|>
stop word is <|end|>
for system use: <|system|>
I think you can adapt easily llava-1-5 handler for phi3:
class Llava15ChatHandler:
    DEFAULT_SYSTEM_MESSAGE: Optional[str] =  "A chat between a curious human and an artificial intelligence assistant.  The assistant gives helpful, detailed, and polite answers to the human's questions."

    CHAT_FORMAT = (
        "{% for message in messages %}"
        "{% if message.role == 'system' %}"
        "{{ message.content }}"
        "{% endif %}"
        "{% if message.role == 'user' %}"
        "{% if message.content is string %}"
        "\nUSER: {{ message.content }}"
        "{% endif %}"
        "{% if message.content is iterable %}"
        "\nUSER: "

        "{% for content in message.content %}"
        "{% if content.type == 'image_url' and content.image_url is string %}"
        "{{ content.image_url }}"
        "{% endif %}"
        "{% if content.type == 'image_url' and content.image_url is mapping %}"
        "{{ content.image_url.url }}"
        "{% endif %}"
        "{% endfor %}"

        "{% for content in message.content %}"
        "{% if content.type == 'text' %}"
        "{{ content.text }}"
        "{% endif %}"
        "{% endfor %}"

        "{% endif %}"
        "{% endif %}"
        "{% if message.role == 'assistant' and message.content is not none %}"
        "\nASSISTANT: {{ message.content }}"
        "{% endif %}"
        "{% endfor %}"
        "{% if add_generation_prompt %}"
        "\nASSISTANT: "
        "{% endif %}"
    )

Take a look: @abetlen

@CaptainOfHacks thank you I'll take a look, can you link to a huggingface model I can use for testing this?

Forgive me if I'm intruding but this one looks like it'll work: https://huggingface.co/xtuner/llava-phi-3-mini-gguf

yes, support for this model we want, thank you @xBelladonna for reference to huggingface.

some updates about phi3 template? @abetlen

I can't get phi-3 to work with -ngl flag. It seems that offloading any layers results in a crash in llama_decode_internal


Make controller configurable via ConfigMap

Describe the enhancement you'd like to see
Controller configuration is currently hardcoded in main.go, despite Kubebuilder generating manifests with ConfigMap.
We need to properly configure it via config file.

@olensmar @jasmingacic
Added requires discussion to this. We need to discuss exactly what we will expose via flags and what via environment variables set via the configmap and then document it here.

The following are exposed as flags as standard by Kubebuilder
	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")

The other flags are ones we added
flag.StringVar(&envoyControlPlaneAddr, "envoy-control-plane-bind-address", ":18000", "The address Envoy control plane XDS server binds to.")
	flag.StringVar(&agentManagerAddr, "agent-manager-bind-address", ":18010", "The address Agent Manager service binds to.")
flag.BoolVar(&development, "development", false, "enable development mode")
	flag.StringVar(&logLevel, "log-level", "INFO", "level of log detail [DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL]")
	// This one is configurable for the local development
	flag.StringVar(&webhookCertsDir, "webhook-certs-dir", "/opt/manager/webhook/certs", "The directory where webhook certificates will be generated.")

Do we want to move ALL to env variables, or just the ones we added?

I'd add all of them because there is a great change that port 8080 for metrics is already taken.