Add support for Bilibili international (and download subtitles)

This is the issue #29279 from the main youtube-dl repo.

Checklist

[x] I'm reporting a new site support request
[x] I've verified that I'm running youtube-dl version 2021.06.06
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that none of the provided URLs violate any copyrights
[x] I've searched the bug tracker for similar site support requests including closed ones

Example URLs

Single video: https://www.biliintl.com/en/play/35452
Homepage: https://www.biliintl.com/

Description

Bilibili is like Youtube but in China. It also has an international site, which is used for Chinese animation and anime. The international version doesn't have a search bar, and you can't switch the subtitle on your own. "What's Wrong with My Big Brother" is the series name of the example URLs. It was used as an example since it's not geo-restricted. To get to my point, the Bilibili app on Android and iOS has more features such as switching subtitles, commenting and searching series. I've looked for many ways to switch the subtitle language to another one to download it, but it always ends up with the top one (when sorted by A-Z). Therefore, along with video downloading, I would like someone with great knowledge in this field to implement a multilanguage subtitle download solution. Note that Bilibili subtitles are usually in the JSON format (there is a converter for this) or CC and BCC (related repo).

Citing comments from SuperSonicHub1:

@minhngoc25a Doing some early research, the single video link isn't a single video, it's a season. This is a single video: https://www.biliintl.com/en/play/35452/357787

@minhngoc25a

I've just recently found out that Subtitle Edit support Bilibili JSON subtitle format (.bcc or .json).

youtube-dl should still be able to parse and convert it without outside help, and we should also include the original JSON just for good measure.

@minhngoc25a The site also supports Siamese, using the URL scheme https://www.biliintl.com/th/

How to Crack Bilibili International
Parse URL
https://www.biliintl.com/en/play/35452/357787
https://www.biliintl.com/th/play/35452/357787
https://www.biliintl.com/<lang>/play/<season_id>/<episode_id:optional>

== OR ==

https://www.biliintl.com/en/play/35452/357787?s_locale=en_US
https://www.biliintl.com/en/play/35452/357787?s_locale=th_TH
https://www.biliintl.com/en/play/<season_id>/<episode_id:optional>?s_locale=<locale>

Season

Get youtube-dl metadata from HTML page

title => meta[itemprop=name] || meta[property='og:title'] (remove `_bilibili`)
description => meta[itemprop=description] || meta[property='og:type']

Download JSON from https://api.biliintl.com/intl/gateway/web/view/ogv_collection?s_locale=<locale>&season_id=<season_id>

if lang == 'th': locale == 'th_TH'
if lang == 'en': locale == 'en_US'

Iterate over each episode in root["data"]["episodes"]

use episode['ep_id'] and episode['long_title'] to create a URL result

InfoExtractor.url_result("https://www.biliintl.com/{lang}/play/{season_id}/{episode_id:optional}", video_title=long_title, video_id=ep_id)

Episode

Get youtube-dl metadata through methods referenced in the season section

This website's API and HTML are surprisingly lean, so there isn't a lot of meat to pick off the bone.

Download JSON from https://api.biliintl.com/intl/gateway/web/playurl?ep_id=<episode_id>&s_locale=<locale>&device=wap&platform=web&qn=64&tf=0&type=0

Look to the season section for determining locale

Set playurl to root['data']['playurl']

Get youtube-dl metadata from playurl

"duration"

Get audio streams by iterating over playurl['audio_resource']

"url"

metadata

"codecs"
"mimetype"
"size"

Get video streams by iterating over playurl['video']

"video_resource"

"url" (sometimes)

"id"

Combine the path with domain upos-bstar-mirrorakam.akamaized.net to get a URL
Requires the URL param hdnts; I don't get how it works: exp=1623873893~hmac=b6f9ae587367bce35a28498735e7dcdf68795003b3f61ebf4f8387ac171c7dd2

"codecs"

"mimetype"

"size"

"width"

"height"

"stream_info"

"desc_words", "desc_text"

youtube-dl will have to mux the video and audio streams in post using a post-processor, a la the YouTube extractor.
Input would be heavily appreciated on generating URLs for the other video streams. I'll probably not make the extractor myself, but I hope this is helpful to whoever does.

Oh, totally forgot subtitles! Those can be requested with https://api.biliintl.com/intl/gateway/m/subtitle?ep_id=<episode_id>&s_locale=<locale>. They're in a proprietary JSON format; look to OP for parsing them.

Another amendment; in order to get the proper subtitle URL, you'll need to iterate over each item in root["data"]["subtitles"] to get the language (item["key"]) you want and request item["url"]. item["title"] gives a more formal subtitle name. Didn't want to forget this since downloading subtitles is the reason @minhngoc25a made this request in the first place!

Yet another amendment; when doing that season request, if a "show" on BiliIntl has more than one season, there will be a seasons array within root["data"] giving us all the season IDs of a show.
{
    "seasons": [
      {
        "title": "S1",
        "season_id": 37653
      },
      {
        "title": "S2",
        "season_id": 1005556
      }
    ]
}

The episodes array will only contain the episodes relevant to the season ID requested. When a youtube-dl user gives us a URL that contains more than one season, should we download only the season from the season ID in the URL, or should we iterate over the seasons array and make multiple requests?

Citing comment from arnoldsecret:

there's also global.bilibili.com, which is more of the same, best include that too.

add json to srt subtitle with this script
https://github.com/y2361547758/bcc2ass

@riyuejz Thanks for referring to that repo. I'll open another issue for that.

Hello
How to get resolution 1080p? I always get 480p resolution.

Hello
How to get resolution 1080p? I always get 480p resolution.

360p to 720p should be available for every non-VIP user. 1080p is locked on the VIP tier. Also, some series will restrict access to episodes if you're not logged in, so remember to pass in a cookie file.


How to close gatt connection on writeCharacteristic and retry 2 times on write error?

       Subscription subscription = device.establishConnection(false)
                    .flatMap(rxBleConnection ->
                            rxBleConnection.writeCharacteristic(fromString("525255-0000-1000-8000-252525255"), hexToBytes(data)))
                    .subscribe(

                            characteristicValue -> {

                                Log.e(TAG, "write success " + characteristicValue + " " + device.getMacAddress());

                            },
                            throwable -> {
                                Log.e(TAG, "write error " + throwable);
                            }
                    );

I am using the above code to make an connection and write. However i will get " write error com.polidea.rxandroidble.exceptions.BleDisconnectedException: Disconnected from" if i connect to too many devices without disconnecting the previous connection. I am wondering how do i do that?
Also, how do i retry writeCharacteristic  if there is an error in writeCharacteristic ?

Hello!
Thank you for using the library.
This is a general usage question and as such should be posted on stackoverflow with rxandroidble tag—it will be easier to find by other users.
Best Regards

do you know the way?

Yes. I suppose you have already asked a question on Stackoverflow and Rob is already looking on your case


pull request for e2e test at 11/08/2016 06:45:09

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

Details

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


WSL commands/processes logging to event viewer

More of a question than an issue, but can't find any reference....
Is is possible for the WSL to log all user executed processes to the windows logging subsystem, either for shipping to a SIEM, ATA, ATP or similar?

you'll need to do the syslog to (im_msvistalog) eventlog from https://www.loggly.com/ultimate-guide/centralizing-windows-logs/ or use splunk or some type of central logging system. keep in context I'm just guessing here....

Hi @c3101 ,
I believe you can do this with command line auditing.
Here is a blog post on how to set up command line auditing for the Windows Console, and you can use the info there to help log process creation and filter down to see what was running in WSL. :)

Isn't the easiest way to parse the event logs from LxCore. Details here: https://github.com/Microsoft/WSL/blob/master/CONTRIBUTING.md#8-detailed-logs

Looks like this got all the answers forthcoming.


Coordinate system not update with landscape orientation

Current bug behavior
When you assign an landscape orientation, the coordinate system is not updated and x and y remain the same as in portrait orientation. This causes difficulties when creating any calculations related to coordinates because when running on a device other than a phone, x and y will be normal and so you will have to do calculations for phones and tablets and separately for other devices
Expected behavior
I expected that with a landscape orientation x would be greater than y, i.e. x in landscape orientation will be equal to y in portrait orientation, and y in landscape orientation will be equal to x in portrait orientation
Steps to reproduce
To reproduce the bug, just set the orientation to the landscape via
Flame.device.setLandscape();
And after that, in any component, get the size and display x and y
Flutter doctor output
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 3.16.2, on Microsoft Windows [Version 10.0.22621.1848], locale ru-RU)
[√] Windows Version (Installed version of Windows is version 10 or higher)
[√] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
[√] Chrome - develop for the web
[√] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.5.4)
[√] Android Studio (version 2022.2)
[√] Connected device (3 available)
! Device <IP_ADDRESS>:37509 is offline.
[√] Network resources
Output of: flutter doctor -v
[√] Flutter (Channel stable, 3.16.2, on Microsoft Windows [Version 10.0.22621.1848], locale ru-RU)
• Flutter version 3.16.2 on channel stable at D:\Flutter\flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 9e1c857886 (6 days ago), 2023-11-30 11:51:18 -0600
• Engine revision cf7a9d0800
• Dart version 3.2.2
• DevTools version 2.28.3
[√] Windows Version (Installed version of Windows is version 10 or higher)
[√] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
• Android SDK at C:\Users\User\AppData\Local\Android\sdk
• Platform android-33, build-tools 33.0.2
• Java binary at: D:\Android Studio\jbr\bin\java
• Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-9586694)
• All Android licenses accepted.
[√] Chrome - develop for the web
• Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe
[√] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.5.4)
• Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community
• Visual Studio Community 2022 version 17.5.33530.505
• Windows 10 SDK version 10.0.22000.0
[√] Android Studio (version 2022.2)
• Android Studio at D:\Android Studio
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-9586694)
[√] Connected device (3 available)
• M2012K11AG (mobile) • <IP_ADDRESS>:40071 • android-arm64  • Android 13 (API 33)
• Windows (desktop)   • windows             • windows-x64    • Microsoft Windows [Version 10.0.22621.1848]
• Chrome (web)        • chrome              • web-javascript • Google Chrome 119.0.6045.200
! Device <IP_ADDRESS>:37509 is offline.
[√] Network resources
• All expected network resources are available.
• No issues found!
More environment information

Flame version: 1.11.0
Platform affected: android, windows
Platform version affected: android 13,  windows 11

I think you must be doing something wrong, print the output in onGameResize and you will see that the x and y switches their values.
Possibly you look at the values before the resize has happened?

I expected that with a landscape orientation x would be greater than y, i.e. x in landscape orientation will be equal to y in portrait orientation, and y in landscape orientation will be equal to x in portrait orientation

This is a bit confusing. Why would x be greater than y in landscape? Are you talking about (x, y) or (size.x, size.y)? Assuming this is about the size, even then, size of any component won't depend on the orientation. It will stay whatever it was set to.
Yeah, size.x > size.y in landscape will be true for the game object. It will also be true of any component that is set to resize proportionally when the game resizes. But it is not true for all component. For example, if a component's size is (10, 20), I would expect it to stay the same after changing orientation.

This is a bit confusing. Why would x be greater than y in landscape?

Ooh, are they not talking about the size? Then it is indeed very confusing...

@neverewer can you clarify a bit?

Closing this as stale.
@neverewer you can write here again if you want it re-opened.


Bump nor_dev_support to latest version

The nor_dev_support gem is out of date (currently 3.1.3) and needs to be bumped to the latest version (currently 5.6.0).

This was addressed by 6cb13f43, so can be closed.


The text for assessments (tasks.text) is displaying & instead of an ampersand

Describe the bug
The text for assessments (tasks.text) is displaying & instead of an ampersand.
To Reproduce
Steps to reproduce the behavior:

Go to an effort / assessment
Select the Notes icon at the top right of the page. Result: assessments will be displayed
See error in the text of the assessment

Expected behavior
An ampersand should be displayed instead of &
Screenshots
None
Environment (please complete the following information):
Firefox 78.2.0esr
Additional context
N/A

Duplicate of NCI-Agency/anet#3065


Build error, unresolved identifier, Swift 3

I have successfully installed Sqlite.swift from GitHub and have
been able to successfully open the database and save a row...
Now I am attempting to include code to close the database but getting 2 build errors:
Unresolved identifier on 'SQLITE_OK'
Unresolved identifier on 'sqlite3_close'
It would have though adding the 'import SQLite' would import any code in the installed pod.
however I must not be importing some module correctly, how do I fix this?
Here is some of the code I have in ViewController:
import UIKit
import SQLite
class ViewController: UIViewController {
@IBOutlet weak var cmdSave: UIButton!
@IBOutlet weak var cmdClose: UIButton!
private var dbs: Connection?
let users = Table("users")
let id = Expression("id")
let email = Expression("email")
let name = Expression<String?>("name")  //allow null
//--------------------------
@IBAction func cmdClose(_ sender: AnyObject) {
if sqlite3_close(dbs) != SQLITE_OK {
    print("error closing database")
}

dbs  = nil

}

I added  'libsqlite3.dylib'   >>> in Build Phases >>> Link Binaries with Libraries >>> Add Other >>>. Ctrl Shift G >>> /usr/lib/
As recommended in my other post but it did not correct the problem :(

You'll most likely need a bridging  header with
#import "sqlite3.h"

in it.

inactive


Fallback to lstat() when d_type is DT_UNKNOWN

By default, XFS doesn't store inode information in directory structure,
and readdir() always return DT_UNKNOWN as d_type.
So hw can't scan directories on XFS.

LGTM :smile:


#1378 Replaced WebL10n to i18next in Physics JS Activity

@llaske pls review this pr
attaching ss of it
Replaced WebL10n to i18next in Physics JS Activity

Nice. Thanks


Buffer updates to plumtree.

Plumtree can be easily overloaded; so periodically buffer updates and
broadcast them in an effort to reduce this problem.

The change looks very good.


Protractor waits for pending 101 requests

For some reason in Chrome console network tab there is a websocket request as "Pending" so apparently Pratractor keeps waiting this request to finish. This request has 101 HTTP status after a quick search I learned that it is suposed to be marked as "Pending" in Chrome console.
I believe Protractor should not wait for this request to finish once it will never finish. A workaround is to prevent syncronization:
browser.ignoreSynchronization = true;
And defined a timeout in configuration file:
allScriptsTimeout: 50000
I am using Protractor 5.1.1, Node 6.9.4, NPM 4.0.5; the application uses Angular 2.

Hi there!
Your question is better suited for StackOverflow or Gitter. Please ask a question there with the 'protractor' tag or post in the Gitter Channel to get help.
From the the getting help section of the README:

Please ask usage and debugging questions on StackOverflow (use the "protractor" tag) or in the Angular discussion group. (Please do not ask support questions here on Github.)

Thanks!

Actually is not a question, I believe it is a bug.
But if it is the expected behavior that's ok! ;)

Hi there,
from your issue it's unclear that this is a bug with protractor itself. It sounds like the site your are testing is making a websocket request (or are you saying that this is happening by default because you are opening the chrome network tab?) and causing protractor to hang. Because of this, it may be an issue isolated to your setup and not something it is necessarily wrong withProtractor.
If you can produce a small reproducible example and believe this is an issue with protractor itself i'd be happy to re-open the issue. Otherwise, it sounds like it's something that you'll need to resolve on your own.
I hope that makes sense, thanks!

Hello,
Sure, no problem I understand... I believe it is an error because the request responds with 101 HTTP Code and as far as I researched it is expected that Chrome console indicates it as "processing" forever even though the request is completed.
However it can be a problem in my application. Actually I am new in Angular and using a template so I don't know exactly why and where this request is made. Since I got this "processing forever" information on Google I thought maybe Protractor could threat. But, again, I'm new at this and may be a problem in my application. ;)
Thanks!


Update pt/lessons/basics/pattern-matching.md

Hi, I've seen on ElixirWeekly (Issue 20) there's help wanted for translating some lessons.
I've updated the Portuguese version mentioned on #809

Thank you @andresilveira!  @pragmaticivan could you take a peek at this when you have a moment?  Thank you.

Hey @andresilveira what do you think about replacing "Vamos dar uma olhada" for "Vejamos a seguir:"

Thanks for the feedback @pragmaticivan. Can you have another look?


VMware Workstation driver

First attempt at a VMware Workstation driver. Most of the code is direct duplicate of the Fusion driver with minor tweaks.
Current limitations:

Only works on Windows version of Workstation
Paths to vmrun, DHCP client table, and shared "Users" folder are hard-coded (with sensible values)
Assumes 64-bit Windows

+1
nice work. There should be a way to provide VMWare installation path on command line and hence use relative paths to it or fall back to default hard-coded values :)

@emsi great idea! I will work on that.

@emsi I've added you as a collaborator in my fork in case you would like to contribute. I'm sort of learning Go as I... "go" :smile: so if you have more experience it might be faster.

We have a CoreOS provisioner already IIRC, so that addition shouldn't be neccessary.
I'm going to close this for now (we're not accepting drivers into the core, we are moving to a pluggable model), but stayed tuned to make this into a driver plugin.  I have the initial versions of that (plugin) code done now too so contact me<EMAIL_ADDRESS>if you'd be willing to try it out :)

@zaps: thanks. I'm new to golang as well (just did few obvious hacks) but I found your patch extremely useful.  :)

@nathanleclaire Is this closed in favor of some other issue for the plugin version?

@masaeedu Perhaps @zaps can comment on whether they have continued development in their own repo.  Feel free to PR and add it to the AVAILABLE_DRIVER_PLUGINS.md as well if so.


Branch deploy updates

Branch Deploy Updates
This pull request does the following:

Uses the "merge commit" mode
Updates branch deploy to v6.0.0
Uses the "unlock on merge" mode

.deploy

.lock

.lock


Sometimes Sample Project Disappear in Visual Studio 2017

Not Completely Sure how to replicate yet.  but after switching the active project in the generated visual studio solution - the samples project directory disappears and you can no longer build the samples.

Closing Because I think this was due to changing cmake config outside of visual studio - so thats expected.


Pyramid support and using extends/block

I'm getting a pyramid_mako.MakoRenderingException when I try to extend a template and fill a block. Is this actually supposed to work or is this functionality unsupported when using jade templates in Pyramid?

Never mind, please. I guess this happened due to a misconfiguration in my project.


Syntax colouring fail on PHP v7.3 HEREDOC / NOWDOC termination

Issue Type: Bug
Since PHP v7.3 the HERE/NOWDOC syntax allows indentation of the terminator, and the semicolon is no longer required, even allowing for other symbols (comma in the case of docstrings in arrays).
The following code no longer colours correctly
<?php
    echo <<<NOWDOC
The quick brown fox
    NOWDOC
    ;
?>

VS Code version: Code - Insiders 1.31.0-insider (bae6c84d46cd85c911db7e166a74e698a54c8644, 2019-01-04T06:17:20.719Z)
OS version: Windows_NT x64 10.0.17134
Extensions (17)

Extension
Author (truncated)
Version

better-comments
aar
2.0.3

spellright
ban
3.0.22

vscode-icontheme-nomo-dark
be5
1.3.6

unique-lines
bib
1.0.0

markdown-preview-github-styles
bie
0.1.4

insert-unicode
bru
0.2.1

better-toml
bun
0.3.2

gitignore
cod
0.6.0

EditorConfig
Edi
0.12.6

php-intellisense
fel
2.3.10

vscode-phpfmt
kok
1.0.30

vscode-apache
mrm
1.1.1

php-docblocker
nei
1.7.0

gitconfig
sid
2.0.0

vscode-hexdump
sle
1.7.2

ignore-gitignore
stu
1.0.1

insert-special-symbol
swi
1.3.0

(2 theme extensions excluded)

This issue was moved to atom/language-php#346


Redisson Map Cache Event Notification doesn't work with HPE's NONSTOP OS

Redisson Map Cache Event Notification doesn't work with HPE's NONSTOP OS.
This is because of the endianness.
Below logic is there in MapCacheEventCodec.decode(ByteBuf buf, State state, Decoder<?> decoder) method,
if (isWindows) {
keyLen = buf.readIntLE();
} else {
keyLen = (int) buf.readLongLE();
}
Here, if it is windows, buf.readIntLE() is used, or else buf.readLongLE() is used. But for HPE's NONSTOP OS, buf.readLong() has to be used.
To fix this, two classes has to be modified,

RedissonMapCache - Like how isWindows is being set using "Windows" substring, isNonstop boolean also can be initialized using "NONSTOP" substring
if (isNonstop == null) {
RFuture<Map<String, String>> serverFuture = commandExecutor.readAsync((String)null, StringCodec.INSTANCE, RedisCommands.INFO_SERVER);
serverFuture.syncUninterruptibly();
String os = serverFuture.getNow().get("os");
isNonstop = os.contains("NONSTOP");
}
Also, while initializing MapCacheEventCodec, need to pass isNonstop boolean.
new MapCacheEventCodec(codec, isWindows, isNonstop) - in all the 4 places

MapCacheEventCodec - Need to declare isNonstop boolean and initialize(in constructor) based on the isNonstop boolean that is received from ReidssonMapCache

Also, in the decode method, like how isWindows is being checked, need to check isNonstop and use buf.readLong() if it is NONSTOP as given below.
if (isWindows) {
keyLen = buf.readIntLE();
}else if (isNonstop) {
keyLen = buf.readLong();
} else {
keyLen = (int) buf.readLongLE();
}
Could you please fix this in Redisson code?

Fixed! Thanks for report

Thanks for accepting this request.
Everything is proper in the fixed version(3.10.4), except that in MapCacheEventCode, Windows check is placed instead of NONSTOP check.
Below is the code for your reference,
if (osType == OSType.WINDOWS) {
keyLen = buf.readIntLE();
} else if (osType == OSType.WINDOWS) {
keyLen = (int) buf.readLong();
} else {
keyLen = (int) buf.readLongLE();
}
Please change that check to HPNONSTOP. Revised code is given below.
if (osType == OSType.WINDOWS) {
keyLen = buf.readIntLE();
} else if (osType == OSType.HPNONSTOP) {
keyLen = (int) buf.readLong();
} else {
keyLen = (int) buf.readLongLE();
}

Sorry, my mistake. Fixed

No Problem. Thanks.


Resolves #2446

Fix example title
https://github.com/MicrosoftDocs/windows-powershell-docs/issues/2446

Docs Build status updates of commit 93c3ca0:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2012r2-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (winserver2012r2-ps)

docset/winserver2016-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2016-ps)

docset/winserver2019-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2019-ps)

docset/winserver2022-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2022-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 8c7132c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2012r2-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (winserver2012r2-ps)

docset/winserver2016-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2016-ps)

docset/winserver2019-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2019-ps)

docset/winserver2022-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2022-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 2508ac1:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2012r2-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (winserver2012r2-ps)

docset/winserver2016-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2016-ps)

docset/winserver2019-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2019-ps)

docset/winserver2022-ps/activedirectory/Get-ADDomainController.md
:white_check_mark:Succeeded
View (WindowsServer2022-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Cherry-pick failover nil pointer panic fix from #13252

Signed-off-by: Liam White<EMAIL_ADDRESS>

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: liamawhite
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rshriram
If they are not already assigned, you can assign the PR to them by writing /assign @rshriram in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/pkg/networking/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liamawhite: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liamawhite: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-unit-tests

prow/istio-integ-k8s-tests.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liamawhite: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-unit-tests

prow/istio-integ-k8s-tests.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: howardjohn, liamawhite
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rshriram
If they are not already assigned, you can assign the PR to them by writing /assign @rshriram in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/pkg/networking/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liamawhite: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
5c02a544901ce7fdacd29e9f6254958d71887b9f
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-k8s-tests.sh
159815ca58f56cb719aeb57ea6cf55f5c9f291a8
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liamawhite: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
159815ca58f56cb719aeb57ea6cf55f5c9f291a8
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
159815ca58f56cb719aeb57ea6cf55f5c9f291a8
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


A new release for soundfile

This is a continuation of the discussion in #310, where I'm working on packaging a new version of soundfile.
It appears that the packaging problem is more complicated than anticipated. Our existing build system will probably need to be replaced, which will take some time.
If anyone would like to help me package soundfile, I'd be grateful!
The problem is, we need to package four wheels for

Windows 32 Bit
Windows 64 Bit
macOS Intel
macOS Arm

each containing their own libsndfile{.dll,.dylib}, and making sure that the right version gets installed on the right platform.
We have a build script in place for building packages for 1-3, but I don't yet know how to build wheels specifically for macOS Arm, and it seems that we should convert our build system to use PEP 517 (pyproject.toml/build et al).
I only had an hour or so today to work on this, so there wasn't much progress. I'll keep you updated.

Thanks for taking time to solve this, If u need any helping hands let me know

pyproject.toml is super nice but I don't think it will resolve your current issues with wheels.  What it can do is make it so that cffi is installed before the setup.py script is run which would crash otherwise.
I'm not sure about the specific CLI flags.  I think it's adding -arch arm64 to the C/link flags will work it if the tools support it.  Or -arch arm64 -arch x86_64 for a Universal 2 library.
The platform tags would be something like macosx_11_0_arm64.  If the library supports both arm64 and x86_64 then the platform tag would be macosx_10_9_universal2 instead (pulling examples from my own projects, I don't know how to figure out the best version numbers.)
For my own projects I have to rely on GitHub Actions to build MacOS packages/libraries for me as downloadable artifacts.  I can write a workflow for you if you're unable to compile the arm64 library yourself.

Thank you so much for your help!
If you could point me to an example of how to have Github compile the macOS libraries, that would be fantastic! That might actually allow us to compile the wheels separately on each OS, instead of the current franken-system that packages precompiled libraries with the generic python code.

This would be an example of a workflow.  This script would be stored in a path like .github/workflows/build-libs.yml in the repository, and will take affect as soon as it's pushed to the repo.
name: Compile libraries

on:
  push:
  pull_request:

jobs:
  build-libs:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ["macos-11"]
      fail-fast: true

    steps:
      - uses: actions/checkout@v2
      - name: Compile library
        run: ./mac_build.sh
      - uses: actions/upload-artifact@v2
        with:
          name: libs
          path: "*.dylib"
          retention-days: 7
          if-no-files-found: error

I imagine this running the mac_build.sh script from bastibe/libsndfile-binaries, so this workflow would be added there for this example.  You can also see the docs for actions/upload-artifact.
GitHub has good documentation on making workflows.  Build artifacts uploaded this way can be downloaded from another job, such as a job that builds the wheel.

This worked amazingly well, thank you a whole lot!
At the moment, Github runners do not yet provide M1 machines, but that's not super important at the moment, since @chuma9615 and @jurihock have already gracefully provided pre-build libraries in libsndfile-binaries#7 and libsndfile-binaries#8, respectively.
I'll investigate cross-compilation once I've managed to build wheels.

I have drafted a new release at releases/0.11.0b1.
There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.
If anyone would like to contribute up-to-date Python libraries compiled with Visual Studio 14+, I'd be grateful for a pull request in bastibe/libsndfile-libraries. Or better yet, integrate it into the CI scripts to build the binaries automatically.
Also, you would help me tremendously if anyone could test the provided wheels on macOS and M1!

Will you provide source distribution archives (tarballs) for the latest and upcoming releases or should Linux distro packagers use the source archives automatically created by Github?

I will upload a source distribution.
The reason I didn't attach it to the draft release linked above is that the current build script still bundles all the libraries with the source dist, making it unnecessarily big. This will need fixing before the final release is cut.

There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.

Looks like if you compile to v140 then that binary will support all versions of Python after 3.5.  The instructions I've found for telling MinGW to use specific runtimes seems convoluted so maybe it would be easier to use Vcpkg to get these builds.

Or better yet, integrate it into the CI scripts to build the binaries automatically.

I'm inexperienced with all the required compilation steps and CI. But I remembered that Mathias uses a script which builds a small version of libsndfile on Mac during some Github actions, build-small-libsndfile.sh. Forgive me if this doesn't help at all. :)

@SpotlightKid I just uploaded a fixed sdist to the latest tag.
Right now, all that's missing for a release is

someone should test the M1 wheels.
I just noticed that the macos x64 wheel does not install. Not sure yet what that is about.
I've opened a pull request for the new release in #326 and noticed that the automated tests no longer run. Does anyone know why?

@HaHeho thank you for your response. Thanks to @HexDecimal, we now have automated builds for Windows and macOS! That's wild!

... and noticed that the automated tests no longer run. Does anyone know why?

If you check the commits on GitHub then TravisCI's last status check was Nov 26, 2019. It stopped working sometime after that, probably when they dropped support for the travis-ci.org site.  You'd have to migrate to travis-ci.com, but I don't think that site actually gives free time for FOSS projects.  You might get a one time allotment to work with while you move to something else.  I remember dropping TravisCI for GitHub Actions over this.
More info: https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567

@SpotlightKid I just uploaded a fixed sdist to the latest tag.

Thanks for the heads up. The URLs for the attached files still have 0.11.0b1 in the path, which is slightly inconvenient. But I'm assuming this will be correct when a proper release is made.

Tested the wheel on a M1 machine and it's working

@chuma9615 that's the source-only wheel. Could you try the M1-specific soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl?

Could you try the M1-specific soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl?

Note that macosx-10.x-arm64 isn't a valid platform tag since the dash - is used to separate tags from each other and this should all be one platform tag.  The x and . in 10.x also isn't valid.  Pip and PyPI might panic when seeing the tags in this state.
The libs were built with MACOSX_DEPLOYMENT_TARGET=10.9 so maybe you meant this tag to be macosx_10_9_arm64 and so the wheel would be called soundfile-0.11.0b2-py2.py3-none-macosx_10_9_arm64.whl
How wheel tags work is clear, but MacOS specific info is harder to find:
https://www.python.org/dev/peps/pep-0425/
https://github.com/MacPython/wiki/wiki/Spinning-wheels
https://lepture.com/en/2014/python-on-a-hard-wheel
https://snarky.ca/the-challenges-in-designing-a-library-for-pep-425/

@bastibe When using the macOS ARM wheel it crashes
ERROR: soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl is not a supported wheel on this platform.

@bastibe there was a tiny typo bug in the code when importing sndfile on Mac. I fixed this in #328.

I was trying to package python-soundfile 1.11.0 for Nix on macOS arm64, and encountered the not a supported wheel problem:

Full log
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/vrckfw0c4wg41ak9nik5gymaigx6i11j-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/test_soundfile.py
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
running build_py
file _soundfile.py (for module _soundfile) not found
creating build
creating build/lib
copying soundfile.py -> build/lib
package init file '_soundfile_data/__init__.py' not found (or not a regular file)
file _soundfile.py (for module _soundfile) not found
generating cffi module 'build/lib/_soundfile.py'
installing to build/bdist.macosx-10.6-arm64/wheel
running install
running install_lib
creating build/bdist.macosx-10.6-arm64
creating build/bdist.macosx-10.6-arm64/wheel
copying build/lib/soundfile.py -> build/bdist.macosx-10.6-arm64/wheel
copying build/lib/_soundfile.py -> build/bdist.macosx-10.6-arm64/wheel
running install_egg_info
running egg_info
creating soundfile.egg-info
writing soundfile.egg-info/PKG-INFO
writing dependency_links to soundfile.egg-info/dependency_links.txt
writing requirements to soundfile.egg-info/requires.txt
writing top-level names to soundfile.egg-info/top_level.txt
writing manifest file 'soundfile.egg-info/SOURCES.txt'
reading manifest file 'soundfile.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'soundfile.egg-info/SOURCES.txt'
Copying soundfile.egg-info to build/bdist.macosx-10.6-arm64/wheel/soundfile-0.11.0-py3.9.egg-info
running install_scripts
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.macosx-10.6-arm64/wheel/soundfile-0.11.0.dist-info/WHEEL
creating 'dist/soundfile-0.11.0-py2.py3-none-macosx-10.x-arm64.whl' and adding 'build/bdist.macosx-10.6-arm64/wheel' to it
adding '_soundfile.py'
adding 'soundfile.py'
adding 'soundfile-0.11.0.dist-info/LICENSE'
adding 'soundfile-0.11.0.dist-info/METADATA'
adding 'soundfile-0.11.0.dist-info/WHEEL'
adding 'soundfile-0.11.0.dist-info/top_level.txt'
adding 'soundfile-0.11.0.dist-info/RECORD'
removing build/bdist.macosx-10.6-arm64/wheel
Finished executing setuptoolsBuildPhase
@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing pipInstallPhase
/private/tmp/nix-build-python3.9-soundfile-0.11.0.drv-0/source/dist /private/tmp/nix-build-python3.9-soundfile-0.11.0.drv-0/source
ERROR: soundfile-0.11.0-py2.py3-none-macosx-10.x-arm64.whl is not a supported wheel on this platform.

 You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

soundfile.py:1217: MemoryError

According to https://cffi.readthedocs.io/en/latest/using.html#callbacks, ffi.callback() is old style callback, and has many drawbacks, should be replaced by new style callback def_extern().

Thank you all so much for your help with this release!
And especially thank you @HexDecimal, for implementing the CI builds and tests!
I've uploaded a new set of pre-release wheels: 0.11.0b3.
I hope that these wheels now work on macOS M1. Again, I'd be grateful if anyone with access to an M1 Mac could test the macOS ARM wheel.

Thank you all so much for your help with this release!
And especially thank you @HexDecimal, for implementing the CI builds and tests!
I've uploaded a new set of pre-release wheels: 0.11.0b3.
I hope that these wheels now work on macOS M1. Again, I'd be grateful if anyone with access to an M1 Mac could test the macOS ARM wheel.

$ pip install https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

According to https://cffi.readthedocs.io/en/latest/using.html#callbacks, ffi.callback() is old style callback, and has many drawbacks, should be replaced by new style callback def_extern().

I've looked into this, and actually drafted an implementation of the callbacks locally. However, there is a problem with this approach: Currently, we are using the ABI mode of CFFI. The "new style" callbacks you mentioned require that soundfile uses the API mode of CFFI, which means compiling a bespoke version of soundfile for every operating system and version of Python we support.
While I am certainly open to doing this, especially now that we have CI runners to automate the task, I think it is out-of-scope for this pull request. Please raise the issue in a new issue, or preferably, try to draft a pull request for it.

$ pip install https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

MacBook Pro (13-inch, M1, 2020), macOS 12.2.1

I've seen a similar error on my wife's old mac. But it seemed to be related to her old version of pip. Perhaps that was wrong. Does your error go away if you install a current version of pip?
Regardless, the issue still needs fixing. I set the OS tag to macosx_10_9_XXX since it seemed to work on my wife's mac, and seemed preferable to macosx_10_9_XXX.macosx_10_10_XXX.macosx_11_0_XXX.future_macosx_that_we_dont_even_know_yet_XXX. But if that's what it takes then I'll do that instead. How annoying.
Could you confirm the correct OS tag to use? Perferably with some measure of backwards and forwards compatibility.
Anyway, my time to work on this for this week is up, so I'll have to defer the issue for next wednesday.

I've seen a similar error on my wife's old mac. But it seemed to be related to her old version of pip. Perhaps that was wrong. Does your error go away if you install a current version of pip?

No.
bash-3.2$ pip install -U pip
Requirement already satisfied: pip in ./lib/python3.8/site-packages (22.0.3)
bash-3.2$ curl -sLO https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
bash-3.2$ pip install soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

Could you confirm the correct OS tag to use? Perferably with some measure of backwards and forwards compatibility.

macosx_11_0_arm64 works for arm64.
bash-3.2$ mv soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
bash-3.2$ pip install soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
Processing ./soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
Requirement already satisfied: cffi>=1.0 in ./lib/python3.8/site-packages (from soundfile==0.11.0b3) (1.15.0)
Requirement already satisfied: pycparser in ./lib/python3.8/site-packages (from cffi>=1.0->soundfile==0.11.0b3) (2.21)
Installing collected packages: soundfile
Successfully installed soundfile-0.11.0b3

Does anyone know why we need an up-to-date pip for this?

Does anyone know why we need an up-to-date pip for this?

How tags are handed can change from version to version.  My guess is that older versions of Pip are looking for exact MacOS versions rather than compatible versions or something similar.  If that's the case then adding redundant versions would solve the issue for older versions of pip.
The links I've posted previously should help: https://github.com/bastibe/python-soundfile/issues/325#issuecomment-1044392935, mostly with which MacOS versions Python might be expecting, which could be 10.6, 10.7, or 10.9.
I think adding tags for 10.9 in addition to 11.0 might help.  For example the platform tag macosx_10_9_arm64.macosx_11_0_arm64 instead of macosx_11_0_arm64.

I will try that. Thank you.

libsndfile 1.1.0 just released, so now it's time to rebuild all the libraries with MP3 support, and then cut the release.

libsndfile 1.1.0 just released, so now it's time to rebuild all the libraries with MP3 support, and then cut the release.

Can't wait to benchmark mp3 performance!

I probably won't have time to look into this in-depth until next week. If anyone wants to take a stab at providing rebuilt libsndfile binaries, or updating our automated CI builds, please open a pull request in libsndfile-binaries.

Thank you!
I had searched the repo for open pull requests earlier, but only for "sndfile", not "libsndfile" 🙄. My bad.
At any rate, I think we can wait for the vcpkg to update, at which point our CI builds should be able to build the new version automatically. Do you happen to know how quickly these vcpkg pull requests are accepted?

I've worked on the macOS build script:
https://github.com/bastibe/libsndfile-binaries/blob/bastibe/1.1.0-build/mac_build.sh
It now seems to compile mpg123 and liblame correctly, but somehow none of our custom-built libraries are picked up by the libsndfile configure script any longer. Any help on this would be greatly appreciated! Fighting compilers is not my strong suite.

@bastibe m1 mac is not currently supported, so I guess one would have to build locally :-/ https://github.com/actions/virtual-environments/issues/2187

It should be possible, in theory, to cross-compile M1 binaries from an Intel Mac. But given that the one Mac I have access to is somewhat unreliable and not mine to tinker with, that's not something I can implement 🤷♀. If someone else wants to take a stab at it, I'd be very interested!

Alternatives to VCPKG that update frequently:
https://winlibs.com/    Use UCRT for Python 3.5+
https://www.msys2.org/    Easy updates
I don't have experience with compiling for Windows, but I am trying to learn.  I saw these projects recommended.  Hope they are helpful.

https://github.com/libsndfile/libsndfile/ says "Autotools is the primary and recommended build toolchain".  WinLibs and MSYS2 both support GNU Autotools.

@cgohlke
You have a link to https://github.com/bastibe/PySoundFile at https://www.lfd.uci.edu/~gohlke/pythonlibs
Please update your link to the renamed project: https://github.com/bastibe/python-soundfile
As you know, there are various problems when Python and the libsndfile DLL are linked to different versions of the Microsoft C runtime DLL.  See https://github.com/libsndfile/libsndfile/
This team is trying to correct the problem for Python v3.5+.  However, we Mac, Linux and inexperienced Windows users are having difficulty navigating the subtleties of Windows building for Python.
We would greatly appreciate you advice and help on how to design the CI scripts for Windows building at https://github.com/bastibe/libsndfile-binaries
A problem with VCPKG is that the version in the CI runner is not up-to-date or slow to update for the latest libsndfile release.
Is there a way to use build tools such that one libsndfile DLL will work with all Python versions starting with v3.5?

Good news: the CI runners were updated, and are building libsndfile binaries with MP3 support now!
Which means we now have up-to-date binaries for Windows 32/64, and macOS Intel over at https://github.com/bastibe/libsndfile-binaries/pull/11. We're still lacking macOS ARM binaries, however, and I don't have an appropriate computer to compile them on. If anyone with an M1 Mac would like to help, please open a pull request at https://github.com/bastibe/libsndfile-binaries/

We're still lacking macOS ARM binaries, however, and I don't have an appropriate computer to compile them on.

Maybe this can help?
https://github.com/quickemu-project/quickemu

I now have a full set of binaries available! Thank you very much to everyone who helped make this happen. I'll soon cut new wheels to beta-test, and then we can hopefully release the new version!

If you can, please make sure the wheels work on your platform, and can indeed open MP3 files!

could you build a new version for linux platform? thank you very much.

A new beta release is available here: https://github.com/bastibe/python-soundfile/releases/tag/0.11.0b4

@bastibe
I tried this beta on MacOS with M1 and arm64 version of Python 3.9.12.
Generated an mp3: sox -n sine.mp3 synth 10 sine 1000
Upon loading, I got an error Format not recognised
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 sf.read("sine.mp3")

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:282, in read(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)
    196 def read(file, frames=-1, start=0, stop=None, dtype='float64', always_2d=False,
    197          fill_value=None, out=None, samplerate=None, channels=None,
    198          format=None, subtype=None, endian=None, closefd=True):
    199     """Provide audio data from a sound file as NumPy array.
    200
    201     By default, the whole file is read from the beginning, but the
   (...)
    280
    281     """
--> 282     with SoundFile(file, 'r', samplerate, channels,
    283                    subtype, endian, format, closefd) as f:
    284         frames = f._prepare_read(start, stop, frames)
    285         data = f.read(frames, dtype, always_2d, fill_value, out)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:655, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    652 self._mode = mode
    653 self._info = _create_info_struct(file, mode, samplerate, channels,
    654                                  format, subtype, endian)
--> 655 self._file = self._open(file, mode_int, closefd)
    656 if set(mode).issuperset('r+') and self.seekable():
    657     # Move write position to 0 (like in Python file objects)
    658     self.seek(0)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:1213, in SoundFile._open(self, file, mode_int, closefd)
   1210 if file_ptr == _ffi.NULL:
   1211     # get the actual error code
   1212     err = _snd.sf_error(file_ptr)
-> 1213     raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1214 if mode_int == _snd.SFM_WRITE:
   1215     # Due to a bug in libsndfile version <= 1.0.25, frames != 0
   1216     # when opening a named pipe in SFM_WRITE mode.
   1217     # See http://github.com/erikd/libsndfile/issues/77.
   1218     self._info.frames = 0

LibsndfileError: Error opening 'sine.mp3': Format not recognised.

I can't see MPEG/MP3 in the output of sf.available_formats()
{'AIFF': 'AIFF (Apple/SGI)',
 'AU': 'AU (Sun/NeXT)',
 'AVR': 'AVR (Audio Visual Research)',
 'CAF': 'CAF (Apple Core Audio File)',
 'FLAC': 'FLAC (Free Lossless Audio Codec)',
 'HTK': 'HTK (HMM Tool Kit)',
 'SVX': 'IFF (Amiga IFF/SVX8/SV16)',
 'MAT4': 'MAT4 (GNU Octave 2.0 / Matlab 4.2)',
 'MAT5': 'MAT5 (GNU Octave 2.1 / Matlab 5.0)',
 'MPC2K': 'MPC (Akai MPC 2k)',
 'OGG': 'OGG (OGG Container format)',
 'PAF': 'PAF (Ensoniq PARIS)',
 'PVF': 'PVF (Portable Voice Format)',
 'RAW': 'RAW (header-less)',
 'RF64': 'RF64 (RIFF 64)',
 'SD2': 'SD2 (Sound Designer II)',
 'SDS': 'SDS (Midi Sample Dump Standard)',
 'IRCAM': 'SF (Berkeley/IRCAM/CARL)',
 'VOC': 'VOC (Creative Labs)',
 'W64': 'W64 (SoundFoundry WAVE 64)',
 'WAV': 'WAV (Microsoft)',
 'NIST': 'WAV (NIST Sphere)',
 'WAVEX': 'WAVEX (Microsoft)',
 'WVE': 'WVE (Psion Series 3)',
 'XI': 'XI (FastTracker 2)'}

Bugs so far:

soundfile.available_formats (and therefore, libsndfile's sf_command(NULL, SF_FORMAT_INFO, ...) does not seem to report MP3 as an available format.
soundfile.write does not know about .mp3 file extensions, but will happily save an mp3 if you supply format='MPEG', subtype='MPEG_LAYER_III', or use the extension .mpeg.

could you build a new version for linux platform? thank you very much.

The none-any wheel should work on Linux. But you'll have to provide your own libsndfile with MP3 support.

I tried this beta on MacOS with M1 and arm64 version of Python 3.9.12. Generated an mp3: sox -n sine.mp3 synth 10 sine 1000
Upon loading, I got an error Format not recognised

Does it work if you rename the file to $filename.mpeg, or supply the format/subtype manually (format='MPEG', subtype='MPEG_LAYER_III')?

I opened an issue at libsndfile to investigate the latter issue: https://github.com/libsndfile/libsndfile/issues/837
(Or is this a soundfile-problem? It doesn't seem to originate from the Python code, though)

Trying renaming to .mpeg gives a similar error:
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 sf.read("sine.mpeg")

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:282, in read(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)
    196 def read(file, frames=-1, start=0, stop=None, dtype='float64', always_2d=False,
    197          fill_value=None, out=None, samplerate=None, channels=None,
    198          format=None, subtype=None, endian=None, closefd=True):
    199     """Provide audio data from a sound file as NumPy array.
    200
    201     By default, the whole file is read from the beginning, but the
   (...)
    280
    281     """
--> 282     with SoundFile(file, 'r', samplerate, channels,
    283                    subtype, endian, format, closefd) as f:
    284         frames = f._prepare_read(start, stop, frames)
    285         data = f.read(frames, dtype, always_2d, fill_value, out)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:655, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    652 self._mode = mode
    653 self._info = _create_info_struct(file, mode, samplerate, channels,
    654                                  format, subtype, endian)
--> 655 self._file = self._open(file, mode_int, closefd)
    656 if set(mode).issuperset('r+') and self.seekable():
    657     # Move write position to 0 (like in Python file objects)
    658     self.seek(0)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:1213, in SoundFile._open(self, file, mode_int, closefd)
   1210 if file_ptr == _ffi.NULL:
   1211     # get the actual error code
   1212     err = _snd.sf_error(file_ptr)
-> 1213     raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1214 if mode_int == _snd.SFM_WRITE:
   1215     # Due to a bug in libsndfile version <= 1.0.25, frames != 0
   1216     # when opening a named pipe in SFM_WRITE mode.
   1217     # See http://github.com/erikd/libsndfile/issues/77.
   1218     self._info.frames = 0

LibsndfileError: Error opening 'sine.mpeg': Format not recognised.

Forcing the format (format='MPEG', subtype='MPEG_LAYER_III') does not work either
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [14], in <cell line: 1>()
----> 1 sf.read("sine.mp3", format="MPEG", subtype="MPEG_LAYER_III")

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:282, in read(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)
    196 def read(file, frames=-1, start=0, stop=None, dtype='float64', always_2d=False,
    197          fill_value=None, out=None, samplerate=None, channels=None,
    198          format=None, subtype=None, endian=None, closefd=True):
    199     """Provide audio data from a sound file as NumPy array.
    200
    201     By default, the whole file is read from the beginning, but the
   (...)
    280
    281     """
--> 282     with SoundFile(file, 'r', samplerate, channels,
    283                    subtype, endian, format, closefd) as f:
    284         frames = f._prepare_read(start, stop, frames)
    285         data = f.read(frames, dtype, always_2d, fill_value, out)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:653, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    651 mode_int = _check_mode(mode)
    652 self._mode = mode
--> 653 self._info = _create_info_struct(file, mode, samplerate, channels,
    654                                  format, subtype, endian)
    655 self._file = self._open(file, mode_int, closefd)
    656 if set(mode).issuperset('r+') and self.seekable():
    657     # Move write position to 0 (like in Python file objects)

File ~/miniforge/lib/python3.9/site-packages/soundfile.py:1480, in _create_info_struct(file, mode, samplerate, channels, format, subtype, endian)
   1477 else:
   1478     if any(arg is not None for arg in (
   1479             samplerate, channels, original_format, subtype, endian)):
-> 1480         raise TypeError("Not allowed for existing files (except 'RAW'): "
   1481                         "samplerate, channels, format, subtype, endian")
   1482 return info

TypeError: Not allowed for existing files (except 'RAW'): samplerate, channels, format, subtype, endian

I created a new environment to play with. Looks like a correct library is being loaded:
>>> sf._snd
<Lib object for '[...]/miniforge/envs/test/lib/python3.9/site-packages/_soundfile_data/libsndfile_arm64.dylib'>

Version is also correct:
>>> sf.__libsndfile_version__
'1.1.0'

@bastibe Some more debugging. I discovered that the standard version of libsndfile installed on M1 mac via brew does not have mp3 support enabled.
I edited the formula (brew edit libsndfile) by adding depends_on "mpg123". Upon re-building (brew reinstall --build-from-source libsndfile) I was finally able to use the in-built command (sndfile-info sine.mp3 ) which gave me the correct info. Before I would get something along the lines of "This version was built without mp3 support".
========================================
File : sine.mp3
Length : 80256
MPEG-1/2 Audio
----------------------------------------
  MPEG version   : MPEG 1.0
  layer          : 3
  rate           : 48000
  mode           : mono
  mode ext       : 0
  framesize      : 192
  crc            : 0
  copyright flag : 0
  private flag   : 0
  original flag  : 1
  emphasis       : 0
  bitrate mode   : constant
  bitrate        : 64 kbps

----------------------------------------
Sample Rate : 48000
Frames      : 481536
Channels    : 1
Format      : 0x00230082
Sections    : 1
Seekable    : TRUE
Duration    : 00:00:10.032
Signal Max  : 31688.3 (-0.29 dB)

Next, I removed the the libsndfile_arm64.dylib which was bundled with pysoundfile, in order to force pysoundfile to look for homebrew version of libsndfile - it got it.
sf._snd
<Lib object for '/opt/homebrew/lib/libsndfile.dylib'>

And now, finally, it reads the mp3 file and MPEG*_ appears in the list of available subtypes.
sf.read("/Users/as2/sine.mp3")

(array([ 0.00000000e+00,  2.81360957e-09,  3.50615470e-09, ...,
         4.67424543e-05,  2.53865019e-05, -9.59571480e-07]),
 48000)

I am just wondering whether the bundled libsndfile was compiled with mp3 support?

I am just wondering whether the bundled libsndfile was compiled with mp3 support?

I actually don't know, and have no way of checking: It was provided by another user, as I don't have an M1 mac myself.
Thus, I'll have to ask for help once again: If you have an M1 mac, could you run the version of mac_build.sh from the 1.1.0 branch of libsndfile-binaries to produce an M1 build of libsndfile with MP3 support?

could you build a new version for linux platform? thank you very much.

The none-any wheel should work on Linux. But you'll have to provide your own libsndfile with MP3 support.

After I installed libsndfile 1.1.0 in the centos server, I checked that version by sf.libsndfile_version, I get 1.0.25, not 1.1.0. How can I link the new libsndfile to the soundfile? Thank you very much.

could you build a new version for linux platform? thank you very much.

The none-any wheel should work on Linux. But you'll have to provide your own libsndfile with MP3 support.
I have installed libsndfile 1.1.0  and the none-any wheel in the centos server. I used the code to check:
sf.libsndfile_version
it is 1.1.0.
used bellow:
audio = BytesIO(request.body)
audio.name = request.headers.get("filename")
y, r = sf.read(data)
read the audio.
when It read a wav, It worked fine. But if read a mp3 audio. have follow error:
y, r = sf.read(data)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/soundfile.py", line 283, in read
subtype, endian, format, closefd) as f:
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/soundfile.py", line 655, in init
self._file = self._open(file, mode_int, closefd)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/soundfile.py", line 1213, in _open
raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7fc592257a40>: File contains data in an unimplemented format.
Could you give some help, Thank you very much!

@BobCN2017 this is known. Mp3 support isn't compiled in right now.
@bastibe Maybe it's better to remove the beta before we confuse more people. I will try more things this weekend

Follow this.

Could anyone else try to compile a version of libsndfile with MP3 support on an M1 Mac? Without your help, I won't be able to ship a soundfile wheel for M1.

@bastibe https://github.com/bastibe/libsndfile-binaries/pull/13

Thank you, @faroit, for providing updated binaries for M1!
There's a new beta release at https://github.com/bastibe/python-soundfile/releases/tag/0.11.0b5 for you to try!
This time, *.mp3 files should open correctly by default (no more shenanigans necessary such as renaming to *.mpeg, or supplying a format and subtype manually), and we should have MP3 support on M1!
I'd be grateful if you could, once again, test the wheels and see if they finally work on all platforms.

@bastibe success
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)
[Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf.info("testcase.mp3")
testcase.mp3
samplerate: 44100 Hz
channels: 2
duration: 25000 samples
format: MPEG-1/2 Audio [MP3]
subtype: MPEG Layer III [MPEG_LAYER_III]
>>> sf.read("testcase.mp3")
(array([[-0.02847968, -0.02057419],
       [-0.02475205, -0.02332359],
       [ 0.00175894, -0.01752098],
       ...,
       [-0.04847291, -0.07572973],
       [-0.07998063, -0.0545641 ],
       [-0.06100846, -0.04042743]]), 44100)
>>> sf.
sf.LibsndfileError(        sf.SEEK_SET                sf.SoundFileRuntimeError(  sf.blocks(                 sf.info(
sf.SEEK_CUR                sf.SoundFile(              sf.available_formats(      sf.check_format(           sf.read(
sf.SEEK_END                sf.SoundFileError(         sf.available_subtypes(     sf.default_subtype(        sf.write(
>>> sf.__version__
'0.11.0'

Thanks for your work and I'm looking forward to the new release!
Just a quick note: it looks like the 0.11.0b5 release sets the name of the package in setup.py to soundfile (whereas it was SoundFile in 0.10.0.)
I'm not certain, but it looks as though changing the capitalization of the package name might cause some headaches for other tools, such as poetry - would you please consider changing it back?

I'm not certain, but it looks as though changing the capitalization of the package name might cause some headaches for other tools, such as poetry - would you please consider changing it back?

Yes, we changed the name from the nonstandard "SoundFile" to a more common "python-soundfile", and the package name from "SoundFile" to "soundfile".
Is this actually causing problems, or is this just a worry at this point? As far as I know, pip does not care about capitalization.

Windows dlls are available:
libsndfilewin.zip

Currently I can't help with testing M1. But Wheel for macOS x86_64 can't read MP3 files. Wheel for Windows x64 is OK.

Hi @bastibe. Any updates on this? We are looking forward to use this new version. Thanks.

It seems that the macOS x64 wheel somehow links to some system-installed /usr/local/opt/lame/lib/libmp3lame.0.dylib, instead of our own libmp3lame.la.
I've tried a fix in libsndfile-binaries#14, but apparently that didn't work as intended and still links against /usr/local/opt/lame. Does anyone understand what's going on here?
As far as I'm aware that's the last blocker for the release.

I can help test on either Darwin arch if that's useful

Thank you for the offer!
The build issue on macOS/Intel is still blocking, and I haven't had time to work on it. Currently, libsndfile-binaries@11 links against /usr/local/opt/lame instead of the self-compiled liblame from build_mac.sh.
If anyone wants to try to fix this issue in the build script, or contribute a self-contained libsndfile with full MP3-support for macOS/Intel, we could get this ready for release.

This issue may be related and one to watch:
https://github.com/libsndfile/libsndfile/issues/852

Thanks to the outstanding work of @joetoddsonos in libsndfile-binaries@15, we should now have working macOS/Intel binaries with statically-linked MP3 support.
As far as I know, this was the last blocking problem for the 0.11 release 🤞.
Once again, I ask for your help, and try out the new wheels of python-soundfile-0.11.0b6. If these finally work on all supported platforms, we can publish the next release of soundfile with MP3 support!
Thank you all so much for your help and patience! This release wouldn't have been possible without your invaluable help!

I just uploaded the wheels to Pypi, finally! We should have working MP3 support in soundfile, now!

@bastibe success 💪
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)
[Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf.info("testcase.mp3")
testcase.mp3
samplerate: 44100 Hz
channels: 2
duration: 25000 samples
format: MPEG-1/2 Audio [MP3]
subtype: MPEG Layer III [MPEG_LAYER_III]
>>> sf.read("testcase.mp3")
(array([[-0.02847968, -0.02057419],
       [-0.02475205, -0.02332359],
       [ 0.00175894, -0.01752098],
       ...,
       [-0.04847291, -0.07572973],
       [-0.07998063, -0.0545641 ],
       [-0.06100846, -0.04042743]]), 44100)
>>> sf.
sf.LibsndfileError(        sf.SEEK_SET                sf.SoundFileRuntimeError(  sf.blocks(                 sf.info(
sf.SEEK_CUR                sf.SoundFile(              sf.available_formats(      sf.check_format(           sf.read(
sf.SEEK_END                sf.SoundFileError(         sf.available_subtypes(     sf.default_subtype(        sf.write(
>>> sf.__version__
'0.11.0'

It's still failing for me :(
Python 3.10.4 [conda env.]
Using latest pull from master branch
Case 1
import io
import soundfile as sf
from urllib.request import urlopen

url = "https://dl.espressif.com/dl/audio/ff-16b-1c-16000hz.mp3"
data, samplerate = sf.read(io.BytesIO(urlopen(url).read()))

Output
Traceback (most recent call last):
  File "/Users/apple/PycharmProjects/Outplay/Test/api.py", line 8, in <module>
    data, samplerate = sf.read(io.BytesIO(urlopen(url).read()))
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 282, in read
    with SoundFile(file, 'r', samplerate, channels,
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 655, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 1213, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
python_soundfile.soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x102ff4680>: Format not recognised.

Case 2
import soundfile as sf
sf.read("/Users/apple/PycharmProjects/Outplay/Test/test.mp3")

Output
Traceback (most recent call last):
  File "/Users/apple/PycharmProjects/Outplay/Test/api.py", line 13, in <module>
    sf.read("/Users/apple/PycharmProjects/Outplay/Test/test.mp3")
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 282, in read
    with SoundFile(file, 'r', samplerate, channels,
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 655, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/Users/apple/PycharmProjects/Outplay/Test/python_soundfile/soundfile.py", line 1213, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
python_soundfile.soundfile.LibsndfileError: Error opening '/Users/apple/PycharmProjects/Outplay/Test/test.mp3': Format not recognised.

CC: @bastibe

How did you install soundfile? You will need both soundfile 0.11.0, and a version of libsndfile that supports MP3. The wheels from Pypi do contain such a libsndfile, but your system's libraries or conda's might not.

How did you install soundfile? You will need both soundfile 0.11.0, and a version of libsndfile that supports MP3. The wheels from Pypi do contain such a libsndfile, but your system's libraries or conda's might not.

I tried 3 ways to install:-

Cloning latest master branch
Installing pre-release build
Directly installing from PyPi

It's working for python 3.85 :). I can settle with this.

Please manually download soundfile-0.11.0-py2.py3-none-macosx_10_9_arm64.macosx_11_0_arm64.whl from https://pypi.org/project/soundfile/#files. Then install the wheel using python -m pip install $filename.whl (use python -m pip instead of pip to make sure you're installing into the correct interpreter). Within that python, it should then work.
Otherwise, check if soundfile is picking up your system's libsndfile instead of the wheel-provided one. Library load order is a bit of a fickle beast sometimes.

Thanks @bastibe & co for this update!  I was looking at opening a PR to update the conda-forge recipe, and for that, it would be most helpful if there was also a source dist (tar.gz) on pypi.  Would it be possible to include that in a post-release?

Dear @bmcfee, thank you for alerting me to that. I simply forgot to upload the source dist. It's available now.

Excellent, thank you!

It seems that the macOS x64 wheel somehow links to some system-installed /usr/local/opt/lame/lib/libmp3lame.0.dylib, instead of our own libmp3lame.la.
I've tried a fix in libsndfile-binaries#14, but apparently that didn't work as intended and still links against /usr/local/opt/lame. Does anyone understand what's going on here?
As far as I'm aware that's the last blocker for the release.

Please manually download soundfile-0.11.0-py2.py3-none-macosx_10_9_arm64.macosx_11_0_arm64.whl from https://pypi.org/project/soundfile/#files. Then install the wheel using python -m pip install $filename.whl (use python -m pip instead of pip to make sure you're installing into the correct interpreter). Within that python, it should then work.
Otherwise, check if soundfile is picking up your system's libsndfile instead of the wheel-provided one. Library load order is a bit of a fickle beast sometimes.

I've tried everything else and still can't load mp3 files. How do I check if soundfile is picking up the right libsndfile version?

System error almost always means your file name is incorrect. It seems you are missing the backslash after C:.

System error almost always means your file name is incorrect. It seems you are missing the backslash after C:.

Omg thanks, didn't see that.
torchaudio.load is working now, but it's returning an empty tensor as waveform when it's called on mp3 files, although working fine with flac files. Any idea why that might be happening?

Try opening the file with straight soundfile instead of going through torch. Who knows what torch is doing in the background.

I'm facing the same issue: Here is my code:
# iterate over the XML files in the folder
    for xml_file in os.listdir(folder_path):
        if xml_file.endswith(".xml"):
            # parse the XML file
            tree = ET.parse(os.path.join(folder_path,xml_file))
            root = tree.getroot()

            # get the filename of the wav file
            filename = './wav/'+root.find('./head/recording').attrib['filename']
            audio_file = AudioSegment.from_file(f'{filename}.wav')

            # iterate over the segments
            for segment in root.findall('./body/segments/segment'):

                start = float(segment.attrib['starttime']) * 1000  # convert to milliseconds
                end = float(segment.attrib['endtime']) * 1000  # convert to milliseconds
                snippet = audio_file[start:end]
                snippet.export(f'./snippets/{segment.attrib["id"]}.mp3', format='mp3')

                # extract the text from the element tags
                text = ' '.join([element.text for element in segment.findall('./element')])

                # create a unique filename
                new_filename = f'./snippets/{segment.attrib["id"]}.mp3'

                audio, sr = sf.read(new_filename)
    ```

This works perfectly on OSX but not on ubuntu 22.04, any idea why?

@asennoussi Ubuntu 22.04 still packages libsndfile version 1.0.31, which, AFAIK, does not have MP3 support yet.
Also note that, due to errors in the CMake build files, the pen-ultimate release of libsndfile (1.1.0) does not have MP3 support in most Linux distribution packages either, even though this version does support MP3 in theory, but due to said errors it will not be compiled in. The latest version (1.2.0) fixes that, but this still has to arrive in Ubuntu.

Makes sense?
I just used  pydub's  AudioSegment.from_file method and then
# get the raw audio data as a one-dimensional array
data = audio.get_array_of_samples()
# get the sample rate
sample_rate = audio.frame_rate

Instead. I hope this helps out someone else.

I have compiled a preliminary binary wheel for Ubuntu recently, that I'll try to release for testing soon. This should allow MP3 support on Ubuntu if installed with pip, and if no system-libsndfile is present.

Please check out https://github.com/bastibe/python-soundfile/pull/364 for a beta-release of a binary wheel for Linux.

@bastibe  hello, I am so fraisated keep got this error:
soundfile.py", line 1226, in _init_virtual_io
def vio_get_filelen(user_data):
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
I installed from pypi, tried your M1 arm wheel, all fails.
Why? I also tried unsintall previous just keep error

I am still getting the same error even with version 0.12.1. Is there a solution to this issue?


[Event Hubs] Map all logging to log levels

In order to support the feature where we let users set loglevels, we need to re-visit all the logging we do. They need to be updated such that we know what combination of different log modules should be used in the enable() method of the debug package for each log level.
For more context on this, please see the plan outlined in https://github.com/Azure/azure-sdk-for-js/issues/2662#issuecomment-491648021
Log levels generally are

None
Error
Warning
Info
Verbose

For example,

Verbose would map to *
we dont have anything for "warning" at the moment, we need to review what exactly will fall under the "warning" category and make the necessary changes
Info would map to azure:event-hubs:*
Error would map to azure:event-hubs:error. This needs review as well, because today we use log.error() to log non errors as well.

Please revisit the latest guidelines around logging before starting making any code changes here

Summary
The @azure/event-hubs library currently supports logging through the debug module.
The library implemented logging prior to the general logging guidelines were published.
The areas before will describe an attempt to bring @azure/event-hubs logging in line with out guidelines.
Supported Log Channels
Proposal
The event-hubs library should only support the following log channels and they must be exposed to the user:
export const logError = debug("azure:event-hubs:error");
export const logWarning = debug("azure:event-hubs:warning");
export const logInfo = debug("azure:event-hubs:info");
export const logVerbose = debug("azure:event-hubs:verbose");

A user likely doesn't know or care about individual components of a library.
Each feature-level log channel we create also needs seperate channels for each log level.
They also need to be exposed directly to the user so they can configure custom logging.
Also with the current SDK log channels, the service-level log channels would never be used
because only the feature-level log channels would be written to.
This can become confusing to a customer then that wants to customize logging for the library.
Using only a service-level log channel keeps the number of channels we expose relatively small
and makes it easier to enable/customize logging for the library.
Background
The event hubs sdk has existing log channels that are used by the SDK today:
https://github.com/Azure/azure-sdk-for-js/blob/%40azure/event-hubs_5.0.0-preview.2/sdk/eventhub/event-hubs/src/log.ts
The guidelines specify the following with regards to naming and creating log channels:

DO prefix channel names with azure:<service-name>.
DO create log channels for the following log levels with the following channel name suffixes:

Error: :error
Warning: :warning
Info: :info
Verbose: :verbose

YOU MAY have additional log channels, for example, to log from separate components. However, these channels MUST still provide the three log levels from above for each subchannel.

Enabling Log Levels
Proposal
The guidelines around enabling log levels are not yet finalized.
With this in mind, I'm not sure we should attempt to enable logging for log levels other than those explicitly defined.
Take the following log levels defined in the guidelines:

Error: :error
Warning: :warning
Info: :info
Verbose: :verbose

We might expect that if a user enables azure:event-hubs:warning, that azure:event-hubs:error should also be enabled.
To enable both of these, users can instead specify the azure:event-hubs:(warning|error) as their DEBUG environment variable.
Using Log Levels
Proposal
Error
I'm not aware of any known scenarios in event-hubs where we could detect an error the user's application
could not recover from.
As such, I don't think we should be using the error log channel at this time.
Warning
Most places where we use log.error today would be good candidates for log.warning, with a few exceptions.
We should use the warning log level anywhere in our code that we catch an error, provided it:

will not be automatically retried. These should be sent to the verbose channel.
is not an AbortError. These should be sent to the info channel.

Also note that stack traces should be sent to the verbose channel.
Example:
async function foo(abortSignal?: AbortSignalLike): Promise<void> {
  try {
    return await bar(abortSignal);
  } catch (err) {
    if (err.name === 'AbortError') {
      log.info(err.message);
    } else if (err.retryable && canStillRetry) {
      // assuming we will automatically retry this error...
      log.verbose(err.message, err.stack);
    } else {
      log.warning(err.message);
      log.verbose(err.stack);
    }
    throw err;
  }
}

Info
Many places where we currently use a feature log channel should use the info log channel.
As mentioned in the warning section, AbortError should also use the info log channel.
This channel is meant to be used when a function operates normally.
For example, the onMessage handler used by receiveBatch logs a message just before returning a successful response
and is a good candidate for the info channel.
https://github.com/Azure/azure-sdk-for-js/blob/404347b24cadefb7c5eba817f3d5260cac2bd966/sdk/eventhub/event-hubs/src/receiver.ts#L348-L354
Verbose
The verbose channel is meant for logging information that's useful for troubleshooting.
Many places where we currently use a feature log channel for non-happy-path scenarios should use the verbose channel.
Stack traces for errors or warnings should also be sent to the verbose channel.
For example, the setTimeout invocation used by receiveBatch logs a message.
This is useful to know when troubleshooting but doesn't provide value in an application that is running normally.
https://github.com/Azure/azure-sdk-for-js/blob/404347b24cadefb7c5eba817f3d5260cac2bd966/sdk/eventhub/event-hubs/src/receiver.ts#L374-L380

Background
The SDK guidelines specify the following rules for determining which log level to write to:

DO use the Error channel for failures that the application is unlikely to recover from (out of memory, etc.).
DO use the Warning channel when a function fails to perform its intended task.
This generally means that the function will raise an exception.
Do not include occurrences of self-healing events (for example, when a request will be automatically retried).
DO use the Info channel when a function operates normally.
DO use the Verbose channel for detailed troubleshooting scenarios.
This is primarily intended for developers or system administrators to diagnose specific failures.
DO log to the Info channel if a service call is cancelled.
DO log exceptions thrown to the Warning channel.
Additionally, send stack trace information to the Verbose channel.

Work has been started in #5401 but needs work due to some major refactoring that's occurred.


[HostListener]: Add ability to bind this to window: and document: listeners

The problem
There's occasionally need for a instance of a component to listen for an event outside of itself  (e.g. on document or window), that is dispatched from outside of itself (another component, some 3rd party JS). In this case, instances have to bind themselves to the event listener so we can get back to the component internals.
In the case of #11085, we bound the instance to the event listener, and typescript had issues with the fact that we were listening for a CustomEvent:
No overload matches this call.

  Overload 1 of 2, '(type: keyof DocumentEventMap, listener: (this: Document, ev: Event | ClipboardEvent | UIEvent | AnimationEvent | MouseEvent | ... 13 more ... | ErrorEvent) => any, options?: boolean | ... 1 more ... | undefined): void', gave the following error.
    Argument of type '"cm-app-pane-displayed"' is not assignable to parameter of type 'keyof DocumentEventMap'.

  Overload 2 of 2, '(type: string, listener: EventListenerOrEventListenerObject, options?: boolean | AddEventListenerOptions | undefined): void', gave the following error.
    Argument of type '(event: CustomEvent<any>) => void' is not assignable to parameter of type 'EventListenerOrEventListenerObject'.
      Type '(event: CustomEvent<any>) => void' is not assignable to type 'EventListener'.
        Types of parameters 'event' and 'evt' are incompatible.
          Type 'Event' is missing the following properties from type 'CustomEvent<any>': detail, initCustomEventts(2769)
(method) Document.addEventListener<keyof DocumentEventMap>(type: keyof DocumentEventMap, listener: (this: Document, ev: Event | ClipboardEvent | UIEvent | AnimationEvent | ... 14 more ... | ErrorEvent) => any, options?: boolean | ... 1 more ... | undefined): void (+1 overload)

These overload issues don't seem to occur when using the HostListener, so I propose we add the ability for the HostListener property to bind this to the callback function
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

This can be done by defining methods using arrow functions:
@hostlistener('document:click')
protected handleClick = (evt) => {
  console.assert(this !== window) // passes
}

In contrast, defining methods as named functions will assume this is equal to globalThis (window)
@hostlistener('document:click')
protected handleClick(evt) {
  console.assert(this !== window) // assertion failed!
}


Does the current model support multi-round dialogue capabilities?

Does the current model support multi-round dialogue capabilities?
If you train such a model with your own data, are there any special requirements for your own data set? For example, multiple rounds of dialogue training samples?
thanks！

You can interact with the 13B demo here: https://huggingface.co/spaces/chansung/Alpaca-LoRA-Serve
You can see examples of the training set here: https://github.com/tloen/alpaca-lora/blob/main/alpaca_data_cleaned.json
And here: https://github.com/gururise/AlpacaDataCleaned

Thank you very much！
If I want to train a model with a 13B, what is the minimum configuration for my GPU specifications?

Thank you very much！
If I want to train a model with a 13B, what is the minimum configuration for my GPU specifications?

You can interact with the 13B demo here: https://huggingface.co/spaces/chansung/Alpaca-LoRA-Serve
You can see examples of the training set here: https://github.com/tloen/alpaca-lora/blob/main/alpaca_data_cleaned.json And here: https://github.com/gururise/AlpacaDataCleaned

Thank you very much！
If I want to train a model with a 13B, what is the minimum configuration for my GPU specifications?


AssertionError / read_raw_brainvision

cc @teonlamont
I get an AssertionError while loading binary file from BrainVision - Version 2.0.
 398         # extract the first section to resemble a cfg
    399         l = f.readline().strip()
--> 400         assert l == 'Brain Vision Data Exchange Header File Version 1.0'
    401         settings = f.read()
    402
AssertionError:

Any idea about the reason?

because when I wrote it, I didn't know if the header would be different across the version. I wasn't sure if I could make it general. If they have the same pattern, then I could lax the requirement.

This has been fixed


add a static IP address to an existing AG listener

will the client/applications connection with SQL server be broken when we adds a static IP address to an existing availability group listener ?
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b679be86-06b0-b425-cd4f-d7b3ea104693
Version Independent ID: aec2cf5b-5bf9-647e-6b0f-9fab6fca3a79
Content: Configure availability group listener - SQL Server Always On
Content Source: docs/database-engine/availability-groups/windows/create-or-configure-an-availability-group-listener-sql-server.md
Product: sql
Technology: high-availability
GitHub Login: @MashaMSFT
Microsoft Alias: mathoma

@declare2006 -- thank you for your question.  Please try these resources:

DBA Stack Exchange
Stack Overflow


re-remove rails things

Does this about cover it @Zogoo?
Feel to merge if so.
I think you have push access too, you can push to this if there were other changes you had in mind.
I had those other merges in v0.15.0 so I can push 0.16.0 after this.

@mjobin-mdsol idea of removing rails-related things is to avoid having questions on the issues like how to override methods for this controller etc.

@mjobin-mdsol if you still want to put back the rails app let's discuss over the issue meanwhile let me merge this PR because this was removed before.


What is the recommended way to aggregate documentation?

I'm using slate to document multiple micro-services whose code resides in different Github repositories.  I would like to keep documentation for each service in it's own repository in order to keep the documentation and the code synched.   What is the recommended way of aggregating documentation from each code repository into one slate repo?
I can obviously create a Makefile which does this, but was looking for a more elegant solution.
I was thinking of maybe being able to add links to documentation in /source/index.md.  I'm not sure if that's possible though.

There's no official way to do this, but maybe you could have every other repo be a subrepo of your documentation repository, and then write a custom helper method that reaches in and pulls out the docs?


Feature/central bot kube

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@shapirov103 Good with this PR


Increase WEBrick request timeout to 300 seconds. Fixes #137

Please see issue #137

@jkeiser, @tyler-ball Any chance you could make a decision on this PR?

👍

👍


Update Build.OpenJDK_generic.cmd

Fix translation lost encoding

for #301

I have just push-force same commit with better commit log message


ITK template export

Hi Viktor, can you please make it one commit? (By squashing locally + git push --force-with-lease).

Yes, if it passes, I will.

I guess it's a follow-up to pull request https://github.com/SuperElastix/elastix/pull/412 ('fixed_template_export_macro_in_classes') by @thewtex right?

@N-Dekker Do you want to check this PR or shall I merge it?

Could you possibly still extend the commit message to clarify the purpose of the commit? Also mention that it's a follow-up to
Revision: df2a3ce40c34e94abf0613e5f7f113bf1ca97b0f
Author: ViktorvdValk
Date: 26 Nov 2020 14:43:16
Message: COMP: Fix template export macro in classes

@N-Dekker Shall I leave ITK_TEMPLATE_EXPORT on the templates you just mentioned? Since there is currently no way to verify if the macros are needed or not.

For the record,
Related ITK commits by MayeulChassagnard and Matt McCormick, 5 Dec 2016:

https://github.com/InsightSoftwareConsortium/ITK/commit/be14aef1ec8646982fa717b7a88b8f1b60558717 "BUG: Add ITK_TEMPLATE_EXPORT to template classes"

https://github.com/InsightSoftwareConsortium/ITK/commit/8bc35937c09657002bfd05517a0edf5dec736695 "ENH: Add tests to demonstrate OSX dynamic_cast failure"

I'll extend the merge message and clarify the purpose of the commit, once I merge the PR, is that oke?

I'll extend the merge message and clarify the purpose of the commit, once I merge the PR, is that oke?

OK, no problem, but why not just do a git amend + git push --force-with-lease from your pull-request-branch?


Document SQL shell help options

Fixes #1810

This change is

@knz, PTAL. Suspect I might have missed a few aspects of the inline help functionality...

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/1901453bee9a26dbd6f4ae8e622840bc95f2800e/

 with nit about ctrl+c

Reviewed 1 of 1 files at r1, 1 of 1 files at r2, 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

v1.1/use-the-built-in-sql-client.md, line 86 at r1 (raw file):

Command | Usage
--------|------------
`\q`
`ctrl-d`
`ctrl-c` | Exit the shell.

ctrl-c does not only exit the shell. If you have entered some text on a line already, it will clear the line first without exiting. Only on a line on its own does it exit the shell. (This is not new)

Comments from Reviewable

Review status: 2 of 4 files reviewed at latest revision, 1 unresolved discussion.

v1.1/use-the-built-in-sql-client.md, line 86 at r1 (raw file):
Previously, knz (kena) wrote…
ctrl-c does not only exit the shell. If you have entered some text on a line already, it will clear the line first without exiting. Only on a line on its own does it exit the shell. (This is not new)

Done.

Comments from Reviewable

TFTR, @knz

Comments from Reviewable

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/ca7e5cb8974f71ddc368572c93f968fa364ca37b/