Fix enum to be more conventional and simplify AblyMessage structure

I simplify the AblyMessage structure, so that there are fewer nested messages. To do this, I add a handle field to AblyEventMessage directly. This also allows me to stop using ablyDo, IMHO this extra level of indirection was confusing to me when first joining (it certainly confused Android Studio, since it caused a lot of linter errors). What makes it extra confusing was the logic that only sometimes created nested messages.

Most of the changes in this PR is probably due to this change.

I rename the cases in the CodecType enum to be the standard naming convention. This means when it used in Swift, it will be converted nicely: e.g. CodecTypeAblyMessage in Objective-C will appear to be CodecType.AblyMessage. Currently, AblyMessageCodecType will appear as _Value.ablyEventMessageCodecType in Swift, because the enum is declared as typedef NS_ENUM(UInt8, _Value).
I rename classes in both Android and iOS to AblyClientStore. On iOS it was called AblyFlutter (confused with AblyFlutterPlugin), and in Android it was called AblyLibrary (vague). What does it do? It stores ably client instances. I've actually mostly rewritten this class in Swift, but that is for a future PR. This also allows me (in a future PR) to rename AblyFlutterPlugin to AblyFlutter: this is the main class for the plugin.
I rename a method from realtimeWithHandle to getRealtime to be consistent with the rest equivalent: getRest.
Removed a file that was duplicated from the Android library (BiConsumer) and just importing it instead

PS: Please excuse the identation / formatting. I did not want to auto format iOS and Android code since they have not been formatted for a while, and the diff would be unnecessarily large. We will tackle this in a separate PR https://github.com/ably/ably-flutter/issues/216

Closing this PR because it mixes up too many things: https://github.com/ably/ably-flutter/issues/245


abbr extension breaks title attribute added using attr_list

Bug description
*[abbr]: Abbreviation Definition

![Image with abbr in title](abbr.png){title="Image with abbr in title"}

This renders the attribute code as plain text because the Abbreviation extension modifies the text in the attribute causing it to no longer be a valid attribute list.
Reproduction
9.5.18-abbr-attr_list-conflict.zip
Steps to reproduce

Enable the abbr and attr_list extensions
Include an abbreviation definition on a page
Use the abbreviation in a title attribute for an object using the attr_list syntax

I can confirm this is an issue.
>>> import markdown
>>> src = '''
... *[abbr]: Abbreviation Definition
...
... ![Image with abbr in title](abbr.png){title="Image with abbr in title"}
... '''
>>> markdown.markdown(src, extensions=['abbr', 'attr_list'])
'
{title="Image with abbr in title"}
'

The cause of the issue is that the abbr extension runs an inline processor while the attr_list extension runs a treeprocessor. If both extensions used the same type of processor we would just ensure they were ordered so that the attr_list processor ran first.  But being different processor types, it is not so easy. Especially because we generally want inline processors to run before attr_list so that attribute lists can be applied to inline elements.
This may require use of a custom treeprocessor for the addr extension which runs later. I will have to think about that.


Handle UnionArray in get_fb_field_type

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-8546
None

Resolved by #1135
(Btw @alamb or @tustvold, is there a particular preference to how to close these old issues that were already resolved, if it makes a difference to how changelog is generated for releases? Completed vs not planned?)

I suppose you could put the development process label on them, so they're excluded, but I haven't been doing so

I added the label for now -- I agree it would be nice to exclude them from the changelog, but also it isnt' the end of the world if they are in there too


Refactor http response status equatable

Refactored extension HTTPResponseStatus: Equatable, no change to the efficacy of the code.
Motivation:
Code readability is essential to a long-lasting maintainable project, the code this PR changes was "readable" however this PR will enhance the readability by removing excess lines of code.
Modifications:
Changed a long case statement, to a much shorter one.
Result:
There will be fewer lines of code, no change to the efficacy of product should occur. I am not familiar enough with the compiler to know if this will have a deleterious effect on performance I don't think it should be based on the way public var code: UInt { get } is implemented.

@helje5 pulling this convo in from #327.
so custom(418) is equal to .teapot, but custom(418,”earl gray”) differs to custom(418,”whatever”) 🤔

are you stating that custom anything lhs/rhs should be cased out, I can see the motivation for that. It is still much shorter then long case statement. I will make change.

I think what we have here is mimic the same as we had before.  If we want to change this another PR should be done IMHO.

@swift-nio-bot test this please

@pedantix IMO using a closed enum in a open API is deeply flawed in the first place, as demonstrated by the teapot (apart from the SemVer major source incompatibility you introduce by adding a case, the ugliness of custom, and all the bugs revolving around the .custom pattern).
Wrt the specific question: yes, the reason must be ignored. Why? Because it is already is stripped by the parser! I.e. you don't actually get the original reason for "builtin codes" and hence can't know whether it matches some other reason you got.

What could be done is to group all the cases that return true to one, and have only one line that returns true, without resorting to a default case.

@Lukasa does the code in my PR not keep the current behavior? As I read it, there is an ever-growing number of cases that pattern matches on if custom compares stored values, if not compare match on equal symbols(which is the source of the growth) if not default to false. I may be missing a swift subtlety but my PR is not aimed at changing functionality just reducing growth. There is definitely a weakness with using a default case.  As @BasThomas says all the true pairs could be reduced to a single case, I am not sure if this is any better in terms of readability.

Sorry, you're right. :)

@swift-nio-bot test this please

@pedantix thanks!

@normanmaurer @weissi @Lukasa  thank you! -119 lines of code as a net gain I think is just cool. This project is so great to make use of thanks!!!


Removing lingering GopherJS support debt

This removes some remaining / lingering GopherJS support debt - which exists primarily due to supporting older GopherJS versions. https://github.com/gopherjs/vecty/issues/264
Fixes:

https://github.com/gopherjs/vecty/issues/266
https://github.com/gopherjs/vecty/issues/261

Codecov Report

Merging #267 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #267   +/-   ##
=======================================
  Coverage   59.55%   59.55%
=======================================
  Files           4        4
  Lines         675      675
=======================================
  Hits          402      402
  Misses        215      215
  Partials       58       58

Impacted Files
Coverage Δ

dom_native.go
57.89% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3dfb52c...f31a9d8. Read the comment docs.


Prisma Client extensions docs improvement

[ ] Review current docs and find a way to make them "approachable"
[ ] Create a starter repository for users to build their own extensions
[ ] Add an extension example for adding field properties on a query method

Closing it in favor of https://github.com/prisma/docs/issues/4842


[DOC] Convert ingest processor supports ip type

What do you want to do?

[x] Request a change to existing documentation
[ ] Add new documentation
[ ] Report a technical problem with the documentation
[ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
In release 2.14.0, the convert ingest processor will support ip type, we need to document that.
What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
https://github.com/opensearch-project/OpenSearch/pull/12818

Hi @gaobinlong , what is your ETA for producing the doc PR? To meet entrance criteria, the PR must be open and in review by 4/30. Thanks!


infinite recursion encountered

Hi all,
with current master branch my nixos modules do not work anymore.
My modules look like this:
http://nixpaste.noip.me/raw/Ulg5mEaj
And here is the --show-trace output:
http://nixpaste.noip.me/raw/Oa5NzJIt
I import the module in my configuration.nix like this:
imports =
    [
      ./hardware-configuration.nix
      ./snmpd.nix
      ./kled-hdd.nix
      ./kled-net.nix
      ./btrsnap.nix
      ./dovecot.nix
      ./fetch-nixos-irc.nix
      ./htpasswd-file.nix
      ./containers.nix
      ./mosix.nix
    ];

@shlevy

OK, after reading PR #6794 I changed {config, pkgs, ...}: to {config, pkgs, lib, ...}: and with pkgs.lib; to with lib;
Now I get a new error: error: anonymous function at "/etc/nixos/nixpkgs/nixos/modules/services/networking/ssh/sshd.nix":382:29 called with unexpected argument ‘_module’, at undefined position
Is this another issue or a new follow-up error.

This is how I have sshd set up in configuration.nix:
# Enable the OpenSSH daemon.
    openssh = {
      enable = true;
      listenAddresses = [ { addr = "<IP_ADDRESS>"; port = 22; } { addr = "<IP_ADDRESS>"; port = 64022; } ];
    };

@nbp

I'm also getting infinite recursion on a config running against then nixos-unstable branch (on the channel branches). The trace looks different, so perhaps it's a different bug. I'll look into it some more.

Yeah, pretty sure anything running config against nixos-unstable just unconditionally gets infinite recursion. I thought there were tests?

Adding the blocker tag because I think it basically breaks all of nixos-unstable. Tell me if you have it working and we can try something else.

@robberer, This is sadly an expected behaviour which is that _module.check (or its old alias) should not be defined with any other option definitions, but only with an unconditional value.

@copumpkin @robberer  can you link to a minimal configuration which reproduce this issue?

@nbp here is a minimal config that reproduce the sshd error. no custom modules included. I can reproduce this on multiple real hosts.
http://nixpaste.noip.me/raw/rxWrMt9u

But I guess, just copy and paste the openssh service part to any config is enough to reproduce

@copumpkin Do you get "infinite recursion encountered" with self written modules outside of nixpkgs tree ? If so, did you applied the changes mentioned in PR #6794 ?
With those hints - especially the "mkOption, mkIf" part, I'm not getting "infinite recursion encountered" with other "custom" modules so far.
@nbp
But I noticed at least one more error related to #6794.
Hint: Apache SSL Virtual Hosts .
So I guess the mentioned PR is bigger regression than I thought. It seems to break almost any common config out there.

So I guess the mentioned PR is bigger regression than I thought. It seems to break almost any common config out there.

I don't think so, at least hydra VM tests in addition to many other configuration files are working fine.
I will fix it as soon as I find some time to do it. (hopefully this week)
Sorry for the delay.

Don't worry. I'm glad to hear that someone is about to investigate this issue. :+1:

For what it's worth, I have a fairly non-trivial config with many derivations defined outside of nixpkgs and everything is still working fine here.
However, I don't have any SSH support defined in my config.

Here is the minimal configuration with which the module system fails:
{
  boot.loader.grub.device = "/dev/sda";
  fileSystems."/".label = "root";
  services.openssh = {
    enable = true;
    listenAddresses = [ { addr = "<IP_ADDRESS>"; port = 22; } { addr = "<IP_ADDRESS>"; port = 64022; } ];
  };
}

Removing any line in this configuration change the error message, or make the configuration work.

Ok, I think I found the source of the issue.  I will make a patch tomorrow as I have to catch a train.
To make it short, in sshd.nix we map the result of  listenAddresses with a function which explictly expects 2 arguments, but no _module argument.
The solution is to strip the _module attribute, or to use ellipsis.

My failure didn't explicitly touch sshd but I did have a an extraUsers
entry with an ssh key. Would that also infinitely recurse?
On Tuesday, April 14, 2015, Nicolas B. Pierron<EMAIL_ADDRESS>wrote:

Ok, I think I found the source of the issue. I will make a patch tomorrow
as I have to catch a train.
To make it short, in sshd.nix we map the result of listenAddresses with a
function which explictly expects 2 arguments, but no _module argument.
The solution is to strip the _module attribute, or to use ellipsis.
—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/7354#issuecomment-93089865.

@robberer, I pushed the following branch in the mean time.
https://github.com/nbp/nixpkgs-2/tree/issue-7354
@copumpkin , I will look at your issue with extraUsers, but this is likely similar.

@nbp thanks! if it ends up being hard to repro I'll try to trim down my configuration to a useful case, but it sounds like we're on the right path already

I also got an "infinite recursion encountered" error after trying to upgrade to nixos-15.05pre60707.4e51a46 but it was caused by the following entry in my hardware-configuration.nix:
require = [
  "${modulesPath}/installer/scan/not-detected.nix"
];

This was fixed by changing it to:
imports = [
  <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
];

I have no idea if this is related, but perhaps someone else runs into this too.

@copumpkin , I checked other use cases, I was not able to find a similar pattern where we use an attribute set deconstruction on a subsmodule/optionSet.  Would you be able to make a minimal test case out of your configuration?

@cillianderoiste , hum this is unfortunate, knowing that we were generating such files.  This fix was suggested as part of #6794.  I will add this to the release notes, but then we should probably revert the modulePath change at least to display a message when it is used.

@copumpkin , I will close this issue, feel free to open a new one for extraUsers.


Can we validate that the GVR of *unstructured.UnstructuredList matches with deploymentConfig GVR, and then start to validate.

Can we validate that the GVR of *unstructured.UnstructuredList matches with deploymentConfig GVR, and then start to validate.
Originally posted by @rahulchheda in https://github.com/litmuschaos/chaos-operator/pull/199

Closing this as this is no longer in use.


upgrade fails

Problem seems to be related to conda (not bcbio).
Here's the output / traceback.
Fetching package metadata: Could not connect to https://repo.continuum.io/pkgs/pro/noarch/
.Could not connect to https://repo.continuum.io/pkgs/pro/linux-64/
.Could not connect to https://repo.continuum.io/pkgs/free/noarch/
.Could not connect to https://repo.continuum.io/pkgs/free/linux-64/
.
Error: No packages found in current linux-64 channels matching: numpy
You can search for this package on Binstar with
binstar search -t conda numpy

You may need to install the Binstar command line client with
conda install binstar

Traceback (most recent call last):
File "/usr/local/bin/bcbio_nextgen.py", line 199, in
install.upgrade_bcbio(kwargs["args"])
File "/usr/local/share/bcbio/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 50, in upgrade_bcbio
_update_conda_packages()
File "/usr/local/share/bcbio/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 192, in _update_conda_packages
subprocess.check_call([conda_bin, "install", "--yes", "numpy"])
File "/usr/local/share/bcbio/anaconda/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/share/bcbio/anaconda/bin/conda', 'install', '--yes', 'numpy']' returned non-zero exit status 1

Par;
Thanks for the report, is this still failing for you? I tried and can't replicate this. I wonder if it was a temporary download or if https://repo.continuum.io/pkgs/free/linux-64/ is blocked from your machine for some reason. Sorry to not have a better idea.

You're right, it worked on a different machine and disabling ssl for conda solved the problem.
Thanks,
Pär

Great, glad it worked for you. Thanks for following up.


O(1) to O(reasonable) text insertions

This is a naïve text-layout-reduction PR: only laying out text after a "placement index" aka the insertion point. There are a couple of unfinished edge cases, but I may leave it since you typically don't have aggressive text formatting when you also are inserting text frequently. More people complain about input text being unbearable, and this helps especially if the cursor is near the end of the text.
It's a little hard to evaluate the "state" of text layout at any index, so it's harder to resume from that state. There are some bigger changes I have planned that address this, but those are a ways away.
Related to #2377, #2229. I'll decide if the latter can be closed when I merge.

The word cache PR nicely addresses performance issues, and this one is too fussy for where the layout code is at right now

The word cache PR nicely addresses performance issues, and this one is too fussy for where the layout code is at right now


Update Buildings in Map

Update the web map to use the 2024 building layer (added in the data folder).

https://github.com/datalab-dev/covid_worksite_exposure/pull/52 128/238 left for manual input


Hitting the API limit

I had the same issue.  The odd part is that I've had this integration set up exactly the same way since December, and this is the first month I've run into the API limit.
Originally posted by @mermelmadness in https://github.com/Pirate-Weather/pirate-weather-ha/issues/218#issuecomment-2069435253

Hi @mermelmadness I created this issue for you from the comment. You're saying that you hit the API key limit for the month when you didn't before?

Thanks for this.
Back in November this forum helped me re-setup the integration using the new GET weather protocol used in HA.  Ever since then the configuration has remain unchanged on my end.  Yet in April I hit the API limit.
The e-mail I used is<EMAIL_ADDRESS>These are my current versions:

I'm going to assume you've already queried the browser to confirm that you've hit the API limit? Since you haven't hit the limit before you have your update interval low enough to not hit the limit? You could also try to regenerate your API key to see if it solves things though you'd have to setup the integration again if you want to change your API key.
Here is a list of all available plans that you could subscribe to all of which give a higher API key limit.

Correct

I think I see what happened.  The plans changed.  The free version used to allow 20,000 calls per month.  I know this because I set my interval to be 00:02:15 because that would make 19,840 calls in a 31-day month.
Now I'll set my interval to 00:04:30 which will make 9,920 calls per 31-day month.

That makes sense why you hit the limit this month when you weren't before. @alexander0042 must have introduced a new tier with the same amount of calls for $2 and reduced the number of calls in the free tier.

Hi, thanks for reaching out and opening this issue, I haven't changed anything recently, so the timing for this is sort of strange; and I'm wondering when you started seeing these errors show up- I'm seeing an increase in 429s (quota) errors starting April 15, so wondering if you aligned with that?

Not exactly.  Looks like the entities stopped populating data on April 21st

@alexander0042 The API plans definitely changed, though.  I distinctly remember setting my interval to come just under the free limit, so it was 20,000 at one point.  Maybe AWS changed something under the hood?

I got my entities back.  Did something change?  Wasn't expecting any data until May 1st.

Maybe @alexander0042 upped your quota for this month to get things working again?

That was my guess originally, but the entities haven't updated in 6 hours, so not sure.

And now it's unavailable again.

I'm ok waiting for May 1st for the quota to reset.

Do you get the message about your API quota being reached when you visit the browser or do you get a different error? Anything in the logs?

Yea, this is definitely a weird one! Is there any chance you signed up using a different e-mail address? I can't even find that one in the system

Wait, found it! I might know what happened here, give me 5 min

Update, I've got no idea here, since everything looks ok on my end. You have one of the OG legacy keys, so actually set at a 25k cap, since I started very untrustworthy of my quota system. It's showing it entirely used, but I don't have enough logging to see exactly when.
I upped the limit to 50k and enabled logging, so I'll be able to see exactly what calls are made for now. This will give a nice view of what's happening going forward, so should get to the root cause

Do you get the message about your API quota being reached when you visit the browser or do you get a different error? Anything in the logs?

In the browser, yes.  I didn't really check the logs since it seemed like my issue was the same as someone else's.

Update, I've got no idea here, since everything looks ok on my end. You have one of the OG legacy keys, so actually set at a 25k cap, since I started very untrustworthy of my quota system. It's showing it entirely used, but I don't have enough logging to see exactly when.
I upped the limit to 50k and enabled logging, so I'll be able to see exactly what calls are made for now. This will give a nice view of what's happening going forward, so should get to the root cause

Thank you.  I look forward to see if any results show themselves.  I can confirm the integration is working again on my end.  I'll leave my interval as such for now:
  - platform: pirateweather
    api_key: <APIKEY>
    scan_interval: '00:04:28'

Since everything seems to be sorted I'm going to close this for now. If you encounter more issues you can re-open this issue or create a new one.

Dang, was about to sign off of work and post that this issue could be closed.  Beat me to it by 2 minutes.
Thanks for all your help.  Everything seems to be working as expected.

Ok, since it's been a couple days and I've got some new data, I think we should re-open this, since it might be a larger issue with the HA integration. The simple reason you hit the limit is because I'm seeing calls every ~2 minutes from your API key, but th harder one is why:

First and foremost, can you double check you only have one version of the integration instated but looking in your entities page? And do you know if it was installed via the yaml file or the UI? I'm wondering if somewhere along the line HA imported the YAML again, which is why you're getting 2x the calls that should be happening

I have 57 entities for the PirateWeather integration, but no duplicates.  Could it be it's going twice because I have two entries, one for weather and one for forecast?
Oddly, 3 entries are showing up:

Originally it was YAML based, but ever since the new GUI option was implemented I re-set it up that way to take advantage of additional data.

That makes sense since the entities match the YAML:

platform: pirateweather
api_key:
scan_interval: '00:04:28'
monitored_conditions:

temperature
precip_probability
precip_type
humidity
cloud_cover
nearest_storm_distance
precip_intensity
wind_speed

Commenting out and restarting...

Had to manually delete the extra entry, but it did not automatically reappear.

Please monitor my API calls and see if the interval has changed....though I think I read that in order to change the interval I need to redo the integration and I'm not prepared to do that right now as I have several automations that use this integration.

Yea, that must be it. The entities and forecast both use the same data update coordinator, so share API calls, but I suspect that rogue "8 entities" one is the issue here. Can you try deleting that, as well as (or commenting out) your YAML, and we'll see what that does.

Hi @alexander0042 - could you please clarify this a bit more?  I previously had my integration set up similarly to @mermelmadness with one instance for Entities and another for Weather.  Honestly, I can't recall why I had done that.  But when I ran into an API Limit issue this month, @cloneofghosts suggested (#218) consolidating to a single instance of the integration.  My conclusion from that was that the former configuration was causing twice the API calls of the latter.  Is that how it works?

Hi @mermelmadness and @alexander0042 just checking in to see if this issue has been solved and can be closed. The last update was just over a month ago and there has been no updates since which leads me to believe this has been solved.

Hi @mermelmadness and @alexander0042 just checking in to see if this issue has been solved and can be closed. The last update was just over a month ago and there has been no updates since which leads me to believe this has been solved.

My issue is resolved.  I cannot speak for everyone who chimed in on this issue.

I created a separate issue for the comment left by @jhemak so this can be closed. Thanks.

@alexander0042 - I think I am having a similar issue. My entities are all reporting errors with "429 - too many requests". The only change I did this month was upgrade Home Assistant and other HA modules.
Can I DM you my API key or email to look up what's going on? My scan interval when it was YAML based was 5minutes, I guess it's the same when it was migrated to the UI (although I can't recall and am not sure where that is stored now).

Hi @catchdave You can always email @alexander0042 at<EMAIL_ADDRESS>with the information needed to look into this issue.

Hi @catchdave are you still having issues with hitting the API limit? V1.5.5 introduced the option to change the scan interval after you configured the integration so you should be able to change it if you're still having issues.

I was able to get it working. Alexander mentioned that the previous version (when it was configured in yaml) had HA reusing API calls which is probably the reason why I never hit limits previously.

Good to know you got this working. Will close this issue but if it pops up again feel free to reply or create a new issue.


Fix the compile errors from the Markdown

It looks like what is happening is that the markdown renderer extension to code is triggering compilation on code snippets which are not supposed to be compiled.
The compile errors don't prevent the server for running but are annoying and make testing changes that happen at initial load time harder since so much text is outputted by default. I comment out the error logging locally but would be good to have a proper fix.
https://github.com/imba/imba.io/blob/master/src/util/markdown.imba#L146-L150
Related-to: #28

So that code is being abandoned. Won't fix.


TOC Candidate Nomination for Jesse Alford

Candidate Name: Jesse Alford
Candidate GitHub Username: @anEXPer
Affiliation: none
Nomination Statement: This being a self nomination, I'll speak in the first-person. I've worked on CF for almost ten years. I was anchor of the release integration team that conceived of and developed cf-deployment, guided the design of the v6 CLI, orchestrated the transition of Pivotal Web Services from cf-release to multi-deployment/cell-block cf-deployment, and have recently resigned from VMware after years of being a Technology Lead focused on release, packaging, and upgrade related issues for TAS, VMware's commercialized distribution.
I care deeply about Cloud Foundry and related technologies like Concourse, and bring significant experience and expertise in transitions and roadmaps through eras of technological and organizational change.

@rkoster @stephanme @emalm @AP-Hunt @beyhan
Per the section 7(e)(i) in the CFF Charter, we'll need a majority of the TOC to approve an exemption for Jesse to run for a TOC seat. If you could, please comment your approval (or disapproval) on this issue.

TOC, please indicate your approval or objection via 👍 / 👎 reaction on this comment or reply on the discussion

Closing as the 2023 TOC election has concluded.


Train/inference job 'app' attribute does not support chinese

We can create train job in 'app' attribute with Chinese, but can't visit other url including '//', because url is not supporting Chinese. We suggest that maybe using 'train job id' and 'inference job id' in url is better. But if do this, maybe it's a large change for both backend and frontend.

We found it depends on the format of python in docker containers. If using Chinese and request posted by frontend is in one format such as 'ascii', then the container python format must be 'ascii'. So just set the format to the same both in backend and frontend. I think this issue can be closed.


Where is ffmpeg location located?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

How can I define ffmpeg-location ?
mypath\ffmpeg/bin/ffmpeg.exe isn't work for me
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[thepath] C:\Users\HP\Downloads\kaicheng music
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[392389b7d] (pip) API
[debug] params: {'format': 'bestaudio', 'outtmpl': 'C:\\Users\\HP\\Downloads\\kaicheng music\\%(title)s.%(ext)s', 'extractaudio': True, 'addmetadata': True, 'nocheckcertificate': True, '--windows-filenames': True, 'default_search': 'auto', 'verbose': True, 'source_address': '<IP_ADDRESS>', 'writethumbnail': True, 'ffmpeg_location': './ffmpeg.exe', 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'mp3'}, {'key': 'FFmpegMetadata', 'add_metadata': True}], 'compat_opts': set()}
[debug] Python 3.10.6 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1n  15 Mar 2022)
WARNING: ffmpeg-location ./ffmpeg.exe does not exist! Continuing without ffmpeg
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[youtube:search] Extracting URL: ytsearch:5 min later
[download] Downloading playlist: 5 min later
[youtube:search] query "5 min later": Downloading web client config
[youtube:search] query "5 min later" page 1: Downloading API JSON
[info] There's no playlist thumbnails to download
[youtube:search] Playlist 5 min later: Downloading 1 items of 1
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 1
[youtube] Extracting URL: https://www.youtube.com/watch?v=wiHYx9NX4DM
[youtube] wiHYx9NX4DM: Downloading webpage
[youtube] wiHYx9NX4DM: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] wiHYx9NX4DM: Downloading 1 format(s): 251
[info] Downloading video thumbnail 46 ...
[info] Writing video thumbnail 46 to: C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webp
[debug] Invoking dashsegments downloader on "https://rr2---sn-uh-h31s.googlevideo.com/videoplayback?expire=1685607775&ei=_wB4ZMyQHenDrtoP-Ky38A8&ip=<IP_ADDRESS>&id=o-AJHOex3_JopOrgmxBiv-uZo4d8iUngFwq5gKUEC7itZX&itag=251&source=youtube&requiressl=yes&mh=rW&mm=31%2C29&mn=sn-uh-h31s%2Csn-30a7rnek&ms=au%2Crdu&mv=m&mvi=2&pcm2cms=yes&pl=21&initcwndbps=1335000&spc=qEK7Bxw5vAC61Uv23_qqb8XkWJ3MklY&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=26963&dur=1.521&lmt=1510051024673503&mt=1685585812&fvip=5&keepalive=yes&fexp=24007246%2C51000023&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAP6RkLNO6f88pN8JOyThpX8hG4hnxTMMNPPFTgt_bYSVAiEA2ft6Wu8TzvJOgneGVgczoytSYCVU5xPutJriRt5Wj0M%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhALMvHzUJZtw_3paKxHOl1BLy4NSjIRRQ31wEfyrKPoFEAiA-KzpArLjcB3oGpPYPE8r4ISSb413CPSpJcDyQpYYqfw%3D%3D"
[dashsegments] Total fragments: 1
[download] Destination: C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webm
[debug] File locking is not supported. Proceeding without locking
[download] 100% of   26.33KiB in 00:00:00 at 54.25KiB/s
ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
Traceback (most recent call last):
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
    info = self.run_pp(pp, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
    return func(self, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
    filecodec = self.get_audio_codec(path)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
    raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

Traceback (most recent call last):
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
    info = self.run_pp(pp, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
    return func(self, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
    filecodec = self.get_audio_codec(path)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
    raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

During handling of the above exception, another exception occurred:
    ie_result = self.process_video_result(ie_result, download=download)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2779, in process_video_result
    self.process_info(new_info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3325, in process_info
    self.report_error('Postprocessing: %s' % str(err))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1015, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 955, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

It's ffmpeg_location. Use https://github.com/yt-dlp/yt-dlp/blob/master/devscripts/cli_to_api.py as the pinned issue suggests.

Technically I think you're meant to do the folder name, but I doubt it would make a difference

Also, please post a verbose log!

and pls don't skip the top lines unlike in OP.

Also, please post a verbose log!

Can u please see my edit?

WARNING: ffmpeg-location ./ffmpeg.exe does not exist! Continuing without ffmpeg

Is there really an FFmpeg binary located in C:\Users\HP\Downloads\kaicheng music, and if yes, why do you even need ffmpeg_location? It might be looking in C:\Users\HP\Downloads, but I believe that requires two dots

my opt:
'ffmpeg_location':os.path.realpath('D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe'),

but there error is always occur
i cant figure out why

Please post a new verbose log

Please post a new verbose log

[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[392389b7d] (pip) API
[debug] params: {'final_ext': 'mp3', 'ffmpeg_location': 'D:\\Desktop\\coding\\youtube\\src\x0cfmpeg\x08in\x0cfmpeg.exe', 'format': 'bestaudio', 'outtmpl': 'C:\\Users\\HP\\Downloads\\kaicheng music\\%(title)s.%(ext)s', 'postprocessors': [{'key': 'FFmpegExtractAudio', 'nopostoverwrites': False, 'preferredcodec': 'mp3', 'preferredquality': '320'}, {'add_chapters': True, 'add_infojson': 'if_exists', 'add_metadata': True, 'key': 'FFmpegMetadata'}, {'already_have_thumbnail': False, 'key': 'EmbedThumbnail'}], 'writethumbnail': True, '--windows-filenames': True, 'default_search': 'auto', 'verbose': True, 'source_address': '<IP_ADDRESS>', 'compat_opts': set()}
[debug] Python 3.10.6 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1n  15 Mar 2022)
WARNING: ffmpeg-location D:\Desktop\coding\youtube\src
fmpein
fmpeg.exe does not exist! Continuing without ffmpeg
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[youtube:search] Extracting URL: ytsearch:5 min latwer
[download] Downloading playlist: 5 min latwer
[youtube:search] query "5 min latwer": Downloading web client config
[youtube:search] query "5 min latwer" page 1: Downloading API JSON
[info] There's no playlist thumbnails to download
[youtube:search] Playlist 5 min latwer: Downloading 1 items of 1
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 1
[youtube] Extracting URL: https://www.youtube.com/watch?v=wiHYx9NX4DM
[youtube] wiHYx9NX4DM: Downloading webpage
[youtube] wiHYx9NX4DM: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] wiHYx9NX4DM: Downloading 1 format(s): 251
[info] Downloading video thumbnail 46 ...
[info] Writing video thumbnail 46 to: C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webp
[debug] Invoking dashsegments downloader on "https://rr2---sn-uh-h31s.googlevideo.com/videoplayback?expire=1685728355&ei=A9h5ZJ60H_HlrtoPu4CewAY&ip=<IP_ADDRESS>&id=o-AFaF1uXhVr35yw0yVpn36GnER7WhC1puRWAPh_8CQJTt&itag=251&source=youtube&requiressl=yes&mh=rW&mm=31%2C29&mn=sn-uh-h31s%2Csn-30a7ynl7&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=1716250&spc=qEK7B2qKHKjJWWUG0-7R8XWA19LeR0E&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=26963&dur=1.521&lmt=1510051024673503&mt=1685706316&fvip=1&keepalive=yes&fexp=24007246&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAPtKU4i_27JL1vgzWLtl5A7W_nE8hpzD0w8LfJIX6rYhAiAWcqqMF7eS-33u0127hIwvpDPbr5nUrVL0JG8Ws3xCEA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgReG4wscEObAcSfCT5Pk8C4EZKWagikU6VuN2aBhfeQwCIQCWGmkWCRrlrjpoq-mHAzu1PfXCBqbCi5V_xE5zXr7SAA%3D%3D"
[dashsegments] Total fragments: 1
[download] Destination: C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webm
[debug] File locking is not supported. Proceeding without locking
[download] 100% of   26.33KiB in 00:00:00 at 297.03KiB/s
ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
Traceback (most recent call last):
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
    info = self.run_pp(pp, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
    return func(self, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
    filecodec = self.get_audio_codec(path)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
    raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

Traceback (most recent call last):
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
    info = self.run_pp(pp, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
    return func(self, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
    filecodec = self.get_audio_codec(path)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
    raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

During handling of the above exception, another exception occurred:
    ie_result = self.process_video_result(ie_result, download=download)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2779, in process_video_result
    self.process_info(new_info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3325, in process_info
    self.report_error('Postprocessing: %s' % str(err))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1015, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 955, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

change 'D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe'
to
'D:\\Desktop\\coding\\youtube\\src\\ffmpeg\\bin\\ffmpeg.exe'

Unfortunately still not work

verbose log
   [debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[392389b7d] (pip) API
[debug] params: {'final_ext': 'mp3', 'ffmpeg_location': 'D:\\Desktop\\coding\\youtube\\src\\ffmpeg\\bin\\ffmpeg.exe', 'format': 'bestaudio', 'outtmpl': 'C:\\Users\\HP\\Downloads\\kaicheng music\\%(title)s.%(ext)s', 'postprocessors': [{'key': 'FFmpegExtractAudio', 'nopostoverwrites': False, 'preferredcodec': 'mp3', 'preferredquality': '320'}, {'add_chapters': True, 'add_infojson': 'if_exists', 'add_metadata': True, 'key': 'FFmpegMetadata'}, {'already_have_thumbnail': False, 'key': 'EmbedThumbnail'}], 'writethumbnail': True, '--windows-filenames': True, 'default_search': 'auto', 'verbose': True, 'source_address': '<IP_ADDRESS>', 'compat_opts': set()}
[debug] Python 3.10.6 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1n  15 Mar 2022)
WARNING: ffmpeg-location D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe does not exist! Continuing without ffmpeg
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[youtube:search] Extracting URL: ytsearch:5 min later
[download] Downloading playlist: 5 min later
[youtube:search] query "5 min later": Downloading web client config
[youtube:search] query "5 min later" page 1: Downloading API JSON
[info] There's no playlist thumbnails to download
[youtube:search] Playlist 5 min later: Downloading 1 items of 1
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 1
[youtube] Extracting URL: https://www.youtube.com/watch?v=wiHYx9NX4DM
[youtube] wiHYx9NX4DM: Downloading webpage
[youtube] wiHYx9NX4DM: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] wiHYx9NX4DM: Downloading 1 format(s): 251
Deleting existing file C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webp
[info] Downloading video thumbnail 46 ...
[info] Writing video thumbnail 46 to: C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webp
[debug] Invoking dashsegments downloader on "https://rr2---sn-uh-h31s.googlevideo.com/videoplayback?expire=1685728754&ei=ktl5ZMeNKb69rtoP65ydiAQ&ip=<IP_ADDRESS>&id=o-AE3z8zOEi5XZi20LxXUo-_GpvN8mEp5RX1K8zHg7F_Rr&itag=251&source=youtube&requiressl=yes&mh=rW&mm=31%2C29&mn=sn-uh-h31s%2Csn-30a7rnek&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=1700000&spc=qEK7B1fXUjUcLWngakmiVOxIWP1RNlM&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=26963&dur=1.521&lmt=1510051024673503&mt=1685706819&fvip=5&keepalive=yes&fexp=24007246%2C51000014%2C51000023&beids=24350017&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAKoHvqwphKaBXhn1ZHlQ0APVtu-yjlunydHeomL3adPlAiAxFluucS7GzKPXHTRY_VfICOueNxFcFW4gpsr5f44fiw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIU4Ybrgyf_rzXmA229mmjqGxI0MHMi_xxTBvLvd7rz4AiEApoj5deY6VDShWb66pmzIyeM_Pg3fvDYjC0EehZQRS3Q%3D"
[download] C:\Users\HP\Downloads\kaicheng music\5 Minutes Later... ｜ SpongeBob Time Card #64.webm has already been downloaded
[download] 100% of   26.33KiB
ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
Traceback (most recent call last):
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
    info = self.run_pp(pp, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
    return func(self, info)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
    filecodec = self.get_audio_codec(path)
  File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
    raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
Traceback (most recent call last):
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3323, in process_info
replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3502, in post_process
info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3484, in run_all_pps
info = self.run_pp(pp, info)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3462, in run_pp
files_to_delete, infodict = pp.run(infodict)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 24, in run
ret = func(self, info, *args, **kwargs)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\common.py", line 129, in wrapper
return func(self, info)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 488, in run
filecodec = self.get_audio_codec(path)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\postprocessor\ffmpeg.py", line 241, in get_audio_codec
raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
yt_dlp.utils.PostProcessingError: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "d:\Desktop\coding\youtube\src\main.py", line 48, in err_inner
func(*args,**kwargs)
File "d:\Desktop\coding\youtube\src\main.py", line 88, in downloadyoutube
result = ydl.extract_info(f"ytsearch:{arg}", download=True)['entries'][0:1][0]
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1507, in extract_info
return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1518, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1615, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1744, in process_ie_result
return self.__process_playlist(ie_result, download)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1888, in __process_playlist
entry_result = self.__process_iterable_entry(entry, download, collections.ChainMap({
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1518, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1920, in __process_iterable_entry
return self.process_ie_result(
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1694, in process_ie_result
return self.extract_info(
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1507, in extract_info
return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1518, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1615, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1674, in process_ie_result
ie_result = self.process_video_result(ie_result, download=download)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2779, in process_video_result
self.process_info(new_info)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3325, in process_info
self.report_error('Postprocessing: %s' % str(err))
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1015, in report_error
self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
File "C:\Users\HP\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 955, in trouble
raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location

I solved it after restarting my PC 🥴. Thanks a lot to those who are willing to help me 🙏🏻🙏🏻

change 'D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe' to 'D:\\Desktop\\coding\\youtube\\src\\ffmpeg\\bin\\ffmpeg.exe'

I think this works for me. I tried to restart my pc for a few times but only works after I changed \ to \

change 'D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe' to 'D:\\Desktop\\coding\\youtube\\src\\ffmpeg\\bin\\ffmpeg.exe'

Not only this has fixed ffmpeg job, it also made --download-archive to actually work.
--download-archive ignored 90% of time if you put ffmpeg_location with single backslashes.
I guess if yt-dlp can't merge downloaded video and audio files, because of wrong syntax in ffmpeg_location, it just stops the next job of --download-archive.

I encountered the same problem and managed to resolve it on Ubuntu by creating symbolic links for ffmpeg and ffprobe within my virtual environment. This approach ensures that the executables are directly accessible from the virtual environment's bin directory, which is often the first place looked at by scripts or Python libraries that depend on these binaries.
ln -s /usr/bin/ffmpeg /home/username/Desktop/virtual_env/llm/bin/ffmpeg
ln -s /usr/bin/ffprobe /home/username/Desktop/virtual_env/llm/bin/ffprobe

Use the code below.
from yt_dlp.postprocessor import FFmpegPostProcessor
FFmpegPostProcessor._ffmpeg_location.set(R'C:\ffmpeg\bin')

change 'D:\Desktop\coding\youtube\src\ffmpeg\bin\ffmpeg.exe' to 'D:\\Desktop\\coding\\youtube\\src\\ffmpeg\\bin\\ffmpeg.exe'

It works for me too.


Fix for issue #609 and a resolve for the request #135

Thank you for the PR. We will review it as soon as possible!

ok from me


write test cases for progress slider

write test cases for progress slider

issue already created https://github.com/Real-Dev-Squad/website-status/issues/683

issue already created https://github.com/Real-Dev-Squad/website-status/issues/683


Factor out execforward and annotation packages

Factors out annotation handling to package annotation
Factors out the main bits to execforward, which itself needs refactoring and tests
Splits up Config into versions for the command and execforward package. In a future PR I'll remove config from these packages in place of Options. For now I left the name to minimize the diff.
Rewrites ParseCommands tests as table tests

This should mostly if not completely finish the breakup of the overloaded command package, which now stands at 100% coverage (as does annotation). Packages forwarder and execforward have near-zero coverage and will still need their own internal refactors.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@012b6ae). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head b2fcf97 differs from pull request most recent head 6aa9c81. Consider uploading reports for the commit 6aa9c81 to get more accurate results

@@            Coverage Diff            @@
##             master      #74   +/-   ##
=========================================
  Coverage          ?   41.59%
=========================================
  Files             ?       17
  Lines             ?      363
  Branches          ?        0
=========================================
  Hits              ?      151
  Misses            ?      198
  Partials          ?       14

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 012b6ae...6aa9c81. Read the comment docs.


[AMBARI-23494] Incorrect servicerpc address configs being generated by NN Federation wizard. Second patch.

results of running unit tests locally:
21529 passing (32s)
48 pending

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1754/
Test FAILed.
Test FAILured.


Updates and changes to match ultra96-v2 board

Did the commit and push from the master branch this time.  Hopefully this works.  Let me know.

can you  open the pull request against ultra96v2 branch

I opened this pull request in my web browser.  Was there something you wanted me to do here?
--Tom
From: mharth<EMAIL_ADDRESS>Sent: Monday, July 1, 2019 3:50 PM
To: Xilinx/ultra96-startup-pages<EMAIL_ADDRESS>Cc: Curran, Tom (EM<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [Xilinx/ultra96-startup-pages] Updates and changes to match ultra96-v2 board (#2)
can you open the pull request against ultra96v2 branch
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Xilinx/ultra96-startup-pages/pull/2?email_source=notifications&email_token=ADCNSOCOU7WBXP5O7RRDO6LP5JNV7A5CNFSM4H3KW7GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7FECA#issuecomment-507400712, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADCNSOAFEVAUAN55NAPHBWTP5JNV7ANCNFSM4H3KW7GA.

open the pull request to this branch : https://github.com/Xilinx/ultra96-startup-pages/tree/ultra96v2 not master

OK, I have it opened now
--Tom
From: mharth<EMAIL_ADDRESS>Sent: Monday, July 1, 2019 5:29 PM
To: Xilinx/ultra96-startup-pages<EMAIL_ADDRESS>Cc: Curran, Tom (EM<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [Xilinx/ultra96-startup-pages] Updates and changes to match ultra96-v2 board (#2)
open the pull request to this branch : https://github.com/Xilinx/ultra96-startup-pages/tree/ultra96v2 not master
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Xilinx/ultra96-startup-pages/pull/2?email_source=notifications&email_token=ADCNSOGSHAXJNKIVIF2XQGDP5JZJNA5CNFSM4H3KW7GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7M6AA#issuecomment-507432704, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADCNSOA4JQ4N7CCSOZ26DO3P5JZJNANCNFSM4H3KW7GA.


ESP32-S3 WiFi does not work if external SPIRAM is enabled

Describe the bug
If SPIRAM is enabled, WiFi does not work on ESP32S3 WROOM N16R8.
WiFi Works if SPIRAM is disabled:
west build --pristine -b xiao_esp32s3/esp32s3/procpu samples/net/wifi -DCONFIG_ESP_SPIRAM=n && west flash && west espressif monitor
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x28 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40048df9
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fc94170,len:0x3804
load:0x40374000,len:0x1014c

SHA-256 comparison failed:
Calculated: 8923cad87525e34488aa27e9abb59530d31d7a56f3c68686442b1c27b70b23be
Expected:<PHONE_NUMBER>c60000000000000000000000000000000000000000000000000000
Attempting to boot anyway...
entry 0x4037cdd8

I (94) boot: ESP Simple boot
I (94) boot: compile time Jun 14 2024 01:20:43
W (95) boot: Unicore bootloader
I (95) spi_flash: detected chip: gd
I (96) spi_flash: flash io: dio
W (99) spi_flash: Detected size(16384k) larger than the size in the binary image header(2048k). Using the size in the binary image header.
I (111) boot: chip revision: v0.2
I (114) boot.esp32s3: Boot SPI Speed : 40MHz
I (118) boot.esp32s3: SPI Mode       : DIO
I (122) boot.esp32s3: SPI Flash Size : 8MB
I (126) boot: Enabling RNG early entropy source...
[esp32s3] [INF] DRAM: lma 0x00000020 vma 0x3fc94170 len 0x3804   (14340)
[esp32s3] [INF] IRAM: lma 0x0000382c vma 0x40374000 len 0x1014c  (65868)

[esp32s3] [INF] padd: lma 0x00013988 vma 0x00000000 len 0xc670   (50800)
[esp32s3] [INF] IMAP: lma 0x00020000 vma 0x42000000 len 0x53260  (340576)
0x42000000: _stext at ??:?

[esp32s3] [INF] padd: lma 0x00073268 vma 0x00000000 len 0xcd90   (52624)
[esp32s3] [INF] DMAP: lma 0x00080000 vma 0x3c060000 len 0x131ec  (78316)
[esp32s3] [INF] Image with 6 segments
[esp32s3] [INF] DROM segment: paddr=00080000h, vaddr=3c060000h, size=131F0h ( 78320) map
[esp32s3] [INF] IROM segment: paddr=00020000h, vaddr=42000000h, size=5325Eh (340574) map

*** Booting Zephyr OS build v3.6.0-5736-gd78a9429450f ***
uart:~$ wifi scan
Scan requested

Num  | SSID                             (len) | Chan (Band)   | RSSI | Security        | BSSID             | MFP
1    | IoT                              3     | 1    (2.4GHz) | -67  | WPA2-PSK        |                   | Disable
2    | IoT                              3     | 11   (2.4GHz) | -74  | WPA2-PSK        |                   | Disable
3    | IoT                              3     | 6    (2.4GHz) | -87  | WPA2-PSK        |                   | Disable
Scan request done
uart:~$

If SPIRAM is enabled, then WiFi returns nothing for wifi scan and will also not connect.  I only have octal SPI devices, so I am unable to determine if this is just an octal SPI issue or it it applies to quad SPI as well.
west build --pristine -b xiao_esp32s3/esp32s3/procpu samples/net/wifi -DCONFIG_ESP_SPIRAM=y -DCONFIG_SPIRAM_MODE_OCT=y && west flash && west espressif monitor
I (97) boot: ESP Simple boot
I (97) boot: compile time Jun 14 2024 01:13:09
W (97) boot: Unicore bootloader
I (97) spi_flash: detected chip: gd
I (98) spi_flash: flash io: dio
W (101) spi_flash: Detected size(16384k) larger than the size in the binary image header(2048k). Using the size in the binary image header.
  In addition, WiFi will not connect to any known access points

Expected behavior
WiFi scan finds access points and allows connecting.
Impact
Complete failure of WiFi
Environment (please complete the following information):
Zephyr 3.6.99 main @ d78a9429450f89de7829c8af7efe41fdb692fd50

Doing a bit more debugging, the issue is that the ESP32-S3 WiFi stack does not support buffers allocated in SPIRAM and the minimum heap size of 51200 is too small.
*** Booting Zephyr OS build v3.6.0-5736-gd78a9429450f ***
uart:~$ kernel heap
free:           12
allocated:      50904
max. allocated: 50904

Increasing the heap size to 100KB allows the scan to work and shows that the maximum heap usage is 83400 bytes.
uart:~$ kernel heap
free:           21068
allocated:      80892
max. allocated: 83400

The HEAP_MEM_POOL_ADD_SIZE_BOARD should be increased to 86016 or similar to allow WiFi to work.
However, this increased memory requirement makes the ESP32-S3 target very difficult to use since the internal SRAM is already in short supply with just this example already using 73% of the available memory.  Is there any chance that the ESP32-S3 WiFi stack could be fixed to support using buffers from SPIRAM?  That has been a long-term issue.
Memory region         Used Size  Region Size  %age Used
           FLASH:      603244 B    8388352 B      7.19%
     iram0_0_seg:       68148 B       329 KB     20.23%
     dram0_0_seg:      234376 B       313 KB     73.13%
     irom0_0_seg:      341706 B         8 MB      4.07%
     drom0_0_seg:      472172 B         8 MB      5.63%
     ext_ram_seg:       1536 KB    8388544 B     18.75%
    rtc_iram_seg:          0 GB         8 KB      0.00%
    rtc_data_seg:          0 GB         8 KB      0.00%
    rtc_slow_seg:          0 GB         8 KB      0.00%
        IDT_LIST:          0 GB         8 KB      0.00%

@sylvioalves

Experiencing a similar issue - given the SRAM is quite limited, it'd be great to be able to utilize the SPIRAM for most memory allocations (performance is not [yet] an issue).
As I was experimenting with CONFIG_ESP_SPIRAM=y, I noticed that WiFi scanning and connection works when CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD is set to 1608, that is only allocations above 1608 bytes are served from the SPIRAM. Logging various allocations, I noticed a set of allocations during startup as wifi was initialized of 1604 bytes which fits very well with the threshold number above. The call stack originates in one of the espressif blobs for which I wasn't able to find the source code:
#0  __wrap_k_malloc (size=1604) at modules/hal/espressif/zephyr/port/heap/heap_caps_zephyr.c:348
#1  0x40376e5c in wifi_malloc (size=1604)
    at modules/hal/espressif/zephyr/esp32s3/src/wifi/esp_wifi_adapter.c:69
#2  0x40376e9c in malloc_internal_wrapper (size=1604)
    at modules/hal/espressif/zephyr/esp32s3/src/wifi/esp_wifi_adapter.c:524
#3  0x42078c71 in wDev_Rxbuf_Init ()
#4  0x420734a0 in ic_init ()
#5  0x420630d1 in wifi_lmac_init ()
#6  0x4206317a in wifi_init_process ()
#7  0x420618e1 in ieee80211_ioctl_process ()
#8  0x40388278 in ppTask ()
#9  0x42009b84 in z_thread_entry (entry=0x403881d4 <ppTask>, p1=0x0, p2=0x0, p3=0x0)
    at zephyr/lib/os/thread_entry.c:48

After wifi connection is established, my app tries to connect to Azure IoT hub. It appears that to the threshold must be raised to 1620 bytes to have a successful connection (otherwise -116 error is reported by mqtt_connect). The callstack of the allocation still points to wifi stack, but I wasn't able to get any additional symbols:
#1  0x4200f543 in __wrap_k_malloc (size=1617)
    at modules/hal/espressif/zephyr/port/heap/heap_caps_zephyr.c:351
#2  0x40376e5c in wifi_malloc (size=1617)
    at modules/hal/espressif/zephyr/esp32s3/src/wifi/esp_wifi_adapter.c:69
#3  0x40376e9c in malloc_internal_wrapper (size=1617)
    at modules/hal/espressif/zephyr/esp32s3/src/wifi/esp_wifi_adapter.c:524
#4  0x4003ec98 in ?? ()
#5  0x4003ee9e in ?? ()
#6  0x400398b8 in ?? ()
#7  0x400399c4 in ?? ()
#8  0x40388290 in ppTask ()
#9  0x42009b84 in z_thread_entry (entry=0x403881d4 <ppTask>, p1=0x0, p2=0x0, p3=0x0)
    at zephyr/lib/os/thread_entry.c:48

With the setting above, I ended up using only 24964 bytes of SRAM. Raising the threshold to 1704 bytes (there are bunch of allocations of 1700 bytes in wifi stack) ends up consuming 79404 bytes of SRAM. But I'd like to keep the threshold at 1024 bytes to maximize hardware capabilities.
Hope this helps find the bug!

@vigen1980 - Similar to your experiments, I finally determined that the ._malloc_internal needs to allocated internal RAM (hence the name which seems obvious in hindsight) and .wifi_malloc can use SPIRAM.  I created two buffer pools for the WiFi and that used 26KB of internal RAM and 85KB of SPIRAM.
I'm really busy for a next week or so, but could share these patches if there is a chance they would be merged.  This can be a configuration option to use the internal buffer pools or continue using the heap as is currently done.

@EricNRS Great find! I updated the impl of *alloc_internal_wrapper functions in esp_wifi_adapter.c to use k_aligned_alloc from zephyr/kernel/mempool.c and reduced the CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD to 1024 bytes, and got WiFi scanning and IoT Hub connection working with only 24452 bytes used from SRAM.
Here's what I've got so far:
static void *IRAM_ATTR malloc_internal_wrapper(size_t size)
{
  return k_aligned_alloc(sizeof(void*), size);
}

static void *IRAM_ATTR calloc_internal_wrapper(size_t n, size_t size)
{
  size_t bounds;
  if (__builtin_mul_overflow(n, size, &bounds)) {
    return NULL;
  }

  void* ret = k_aligned_alloc(sizeof(void*), size);
  if (ret != NULL) {
    (void)memset(ret, 0, bounds);
  }

  return ret;
}

static void *IRAM_ATTR zalloc_internal_wrapper(size_t size)
{
  return calloc_internal_wrapper(1, size);
}

@EricNRS Do you have something similar as well?
With all that - this solution does not seem like a bug fix, but more avoidance of the bug. I would love to still learn  why use of SPIRAM causes issues in WiFi stack.

@vigen1980 - I used two different heaps and tracked which heap the allocated block came from since the same free call is used regardless of the heap allocation.  I have uploaded the code here if you want to give it a try:  https://github.com/zephyrproject-rtos/hal_espressif/compare/zephyr...EricNRS:hal_espressif:2024-08-04-wifi-spiram-heap

this solution does not seem like a bug fix, but more avoidance of the bug. I would love to still learn why use of SPIRAM causes issues in WiFi stack.

It may just be a hardware requirement of the WiFi firmware.  Ideally, the driver should fail an assert or otherwise warn that it needed an internal buffer but got an SPIRAM change which it could do with pointer comparisons.

I have the same issue on a ESP32-WROOM-32E-N8R2 (Zephyr 3.7.0), can't scan/connect when SPIRAM is enabled.
Increasing CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD to 16xx bytes also work for me. It doesn't seem like CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM is either affecting the memory regions?
As you said, I would be great if WiFi could utilize the SPIRAM more.
@EricNRS I will try your fix!

@EricNRS, @vigen1980  can you please try to disable the CONFIG_FEATURE_CACHE_TX_BUF_BIT flag from the g_wifi_feature_caps (in the esp_wifi_adapter.c) and give it a try? In my case, allocating those extra TX buffers renders the wifi to not working.

@marekmatej  - are you saying that all of the heap memory could be in SPIRAM with CONFIG_FEATURE_CACHE_TX_BUF_BIT removed?  I will give it a try when I get a chance.
My changes in https://github.com/zephyrproject-rtos/hal_espressif/compare/zephyr...EricNRS:hal_espressif:2024-08-04-wifi-spiram-heap allocate using internal RAM for all *_internal_wrapper() functions.  It works well.  Sorry, there are some other debug changes in the patch.  With that change, I was able to reduce internal DRAM usage to 32KB and provide a generous 120KB from SPIRAM.  WiFi has been working well for months with these changes.

@marekmatej - are you saying that all of the heap memory could be in SPIRAM with CONFIG_FEATURE_CACHE_TX_BUF_BIT removed? I will give it a try when I get a chance.

No, I meant to say that enabling this flag will ask for additional dozens of kB of memory - it is automatically enabled when CONFIG_ESP_SPIRAM is enabled.

No, I meant to say that enabling this flag will ask for additional dozens of kB of memory - it is automatically enabled when CONFIG_ESP_SPIRAM is enabled.

Ah, nice.  With it disabled, it doesn't change the amount of internal RAM used, but does reduce the SPIRAM usage.  I'm not concerned about SPIRAM usage since I have over 8 MB of that.
# With CONFIG_FEATURE_CACHE_TX_BUF_BIT enabled (default) in g_wifi_feature_caps
uart> esp32_wifi heap_stats
SPIRAM Heap: used 58 KB of 120 KB; max allocated 75 KB
DRAM Heap: used 20 KB of 32 KB; max allocated 21 KB

# With CONFIG_FEATURE_CACHE_TX_BUF_BIT removed from g_wifi_feature_caps
uart> esp32_wifi heap_stats
SPIRAM Heap: used 5 KB of 120 KB; max allocated 19 KB  <-- reduced
DRAM Heap: used 20 KB of 32 KB; max allocated 21 KB  <-- DRAM still the same

I think this issue has been resolved as part of other changes, but it needs a retest and hopefully a reference to the fix(es).

I think this issue has been resolved as part of other changes, but it needs a retest and hopefully a reference to the fix(es).

It is not fully fixed yet.

@sylvioalves does this mean that its coming soon? :)


Add link_to erb helper to render links to the artifact reference.

~Test that actually works. (help appreciated)~
~This won't work for now as we cannot serve static files, I'll add it in a follow up PR but this is mergeable, to open the artifact just copy the link target in a new tab et voilà~
Not it works 😄 mergeable!! 🚀

This is mergeable btw!


[1.7.0-dev][graphdriver] prior storage driver "devicemapper" failed: driver not supported

Running docker-1.7.0-dev from up-to-date git repo fails on Fedora 22 (kernel-4.0.0-1.fc22.x86_64) with:
$ ./docker-1.7.0-dev -d -D
DEBU[0000] waiting for daemon to initialize
ERRO[0000] Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/cli/#daemon-storage-driver-option
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: driver not supported
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering DELETE, /containers/{name:.}
DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] Registering GET, /containers/{name:.}/changes
DEBU[0000] Registering GET, /containers/{name:.}/logs
DEBU[0000] Registering GET, /exec/{id:.}/json
DEBU[0000] Registering GET, /containers/{name:.}/attach/ws
DEBU[0000] Registering GET, /images/search
DEBU[0000] Registering GET, /containers/json
DEBU[0000] Registering GET, /containers/{name:.}/export
DEBU[0000] Registering GET, /containers/{name:.}/json
DEBU[0000] Registering GET, /containers/{name:.}/top
DEBU[0000] Registering GET, /containers/{name:.}/stats
DEBU[0000] Registering GET, /info
DEBU[0000] Registering GET, /version
DEBU[0000] Registering GET, /images/viz
DEBU[0000] Registering GET, /containers/ps
DEBU[0000] Registering GET, /images/{name:.}/get
DEBU[0000] Registering GET, /images/{name:.}/history
DEBU[0000] Registering GET, /images/{name:.}/json
DEBU[0000] Registering GET, /_ping
DEBU[0000] Registering GET, /events
DEBU[0000] Registering GET, /images/json
DEBU[0000] Registering GET, /images/get
DEBU[0000] Registering POST, /containers/{name:.}/copy
DEBU[0000] Registering POST, /containers/{name:.}/exec
DEBU[0000] Registering POST, /exec/{name:.}/start
DEBU[0000] Registering POST, /auth
DEBU[0000] Registering POST, /build
DEBU[0000] Registering POST, /containers/{name:.}/wait
DEBU[0000] Registering POST, /containers/{name:.}/attach
DEBU[0000] Registering POST, /images/{name:.}/tag
DEBU[0000] Registering POST, /containers/{name:.}/pause
DEBU[0000] Registering POST, /containers/{name:.}/rename
DEBU[0000] Registering POST, /commit
DEBU[0000] Registering POST, /images/create
DEBU[0000] Registering POST, /images/load
DEBU[0000] Registering POST, /images/{name:.}/push
DEBU[0000] Registering POST, /containers/{name:.}/unpause
DEBU[0000] Registering POST, /containers/{name:.}/restart
DEBU[0000] Registering POST, /exec/{name:.}/resize
DEBU[0000] Registering POST, /containers/{name:.}/resize
DEBU[0000] Registering POST, /containers/create
DEBU[0000] Registering POST, /containers/{name:.}/kill
DEBU[0000] Registering POST, /containers/{name:.}/start
DEBU[0000] Registering POST, /containers/{name:.*}/stop
DEBU[0000] docker group found. gid: 1001
FATA[0000] Shutting down daemon due to errors: error intializing graphdriver: driver not supported
1.6.0 runs OK:
$ docker -d -D
DEBU[0000] waiting for daemon to initialize
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering GET, /containers/{name:.}/changes
...
DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] docker group found. gid: 1001
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm version   OF   [16384] (*1)
DEBU[0000] devicemapper: udev sync support: true
DEBU[0000] Generated prefix: docker-253:22-524289
DEBU[0000] Checking for existence of the pool 'docker-253:22-524289-pool'
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm info docker-253:22-524289-pool  OF   [16384] (*1)
DEBU[0000] [deviceset] constructDeviceIdMap()
DEBU[0000] Loading data for file /var/lib/docker/devicemapper/metadata/00a0c78eeb6d81442efcd1d7c02e8b141745e3a06f1ee3458e1bae628e0067d3
...
DEBU[0000] Added deviceId=409 to DeviceIdMap
DEBU[0000] Loading data for file /var/lib/docker/devicemapper/metadata/transaction-metadata
DEBU[0000] Added deviceId=659 to DeviceIdMap
DEBU[0000] [deviceset] constructDeviceIdMap() END
DEBU[0000] Using graph driver devicemapper
DEBU[0000] Creating images graph
DEBU[0000] Restored 103 elements
DEBU[0000] Error restoring volume: Volume exists: ee38a6a63aaefed30d3b26fb31151f5fb9bf93c7e8a7bc6514d621746735e33f
DEBU[0000] Creating repository list
DEBU[0000] Reloaded graph with 3 grants expiring at 2017-03-22 19:04:46.713978458 +0000 UTC
INFO[0000] +job init_networkdriver()
...

Because you are running the static binary, this is expected and working
correctly
On Friday, April 17, 2015, Jiri Popelka<EMAIL_ADDRESS>wrote:

Running docker-1.7.0-dev from up-to-date git repo fails on Fedora 22
(kernel-4.0.0-1.fc22.x86_64) with:
$ ./docker-1.7.0-dev -d -D
DEBU[0000] waiting for daemon to initialize
ERRO[0000] Udev sync is not supported. This will lead to unexpected
behavior, data loss and errors. For more information, see
https://docs.docker.com/reference/commandline/cli/#daemon-storage-driver-option
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed:
driver not supported
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering DELETE, /containers/{name:.
} DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] Registering GET, /containers/{name:.
}/changes DEBU[0000] Registering GET, /containers/{name:.}/logs
DEBU[0000] Registering GET, /exec/{id:.
}/json DEBU[0000] Registering GET, /containers/{name:.}/attach/ws
DEBU[0000] Registering GET, /images/search
DEBU[0000] Registering GET, /containers/json
DEBU[0000] Registering GET, /containers/{name:.
}/export DEBU[0000] Registering GET, /containers/{name:.}/json
DEBU[0000] Registering GET, /containers/{name:.
}/top DEBU[0000] Registering GET, /containers/{name:.}/stats
DEBU[0000] Registering GET, /info
DEBU[0000] Registering GET, /version
DEBU[0000] Registering GET, /images/viz
DEBU[0000] Registering GET, /containers/ps
DEBU[0000] Registering GET, /images/{name:.
}/get DEBU[0000] Registering GET, /images/{name:.}/history
DEBU[0000] Registering GET, /images/{name:.
}/json DEBU[0000] Registering GET, /_ping DEBU[0000] Registering GET,
/events DEBU[0000] Registering GET, /images/json DEBU[0000] Registering
GET, /images/get DEBU[0000] Registering POST, /containers/{name:.}/copy
DEBU[0000] Registering POST, /containers/{name:.
}/exec DEBU[0000] Registering POST, /exec/{name:.}/start
DEBU[0000] Registering POST, /auth
DEBU[0000] Registering POST, /build
DEBU[0000] Registering POST, /containers/{name:.
}/wait DEBU[0000] Registering POST, /containers/{name:.}/attach
DEBU[0000] Registering POST, /images/{name:.
}/tag DEBU[0000] Registering POST, /containers/{name:.}/pause
DEBU[0000] Registering POST, /containers/{name:.
}/rename DEBU[0000] Registering POST, /commit DEBU[0000] Registering
POST, /images/create DEBU[0000] Registering POST, /images/load DEBU[0000]
Registering POST, /images/{name:.}/push
DEBU[0000] Registering POST, /containers/{name:.
}/unpause DEBU[0000] Registering POST, /containers/{name:.}/restart
DEBU[0000] Registering POST, /exec/{name:.
}/resize DEBU[0000] Registering POST, /containers/{name:.}/resize
DEBU[0000] Registering POST, /containers/create
DEBU[0000] Registering POST, /containers/{name:.
}/kill DEBU[0000] Registering POST, /containers/{name:.}/start
DEBU[0000] Registering POST, /containers/{name:.*}/stop
DEBU[0000] docker group found. gid: 1001
FATA[0000] Shutting down daemon due to errors: error intializing
graphdriver: driver not supported
1.6.0 runs OK:
$ docker -d -D
DEBU[0000] waiting for daemon to initialize
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering GET, /containers/{name:.
}/changes ... DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] docker group found. gid: 1001
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm version OF
16384 http://*1
DEBU[0000] devicemapper: udev sync support: true
DEBU[0000] Generated prefix: docker-253:22-524289
DEBU[0000] Checking for existence of the pool 'docker-253:22-524289-pool'
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm info
docker-253:22-524289-pool OF 16384 http://*1
DEBU[0000] [deviceset] constructDeviceIdMap()
DEBU[0000] Loading data for file
/var/lib/docker/devicemapper/metadata/00a0c78eeb6d81442efcd1d7c02e8b141745e3a06f1ee3458e1bae628e0067d3
...
DEBU[0000] Added deviceId=409 to DeviceIdMap
DEBU[0000] Loading data for file
/var/lib/docker/devicemapper/metadata/transaction-metadata
DEBU[0000] Added deviceId=659 to DeviceIdMap
DEBU[0000] [deviceset] constructDeviceIdMap() END
DEBU[0000] Using graph driver devicemapper
DEBU[0000] Creating images graph
DEBU[0000] Restored 103 elements
DEBU[0000] Error restoring volume: Volume exists:
ee38a6a63aaefed30d3b26fb31151f5fb9bf93c7e8a7bc6514d621746735e33f
DEBU[0000] Creating repository list
DEBU[0000] Reloaded graph with 3 grants expiring at 2017-03-22
19:04:46.713978458 +0000 UTC
INFO[0000] +job init_networkdriver()
...
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/12479.

See https://github.com/docker/docker/pull/11412
On Friday, April 17, 2015, Jessica Frazelle<EMAIL_ADDRESS>wrote:

Because you are running the static binary, this is expected and working
correctly
On Friday, April 17, 2015, Jiri Popelka<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>> wrote:

Running docker-1.7.0-dev from up-to-date git repo fails on Fedora 22
(kernel-4.0.0-1.fc22.x86_64) with:
$ ./docker-1.7.0-dev -d -D
DEBU[0000] waiting for daemon to initialize
ERRO[0000] Udev sync is not supported. This will lead to unexpected
behavior, data loss and errors. For more information, see
https://docs.docker.com/reference/commandline/cli/#daemon-storage-driver-option
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed:
driver not supported
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering DELETE, /containers/{name:.
} DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] Registering GET, /containers/{name:.
}/changes DEBU[0000] Registering GET, /containers/{name:.}/logs
DEBU[0000] Registering GET, /exec/{id:.
}/json DEBU[0000] Registering GET, /containers/{name:.}/attach/ws
DEBU[0000] Registering GET, /images/search
DEBU[0000] Registering GET, /containers/json
DEBU[0000] Registering GET, /containers/{name:.
}/export DEBU[0000] Registering GET, /containers/{name:.}/json
DEBU[0000] Registering GET, /containers/{name:.
}/top DEBU[0000] Registering GET, /containers/{name:.}/stats
DEBU[0000] Registering GET, /info
DEBU[0000] Registering GET, /version
DEBU[0000] Registering GET, /images/viz
DEBU[0000] Registering GET, /containers/ps
DEBU[0000] Registering GET, /images/{name:.
}/get DEBU[0000] Registering GET, /images/{name:.}/history
DEBU[0000] Registering GET, /images/{name:.
}/json DEBU[0000] Registering GET, /_ping DEBU[0000] Registering GET,
/events DEBU[0000] Registering GET, /images/json DEBU[0000] Registering
GET, /images/get DEBU[0000] Registering POST, /containers/{name:.}/copy
DEBU[0000] Registering POST, /containers/{name:.
}/exec DEBU[0000] Registering POST, /exec/{name:.}/start
DEBU[0000] Registering POST, /auth
DEBU[0000] Registering POST, /build
DEBU[0000] Registering POST, /containers/{name:.
}/wait DEBU[0000] Registering POST, /containers/{name:.}/attach
DEBU[0000] Registering POST, /images/{name:.
}/tag DEBU[0000] Registering POST, /containers/{name:.}/pause
DEBU[0000] Registering POST, /containers/{name:.
}/rename DEBU[0000] Registering POST, /commit DEBU[0000] Registering
POST, /images/create DEBU[0000] Registering POST, /images/load DEBU[0000]
Registering POST, /images/{name:.}/push
DEBU[0000] Registering POST, /containers/{name:.
}/unpause DEBU[0000] Registering POST, /containers/{name:.}/restart
DEBU[0000] Registering POST, /exec/{name:.
}/resize DEBU[0000] Registering POST, /containers/{name:.}/resize
DEBU[0000] Registering POST, /containers/create
DEBU[0000] Registering POST, /containers/{name:.
}/kill DEBU[0000] Registering POST, /containers/{name:.}/start
DEBU[0000] Registering POST, /containers/{name:.*}/stop
DEBU[0000] docker group found. gid: 1001
FATA[0000] Shutting down daemon due to errors: error intializing
graphdriver: driver not supported
1.6.0 runs OK:
$ docker -d -D
DEBU[0000] waiting for daemon to initialize
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering GET, /containers/{name:.
}/changes ... DEBU[0000] Registering DELETE, /images/{name:.}
DEBU[0000] Registering OPTIONS,
DEBU[0000] docker group found. gid: 1001
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm version OF
16384 http://*1
DEBU[0000] devicemapper: udev sync support: true
DEBU[0000] Generated prefix: docker-253:22-524289
DEBU[0000] Checking for existence of the pool 'docker-253:22-524289-pool'
DEBU[0000] libdevmapper(6): ioctl/libdm-iface.c:1795 (4) dm info
docker-253:22-524289-pool OF 16384 http://*1
DEBU[0000] [deviceset] constructDeviceIdMap()
DEBU[0000] Loading data for file
/var/lib/docker/devicemapper/metadata/00a0c78eeb6d81442efcd1d7c02e8b141745e3a06f1ee3458e1bae628e0067d3
...
DEBU[0000] Added deviceId=409 to DeviceIdMap
DEBU[0000] Loading data for file
/var/lib/docker/devicemapper/metadata/transaction-metadata
DEBU[0000] Added deviceId=659 to DeviceIdMap
DEBU[0000] [deviceset] constructDeviceIdMap() END
DEBU[0000] Using graph driver devicemapper
DEBU[0000] Creating images graph
DEBU[0000] Restored 103 elements
DEBU[0000] Error restoring volume: Volume exists:
ee38a6a63aaefed30d3b26fb31151f5fb9bf93c7e8a7bc6514d621746735e33f
DEBU[0000] Creating repository list
DEBU[0000] Reloaded graph with 3 grants expiring at 2017-03-22
19:04:46.713978458 +0000 UTC
INFO[0000] +job init_networkdriver()
...
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/12479.

How to work make it run again in Fedora 22?

add  -s graphdriver_you_want_to_use to daemon options
On Mon, May 11, 2015 at 12:37 PM, Ricky Zhang<EMAIL_ADDRESS>wrote:

How to work make it run again in Fedora 22?
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/12479#issuecomment-101026960.

I think it knows its devicemapper.
One thing I don't follow is that why it mention dynamic binary in the comment. Why is it related to udev sync bug and docker binary?

oh ya you need a dynamic binary not a static one
On Mon, May 11, 2015 at 1:55 PM, Ricky Zhang<EMAIL_ADDRESS>wrote:

I think it knows its devicemapper.
One thing I don't follow is that why it mention dynamic binary in the
comment. Why is it related to udev sync bug and docker binary?
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/12479#issuecomment-101043958.

But why?

Forget why. But how to compile it in dynamic binary? I find one (http://sickbits.net/build-a-dynamically-linked-docker/) online. But it doesn't work in FC21.
Are there official document? dynbuild is not included in Makefile. Is it OK to do the following:
diff --git a/Makefile b/Makefile
index d139602..a3ba0a8 100644
--- a/Makefile
+++ b/Makefile
@@ -44,6 +44,9 @@ all: build
binary: build
$(DOCKER_RUN_DOCKER) hack/make.sh binary
+dynbinary: build

  $(DOCKER_RUN_DOCKER) hack/make.sh dynbinary

cross: build
$(DOCKER_RUN_DOCKER) hack/make.sh binary cross

I build one by hacking Makefile.
But there is showstopper.
[root@gtx ~]# ldd /usr/bin/docker
linux-vdso.so.1 =>  (0x00007ffe6edfa000)
libapparmor.so.1 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003c32e00000)
libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x000000315be00000)
libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000003c51600000)
libc.so.6 => /lib64/libc.so.6 (0x0000003c32600000)
/lib64/ld-linux-x86-64.so.2 (0x0000003c32200000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003c33200000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003c34600000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x0000003c4b400000)
libudev.so.1 => /lib64/libudev.so.1 (0x0000003c46000000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c33e00000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x0000003c33a00000)
librt.so.1 => /lib64/librt.so.1 (0x0000003c35200000)
libdw.so.1 => /lib64/libdw.so.1 (0x0000003c42400000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c34200000)
libelf.so.1 => /lib64/libelf.so.1 (0x0000003c42000000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x0000003c35600000)
libz.so.1 => /lib64/libz.so.1 (0x0000003c33600000)
Read Apparmor website. It requires kernel recompile in Centos and Fedora.

@rickyzhang82  try to this, remove apparmor form BUILDTAGS
diff --git a/Dockerfile b/Dockerfile
index 9b6a79b..44f6c70 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -150,7 +150,7 @@ RUN useradd --create-home --gid docker unprivilegeduser

 VOLUME /var/lib/docker
 WORKDIR /go/src/github.com/docker/docker
-ENV DOCKER_BUILDTAGS apparmor selinux
+ENV DOCKER_BUILDTAGS selinux

@coolljt0725 Thanks! It works now.
BTW, do you know why it has to compile in dynamic binary to resolve udev race condition?

@rickyzhang82  There is a long story here #4036


DefaultOAuthInterceptor - allowedUrls should be required when sendAccessToken is true

When providing OAuthResourceServerConfig with sendAccessToken = true the allowedUrls property should be required. The current implementation of DefaultOAuthInterceptor will append your token to EVERY request when allowedUrls is falsy as seen here:
https://github.com/manfredsteyer/angular-oauth2-oidc/blob/master/projects/lib/src/interceptors/default-oauth.interceptor.ts
if (this.moduleConfig.resourceServer.allowedUrls) {
  return !!this.moduleConfig.resourceServer.allowedUrls.find(u =>
    url.startsWith(u)
  );
}

...

if (
  !this.moduleConfig ||
  !this.moduleConfig.resourceServer ||
  !this.checkUrl(url)
) {
  return next.handle(req);
}

This seems like a pretty big security hole - you shouldn't be appending your token to every outgoing request without regard to where it's going, especially when it's so easy to not provide the allowedUrls value.

Furthermore, the custom interceptor example found here is pretty similar to the DefaultOAuthInterceptor and would not have the same bug. It explicitly checks that allowedUrls is truthy before appending the token.

Not sure if I'd call it a "big security hole" on its own, but it does indeed invite people to (possibly without realizing) have a security hole in their applications. I'd personally agree that the default should not be that all URLs are allowed.
Interestingly I found that the commit that introduced this did so intentionally, although I'm not sure if the PR was merged minding this intention too.
Changing this behavior would be a big breaking change though, as people might have come to rely on this 'feature'. It might be wise to have either a deprecation in a version first, or at the very least detect this situation and warn people in the console or at build time about this, if we are to change it.
My 2 cts.

Sure, it's not inherently a problem on it's own but it's far too easy to misconfigure which results in very real security implications (feel free to ask my coworkers about it). People don't pay attention, the code "works" and that's the end of it. I'd go as far as to suggest that the DefaultOAuthInterceptor should never have allowed a falsy value and if a different behavior is desired then just write a custom interceptor.
However, here we are and I understand that it would be a breaking change and requires some thought to implement correctly. Your suggestion of a warning (console or otherwise) would at least point out the likely undesired behavior.
A feature like this should be a deliberate decision, and a missing or empty configuration property does not qualify in my book.


Database Design - Update "PurpleAir Stations"

The database structure needs some slight adaptations for the PurpleAir Stations. This will affect the daily_updates, particularly when adding new sensors.
Additions:
last_elevated datetime, for use when deciding when to message people and migrating alerts, default value upon creation is... 1 day ago?
Irrelevant Information:
firmware_version
modified_since
uptime
Tasks:

update ER Diagram
update initializedb.sql
modify Daily_Updates.py

branch is here: 33-UpdatePurpleAirStations

Mostly done 1a76ee2 just need to change the ER Diagram

Incorporated a regular update of the last_elevated field in this commit e0f992c (see GetSort_Spikes.py)

Finished ER Diagram 781c087 - just need confirmation on e0f992c

I didn't test this intensively, but based on what I noticed in the Modularization branch, it does seem to be working!
I didn't realize it was done already, so I was getting confused as to why I couldn't mock up alert starts/ends just by flipping between limit of 1 vs. limit of 50 in MAIN invocation. But once I realized, I was able to adjust the time from 20 to 2 minutes, then the mocking up started actually ending old alerts & starting new ones.


Main instance should be manually deployed

Currently it's deployed on a push to master. Should be switched to manual to minimize disruption

according to uptime robot, deployment means the server is down for 7mins, which is not good. But this could be to do with how frequently it checks