get_css() takes 3 positional arguments but 4 were given

Hi,
I'm trying to add wagtail-pdf to a site that is just about ready to launch. I tried to add pdf generation to the main app for the site, but I may have misunderstood how to do this. What I did was simply to have my existing Page model inherit from PdfViewPageMixin, and to add ROUTE_CONFIGto the model, as below:
class DocumentPage(PdfViewPageMixin, Page):
    template  = "documents/document.html"
    parent_page_types = ["top.IndexPage", "DocumentList"]

    date = models.DateField("Date", null=True)
    date_precision = models.CharField(

[snip...]

    ROUTE_CONFIG = [
        ("html", r'^$'),
        ("pdf", r'^pdf/$'),
    ]

The "PDF Preview" option shows up correctly in the Wagtail admin interface, but when I select it, I get the error get_css() takes 3 positional arguments but 4 were given. I get the same error when I add /pdf to the end of the URL for the published page. The HTML page still renders correctly.
Can someone help me figure out what I'm doing wrong?
Thanks!

Hi,
cool that you are using this library!
I did not have that error before I can't directly see something in your code that is wrong, so I would guess that this is a bug.
While I just wrote this, I did a quick search on the dependencies of get_css() and I found that django-weasyprint has altered this method
In particular this library still uses get_css(self, base_url, url_fetcher), while the latest version of that library requires get_css(self, base_url, url_fetcher, font_config) which would perfectly match to your reported failure.
To resolve it simply downgrade django-weasyprint to a version that was created before october 21.
Hope that helps & Cheers!

Thanks! I'll try that and report back.

First attempt:
I uninstalled django-weasyprint 2.1.0 (the current version) and weasyprint 54.0, then installled django-weasyprint 2.0 (which installed weasyprint 53.0). Now I get a different error:
function/symbol 'pango_context_set_round_glyph_positions' not found in library '/usr/local/lib/libpango-1.0.dylib': dlsym(0x7f9b9d413720, pango_context_set_round_glyph_positions): symbol not found
I'll try going back another version to see if that helps.

I installed django-weasyprint 1.1.0.post1, the last version before 2.0. Now it doesn't crash, but all I get with PDF Preview in the wagtail admin (and with /pdf at the end of the page URL) is a single page with a rectangle of the background color of my site.
But tomorrow I will try a template dedicated to the pdf to see what happens.
I should have mentioned that I'm using wagtail 2.15.1 and Django 3.2.7.

Hm okay,
from my experience the pango errors are among the most annoying ones, pango itself is a dependency of weasyprint that is used under the hood to render the pdfs.
I've had the case that pango (which is a C library) was highly outdated on the webserver and nothing did work.
In particular updating weasyprint via pip can't solve those issues as pango needs to be upgraded by the systems package manager (e.g. apt-get).
So before you go down all that struggle I would recommend  that you check whether pango is using a recent version, this really saves a lot of frustration ;)
Your last report does not sound too bad, you need to make sure to use a dedicated pdf template if you are using a mixed html/pdf side. I would guess that weasyprint tries to render the full html page and that will not work out.
Maybe a good first step would be to fork the demo folder of this project and to see whether it works out as it's supposed to. Then you at least know, whether you have some package issues or just some mistake in your code.

Thanks for your help. I tried to incorporate elements into my existing model from HtmlAndPdfPage in home.models and pdf_document_base.html from the demo, but so far no luck. I may not correctly understand how this is supposed to work.

Sure, no problem!
In general creating a "plain" PDF page should be relatively easy, you just need to ensure that weasyprint will mess up, if you simply pass the HTML of an entire website to it.
Especially if you using the shared HTML&PDF mode you need to take care of this.
The demo uses document_base.html as base HTML-page layout and pdf_document_base.html as PDF-page layout and then inserts the actual page content into both templates, such that your browser and weasyprint both get their own distinctive HTML-code.
In particular, pdf_document_base.html does not extend the django base template here, such that all the JS stuff and so on is not included.
{% extends 'base.html' %}
Sorry that you are going through so much trouble here, usually this is really rather plug and play ;)
I probably need to add more details on this to the project

Thanks and no problem. I'd like to be able to use your package, so I'd like to do whatever I can to help figure out why it's not working.
I thought that I had already understood everything you wrote in your last response, but I hadn't been able to get it to work. I uninstalled everything, took out all of the relevant code, and today I started over from scratch. I'll give my detailed steps, in case I'm missing something obvious.

I installed wagtail-pdf-view 0.2.0 and django-weasyprint 1.1.0.post1 (which installs weasyprint 52.5).
I added wagtail_pdf_view to my INSTALLED_APPS (my project was already using wagtail.contrib.routable_page, so that was already in INSTALLED_APPS)
In models.py for the app (documents) I'm trying to add pdfs to, I added from wagtail_pdf_view.mixins import PdfViewPageMixin
The existing model I'm trying to add pdfs to is DocumentPage. I modified that model using your HtmlAndPdfPage as my guide:

class DocumentPage(PdfViewPageMixin, Page):
    template  = "documents/document.html"

[snip...]

# HTML first
    ROUTE_CONFIG = [
        ("html", r'^$'),
        ("pdf", r'^pdf/$'),
    ]

[snip]

    pdf_base_template = "document_pdf.html"

    def get_context(self, request, mode=None, **kwargs):
        context = super().get_context(request, **kwargs)

        if mode == 'pdf':
            context["override_base"] = self.pdf_base_template

        return context

[snip]

I created document_pdf.html to be as simple as possible, just to see if Wagtail is even finding it;
{% load wagtailcore_tags %}
<!DOCTYPE html>

    <meta charset="utf-8">
    <title>{{page.title}}</title>

Hello world

In the Wagtail editor, Preview now offers "PDF Preview" as a choice (as expected). "HTML Preview" gives a preview using my normal base.html and document.html at http://<IP_ADDRESS>:8000/admin/pages/26/edit/preview/?mode=html, also as expected.
"PDF Preview" opens http://<IP_ADDRESS>:8000/admin/pages/26/edit/preview/?mode=pdf, but it seems not to find my document_pdf.html, which is currently at the top level of my templates folder (I've also tried it in the documents subfolder under templates). It opens a page in Firefox's pdf reader, but it contains only a rectangle in the content area that is the color of the background from my site (#f6f7f6). This is a surprise, because I can't see why it would be taking anything at all from the main CSS for my site. The "Hello world"  text doesn't appear.
No worries if you haven't got the time to diagnose this now. I just wanted you to have the full information. Let me know if there is anything else I can tell you.

Okay now I think we are getting there!
Again its still looks like you are using this correctly but I suspect that there is a small part missing in your templates/documents/document.html file:
Since you do still get the gray #f6f7f6 box and not a TemplateDoesNotExist error, django does not hook in your PDF-template at all.
From your code, I can see that you've cloned the get_context() method from the demo, which sets the override_base variable in the context.
Importantly, this context variable needs to be incorporated into your document.html (html_and_pdf_page.html in the demo)
{% extends override_base|default:'document_base.html' %}

By this you can selectively embed the content of document.html into either pdf_document_base.html (for PDF) and document_base.html (for HTML)
For the sake of completeness: If your PDF content is supposed to be very different from your HTML page content (e.g. the PDF renders a menu and while HTML page is a delivery service form), you should rather hook in entirely separated templates:
You can simply override PdfViewPageMixin.get_template() in your Page Model (important I did not test this snippet):
class DocumentPage(PdfViewPageMixin, Page):
    # override the default template selection
    def get_template(self, request, extension=None, view_provider=None, **kwargs):

        # Check whether a PDF was requested or not
        if isinstance(view_provider, PdfViewPageMixin.PDF_VIEW_PROVIDER):
            return 'path/to/html/template.html'

        return 'path/to/html/template.html'

Even more info that you wont probably need, but might be relevant to others: The get_template() is called by WagtailAdapterMixin.get_template_names(), from which WagtailWeasyView inherits. On the other hand WagtailWeasyView is specified as DEFAULT_PDF_VIEW_PROVIDER and (assuming you did not override this in the settings), PdfViewPageMixin.PDF_VIEW_PROVIDER is just DEFAULT_PDF_VIEW_PROVIDER.
Hope this suffies for you to get the job done nicely now!
During the next weeks I will add simplification to the two presented methods, this is still to complicated and needs to be more intuitive.
Besides that I will add a fix for the latest weasyprint and django-tex :+1: in the next days

Importantly, this context variable needs to be incorporated into your document.html (html_and_pdf_page.html in the demo)

{% extends override_base|default:'document_base.html' %}

By this you can selectively embed the content of document.html into either pdf_document_base.html (for PDF) and document_base.html (for HTML)

Aha! That's what I overlooked! Now it works!
I guess maybe that ought to be in the README ;-)
I'd be happy to help with the project. My coding skills may not be advanced enough (yet, at least) to help with the code, but I'd be happy to help with the README.
Thanks so much for your help and patience!

Okay nice that it works now!
Contributions of any kind are very welcome, if you want to participate besides coding:

you can help by adding details to the README, (which clearly lacks some information here..)
by reporting bugs and non-intuitive stuff of all kind
or by improving the demo

In case of the README, just add your adjustments and make a pull request
Cheers! :)

The "PDF Preview" option shows up correctly in the Wagtail admin interface, but when I select it, I get the error get_css() takes 3 positional arguments but 4 were given. I get the same error when I add /pdf to the end of the URL for the published page. The HTML page still renders correctly.

This is fixed now in version 0.2.1

Just installed 0.2.1, and everything seems to be working smoothly.

Sorry, I spoke too soon. 0.2.1  seems to have broken pdf generation for me. I reverted to 0.2.0, and it works (with wagtail 2.15.3, django-weasyprint 1.1.0.post1, and WeasyPrint 52.5.

Sorry, I spoke too soon. 0.2.1 seems to have broken pdf generation for me. I reverted to 0.2.0, and it works (with wagtail 2.15.3, django-weasyprint 1.1.0.post1, and WeasyPrint 52.5.

Did you just update this library, or django-weasyprint and weasyprint as well?
I've just tested 0.2.1 against the latest versions of the related libraries and I think the changes are not compatible to the libraries previous versions.
Just in case that you are using Debian 10 (buster): The latest weasyprint requires Debian 11 (see https://github.com/donhauser/wagtail-pdf/issues/8)
If upgrading does not fix your problem with 0.2.1, it would be nice if you could add some further information (error message, ..) on this :+1:

It turned out to be a pango problem, as you had mentioned might be the case. This afternoon I installed the current stable versions of everything: Weasyprint 54.1, django-weasyprint 2.1.0, and wagtail-pdf-view 0.2.1. This crashed on my local machine with a pango-related error. But I pushed it anyway to the live version of my site on Heroku, and it worked fine there. So I updated pango to 1.50 on my Mac via homebrew (which took about an hour!), and now everything seems to work locally too. Fingers crossed!

Nice! I think you should be good now, everything up-to-date and no breaking changes in sight!


Fix comments in bigchaindb/toolbox Dockerfile

The comments were referring to another Docker image (krish7919/toolbox).

Codecov Report

Merging #1276 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1276   +/-   ##
=======================================
  Coverage   97.18%   97.18%
=======================================
  Files          51       51
  Lines        2380     2380
=======================================
  Hits         2313     2313
  Misses         67       67

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1de7ae1...e036646. Read the comment docs.


Implement filtering by relationships

JSON API Relationships will be used by OPTiMaDe at some point (as discussed with @rartino and @muhrin). It would be useful to allow filtering entries by their relationships (as noted at least in #95).
In other words, we'd like to use filter strings to query the relationships. This requires some sort of namespace merge for the attributes and the relationships.

In the discussion with @fawzi and @rartino it was decided to support JSONPath notation, at least to some extent. Concerning the namespace merge for the filter strings, the following four options emerged:

Merge the namespaces of attributes and relationships. This mandates disjoint sets of property and endpoint names.
Use relationships. prefix to distinguish between relationships and attributes (makes relationships a reserved property name).
Use a. and r. prefixes to distinguish between attributes and relationships (accordingly) if-and-only-if a period symbol (.) is used in an identifier.
Use full JSONPaths starting from the root (i.e., $.attributes.formula and $.relationships.references[*].id).


Remove darwin linker hack

What problem is this PR intended to solve?
This PR allows us to continue to support both newer rubies on macOS and
support binary extensions linked against nokogiri.
See comment in body for details on why and how it can be replaced.
The python community has been running into the same issues because
cpython extensions operate very similarly to ruby.  They got an
explanation from Apple on some of how the changes work which can be
found here and gives some additional useful context for understanding
this change:
https://github.com/python/cpython/issues/97524#issuecomment-1458855301
(I do wonder if there should be some upstream conversation with ruby
and rake-compiler-dock to move in this direction?)
Have you included adequate test coverage?
There should be no functional changes here, so I don't think we can do much
better than the existing test suite continuing to pass.B
Does this change affect the behavior of either the C or the Java implementations?
This change should have no behavioral effect.

@flavorjones I'd be interested in your thoughts on if this approach is actually the direction that you want to go.  It obviously is a reversal of the proposal in #2746 , but it does allow nokogiri-xmlsec (and anything else that someone might choose to build in the future) to continue to exist on its own and prevents needing to deal with a lot of the grossness implied in #2896

So if I understand correctly, nokogiri would be compiled with a two-level namespace and using chained fixups except for symbols from ruby/libruby which would be using the standard dynamic symbol resolution. That seems like a win all around.
I'm not thrilled about using nm to get a list of symbols. But I don't have a better suggestion absent something like https://github.com/apple-opensource/ld64/pull/1. I don't think we use a lot of functions from ruby/libruby so we could just maintain a list by hand but that's not a great experience for anyone trying to modify Nokogiri not on macOS.
Does Nokogiri-xmlsec just need symbols from Nokogiri or does it also need symbols from libxml/libxslt? If it's the latter, I would think that'd run into the same issues Nokogiri had with the system libxml being used. And two-level namespaces wouldn't work since Nokogiri can use the system libraries so other extensions wouldn't know if they needed to link against Nokogiri or libxml.

That understanding is correct.
I don't love the nm method either, but without something like that linker method there's not another option I can really come up with. Upstream ruby supports generating a list of those functions (with nm) in its makefile at ruby build time, which would really be ideal, but that's only automatically built and available on AIX, not darwin best I can tell.
nokogiri-xmlsec does use libxml symbols, but it doesn't support precompiled binary extensions, so in practice it doesn't need to worry all the two-level precompiled namepace complications. I agree though that there isn't an elegant solution there short of having 2 precompiled gems, one linked against system libxml and one linked against nokogiri-integrated libxml.

One alternative I just thought of would be to create a separate repository with symbol list files that are autogenerated with GitHub actions and then nokogiri could download them as part of its build process (and ship them with the gen for the shipped gem).  That would mean though that the gem would be uninstallable on a new major version of ruby until it had a release explicitly supporting it, or we'd want to keep the nm logic as a fallback.


Fixing some typos

Line 44 in step1-02/demo/README.md is just a suggestion, might be just me but I got confused wether the text was referring to the HTML tag or a title on the page itself.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: carlosk12 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for catching those typos! I made a few tweaks of my own.


[0.11.0] Time Parsing Extraction Function returning wrong WeekOfTheYear

Time Parsing Extraction Function for input "20171231" returns wrong WeekOfTheYear as "2017-1" instead of "2017-52". Where as it is returning correctly for "20171230".
Input
{
	"type": "extraction",
	"dimension": "stats_date",
	"outputName": "Week",
	"extractionFn": {
		"type": "time",
		"timeFormat": "yyyyMMdd",
		"resultFormat": "yyyy-w"
	}
}

Output
{
  "version" : "v1",
  "timestamp" : "2017-10-02T00:00:00.000Z",
  "event" : {
    "Stats Date" : "20171230",
    "Some Metric" : 0,
    "Week" : "2017-52"
  }
},
{
  "version" : "v1",
  "timestamp" : "2017-10-02T00:00:00.000Z",
  "event" : {
    "Stats Date" : "20171231",
    "Some Metric" : 0,
    "Week" : "2017-1"
  }
}

Hi @pavanab4u, I think you meant to do YYYY-w or YYYY-ww rather than yyyy-w. Y is year of weekyear and y is year of date. So if you did that then 20171231 would give you 2018-01 which does make more sense.
However there is another wrinkle- this formatter uses java's GregorianCalendar which is not strictly ISO: https://docs.oracle.com/javase/9/docs/api/java/util/GregorianCalendar.html#week_year. ISO reckoning would have said 2017-12-31 is weekyear-week 2017-52. Although in any reckoning, weekyear and calendar year are not always the same, so the 31st of a month could be considered to be in the first week of the next year. Exactly which years this happens for differs from reckoning to reckoning.
Probably ISO reckoning would be preferred, since it's more common in Druid. So it'd be nice to add an option for that. Probably also an option to use joda formatter strings rather than SimpleDateFormat strings, since those aren't compatible either, and joda is also more common in Druid.
Going to change this from a bug to an improvement request since it does not seem like a bug to me anymore.

Hi @gianm , makes sense, thanks!

Hi @pavanab4u - I also raised #5448 with a patch to add support for joda formats with ISO reckoning, which may also be interesting to you.

Hi @gianm This is going to be helpful, thanks !


Added Merge Sort [C]

Fixed #17

By submitting this pull request I confirm I've read and complied with the below declarations.

[x] I have read the Contribution guidelines and I am confident that my PR reflects them.
[x] I have followed the coding guidelines for this project.
[x] My code follows the skeleton code structure.
[x] This pull request has a descriptive title. For example, Added {Algorithm/DS name} [{Language}], not Update README.md or Added new code.
[x] This pull request will be closed if I fail to update it even once in a continuous timespan of 7 days.

@singhpratyush Please review.


Adddocblockhandling

Add the handling of a @transformer, @transformercollection or @repsonse tag in the docblock of a method.
This will also generate results for other request methods then GET.
This is a function I like to use and hope it is useful for other people.
I think it can help for #187 , #186, #174

Codecov Report

Merging #192 into master will increase coverage by 1.44%.
The diff coverage is 85%.

@@             Coverage Diff              @@
##             master     #192      +/-   ##
============================================
+ Coverage     79.07%   80.51%   +1.44%
- Complexity      178      187       +9
============================================
  Files             7        7
  Lines           626      621       -5
============================================
+ Hits            495      500       +5
+ Misses          131      121      -10

Impacted Files
Coverage Δ
Complexity Δ

src/Mpociot/ApiDoc/Generators/LaravelGenerator.php
86.48% <84%> (-3.52%)
22 <2> (+6)

...rc/Mpociot/ApiDoc/Generators/AbstractGenerator.php
89.92% <90%> (+1.19%)
88 <3> (+3)
:arrow_up:

...c/Mpociot/ApiDoc/Parsers/RuleDescriptionParser.php
100% <0%> (ø)
12% <0%> (ø)
:arrow_down:

src/Mpociot/ApiDoc/Generators/DingoGenerator.php
0% <0%> (ø)
8% <0%> (ø)
:arrow_down:

src/Mpociot/ApiDoc/Postman/CollectionWriter.php
100% <0%> (ø)
6% <0%> (ø)
:arrow_down:

.../Mpociot/ApiDoc/Commands/GenerateDocumentation.php
72.72% <0%> (+3.59%)
48% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db46be8...88706b0. Read the comment docs.

This are a lot more commits then I intend to add this pull request. If you want I can squash all the changes to one commit and add a new pull request instead of this one

Great, thank you!
It's okay, GitHub let's me squash the commits when I merge the PR :)


DevToys source code failed to compile due to missing ResxHelperGenerator.dll

Current behavior
The current .NET project failed to build with the following logs:
dotnet build /home/shift/projects/devtoys/src/app/dev/platforms/desktop/DevToys.Linux/DevToys.Linux.csproj /property:GenerateFullPaths=true /consoleloggerparameters:NoSummary
MSBuild version 17.8.5+b5265ef37 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  DevToys.Api -> /home/shift/projects/devtoys/bin/Debug/AnyCPU/DevToys.Api/net8.0/DevToys.Api.dll
CSC : error CS0006: Metadata file '/home/shift/projects/devtoys/src/generators/ResxHelperGenerator/bin/netstandard2.0/ResxHelperGenerator.dll' could not be found [/home/shift/projects/devtoys/src/app/dev/DevToys.Localization/DevToys.Localization.csproj::TargetFramework=net8.0]

How to reproduce it (as minimally and precisely as possible)

Make sure to install .NET 8 via Fedora's package manager (dnf install dotnet-sdk-8.0 dotnet-runtime-8.0)
Make sure that the necessary Linux-specific libraries (libadwaita etc.) are also installed
Clone and open the repository in Visual Studio Code
Run bash ./init.sh (without sudo as recommended in CONTRIBUTING.md)
Perform a debug build on either "DevToys CLI" or "DevToys Linux"

Expected behavior
The required DLL should exist to be able to compile for both CLI and GUI app.
Screenshots
No response
Workaround
No response
Affected platforms
Linux
Affected DevToys kind
DevToys (app with GUI), DevToys CLI
DevToys Version
Version: <IP_ADDRESS> (Commit 211c66f) | .NET Version: 8.0.100 | OS Version: Fedora Linux 40
Relevant Assets/Logs
No response

I believe this might also be mentioned on microsoft/vscode-dotnettools#245:

I tried re-targeting it, but then I got some build errors about ResxHelperGenerator.dll.

Hi,
I hit this issue a few random times myself. I noticed the issue vanishes if I build ResxHelperGenerator project alone first, before building the whole solution. Can you please try that? This is just a workaround that works for me and I don't have a good solution so far.

Confirmed that the build now works.

I noticed the issue vanishes if I build ResxHelperGenerator project alone first, before building the whole solution.

Should this be documented first as a temporary workaround in CONTRIBUTING.md? Also, why the use of sudo in bash ./init.sh (this will make the project files failed to build due to "insufficient permissions" issue)?

Thanks for confirming ! yes, i should probably add this as a workaround.
For the sudo command, I recall having trouble when not using it, because dotnet needed it in order to install some tools. But perhaps I'm not remembering correctly and it's not needed anymore. I will have to test on a fresh VM.

As far as I know you can install dotnet tools on a system and user level. The user level should not need increased privileges and should be preferred from my point of view.

As far as I know you can install dotnet tools on a system and user level. The user level should not need increased privileges and should be preferred from my point of view.

Maybe that varies between platforms? I am using the official pre-packaged version of dotnet-tools: dotnet-sdk-8.0 and dotnet-runtime-8.0.

Ah perhaps I misinterpreted global/ local tools: https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-tool-install
Nevertheless there should be no need for sudo.


Review solution from Manu

This PR was created to review the solution from Manu

Generally the code is fine. A bit an odd structure but otherwise nothing big to say about it. Keep it up man.
Regarding the user experience, I have a few ideas to improve it:

The root command should display help by default
The user should be an optional command argument since through the token by default we already know the user and we could still then allow specifying other users to check. Most importantly it really MUST be a command line argument and no input as a second step.
The names are really not ... that good. Here is my outline recommendation:

stars [optional username]
user [optional username]
status [tea | clear]

The help message of setstatus requires improvement.


It should not be possible to praise/diss your own texts

Actually, why not? Facebook and LinkedIn allows this behaviour.


Add capability for automatically downloading (and storing in some fixed location) versions of browsers

And track them somehow, e.g. seltest list-browsers, which should also return the default browsers.

This seems out of scope and horrible.


Allow blackArgs to accept a file path for settings

similar to sorting, allow python.formatting.blackArgs to allow a cfg or toml file to import settings.
    "python.sortImports.args": [
        "--settings-path=${workspaceFolder}/pyproject.toml"
    ]

That would be an ask on Black to support other file paths than pyproject.toml, but I very strongly suggest they will not add such a setting as the tool's opinionated opinion is pyproject.toml is the way to do settings.

What, this is exactly what I’m asking. It doesn’t support any file settings.  Just named arguments

@notacoat Black the tool itself doesn't support any way to specify a file path for your settings. Because of this there's nothing we can do here. The only configuration file that Black supports is pyproject.toml and since there is no way to tell Black to use a different file or specify where it is we can't really do anything here.
Otherwise I don't understand what you're asking of us to provide? If I'm wrong then please show what you would run on the command-line with Black to get the result you want.

you can specify the path for the config file like so
black --config ../pyproject.toml models.py

here I am telling it the pyproject.toml file is one directory up. so you can set the path. but it does need to be pyproject.toml


Disable all tests depending on half_plus_two genrules.

This is a measure to stabilize flaky GPU builds.
This is a fix for the CUDA out of memory error issues we have been seeing.

The CPU failures are known issues.
So this should fix the GPU issues.


when the go sdk will support System instructions setting in GenerativeModel of Client

when the go sdk will support System instructions setting in GenerativeModel of Client,
now i am using 0.11.0

i am also facing an issue where api will throw a 400 error when calling SendMessage after i have set model.systemInstruction

Setting SystemInstruction should work now, you can see an example at https://pkg.go.dev/github.com/google/generative-ai-go/genai#example-GenerativeModel.GenerateContent-Config
Note that you need a Gemini 1.5 model for that, since 1.0 doesn't support system instructions. Set the model name to "gemini-1.5-pro-latest"

Since the example in go package, model := client.GenerativeModel("gemini-1.0-pro") , uses 1.0,  I wasn't aware that the 1.0 doesn't support the system instructions. Also it is not mentioned  in docs that system instruction is exclusive to 1.5. Is it possible to learn if 1.0 will ever get system instructions? If not example in go package is misleading.

#92 fixes the example to use an appropriate model.

Is it possible to learn if 1.0 will ever get system instructions?

Unfortunately, we don't know.


Fix broken link 3650

Category

[x] Content fix
[ ] New article

Related issues:

closes #3650

What's in this Pull Request?

previous PR #3657 caught one link... there were two
this fixes the other link

thx @andrewconnell, appreciated.


list near

https://docs.near.org/docs/develop/basics/getting-started
bridge https://ethereum.bridgetonear.org/
explorer https://explorer.near.org/

https://etherscan.io/address/0x23ddd3e3692d1861ed57ede224608875809e127f

fixed in https://github.com/gasolin/ethhole/commit/f8a5c09f67119e039245f909494af89985a5af15


I keep getting this error Fatal error: Class 'Eher\OAuth\HmacSha1' not found in

I get this error  Fatal error: Class 'Eher\OAuth\HmacSha1' not found in,  I tried to install Eher folder data on windows , but for some reason I have an issue with  the php_imagick.dll when I run the command curl -sS https://getcomposer.org/installer | php
php composer.phar install --no-dev
Is there any way I can install both the Eher and Tumblr PHP files alltogether manually ? and can you explain me the hierarchy of the Tumblr Folder and the Eher , because I installed the Tumblr Foler in my FTP (Tumblr/API/Client.php,RequestException.php,RequestHandler.php) and then I installed the Eher folder in the API folder .. which i thought would work , but it's still giving me those errors ... Any help regarding this issue would be greatly appreciated.... If there's a way I can install both the Tumblr and Eher folders alltogether to avoid these path errors , that'd be great.

I don't see an Eher class anywhere in this project? I have the same error.


Bottom part of the selected dot cuts off

Library version: 1.0.0
Affected device(s): OnePlus 5T with Android 9.0
Bug: The bottom part of the selected dot cuts off.

Padding bottom definitely won't do it, that will cut it off. I'm guessing it's because you're setting an explicit height - it should be wrap_content.

Setting the dots_gravity property to only center fixed the issue. The height had to be set to 8dp though.


PR update

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@6b02468). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #3   +/-   ##
=========================================
  Coverage          ?   42.26%
=========================================
  Files             ?        4
  Lines             ?       97
  Branches          ?        0
=========================================
  Hits              ?       41
  Misses            ?       56
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b02468...68b0fb3. Read the comment docs.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@6b02468). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #3   +/-   ##
=========================================
  Coverage          ?   42.26%
=========================================
  Files             ?        4
  Lines             ?       97
  Branches          ?        0
=========================================
  Hits              ?       41
  Misses            ?       56
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b02468...41d0571. Read the comment docs.


.exe import error: Specified length extends beyond file bytes length

Describe the bug
I'm attempting to import an old Win95 game into Ghidra for disassembly, but Ghidra throws an error when importing the portable executable file into a project.
To Reproduce

Go to 'File/Import'
Select 'TM2.EXE'
Import the file with default settings in the import menu
See error

Expected behavior
Ghidra should successfully import/process the file and allow its disassembly.

Environment (please complete the following information):

OS: Linux Mint 19.2 (kernel version 5.1.16)
Java Version: OpenJDK 11
Ghidra Version: 9.1.2

Additional context
This is an old Win95 game I'm trying to reverse engineer, so there should be no obfuscation or anything.
Let me know if any more information is needed! Thanks.

@abelbriggs1 are you referring to https://oldgamesdownload.com/twisted-metal-2/ correct?
Thrown Location:
https://github.com/NationalSecurityAgency/ghidra/blob/6e6bdffe86e8d7b1d9f58f5f7d738d8fe5be0b23/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/mem/MemoryMapDB.java#L567
Looks like your starting offset + length is greater than it can handle.
https://github.com/NationalSecurityAgency/ghidra/blob/6e6bdffe86e8d7b1d9f58f5f7d738d8fe5be0b23/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/mem/MemoryMapDB.java#L490
There is no issues with the block length.
https://github.com/NationalSecurityAgency/ghidra/blob/6e6bdffe86e8d7b1d9f58f5f7d738d8fe5be0b23/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/mem/MemoryMapDB.java#L1956
Interesting. I'll also look into it a bit more.

That would indeed be the game I'm attempting to disassemble, yes.

I have same problem with an another late 90's game. I investigated this and found that PeLoader tries interprests .bss section as initialized block. Patch below resolved my problem.
diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
index ae17ad4..dad720e 100644
--- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
+++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
@@ -641,8 +641,9 @@
 					SectionFlags.IMAGE_SCN_MEM_EXECUTE.getMask()) != 0x0);

 				int rawDataSize = sections[i].getSizeOfRawData();
+				int pointerToRawData = sections[i].getPointerToRawData();
 				virtualSize = sections[i].getVirtualSize();
-				if (rawDataSize != 0) {
+				if (rawDataSize != 0 && pointerToRawData > 0) {
 					int dataSize =
 						((rawDataSize > virtualSize && virtualSize > 0) || rawDataSize < 0)
 								? virtualSize

I have same problem with an another late 90's game. I investigated this and found that PeLoader tries to load .bss section as initialized block. Patch below resolved my problem.
diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
index ae17ad4..dad720e 100644
--- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
+++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/PeLoader.java
@@ -641,8 +641,9 @@
 					SectionFlags.IMAGE_SCN_MEM_EXECUTE.getMask()) != 0x0);

 				int rawDataSize = sections[i].getSizeOfRawData();
+				int pointerToRawData = sections[i].getPointerToRawData();
 				virtualSize = sections[i].getVirtualSize();
-				if (rawDataSize != 0) {
+				if (rawDataSize != 0 && pointerToRawData > 0) {
 					int dataSize =
 						((rawDataSize > virtualSize && virtualSize > 0) || rawDataSize < 0)
 								? virtualSize

I can confirm that this patch seems to fix the issue and allows me to load the executable into Ghidra.

Just ran into this issue with another late 90s Windows binary. Reproduced with a fresh master build. Successfully applied patch from above. What's the status on this? Is more information needed, is the above patch insufficient?

Bumping this because I run into the same issue.

As far as I can tell this happens with Watcom C(++) binaries, which was the go-to compiler in the 90s. But I guess the NSA doesn't really stumble upon these too often in their day-to-day job.

Duplicate of #2496.


headless-client_1  | Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: Empty key

Hi,
I've been trying to get the token app running on my local machine using Docker (Using toolbox if that's a problem) On windows 10. Ran docker-compose up and then this error appears. I can't seem to find any fixes online. Hoping you might have an idea for a fix. Thank you in advance
           | [nodemon] startingnode src/bot.js config.yml bot_1              | Fiat: Fetching rates bot_1              | Address: headless-client_1  | com.bakkenbaeck.token.headless.TokenHeadlessClientConfiguration@7b1d7fff headless-client_1  | [main] INFO org.flywaydb.core.internal.util.VersionPrinter - Flyway 4.1.0 by Boxfuse headless-client_1  | [main] INFO org.flywaydb.core.internal.dbsupport.DbSupportFactory - Database: jdbc:postgresql://postgres:5432/token (PostgreSQL 9.6) headless-client_1  | [main] INFO org.flywaydb.core.internal.command.DbValidate - Successfully validated 1 migration (execution time 00:00.091s) headless-client_1  | [main] INFO org.flywaydb.core.internal.command.DbMigrate - Current version of schema "public": 1 headless-client_1  | [main] INFO org.flywaydb.core.internal.command.DbMigrate - Schema "public" is up to date. No migration necessary. headless-client_1  | Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: Empty key headless-client_1  |    at org.bitcoinj.crypto.PBKDF2SHA512.derive(PBKDF2SHA512.java:61) headless-client_1  |    at org.bitcoinj.crypto.MnemonicCode.toSeed(MnemonicCode.java:134) headless-client_1  |    at org.bitcoinj.wallet.DeterministicSeed.<init>(DeterministicSeed.java:81) headless-client_1  |    at org.bitcoinj.wallet.DeterministicSeed.<init>(DeterministicSeed.java:54) headless-client_1  |    at com.bakkenbaeck.token.crypto.HDWallet.initFromMasterSeed(HDWallet.java:88) headless-client_1  |    at com.bakkenbaeck.token.crypto.HDWallet.initWallet(HDWallet.java:65) headless-client_1  |    at com.bakkenbaeck.token.crypto.HDWallet.init(HDWallet.java:43) headless-client_1  |    at com.bakkenbaeck.token.headless.TokenHeadlessClient.main(TokenHeadlessClient.java:80) headless-client_1  | Caused by: java.lang.IllegalArgumentException: Empty key headless-client_1  |    at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96) headless-client_1  |    at org.bitcoinj.crypto.PBKDF2SHA512.F(PBKDF2SHA512.java:74) headless-client_1  |    at org.bitcoinj.crypto.PBKDF2SHA512.derive(PBKDF2SHA512.java:56) headless-client_1  |    ... 7 more

This is likely because TOKEN_APP_SEED env var does not contain a value - make sure your docker-compose.yml config contains all the required fields: http://developers.tokenbrowser.com/docs/configuration

Doh!!! Didn't hit save!! Thank you and sorry for wasting your time!


Error when assigning value to non-contiguous sliced views

The snippet below doesn't work with xtensor 0.18.2:
xt::xtensor<int, 2> arr = xt::ones<int>({10, 10});
auto v = xt::view(arr, xt::keep(0, -1), xt::all());
v = 0;

std::cout << arr;

In file included from input_line_7:7:
/Users/bbovy/miniconda3/envs/fastscapelib/include/xtensor/xview.hpp:957:29: error: no matching member function for call to 'storage_begin'
            std::fill(this->storage_begin(), this->storage_end(), value);
                      ~~~~~~^~~~~~~~~~~~~
/Users/bbovy/miniconda3/envs/fastscapelib/include/xtensor/xview.hpp:873:15: note: in instantiation of function template specialization 'xt::xview<xt::xtensor_container<xt::uvector<int, std::__1::allocator<int> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag> &, xt::xkeep_slice<long>, xt::xall<unsigned long>
      >::fill<int>' requested here
        this->fill(e);
              ^
input_line_8:4:3: note: in instantiation of function template specialization 'xt::xview<xt::xtensor_container<xt::uvector<int, std::__1::allocator<int> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag> &, xt::xkeep_slice<long>, xt::xall<unsigned long>
      >::operator=<int>' requested here
v = 0;
  ^
/Users/bbovy/miniconda3/envs/fastscapelib/include/xtensor/xview.hpp:502:9: note: candidate template ignored: requirement 'is_strided_view' was not satisfied [with T = xt::xtensor_container<xt::uvector<int, std::__1::allocator<int> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag>]
        storage_begin();
        ^

This used to work with version 0.17.x. Contiguous sliced views seem to still work fine.

I see that this could result from #1039.
Is this behavior intentional or is it an unwanted side effect?

Yes, we should have used a static_if on the compile-time layout instead of the if statement, to prevent compiling the code of the other branch.

Let me know if this fixes your issue.

I think this was two different PR's -- the first where we propagated the contigouty, and the other where we were cleaning up the storage_begin/end interface!
Thanks for spotting this!

Yes it fixed the issue, thanks!!


New CMS

Find a better CMS that does not use Node.js since it produces docker images far too large. For context, the Strapi CMS image is just over 1 GB. Ideally, it should still support GraphQL and integrate with Gatsby (the framework we use for the website).

Currently evaluating DIrectus (v9) and Ghost. If neither meet my criteria, then it may be worth developing in in-house one or using a commerical service. I am hesistant to use a commerical service since there is usually a "tax" for SSO (sso.tax).
Current evaluation status:

Ghost appears to be geared more towards hosting the website itself, similiar to how wordpress works, but it has a way to disable the frontend.
Directus v9 is currently in release candidate, so stability may not be the greatest, but I will do some more testing.


Fix elision of dot nodes

This changelist fixes a bug in the elision of dot nodes (#1152), where dot nodes with channels attributes would be incorrectly bypassed, causing the shader generator to produce incorrect code.

CC'ing @kwokcb to make sure this change looks reasonable!

Hi @jstone-lucasfilm , I put more details in the code area, but I think it's worthwhile to not allow "channels" on dot nodes and have users do things explicitly as they could never create this in the node editor (from example I added).

If they could then I think along the thoughts of moving towards a code gen friendly graph, then these could be pre-converted
from implicit to explicit.

@kwokcb Although it's not yet possible to create connections via channels strings in the editor, that's just a temporary limitation, rather than a deprecation of the channels string feature.  In the future, the goal is to support channels string connections in the editor, and this ought to work for dot nodes as well, since there is no rule in the specification or codebase to discourage this.


[Bug][Azure DevOps] Build not found

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
Receiving build not found when running AzureDevOps blueprint.

task-293-2-1-azuredevops.log
What do you expect to happen
Run without error
How to reproduce
running AzureDevOps blueprint.
Anything else
Doing a get request to "https://dev.azure.com/shapeDigitalSD/Maintenance Product/_apis/build/builds/" dont return any 4863 buildid.

Version
v0.17.0-beta10
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

How does the devlake get those build ids? Is searched from azure dev ops or based on some devlake db table?

This build id comes from a previous collection.
I suspect that this is a build that first got marked as deleted, and really got deleted after some time (set in the retention policy of your project settings, minimum 30 days). Asking for the timeline of a build that is just marked deleted returns a 204, but for a build that really got deleted, we apparently get a 404.


The Stream API Documentation URL can be linked

In the Product page of a streaming asset the API docs URL is displayed. It would be good to make it a link to itself, like:
https://digitransit.fi/en/developers/apis/4-realtime-api/

If it's an href it'll be magenta, just like the Publisher address.

An example can be seen here
Scrrenshot

Just to clarify, it should be clickable.


Headless rendering of a 2d image

Hello Sascha,
thank you very much for your great examples. This makes it much easier to get started with Vulkan.
For handling 2D images, I would find it great if there was an example where you can render an image over a headless path by piping an input 2d image (e.g. 800 x 600 pixel) and end up with a modified image of size 800 x 600 (e.g. low pass filtered). One idea of mine is to combine the example https://github.com/SaschaWillems/Vulkan/tree/master/examples/renderheadless with the example https://vulkan-tutorial.com/code/25_texture_mapping.cpp by putting an image as texture on a quad and running the processing on it.
I know that theoretically you could also take a computing shader for this. But I find it interesting to possibly augment the image with 3D objects later.
Maybe there is such a 2D image processing example already and I missed it. Every hint is welcome.

This compute shader does some basic image processing, and could be used as a base for such a sample. Making this headless shouldn't be hard, and reading the image back similar to how the other compute headless sample reads back the buffer is also not that hard to add. I don't plan to do such a sample myself, but I hope this helps as a starting point.

This compute shader does some basic image processing, and could be used as a base for such a sample. Making this headless shouldn't be hard, and reading the image back similar to how the other compute headless sample reads back the buffer is also not that hard to add. I don't plan to do such a sample myself, but I hope this helps as a starting point.

Hello Sascha,
Thank you very much for your prompt reply. I will explicitly take a closer look at this example.

Hello Sascha,
Thank you very much for your prompt reply. I will explicitly take a closer look at this example.


Syncronous Interrupt hanging

I'm interacting with an ADC that has a busy signal while it does conversion. I block while the busy pin is high:
let mut busy = Gpio::new()?.get(GPIO_BUSY)?.into_input_pullup();

loop {
    start_conversion();
    while busy.is_high() { /* block while we wait for conversion to finish */}
    ...
    read_data();
}

This gets me the basic functionality, but I wanted to use an interrupt on that pin instead. I set it up like so:
let mut busy = Gpio::new()?.get(GPIO_BUSY)?.into_input_pullup();
busy.set_interrupt(gpio::Trigger::FallingEdge);

loop {
    start_conversion();
    busy.poll_interrupt(false, None);
    ...
    read_data();
}

🤔 Am I doing this right or is this a bug?

Thanks for the compliment! Glad to hear you found the cause of the issue.


Use an array of structures instead of a structure of arrays for histograms

On master (structure of arrays using @njitclass):
$ python benchmarks/bench_histogram.py
Building feature histogram on 10000000 values of 100000000
Built in 2.007s

On this branch / PR (array of structures):
$ python benchmarks/bench_histogram.py
Building feature histogram on 10000000 values of 100000000
Built in 1.655s

So the memory access pattern is probably better this way.
Questions for @seibert:

is there a natural way to use a @jitclass to implement the array of structures pattern.
when using the @jitclass construct, it does not seem to be possible to use introspect_types / asm on the instance methods. Is this is a missing feature in the current implementation or is there a fundamental problem that prevents implementing those?

I have just pushed a new commit with unrolled histogram building as done in LightGBM. Here are the results:
$ python benchmarks/bench_histogram.py
build_histogram on 1e+07 values of 1e+08
Built in 1.771s
build_histogram_unrolled on 1e+07 values of 1e+08
Built in 0.863s


Entropy z crashing 103

Fix bug reported by issue #103

Coverage decreased (-0.5%) to 85.755% when pulling 023aa8dead77ffaea5aefea838a43042392a724c on entropy_z_crashing_103 into 3139beec5fdc363d3a260017659081b1061a6b14 on development.


parking lot is broken with recent nightly

$ rustc --version
rustc 1.25.0 (84203cac6 2018-03-25)
$ rustup --version
rustup 1.13.0 (ea9259c1b 2018-07-16)
$ cargo --version
cargo 0.26.0 (41480f5cc 2018-02-26)

error[E0658]: use of unstable library feature 'thread_local_state': state querying was recently added (see issue #27716)
   --> /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.3.0/src/parking_lot.rs:169:34
    |
169 |     if let Ok(tls) = THREAD_DATA.try_with(|x| x as *const ThreadData) {

You need Rust 1.26, which was released 4 months ago.

Oups, sorry for the noize confused nightly with an MSRV :sweat:


update rollup options to make react and react-dom external

This might help reducing the bundling size.

rollup options are now upto date


Assert::all* not handle

If I do:
$opLimit = $input->getOption('operation-limit');
$timeLimit = $input->getOption('time-limit');
Assert::numeric($opLimit);
Assert::numeric($timeLimit);
$opLimit = (int) $opLimit;
$timeLimit = (int) $timeLimit;

No error outputted. But if I do:
$opLimit = $input->getOption('operation-limit');
$timeLimit = $input->getOption('time-limit');
Assert::allNumeric($opLimit, $timeLimit);
$opLimit = (int) $opLimit;
$timeLimit = (int) $timeLimit;

I have:
src/AppBundle/Command/OperationExecuteCommand.php:105:Cannot cast array<string>|bool|string|null to int.
src/AppBundle/Command/OperationExecuteCommand.php:106:Cannot cast array<string>|bool|string|null to int.

It seems the all* methods are ignored by the plugin.

::all asserts are implemented. They do something else than you probably think. According to documentation, they are meant to be used by arrays and traversables:

All of the above assertions can be prefixed with all*() to test the contents of an array or a \Traversable

Oh right, my bad then. Sorry for that.


Automatically Register/Reload Component Managers and Systems

Right now game code has to manually register all component managers and systems at startup, and has to manually reload them when the DLL is reloaded (if hotloading is enabled). This code is super boilerplate and we should be able to automate it to some extent. The easiest thing to do would be to create a macro that generates the init and and reload glue given a list of managers and systems, but I think we can do one better. I'd like to see if it's possible to create a compiler plugin to automatically generate the glue code without any need for the end use to do any work.
Minimum Feature Set

[ ] Automatically generate the glue code for registering managers and systems.
[ ] Allow for managers or systems to be in some way marked as not automatically added.
[ ] Provide some means to manually register managers and systems.
[ ] Provide some means to enable and disable the feature as a whole (it should probably be on by default, though).

Extra Features to Explore

[ ] Attribute (or something) to mark the priority for systems to ensure a specific ordering can be attained.

Notes

I think we'll have to require that all managers and systems that are auto-registered are Default otherwise we won't have any way of instantiating the objects.
Better than specifying a priority for each system it would be double-extra cool if we could specify a list of dependencies that need to be run first and then automatically generate a processing order. Not only would this allow for more correct code, it would open up possibilities for parallelizing game code.

This task is large enough to be a major milestone of its own. The hotloading system is currently broken anyway so this will likely get started as part of a future overhaul to hotloading.

The ECS is going away (or at least getting majorly reworked) so this issue is no longer relevant.


ref: update stage list refs.

Continues https://github.com/iterative/dvc.org/pull/2245#issuecomment-817176199

Corrections to the synopsis
Clarifies the description
Remove term "group" in the context of foreach stages (avoiding that term was decided early on, see https://github.com/iterative/dvc/issues/5165#issuecomment-751642817 and https://github.com/iterative/dvc/issues/5165#issuecomment-752265954)
Copy edits to the Example

what is the reason/motivation for this PR

@shcheklein basically I didn't have a chance to review #2245 before merge so I did a follow-up PR instead. It includes corrections and copy edits (detailed in the PR desc.)


Added Docker Cloud

@senk Thanks!
I guess we could remove tutum altogether

Or maybe not, tutum website is still on.

I will observe that. Docker Cloud was announced this morning i guess so maybe they will turn off the tutum page later. anyhow they have some github repos which are very tutum branded so we better leave it in the list unless it's finally shut down

Exactly.
Thanks a lot.


Update actions/checkout to v4

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (95d4732) 90.25% compared to head (943a2c8) 90.25%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #393   +/-   ##
=======================================
  Coverage   90.25%   90.25%
=======================================
  Files         141      141
  Lines       19869    19869
=======================================
  Hits        17933    17933
  Misses       1936     1936

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Merge master to apprtc-t

@milkypostman can you double check your changes? Everything looked correct to me.

This merges my most recent change which required manual merging. The source file for the code being modified had changed.
@jiayliu @tkchin

I skimmed the files and nothing stands out. lgtm


No log newlines

This PR harmonizes the use of newlines in log statements.
My custom logger was a little confused by a mix of logging statements with and without newlines.

Yea that's no good!
Does this change the output that we have today? If you do export PION_LOG_TRACE=ice does it look different?
Would it be possible to write a linter/check so this doesn't break again in the future.

Does this change the output that we have today? If you do export PION_LOG_TRACE=ice does it look different?

No it does not. Mainly because log.Logger.Output() will check if the log message ends with a newline or not.
If not, it will add one automatically:
https://cs.opensource.google/go/go/+/refs/tags/go1.18.1:src/log/log.go;l=192
So the change does not change the output for the default logger which uses the stdlib log.Logger

@stv0g That's great news! I am all for merging this.
If you have the bandwidth would you mind writing a sh script that uses grep to catch this? If you can just copy lint-disallowed-functions-in-library.sh
If you don't the bandwidth totally understand. I would just hate to see your work regressed by someone in the future :(

@Sean-Der Sure :) Here it is https://github.com/pion/.goassets/pull/88
I will rebase this PR and enable the linter once the goasset PR has been merged into pion/ice.


Ensure Code style is PR2

90% of these changes are NOT PSR-2 specified.

Actually, it's 99% of this that has nothing to do with PSR-2 at all! Only the change to the namespace to match the folder name is PSR-2. Everything else is not PSR-2.


contact alias not sync after call contact.alias(string) to set alias

Provide Your Network Information

Where is the location of your server? (i.e. City, or In/Out China)
U.S.
Which cloud platform(AliYun/Qcloud/DigitalOcean/etc) are you using?
own machine

Run npm run doctor or wechaty run doctor(for docker user), paste output here
//docker run in container
const old_alias = await contact.alias()
const new_alias= "new_alias"
console.log(old_alias)   // old_alias
await contact.alias(new_alias)
await contact.sync()
await delay(getRandomIntInclusive(3000, 5000)) // delay 3~5 secs
const new_set_alias = await contact.alias()
console.log(contact.name() + ' Alias:'  + new_set_alias ) //contact name Alias: old_alias
Expected behavior
The contact alias should be set to new_alias in the wechaty after call alias()
Actual behavior
The contact alias not be set to new_alias in the Wechaty after call alias()
It still keep the old alias().
NOTE: this also happen when new friend being added, and it only be correct alias after Wechaty docker restart
Steps to reproduce the behavior (and fixes, if any)
None for now.
It may be sync issue with cache.
Full Output Logs
$ docker create -t -i --name=$docker_name \
--mount type=bind,source="${dir}",target=/bot \
-e WECHATY_LOG=silly \
-e TZ="America/Los_Angeles" \
zixia/wechaty:latest mybot.ts
$ docker start -a $docker_name

Show Logs

Paste the full output logs here with WECHATY_LOG=silly set
# PASTE FULL LOG OUTPUT AT HERE:
10:26:26 VERB Puppet selfId()
10:26:26 VERB Contact friend()
10:26:26 SILL Contact alias()
old_alias
10:26:26 SILL Contact alias(new_alias))
10:26:26 VERB PuppetPadchat contactAlias(wxid_, new_alias)
10:26:26 SILL PadchatRpc rpcCall(WXSetUserRemark, ["wxid_","new_alias"])
10:26:26 SILL PadchatRpc WXSetUserRemark result: {"message":"","status":0}
10:26:26 SILL Contact ready() @ Puppet#0<PuppetPadchat>(Ponyo)/PuppetPadchat#0
10:26:26 SILL Contact ready() isReady() true
10:26:29 SILL PadchatRpc onSocketTencent() discard empty message msg_id payoad: {"continue":0,"msg_type":32768,"status":1,"uin":488036821}
10:26:29 SILL PadchatRpc initHeartbeat() throttleQueue.subscribe(ws.on(message))
10:26:29 SILL PuppetPadchat startWatchdog() padchatManager.on(heartbeat)
10:26:29 SILL Wechaty memoryCheck() free: 151 MB, require: 4 MB
10:26:31 SILL Contact alias()
contact name Alias: old_alias
10:26:33 SILL Wechaty start() setInterval() this timer is to keep Wechaty running...
10:26:49 SILL PadchatRpc initHeartbeat() debounceQueue.subscribe(ws.on(message))
10:26:49 SILL PadchatRpc rpcCall(WXHeartBeat, [])
10:26:50 SILL PadchatRpc initHeartbeat() throttleQueue.subscribe(ws.on(message))
10:26:50 SILL PuppetPadchat startWatchdog() padchatManager.on(heartbeat)
10:26:50 SILL Wechaty memoryCheck() free: 152 MB, require: 4 MB
10:26:50 SILL PadchatRpc WXHeartBeat result: {"status":0,"message":"ok"}
10:26:50 SILL PadchatRpc initWebSocket() ws.on(pong)
10:26:51 SILL PadchatRpc onSocketTencent() discard empty message msg_id payoad: {"continue":0,"msg_type":32768,"status":1,"uin":488036821}

Good catch.
Please try v0.19.119 or above, it should be solved.

Another might be slimier issue/bug is if a new friends just being add.
after 5s
await friendship.contact().sync()
let fs = await friendship.contact().friend()
console.log(fs) //false

So it should force refrash contact for new add friends as well.

It's great that you pointed this out.
I suggest we should create another issue for this, and would you like to send a PR to fix this? because you already had the code above.

Please close this issue if it's done, thanks.

Yes, I just confirmed this issue has been fix.
Just also put a note
If bot's friends(some people) change alias during the bot is running, the bot's cache can not detect the contact change alias, but it will be detect when next restart wehchaty

What did you mean bot's friend change alias during the bot is running?
I do not think the alias can be changed other than the bot itself.

I am sorry for mistypo, what I mean is bot's friends change name


ENH: linalg.solve: detect and exploit matrix structure

summary
The solve function is basically one of the most used in the linalg module. Its counterparts in other packages and languages mostly enjoy varying degrees of automation via the so-called polyalgorithms. These basically try the simplest cases (is it diagonal, bidiagonal, tridiagonal, triangular ... so on) such that simple problems are handled much faster than the less structured problems.
The most well-known version of this is from matlab syntax A\b hence the name backslash. Julia, Octave, and many other languages have an attempt of this. And through mathworks the following logic is generally followed

scope
There are basically lots of structure to be tested and I am aiming to address most of them (enumerated below). The main mechanism of the algorithm is to discover structure as quick possible and branch off to the relevant solver. This is made possible via find_array_structure function written in Cython. If the entrywise checking didn't turn out anything we also check for symmetricity/hermitianness and if indeed there is structure we also test for positive/negative definiteness. If all fail then we fall back to the generic no structure solution.
With this PR linalg.solve will be able to automatically distinguish, solve and check for singularity/ill-condition cases for (marked are done)

[x] generic
[x] diagonal
[x] bidiagonal (lower, upper)
[x] triangular (lower, upper)
[x] tridiagonal
[ ] hessenberg (lower, upper)
[ ] symmetric/Hermitian
[ ] pos./neg definite

permuted triangular arrays are out-of-scope on purpose because there is no shuffled-triangular solvers available and reshuffling the arrays back triangular, unfortunately eats up all the benefits of this polyalgorithm and destroys the memory layout. Hence there is no point in that specialization for now.
performance
Main performance gains are through the custom solvers however they are not the only tie-breakers for small n when it comes to the complete performance measurements. Arguably the next most important detail is that now a array can survive without any copies if dtype  is suitable to LAPACK and contiguous since C/F layout is meticulously tracked and their transposes are used whenever possible. This also made it possible to have more possibilities of overwriting the original array. Due to the custom check_finite function the slowdowns of _asarray_validated passes are sped up too. Finally the get_lapack_funcs bottleneck is removed via completely separate (s-d-c-z) type-paths.
I will add the relevant plots here. But as a teaser, it is always faster than current version of scipy.linalg.solve for generic case even though doing lots of extra work. It can go much faster but losing most of the pace in the array regularization at the intake. Compared to numpy.linalg.solve I don't know why exactly but up-to about n=10 NumPy version is very fast with no checks whatsoever. Then the proposed version takes over both SciPy and NumPy versions.
Diagonal case

There is no point in comparing to the current SciPy master since NumPy pretty much smokes it in every case. In the fat B array case on the left, we are only paying the price of copying B into fortran contiguous array. On the right the columns of B is set to 10 and N varied. As you can see in both cases, we are pretty much an order of magnitude faster for large arrays. On very small arrays N=[3, 10] we don't pay any price even though we perform many more checks.
Bi/Tridiagonal case

Similarly, bidiagonal/tridiagonal difference is the same complexity difference hence still no surprise. In fact checking the condition number takes the same amount of time as the solver step (rcond part is not optimized yet). SciPy and NumPy has no special solver for these cases.
Triangular case
For triangular cases, already much faster than the generic solvers in both SciPy and NumPy, and slightly faster than linalg.solve_triangular. As n gets larger the difference grows.
More to come here....
early feedback points needed

[ ] Cython code quality: My Cython skills and typing knowledge is mediocre and some places I have been basically blindly imitating. I would be grateful if as many people as possible take a look for anti-patterns or flat-out stupidity.
[x] Cython templating possibilities; this is something actually I don't care too much. Currently, readability is harder but it is significantly easy to code with rolled out copies. Also makes the compilation bug hunting easier. --> Switched to tempita to stay sane since fused types are too experimental in Cython to be robustly used. In many simple cases I hit compiler crashes
[ ] ILP64 details: This is probably going to cost me a lot of rewrite but it is an accepted cost. I just wanted to see if this project can even fly. Then I would add the details on top.
[ ] Should we incorporate the rectangular case here too as matlab and others do? In other words instead of calling solve and lstsq depending on the matrix size internally this can be decided and branched off.
[ ] More items will be populated here.

backward incompatible details

The function no-longer checks whether the inputs are sparse arrays or not. This had the side effect of importing scipy.sparse through _lib._util._asarray_validated for no reason. Instead it will be discouraged through the documentation.

If everybody is OK with it, I would like to finally zap the disfunctional debug keyword (maybe sym_pos too? ).

TO DO

[ ] megatons of new tests while keeping the existing ones green.
[x] announce in the mailing list
[ ] Finish the solvers
[x] Finish the structure discovery
[ ] Test ill-conditioning behavior before/after
[ ] Get the dreaded transposed keyword handling working properly for complex arrays when array has C-layout
[ ] other tasks...

I would appreciate all the help and all kinds of feedback for this overhaul.
Code for generating various structured a, b pairs

import numpy as np
import scipy.linalg as la

def get_test_matrices(case, n, m=1, a_dtype=float, b_dtype=None):
    """Generate test matrices.

    Parameters
    ----------
    case : str
        which case to be tested;
        'diag', 'ubidiag', 'lbidiag', 'tridiag', 'uptri', 'lotri', 'uhess',
        'lhess', 'sym', 'her', 'pos'
    n : int
        number of rows of the problem
    m : int
        RHS b array number of columns

    Returns
    -------
    a : ndarray
        2D LHS nxn array with the given property defined by case
    b : ndarray
        2D RHS nxm array with random data

    """
    if b_dtype is None:
        b_dtype = a_dtype
    if case == 'diag':
        a = np.diag(np.random.rand(n)).astype(a_dtype)
    elif case == 'ubidiag':
        a = np.triu(np.tril(np.random.rand(n, n), 1)).astype(a_dtype)
        a += n/1.9 * np.eye(n, dtype=a_dtype)
    elif case == 'lbidiag':
        a = np.tril(np.triu(np.random.rand(n, n), -1)).astype(a_dtype)
        a += n/1.9 * np.eye(n, dtype=a_dtype)
    elif case == 'tridiag':
        a = np.tril(np.triu(np.random.rand(n, n), -1), 1).astype(a_dtype)
    elif case == 'uptri':
        a = np.triu(np.random.rand(n, n)).astype(a_dtype)
    elif case == 'lotri':
        a = np.tril(np.random.rand(n, n)).astype(a_dtype)
    elif case == 'uhess':
        a = np.triu(np.random.rand(n, n), -1).astype(a_dtype)
    elif case == 'lhess':
        a = np.tril(np.random.rand(n, n), 1).astype(a_dtype)
    elif case == 'sym':
        a = np.random.rand(n, n).astype(a_dtype)
        a = a + a.T
        if np.iscomplexobj(a):
            a += (1j*np.random.rand(n, n)).astype(a_dtype)
            a = a + a.T
    elif case == 'her':
        a = np.random.rand(n, n).astype(a_dtype)
        a = a + a.T
        if np.iscomplexobj(a):
            a += (1j*np.random.rand(n, n)).astype(a_dtype)
            a = a + a.T.conj()
    elif case == 'pos':
        a = np.random.rand(n, n).astype(a_dtype)
        a = a + a.T
        if np.iscomplexobj(a):
            a += (1j*np.random.rand(n, n)).astype(a_dtype)
            a = a + a.T.conj()
        a += n/1.9 * np.eye(n, dtype=a_dtype)
    elif case == 'gen':
        a = (np.random.rand(n, n).astype(a_dtype) +
             (n/1.9)*np.eye(n, dtype=a_dtype))

    b = np.random.rand(n, m).astype(b_dtype)

    return a, b

I don't get which part of [skip ci] do these webhooks not understand?

Should we incorporate the rectangular case here too as matlab and others do?

Yes. I think so.

C/F layout is meticulously tracked and their transposes are used whenever possible

This means that the issue like gh-10722 will not occur, right? (I guess this PR won't solve that issue because it won't change solve_triangular, but it will essentially make solve_triangular obsolete, right?)

This means that the issue like gh-10722 will not occur, right? (I guess this PR won't solve that issue because it won't change solve_triangular, but it will essentially make solve_triangular obsolete, right?)

Indeed, that's the idea. Maybe not obsolete but a bit laborious.

By "sparse arrays" do you mean whether it's a sparse matrix object, a np.array that is relatively sparse, or actually a sparse nd array? Can't this be checked some other way that doesn't import scipy.sparse?

I mean in this sense:
https://github.com/scipy/scipy/blob/master/scipy/_lib/_util.py#L252-L258
If there are easier ways to check sparseness I can put it back in.

If there are easier ways to check sparseness I can put it back in.

Not sure if it is fast enough, clean enough, or even doable in Cython but how about 'scipy.sparse.base.spmatrix' in repr(A.__class__.__mro__)?

This looks really good already! Cython looks pretty idiomatic to me.

I don't get which part of [skip ci] do these webhooks not understand?

Long-standing issue, only TravisCI has always understood [skip ci]. CircleCI didn't seem to have run here, so maybe it finally started to understand it as well.

Ilhan... If the diagram above is complete, and for non-square systems just QR is called, then this is probably the place to call my fast ill-conditioned test, and if it says the system is ill-conditioned (which is very likely correct) then call  arls() (or aregls, or whatever the final name is.  If you are interested I will promote my latest version of it. I have not been promoting incremental versions so as not to be a nuisance.
In contrast, linalg.lstsq seems like a complicated place to put it.... but that is based on brief examination of the code.
Ron Jones

Ah yes. I didn't mean to ignore I am having very little time to come back to this or any other algo-heavy stuff. And when I do, I am mostly fighting with Cython which turned out to be very laborious to develop anything with it. But finally got some progress. I can't make any promises but if I can stabilize the square case then yes let's try to sneak that in.
The API will be a hot mess for a few versions to accomodate the old and new behavior simultaneously for backwards compatibility though. So we have to think very hard for the least disruption in any case.

Cython code looks reasonable enough to me, I made some comments and suggestions above.  I think this would be a fantastic addition to solve capabilities and performance

@ilayn thought I'd try to help revive this by merging main: ilayn/scipy#8. I took care to preserve the changes to solve that have been made in the past few years. Looks like some basic tests are failing - but it was probably like that before. Looks like not all families are treated, and some keywords are ignored (e.g. transposed, which changes the system to be solved).

I'm wondering if it would it be easier for us if we close this one and start over again properly?
Now that I know much more about the internals of our codebase, I think I can send things to C and be much cleaner about the array handling. Cython makes things quite complicated when we are dealing with multiple dtypes and fused types are conflating the binary sizes, as far as I understand.
Once we get a proper API then the rest is pretty much mapping to <s,d,c,z> dtypes and then choosing the right algorithm.

TBH my motivation was to try a different approach, too. linalg.bandwidth and issymmetric/ishessian are very fast compared to solve for essentially all array sizes, even in the worst case, and those provide enough information to detect most of the structures we're looking for. I think we can speed up solve in all but the general case with just existing functions and without adding much overhead. Would you mind to if I were to try a pure-Python version of this work to establish a better baseline? Then we can see the actually benefit of any low-level improvements. If it's ok with you, I'd just use this PR for development, starting with my merge of main to preserve the history.

Here's a 40-line addition to _basic.py to demonstrate what I'm talking about:
https://github.com/scipy/scipy/compare/main...mdhaber:gh12824?expand=1
The premise is that we're trying to beat scipy.linalg.solve as it is now and that the only way we want to do that is by detecting and taking advantage of the structure of the matrix. I assume that other strategies for improvement (getting the correct underlying LAPACK routine for the dtypes, finiteness checks, dealing with row-major vs column-major ordering) are orthogonal.
Here are the results so far:

For other structures, we'd add new public functions like solve_tridiagonal and just call them. What do you think about getting the logic in place like that and making other improvements separately?

Would you mind to if I were to try a pure-Python version of this work to establish a better baseline?

Please go ahead, more than happy to have that.
The results look really good already so definitely a win.

For other structures, we'd add new public functions like solve_tridiagonal and just call them.

I was actually trying to go the other way by consolidating separate solve_xxxs under solve including the functionality of lstsq (without deprecating it) if the array is not square (basically the top branch). So that it becomes a one-stop-shop; you pass your array and things happen automatically. If you have a specific structure you can use assume_a=xxx and bypass all the checks (as it is currently doing in a limited fashion).
In my head this seems like a more coherent and less complicated API. What do you think?

I know it's much less ambitious, but there could still be big gains in performance for structure matrices, and it would take less time than this PR has : )

Definitely. Multiple things happened during this PR which is a Cython bug, stopped fused types operational, got a new job, then F77 crusade started etc. So it's not that I lost interest but got myself into bigger troubles so anything that moves this PR is much appreciated. I still have sqrtm, and logm overhauls waiting in the backlog.

In my head this seems like a more coherent and less complicated API. What do you think?

Yes, I thought about that, too. In principle I favor consolidated functions like that, although I know that sometimes the interface of the consolidated function can get quite a bit more complicated than any one of the individual functions would be. Here, for instance, if we want to fit a solve_tridiagonal into solve, would the user pass a tuple of three 1-D arrays for the first argument (instead of three separate arguments)? I suppose we can still respect transpose and lower, but they make a bit less sense as options in this case.
However, I'll start with that as the goal, and we'll see how it goes!

So it's not that I lost interest but got myself into bigger troubles

Yeah, I'll say! I think rewriting all Fortran code counts.
Ok, I'll try to pick just the low-hanging fruit here so as not to distract from other efforts!

if we want to fit a solve_tridiagonal into solve, would the user pass a tuple of three 1-D arrays for the first argument (instead of three separate arguments)?

I would say no because solve is for dense rectengular matrices. solve_banded is already there but I think it sticks out as a bit too specific and very thin wrapper over the lapack function does not really provide a lot of convenience. Probably, we should collect them under the same bag as passing a struct or dict or something similar to solve_special or solve_structured or whatever. I don't think bidiag, hess, or any other special case is generic enough to grant separate functions in my opinion, which starts cluttering the whole namespace. I'd rather recommend users how to use the lapack function themselves.

And do you know how to fix the CI errors like:
scipy/linalg/_basic.py:54: in _find_matrix_structure
    below, above = bandwidth(a)
        a          = array([[ 6.40059762e+16+1.20821073e+16j, -4.42911529e+15-1.83036083e+14j,
         2.20093984e+16+1.73855766e+16j, -1....8850e+15-1.55752295e+15j,  3.43482610e+15-2.82709392e+15j,
         7.71433977e+16-4.98039845e+15j]], dtype=complex192)
_cythonized_array_utils.pyx:151: in scipy.linalg._cythonized_array_utils.bandwidth
    ???
_cythonized_array_utils.pyx:160: in scipy.linalg._cythonized_array_utils.__pyx_fused_cpdef
    ???
E   TypeError: No matching signature found

On my Windows machine - and on Windows/macOS CI - we don't get this error.

On Windows complex192 and complex 256 map to complex128, on Linux they don't. You need to explicitly map input arrays to s, d, c, z. Another reason that I don't want to do this with Cython.

Here's what the timing looks like now:

The gap closed a bit because now this goes through the same input validation for all cases.
A few remaining questions from my side:

Re: condition number checks

Is this really needed for the diagonal case? It doesn't affect the accuracy of the solution
Is there a ??con routine for tridiagonal matrices?
It looks we don't offer a Python wrapper of ?trcon; just Cython.

I defined private functions for bandwidth/issymmetric/ishermitian for all types by falling back on a simple implementation if the public one fails. Can/should this be done by the public routines?

Is this really needed for the diagonal case? It doesn't affect the accuracy of the solution

It is a bit of a warning about the problem and not just the problem itself. It is a practice for all linear solve problems regardless of the situation. Ill-conditioned problems don't necessarily imply the solution is wrong but the problem is ill-conditioned. That's what the warning is supposed to warn about. Here is a matlab example

Sounds like not - let's add your algorithm in a separate PR.

OK. I need to dig into harddrives apparently anyways. I feel like I have to rewrite it.

It looks we don't offer a Python wrapper of ?trcon; just Cython

Yes and also if you do this at the python level, you are doing 2 extra f2py trips one to return from trcon and one to resend the result to trtrs. That's why low-level implementation would benefit from this greatly; just push the pointer from one tothe other.

why is lange so slow?

You are not taking the absolute values of the entries for the 1-norm. That would close the time difference.

Re: reciprocal condition number, OK, I will compute it for the diagonal case. I don't think it is currently done for solve_banded (like tridiagonal) or solve_triangular, so I hope it is OK to add use of ?trcon and ?gtcon (tridiagonal) to separate PRs.
And RE: lange got it. Yes, I planned to do the smart thing for diagonal and tri-diagonal arrays, and we can dig into the low-level improvements in a separate PR.

That's why low-level implementation would benefit from this greatly; just push the pointer from one tothe other.

Great. After this pure-Python PR establishes a baseline, we can see how much more improvement we can get by compiling.

Much better with custom lange equivalent.

For positiveness/negativeness it starts with

Check if the array is hermitian (complex) or symmetric (reals) then if it is only symmetric (complex case not hermitian but symmetric), send it to zsycon and zsysv.
If it is hermitian check the main diagonal for negative entries. If all positive send it to Cholesky potrf
If Cholesky passes then it is positive definite -> Send it pocon and then to the Cholesky solver. otherwise send it to hermitian solver sycon/hecon and and the corresponding solver.

For hessenberg and bidiagonals I have to refresh my memory. I need to implement the hessenberg solver from https://dl.acm.org/doi/abs/10.5555/866641 to make ..con, ..sv in one go. For bidiags, it is similar story.

Ok so for symmetric matrices, try Cholesky if it passes a necessary condition, if not, fall back to LDL.
Yeah I saw that Julia used that Hessenberg solver. That does not look like low-hanging fruit : )

try Cholesky if it passes some necessary conditions

No conditions, if info = 0 in portf it is pos def. That's surprisingly the cheapest check for positive definiteness (as far as I know).

No conditions

Right. No sufficient conditions (other than trying Cholesky). But having all positive diagonal entries is a necessary condition, right? (And being Hermitian if complex.)

Yes positive diagonals is a necessary condition for pos def. You can also check if all negative and compute the Cholesky of the negative etc. but that's probably for later. The diagram at the top describes the workflow or in matlab docs

You mean it is already tridiagonal?

I just mean that if we're accepting a full matrix rather than just the diagonals, there are few circumstances under which using a bidiagonal solver rather than a tridiagonal solver will make a significant difference in overall computation time. So personally, writing custom bidiagonal routines would be lower on my priority list than some other things.

Anyway, LMK if there is anything else you wanted here. This is the scope of what I had in mind for a first PR.

I just mean that if we're accepting a full matrix rather than just the diagonals, there are few circumstances under which using a bidiagonal solver rather than a tridiagonal solver will make a significant difference in overall computation time.

I agree. The main use for polyalgorithm is to work automagically regardless whether the user knows it or not. So if they pass a bidiagonal array the machine should recognize it. So we are increasing the structures that the solver can identify. Not necessarily providing separate solvers for each cases. The more the better.

Is the limited benefit of using triangular solve not surprising? (Is LAPACK LU just unreasonably fast?)

The main slow down is to call ??con and then calling ??sv algorithms. That is solving the problem twice. What has to happen  is first hit ??trf get its output to ??con and then send it to ??trs algorithms. ??sv is a higher level front end to this operation without doing ??con hence you are getting closer to generic solver speeds. That's probably what is being done inside solve_triangular.

In follow-ups, we can try Cholesky and fall back to LDL, add the missing reciprocal condition number checks, and then see how much compiling helps?

Sure this is already good progress.

So we are increasing the structures that the solver can identify. Not necessarily providing separate solvers for each cases.

Ok. Just to be explicit, this already identifies bidiagonal matrices as tridiagonal and calls the tridiagonal routines for them.

The main slow down is to call ??con and then calling ??sv algorithms. That is solving the problem twice.

But ??con is not currently being called at all for triangular matrices. _solve_triangular just calls ?trtrs.

But ??con is not currently being called at all for triangular matrices. _solve_triangular just calls ?trtrs.

Hmm then we need to do some surgery. Are we sure that solve_triangular is not doing any slow things?

Hmm then we need to do some surgery. Are we sure that solve_triangular is not doing any slow things?

For a triangular matrix with n=1000 solving with assume_a=None (~3.66 ms):

finiteness check: 7.3%
find matrix structure: 24%
lange: 52%
solve_triangular: 13.3%, 93% of which is trtrs

With assume_a='gen' (~21.1 ms):

finiteness check: 2.2%
lange: 16% (but about the same wall clock time as for triangular, since it's the same LAPACK routine)
getrf: 65.7%
gecon: 12.1%

This brings to mind that we could speed up the finiteness check, especially for the diagonal and tridiagonal cases, by checking only the elements that will be used in the calculation. However, this brings to mind gh-2116.

@ilayn Anything else to do here? Did you want someone else to take a look or does this look safe enough?

No I think the tests are comprehensive enough as far as I can see and we have a good indication that the performance is improved. Let's see if the nightlies cause any issue downstream. Thank you for pushing this over the line. Great stuff.


README refresh

Made a comment on a previous PR and saw that this README could use some copy editing and a few updates:

Updated headers to make it more scannable
Added syntax example and param table
Internal links where appropriate
General copy editing

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@NerdLogic @dmankins @mfelici
I was unable to add reviewers, so tagging you here.


ProducerMessage confusion

Versions
Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version:0.8
Kafka Version:2.10
Go Version:1.8.3
Configuration
What configuration values are you using for Sarama and Kafka?
In kafka part of my server.properties:
      broker.id=0
      num.partitions=10

Logs
When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.
Problem Description
In the type ProducerMessage, after I set the Partition field and the Offset field, I send a message to kafka,but when I consume the message,Partition field and the Offset field not take any effect,I don't know why?  I make sure the message was successfully delivered.But the two field haven't taken any effect.Could you tell me why and what purpose do you set these fields. Looking forward to hearing from you!
my produce code:
 msg := &sarama.ProducerMessage{}
 msg.Topic = "hello"
 msg.Partition = 10
 msg.Key = sarama.StringEncoder("key")
 msg.Value = sarama.ByteEncoder("hello,world")
 msg.Offset= 6

after I run the produce code ,the log show below:
[srama]2017/09/07 16:05:05 producer/broker/0 starting up
[srama]2017/09/07 16:05:05 producer/broker/0 state change to [open] on hello/7
[srama]2017/09/07 16:05:05 Connected to broker at SHA1000048829:9092 (registered as #0)
[srama]2017/09/07 16:05:05 partition=7, offset=2
[srama]2017/09/07 16:05:05 Producer shutting down.
[srama]2017/09/07 16:05:05 Closing Client
I set the Partition=10, Offset=6, but actually is partition=7, offset=2,they seem not to  take any effect.

You cannot set a specific offset; the offset is always assigned by the broker when produced.
You can set a specific partition, but to do so you need to configure your producer with Config.Producer.Partitioner = NewManualPartitioner.

Yeah, I think I got it. The Partition and Offset fields are set because message requires them ,and producer can initial them. Only after I set  config.Producer.Partitioner = sarama.NewManualPartitioner  the field Partition is work, but the field Offset couldn't change wherever  I set it or not.
Thanks a lot for you replying.