Update SecureBootAndBitLocker.md

Add a tip how to allow Microsoft Store apps by including the Microsoft MarketPlace PCA 2011 certificate.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows-iotcore/secure-your-device/SecureBootAndBitLocker.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


docs: move/expand "Keptn for Non-Kubernetes applications"

Originally, this PR moved the "Keptn for Non-Kubernetes applications" page from "Tutorials" to "Implementing" as well as expanding the information.  But the "expanding" work turned into a more complex exercise than anticipated so I am creating this PR to just move the content as is.  We can then.delete the "Tutorials" section with https://github.com/keptn/lifecycle-toolkit/pull/2088 with no problems while we work on expanding this information.
This PR:

and modifies the content a bit
Creates a reference page for KeptnTask in the yaml-crd-ref section.  Normally, KeptnTask is generated automatically but, in this case, the user must populate and maintain it manually so we need to document the YAML syntax.

I discovered what appear to be some serious disconnects between what is in this document and what is in the API reference.  So I added a bunch of TODO statements to the reference page and am putting it out for review in hopes that you guys can straighten me out and then I'll go back and finish this.  Thanks for your help.

https://github.com/keptn/lifecycle-toolkit/pull/2103 supercedes this PR .


fixed: UI BUG : Navigation button text overflows from parant span box…

…, issues - #6460

CLA signed done

Fixed here in a more robust way so it won't word break almost ever (and not at all today): https://github.com/reactjs/react.dev/pull/6519


Tailwind CSS support

It would be great to see Livewire+TailwindCSS support in cray, and also configurable views directory would be great plus.
Thanks for you work on this. 👏

In v3.2.0-beta2 when running php artisan cray Modelname, the user gets a prompt to choose the target for generating views if you have provided multiple view paths in config/view.php
As you can customize the provided stubs, you can use any UI kit. But for now we don't offer built-in support for Tailwind. May be in future we do add the option to choose a preset just like Laravel does.

Tailwind support will save lots of time.


CHROME only - Copying a shareable link wont stay highlighted long enough to share.

STR-
Bookmark three apps from Center
Go to HUD
Drag the two apps on top of the third app in order to create a folder with all three apps
Open folder
Get shareable link
Click on the url
RESULTS - The link will highlight for a second then unhighlight really fast so the user can never actually copy it.
EXPECTED RESULTS - When the user clicks on the URL it should highlight and stay highlighted.

It looks like it is working to me. Is it possible you double clicked the link?

never mind i was able to reproduce this issue, if you don't save the name of the folder first and try to go into it


ENH: The Fortran 77 implementation of COBYLA is buggy and challenging to maintain. Switch to the PRIMA implementation?

Is your feature request related to a problem? Please describe.
Dear SciPy maintainers,
This is Dr. Zaikun Zhang from the Hong Kong Polytechnic University. Together with Professor N.I.M. Gould, I am responsible for maintaining the derivative-free optimization solvers of the late Professor M.J.D. Powell.
Thank you for making COBYLA available in SciPy. I note that the current version is based on the original Fortran 77 implementation, which is not maintained anymore.
Although the Fortran 77 code is truly a masterpiece, it contains many bugs, most of which are due to the language itself. For example, see Section 4.4 of our recent paper and the GitHub issues / requests listed below (not all of them concern SciPy).

The Fortran 77 solvers may get stuck in infinite loops.

optimize: COBYLA hangs / infinite loop #8998

BUG: Scipy.optimize / COBYLA hangs on some CPUs #15527

COBYLA freezes (though maxeval and maxtime are given) #370

COBYLA hangs #118

NEWUOA_BOUND stuck in infinite loop inside MMA #117

Cobyla freezes in 0T1.16rc1 #1651

Optimization freezes #25

BOBYQA gets stuck in infinite loop. #7

Algorithm turns into infinite loop and never finishes #3

The Fortran 77 solvers may crash with segmentation faults
due to uninitialized variables that are used as indices.

Fix all uninitialized variable warnings #134

BOBYQA uninitialised variables in rare cases #133

Use of uninitialized variable in BOBYQA altmov #36

Fortran 77 COBYLA may not return the best point that is evaluated; sometimes, the returned point can have a
large constraint violation even though the starting point is feasible.

nlopt COBYLA optimizer gives unexpected output #182

Last Result Returned Not Optimized Result #110

COBYLA returns last evaluated function which might not be minimum #57

Successful termination when constraints violated #1

Describe the solution you'd like.
To avoid the problems originating from the Fortran 77 code, I suggest you use the PRIMA implementation of Powell's solvers. PRIMA provides the reference implementation for Powell's renowned derivative-free optimization methods, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA. The "P" in the name stands for Powell, and "RIMA" is an acronym for "Reference Implementation with Modernization and Amelioration".
PRIMA provides the modern implementation of the solvers in Fortran 2008. It fixes bugs in the original Fortran 77 code. In addition, it introduces improvements that boost the performance in terms of the number of function evaluations, which is the standard measure of computational costs in derivative-free optimization.
See the GitHub repo of PRIMA for more information. I will be glad to provide assistance if help is needed.
Thanks.
Describe alternatives you've considered.
No response
Additional context (e.g. screenshots, GIFs)

Sadly - I believe we would need someone to change the license, or release it under another license - it's currently under LGPL 3.
This is very topical for us - do you have any comment on this mailing list thread on Powell optimization?

Hi Matthew @matthew-brett ,
Which license is preferred? I will consider releasing it under another license if that is really needed.
I read the discussions on the mailing list thread on Powell optimization.
As a coauthor of PDFO (the predecessor of PRIMA), I am very glad to see that our version of Powell's solvers succeeded in solving this problem, both in the classical mode (highly discouraged) and in the default mode. (BTW, it is PDFO rather than PGFO).
According to my experiences with Powell's code in the past 15 years (especially the past three working on PRIMA), I do not think it is trivial to find out what is wrong with the scipy.optimize.fmin_pow or SciPy version of COBYLA when solving this particular problem. It is simply too hard to debug Powell's Fortran 77 code. That is why I spent three years of  my life working on PRIMA.
Professor Powell was a great mathematician and a pioneer (I was very close to him). His code was great, but it should not be used anymore unless we want to be bitten by bugs.
Thanks.

I'm afraid we absolutely do need a non-viral license - such as MIT or BSD.   (Does anyone have a good link for our licensing requirements?).
Thanks for your comment on the thread - note that we aren't using Powell's code in Scipy, for the "Powell" optimizer, but an adaptation apparently due to to Acton, and that appears in Numerical Recipes.

note that we aren't using Powell's code in Scipy,
I note that the COBYLA optimizer of scipy.optimize is based on Powell's Fortran 77 code:

https://github.com/scipy/scipy/blob/main/scipy/optimize/cobyla/cobyla2.f
It is the main motivation for me to raise the current issue. It is so [widely used](https://www.google.com/search?q=scipy+cobyla --- without exaggerating, it is one of the most widely used optimization solvers in today's science and engineering, but it is so buggy.

for the "Powell" optimizer, but an adaptation apparently due to to Acton, and that appears in Numerical Recipes.

Sure, I know this algorithm. It is called Powell's conjugate direction method. Published in 1964, it was Professor Powell's second published work on optimization, and also the second most cited work  (the first and meanwhile most cited is his paper on the DFP algorithm, co-authored with late Professor Roger Fletcher FRS). See historical remarks in Section 3 and footnote 6 of our paper on PDFO.
BTW, I note that the comment in scipy/scipy/optimize/__init__.py  is incorrect: it says fmin_powell - Powell's (modified) level set method, but there is no such thing as "Powell's (modified) level set method". As mentioned, it is indeed "Powell's (modified) conjugate direction method". I will raise an issue / pull request regarding it.
The word "level set" probably came from a miscopying / misreading / misunderstanding of the word "Direction Set (Powell’s) Methods" in Numerical Recipies. Here, "Direction Set" refers to the set of conjugate directions generated by the algorithm. However, Powell's conjugate direction method is a much more used name --- I have never heard anybody from the optimization community calling it a "Direction Set" method.

Thank you - it would be very helpful indeed to have your expertise in setting that straight.
I don't personally maintain the optimizers module, but it would be very good to have someone who has worked a lot on these methods maintaining them, or replacing them if necessary, in Scipy.

Thank you - it would be very helpful indeed to have your expertise in setting that straight.
I don't personally maintain the optimizers module, but it would be very good to have someone who has worked a lot on these methods maintaining them, or replacing them if necessary, in Scipy.

Sure, it is a pleasure to contribute. I do not really know Python, but I will be glad to help from the optimization and (modern) Fortran side.

@matthew-brett here are our license notes: https://scipy.github.io/devdocs/dev/core-dev/index.html#licensing and this https://scipy.github.io/devdocs/dev/hacking.html#license-considerations
Note that relicensing can be challenging. If you are a single author it can be straightforward, otherwise you need explicit permission from all contributors. And that would only work if the code you wrote in the first place could be license under the new terms. e.g. if you took some code from another GPL code or a snippet from StackOverflow, then you cannot relicense to MIT/BSD like that.

Sure, it is a pleasure to contribute. I do not really know Python, but I will be glad to help from the optimization and (modern) Fortran side.

@zaikunzhang thank you! That would be great to have you on-board.
We are discussing more and more about ways to move away from our old Fortran code. LFortran folks are trying to get SciPy compiling there and we are even thinking about some tools to translate the code into something else like C++ or even just Python for some things.
That being said, I don't think people would mind if you would propose to replace our F77 with a version of your code in modern Fortran. On the contrary 😃

There are two contributors on PRIMA so the relicensing should not be a problem if the authors are in agreement.

I think it needs slightly more as in the description there are statements about other codes like PDFO which is also LGPL. But yes also a few contributors.
Maybe another option, would that work if we just use the binary as a dependency?

Depending on a binary would be a problem would it not?  Won't it mean we'd either have to make everything optional, or provide binaries for every platform?

I have no clue here. If this is something we really want to consider, I suggest this discussion be moved to the mailing list.

Well - if @zaikunzhang and collaborators can and will consider moving to a compatible license - then we don't have the binary question.   So - can I ask - is that possible?  That you can release under a compatible license?  If so, my guess is that the mailing list, like us, would be delighted to include the code.

Note that they can retain their license if they want. This is what we do with UNU.RAN for instance. UNU.RAN is still GPL, but they explicitly allowed us to use some part of it under our license terms. The annoyance is that when they make a change in their code, they need to explicitly allow us to use the change.

We did this relicensing before in Scipy and also . So it's not a problem. Nick Higham and SuitSparse already did this with Python and Julia users too. So let's not confuse ourselves. That's the easiest of the complications.

Oh I am not saying it's impossible, just pointing out things 😃 I am more than happy if that happens here and we can remove old code.

The other contributior, Tom Ragonneau @ragnneau was formerly my Ph.D. student and now a colleague. He is a contributor because PRIMA contains some code from the above-mentioned PDFO package, which was developed during his Ph.D. under LGPL3 and lays the groundwork that made PRIMA possible.
That said, it is not very difficult to get permission from Tom to change the license of PRIMA to, e.g., the BSD 4-clause license.
Indeed, Tom himself seems to prefer BSD to LGPL. For instance, another derivative-free optimization package COBYQA by us is licensed under BSD, which was chosen by Tom.
For PDFO and PRIMA, we currently use LGPL because Professor Powell had chosen LGPL for the original Fortran 77 code of his  solvers before he passed. I am not an expert on law/licenses, but does this prohibate us from licensing PRIMA under BSD? PRIMA is a derived work of Powell's code.
Thanks.

Ouch. If it's a derivative indeed that is a problem but I think @rkern is more knowledgeable in this regard

Ouch. If it's a derivative indeed that is a problem but I think @rkern is more knowledgeable in this regard

In fact I do not know the exact definition of "derived". I mean, PRIMA was written according to Powell's Fortran 77 code, even thoug it does not contain his code anymore.

I would guess that you can ask Powell's heirs if they would allow the code to be relicensed.   I am sure we could help you write up something that was clear enough for them to decide on:
https://copyrightalliance.org/what-happens-to-your-copyrights-after-you-die/

@zaikunzhang thank you! That would be great to have you on-board.

Thank you @tupui ! It will be my pleasure to contribute. It is also my mission and commitment to maintain Professor Powell's solvers, which are too precious to be lost. I had promised to Professor Powell before he passed.

We are discussing more and more about ways to move away from our old Fortran code. LFortran folks are trying to get SciPy compiling there and we are even thinking about some tools to translate the code into something else like C++ or even just Python for some things.
That being said, I don't think people would mind if you would propose to replace our F77 with a version of your code in modern Fortran. On the contrary smiley (especially if you manage to do the change with preserving the user facing API.)

I know LFortran and the main contributors, even though not personally only through Fortran Discourse. I highly appreciate what they are doing and pushing forward. We do not have collaboration but we are going in the same direction concerning the maintenance and modernization of existing Fortran code, although I focus only on Professor Powell's solvers.

I would guess that you can ask Powell's heirs if they would allow the code to be relicensed.

This is possible, as I know them personally. However, I hope not to disturb them because of this issue. Let's say this is the last resort.
The LGPL license was suggested by Professor Nick Gould to Professor Powell. Nick is also a maintainer of Professor Powell's Fortran 77 code, although he is not involved in PDFO or PRIMA. I would also need to consult him if relicensing Professor Powll's code becomes a real need.
Indeed, exactly the same licensing problem exists in the current version of SciPy because of the COBYLA code contained in scipy.optimize. This is obviously a (slightly) modified version of Professor Powell's Fortran 77 code. If LGPL is enforced, there may be a problem. Again, I am not an expert on law/licenses, and I am not a heir of Professor Powell. But since Professor Powell asked me to maintain these solvers and informed me about the license of his choise, I feel obliged to point this out.

I'm sorry for the inconvenience, but my guess is that changing the license will be necessary, to include the code in Scipy.  If I was arguing to Nick Gould, or Professor Powell's heirs, I think I would make the case that Scipy has become the default base library for numerical computing worldwide, and therefore, that his code can have a great deal more impact in terms of good for the world, if they can be included in Scipy, than if they cannot.
For the COBYLA code - ouch - and thanks for pointing that out.  We will have to resolve that.

Yes, looking at how you made the modern Fortran from the Fortran 77, I think it would be considered a derivative work, and the LGPL would still apply. At least for our purposes.
However, we did ask for and receive permission from Professor Powell to include our version of COBYLA under our BSD license. So at least for COBYLA, you could probably double-check that our version is close enough to the version you started from, then it would be clean as far as we're concerned. We should update our license information to include this fact.
For the others, I actually suspect that the copyright ownership of the originals may actually reside with his employer, the University of Cambridge. The copyright statement includes both his name and the University for some reason. It is commonly the case that universities claim ownership of the IP of their employees as part of their employment agreements. I would probably contact their Technology Transfer office first and see if they make any claim over the ownership. If not, then yes, it would be his heirs.

Hi @rkern , thank you for the professional comments.

Yes, looking at how you made the modern Fortran from the Fortran 77, I think it would be considered a derivative work, and the LGPL would still apply. At least for our purposes.

OK. Then it is clear.

However, we did ask for and receive permission from Professor Powell to include our version of COBYLA under our BSD license. So at least for COBYLA, you could probably double-check that our version is close enough to the version you started from, then it would be clean as far as we're concerned. We should update our license information to include this fact.

This is very fortunate! I note that the conversation took place 19 years ago, in 2004. At that time Professor Powell was developing his powerful and renowned NEWUOA solver for unconstrained optimization without derivatives. It is a pity that the inclusion of NEWUOA was not able to be permitted.
Yes, I confirm that the modernized implementation of COBYLA is based on a Fortran-77 version that is essentially the same as the one included in scipy.optimize.

For the others, I actually suspect that the copyright ownership of the originals may actually reside with his employer, the University of Cambridge. The copyright statement includes both his name and the University for some reason. It is commonly the case that universities claim ownership of the IP of their employees as part of their employment agreements. I would probably contact their Technology Transfer office first and see if they make any claim over the ownership. If not, then yes, it would be his heirs.

If the university gets involved, I do not think it would be a trivial discussion. So it turns out much more complicated ...

For what it's worth, the license has:
Copyright (C) 1992 M. J. D. Powell (University of Cambridge)
...
Michael J. D. Powell<EMAIL_ADDRESS>University of Cambridge
Cambridge, UK.

So, I think Powell is just saying that's where he works, not that he's asserting in the license that the University of Cambridge has copyright.     In the linked email thread about COBYLA (thanks!) Jean-Sebastien Roy says he would ask Powell for permission to relicense on the 10 April 2004, and reports back the Powell said yes on the 12 April - I doubt whether Powell had had time to consult and heard back from whatever the technology transfer office was back then, so I take that as more evidence that Powell believed he had sole copyright.  Of course Cambridge may disagree.

My guess is that, if you could persuade Powell's heirs, and I guess, Nick Gould, that Powell would have agreed, and that he would have been right to agree, you could then go to Cambridge with that information.  They'd have to be pretty aggressive to stop you at that point - especially as it is already under a free license, and Powell has already quickly said yes to a similar request.  I can see if I can find anyone who would know the best way to contact the Cambridge people for this, if you would consider going down that route.

For the purposes of this issue about COBYLA, I'm happy with our provenance. If @zaikunzhang and @ragonneau agree with relicensing their modernization contributions to COBYLA to the BSD license that Professor Powell agreed to, then I, at least, am happy to accept them.
For the other algorithms, we can broach the topic of relicensing when someone actually wants to step forward to do the work of making the Python wrappers for them. Right now, no one has.

My guess is that, if you could persuade Powell's heirs, and I guess, Nick Gould, that Powell would have agreed, and that he would have been right to agree, you could then go to Cambridge with that information. They'd have to be pretty aggressive to stop you at that point - especially as it is already under a free license, and Powell has already quickly said yes to a similar request. I can see if I can find anyone who would know the best way to contact the Cambridge people for this, if you would consider going down that route.

Thank you for the suggestion. I will contact Nick about the licensing issue. He is a world-leading British numerical analyst, and hence I suppose he has good connections with British universities including Cambridge. We may also need your help eventually.

For the purposes of this issue about COBYLA, I'm happy with our provenance. If @zaikunzhang and @ragonneau agree with relicensing their modernization contributions to COBYLA to the BSD license that Professor Powell agreed to, then I, at least, am happy to accept them.
For the other algorithms, we can broach the topic of relicensing when someone actually wants to step forward to do the work of making the Python wrappers for them. Right now, no one has.

This is great! I will ask Tom @ragonneau and I presume he will agree to license the COBYLA code under BSD.
In addition, Tom knows Python quite well. He coded the Python part of PDFO and the whole of COBYQA. So I guess he may help with the Python wrapper.

Well @zaikunzhang officially welcome to the funky world of open source 😅

@ilayn and everyone, thank you!
I have prepared a copy of PRIMA COBYLA licensed under the 4-clause BSD.
I confirm the following.
As of commit 0801c02, the repository at https://github.com/libprima/prima_for_scipy contains only code by myself (Zaikun ZHANG) alone. Part of the code is written based on Powell's Fortran-77 version of COBYLA, which is essentially the same as the one included in SciPy at
https://github.com/scipy/scipy/blob/main/scipy/optimize/cobyla/cobyla2.f
In particular, it does not contain code from PDFO.
Thanks.

Thank you! Unfortunately the 4-clause BSD variant is not one that we can use. The advertising clause is incompatible with other open source licenses that we want to allow people to combine scipy with. We use a 3-clause BSD license for scipy itself, and that would be the easiest variant to use. We will make sure to have an explicit entry in the bundled licenses document that will cite and link to the PRIMA project.

Thank you! Unfortunately the 4-clause BSD variant is not one that we can use. The advertising clause is incompatible with other open source licenses that we want to allow people to combine scipy with. We use a 3-clause BSD license for scipy itself, and that would be the easiest variant to use. We will make sure to have an explicit entry in the bundled licenses document that will cite and link to the PRIMA project.

OK. I see. I have changed it to the 3-clause BSD. Thank you @rkern for the quick comment.

Thank you very much! I appreciate your consideration. I know that I would rather be finding new ways to provoke bugs in Fortran compilers than dealing with licensing minutiae, and that's saying something.

I have got a response from Nick. His opinion is summarized as follows. If there is a need, I can show the email in private.

It was Professor Powell's intention to make his software distributed freely. He chose a license to make the distribution easy, rather than the opposite.

Nick suggested LGPL (v2.1 at that time) to Professor Powell because it was the license that Nick was familiar with and uses for his own code. Now Nick has shifted to the 3-clause BSD. The icense was beyond Professor Powell's care due to his health condition, so he adopted directly what was suggested.

According to email exchanges between Nick and Professor Powell, LGPL was attached to Powell's code on April 16, 2015. On the other hand, Profeessor Powell sent me all the code on or before April 6, 2015, asking me to maintain and distribute it without mentioning a license.

For the above reasons, Nick suggests that I can legitimately change the license from LGPL to 3-clause BSD.

Professor Powell mentioned "Cambridge" in the license as an address instead of an attribution of ownership.

If there is any concern from the SciPy maintainer, I may contact Professor Powell's heirs (but it seems unnecessary).
The standard emails that Professor Powell used to distribute his code can be found below. He sent me the code using the same emails.

COBYLA
UOBYQA
NEWUOA
BOBYQA
LINCOA

Thanks.

Just to clarify - each of the files you mention was sent to you by Professor Powell, for the purpose of redistribution, before the LGPL was attached to it?
Each of those files has the following sentence:

There are no restrictions on or charges for its use.

and no other restrictions.   I think these are therefore public domain dedications, along with the assertion of sole copyright, and my opinion would be that these are therefore safe to release under the BSD 3-clause license as they are, given the information you've given.

Just to clarify - each of the files you mention was sent to you by Professor Powell, for the purpose of redistribution, before the LGPL was attached to it?

Yes, I confirm this is true. I still have all the emails saved in my mailbox. The last email, which distributed LINCOA to me was received on December 15, 2013. He asked me to maintain and distribute all his solvers on his behalf on April 6, 2015. Both before April 16, when he adopted LGPL for his code.

Thanks for providing this @zaikunzhang !
Besides bugs in COBYLA, a wrapper for BOBYQA has been on the wish list for many years (see #1477).
Maybe I missed parts of the discussion but would it be easier to use the python wrappers already available in pdfo ? That repo is also BSD-3 licensed.

Hi @dschmitz89 ! Thank you for your comments.
Yes, PDFO provides Python wrappers for all the five derivative-free optimization solvers of Professor Powell. They were written by Tom @ragonneau for Chapter 3 of his Ph.D. thesis.
However, the wrappers were written for the Fortran 77 implementation of the solvers. The PRIMA interpolation adopts a modernized signature, using features like assumed-shape arrays. Therefore, the PDFO wrappers cannot be used directly. However, it would be helpful to use them as a reference.

That repo is also BSD-3 licensed.

FYI, PDFO just changed licenses from LGPL to BSD last night, mere hours before your comment, because of this discussion. Thanks, @ragonneau!
Because of the call signature changes, it's not completely trivial to drop in the PRIMA implementations into PDFO, but PDFO does have a lot of the Python-side implementation of the fmin_*-style interface (e.g. dealing with the various kinds of bounds). Probably a good starting point.

That repo is also BSD-3 licensed.

FYI, PDFO just changed licenses from LGPL to BSD last night, mere hours before your comment, because of this discussion. Thanks, @ragonneau!

Right. I changed the license from LGPL to BSD because of this discussion, after seeking agreement from Professor Nick Gould and Tom @ragonneau.

Because of the call signature changes, it's not completely trivial to drop in the PRIMA implementations into PDFO, but PDFO does have a lot of the Python-side implementation of the fmin_*-style interface (e.g. dealing with the various kinds of bounds). Probably a good starting point.

Exactly.
BTW, three years ago, when Tom and I initiated the PDFO project, I did not post here to suggest the inclusion of Professor Powell's solvers in SciPy because the solvers were still in Fortran 77. Since  [PDFO] has gained decent popularity, with 50,000+ downloads in total (mirror downloads excluded) and recently ~100 downloads per day. The popularity demonstrates that there is a true demand for these solvers from Python users, which is also manifested by the popularity of (COBYLA under SciPy)[https://www.google.com/search?q="cobyla"+"scipy"]. Since the previous request for Powell's solvers was not met only because of the license, the last obstacle has been removed.

Update from the community call: there is interest in adopting this implementation, but little appetite for taking on more Fortran code (because we're already working to get rid of it in several other components). A feasible route forward could be to translate the Fortran implementation to C, using e.g. f2c, and to use that. Or perhaps there is interest in doing an official Cython / Pythran / C / C++ port?

Hi @stefanv . Thank you and the community for the update.
I personally understand the lack of appetite for more Fortran code. Making Professor Powell's solvers easily available to non-Fortran users is the very reason for me to work on PRIMA.
However, f2c is not a feasible solution here, as it is incapable of handling modern Fortran.
Official C++ and Python implementations are being planned, but they will not be delivered in the near future.
That said, I guess the most probable and practical solution is to wrap the modern Fortran implementation of PRIMA using iso_c_bindings + ctypes or similar facilities.
Indeed, there have been some intensive discussion on the inclusion of PRIMA into SciPy in the past few days. See
Hacker News (front page stories):
https://news.ycombinator.com/item?id=35959991 (Front-page Story of 16 May)
https://news.ycombinator.com/item?id=35986906  (Front-page Story of 18 May, right now)
Fortran Discourse:
https://fortran-lang.discourse.group/t/scipy-there-is-interest-in-adopting-this-prima-implementation-but-little-appetite-for-taking-on-more-fortran-code/5821
https://fortran-lang.discourse.group/t/optimization-without-using-derivatives-the-prima-package-its-fortran-implementation-and-its-inclusion-in-scipy/5798

FWIW, the root of much of our woes with the other Fortran code has been that we were effectively the maintainers of that Fortran code. Here, we have an active maintainer of the Fortran code keeping it working well on multiple compilers. That might tip the scales some. Swapping in the improved COBYLA Fortran code for the existing COBYLA Fortran code may be pure profit from a maintenance standpoint (though someone does have to rewrite the wrapper). Adding NEWUOA, BOBYQA, LINCOA, and COBYQA is another matter, but this is somewhat different than our issues with other Fortran code because of the active maintenance.

That said, I guess the most probable and practical solution is to wrap the modern Fortran implementation of PRIMA using iso_c_bindings + ctypes or similar facilities.

The issue isn't the wrapping, but the Fortran of the implementation itself.

FWIW, the root of much of our woes with the other Fortran code has been that we were effectively the maintainers of that Fortran code. Here, we have an active maintainer of the Fortran code keeping it working well on multiple compilers.

Excellent point.   That seems like a big win - better code and a maintainer.

I'd planned to attend the community meeting but missed it due to being wiped out from the flu doing the rounds in my household. I'm +1 for swapping out the existing COBYLA code for this modern implementation. Looking at some of the code it looks very clean and well documented. It's also actively maintained and it's maintenance appears to be well-funded
PRIMA is a long-term project, which would not have been sustainable without the continued funds from the Hong Kong Research Grants Council (ref. PolyU 253012/17P, PolyU 153054/20P, and PolyU 153066/21P) and The Hong Kong Polytechnic University (PolyU), in particular the Department of Applied Mathematics (AMA).
The original Fortran 77 implementation on the other hand is a tangled mess and @zaikunzhang seems to have gone to considerable lengths to untangle it. I agree with @rkern, that after a new wrapper is written, this should only lower our maintenance burden.

@ilayn - if the option was between well-maintained Fortran code and well-maintained C or C++ code, I'm sure we'd all go for the latter - but here the option is between poorly maintained buggy old Fortran code, and well-maintained more correct modern Fortran code.   It's perfectly reasonable to plan to replace the Fortran code over the long hall, but for now, and the medium term, this seems like a much better option than what we have, and what we can reasonably hope to have in the near future.

Fully agree, however this doesn't change the fact that a decade later, someone will be looking at us and shaking their head about the modern fortran code while they are trying to remove fortran code. I am confident that prima is much better than our cobyla code. No question about that. However, I'd like to kindly ask you to think about the time when cobyla was being integrated into SciPy.

I agree that swapping the old for the new COBYLA implementation is a clear win, and if there's a PR we should just merge it. That doesn't block a future translation to Cython/C/C++ if that would be desired, in fact it'll make it easier.

Yes, that goes without saying indeed let's do that. But also let's try to figure out if there is a readable and easily translatable version of the numerical code. Thanks to NumPy we really don't lose any performance if we code it carefully.

Indeed, there has been some intensive discussion on the inclusion of PRIMA into SciPy in the past few days. I put them below for your reference. You may find therein useful information and viewpoints on modern Fortran.

Thanks for sharing @zaikunzhang, nice discussions. I think folks here do agree modern Fortran can be quite nice, and LFortran looks very promising and it'll be a boon for SciPy when it matures. Our hesitation is not about any of that, nor about the quality of the PRIMA optimizers.

FWIW, I'm  +0.7 for replacing the messy original Fortran source with a streamlined version with an active maintainer. Pros:

makes a possible future translation to python/cython/whatever easier
shifts the priorities for the translation effort. If COBYLA stops being a spaghetti mess, Ilhan's (and whoever joins his effort) bandwidth can be spent on more legacy parts of our Fortran load.

The remaining 0.3 is basically twofold:

do we want to add a submodule or vendor the code (and sync periodically)? Maybe the former.
is there a clear picture of how the python bindings are going to look with the new code? (e.g. we have plenty of experience with f2py, even if this experience is often painful; do we have any experience with iso_c_bindings in real world? etc)

And just to make it very obvious, @zaikunzhang you have been doing a thankless job in the last three years, and I am the last person to say anything negative for this effort, especially while I have been wandering in the F77 code. I can't imagine the issues you encountered with original code. Wholeheartedly congratulations.
Side note:
I enjoyed the grilling we are getting from the HN and Fortran folks by the way 😃 I hope my comments don't come as rude to anyone reading these.

@ilayn Yes, I think there is a significant appetite (from @zaikunzhang, no less) to have the algorithms translated into another language. Powell's legacy is his algorithms, not his FORTRAN implementations. I wonder if it would make sense to go straight to COBYQA first in that case.

All good from my side then, I don't want to steer the discussion to fortran lamenting/praising further. So if PRIMA PR is in the works, that would be great and much appreciated. If there is any translation effort needed, then please ping me.

shifts the priorities for the translation effort. If COBYLA stops being a spaghetti mess, Ilhan's (and whoever joins his effort) bandwidth can be spent on more legacy parts of our Fortran load.

+1

@zaikunzhang it looks like you got it preapproved, so go ahead and submit a PR!
I agree with the comments here that using two languages in the same code base is bad, because then you need expertise in both, and the interface has to be maintained and building everything and shipping on all platforms is much harder, etc. I think since SciPy is in Python, all the code should be in Python and NumPy (API). But it still needs to run as fast as the Fortran code. We are working on a solution for that using our Python frontend called LPython: https://github.com/lcompilers/lpython, which compiles regular (subset) of Python/NumPy and is as fast as LFortran (it shares the same intermediate representation, optimizations and backends), but it's not ready for production use in SciPy yet. Once it is ready, I think it should be possible to translate automatically using LFortran to Python that LPython can compile. This is long term obviously, but I think that's the way to go. You can browse what LPython can already compile here: https://github.com/lcompilers/lpython/tree/d731c7a2760a02d0f876212508983ce054ba57e9/integration_tests, all those tests run with CPython as well (so if you don't need speed, you don't need to compile at all and you can use your existing Python tooling), but when compiled with LPython they run as fast as Fortran, no compromise. There are 22 other Python compilers that I know about, we have a list at https://lpython.org/, if I missed any, please let me know. As these Python compilers mature, there might even be a way to agree on some conventions for "fast subset of Python", so that one can use more than just one compiler. I think it's a little too early for that, as all the compilers are still experimenting with the best design. SciPy already uses Pythran, so you have experience with this approach. Let me know how this idea of just compiling a subset of Python works in practice.
@ilayn you will be happy to know that just today we spent several hours designing how to compile common blocks properly with LFortran, I must say I always learn some new surprising things when implementing these old features and I don't recommend to use them. But SciPy uses them, so we have to implement it. :) I have your Fortran TODO list on my mind, thanks again for writing it. Modern Fortran on the other hand is quite a nice language. Today I think is the first time that any of my Fortran codes made it to the front page of Hacker News: https://news.ycombinator.com/item?id=35973653, and it didn't receive any negative comments about Fortran, so I am just very happy about it.
I want to thank the whole SciPy community for this discussion and for being open minded about considering PRIMA. The Fortran community is small, but growing, and excited about this possibility: https://fortran-lang.discourse.group/t/optimization-without-using-derivatives-the-prima-package-its-fortran-implementation-and-its-inclusion-in-scipy/5798. And I am hoping that it will join the maintenance of modern Fortran code in SciPy. At the very least you can always ask a question or report a bug at the Fortran forum and you will get help, which is a progress.

Hello everybody,
Would the team consider seriously integrating the work from @zaikunzhang it might be worth it to consider using the iso_c_binding+ctypes approach. This would minimize the headaches given by f2py imposing heavy restrictions of the Fortran side of things.
You can find a micro test on how this works here : https://fortran-lang.discourse.group/t/optimization-without-using-derivatives-the-prima-package-its-fortran-implementation-and-its-inclusion-in-scipy/5798/19?u=hkvzjal
Regarding the current architecture of @zaikunzhang package, one way of achieving the interface would be to include a module wrapper in his library that could look like (based on his original high-level interface of COBYLA):
  module prima_wrapper
    use iso_c_binding
    use prima
    implicit none

    ! Define the integer kind to be used in the Fortran code.
#if PRIMA_INTEGER_KIND  == 0
    integer, parameter :: CIK = c_int32_t ! IK_DFT
#elif PRIMA_INTEGER_KIND == 16
    integer, parameter :: CIK = c_int16_t
#elif PRIMA_INTEGER_KIND == 32
    integer, parameter :: CIK = c_int32_t
#elif PRIMA_INTEGER_KIND == 64
    integer, parameter :: CIK = c_int64_t
#else
    integer, parameter :: CIK = c_int32_t ! IK_DFT
#endif
! Define the real kind to be used in the Fortran code.
#if PRIMA_real_PRECISION == 0
    integer, parameter :: CRP = c_double
#elif PRIMA_real_PRECISION == 32
    integer, parameter :: CRP = c_float
#elif PRIMA_real_PRECISION == 64
    integer, parameter :: CRP = c_double  ! real64
#elif PRIMA_real_PRECISION == 128
    integer, parameter :: CRP = c_float128
#else
    integer, parameter :: CRP = c_double  ! real64
#endif

    ! C-interoperable interface for the callback
    abstract interface
        subroutine c_objcon_32( x, f, constr ) bind(C)
            use :: iso_c_binding, only: c_float
            real(c_float), intent(in)  :: x(:)
            real(c_float), intent(out) :: f
            real(c_float), intent(out) :: constr(:)
        end subroutine c_objcon_32
        subroutine c_objcon_64( x, f, constr ) bind(C)
            use :: iso_c_binding, only: c_double
            real(c_double), intent(in)  :: x(:)
            real(c_double), intent(out) :: f
            real(c_double), intent(out) :: constr(:)
        end subroutine c_objcon_64
        subroutine c_objcon_128( x, f, constr ) bind(C)
            use :: iso_c_binding, only: c_float128
            real(c_float128), intent(in)  :: x(:)
            real(c_float128), intent(out) :: f
            real(c_float128), intent(out) :: constr(:)
        end subroutine c_objcon_128
    end interface

    contains

    subroutine c_cobyla(calcfc, m, x, f, cstrv, constr, f0, constr0, nf, rhobeg, rhoend, &
                      & ftarget, ctol, cweight, maxfun, iprint, eta1, eta2, gamma1, &
                      & gamma2, xhist, fhist, chist, conhist, maxhist, maxfilt, &
                      & info) bind(c)
#if __INTEL_COMPILER
        !DEC$ ATTRIBUTES DLLEXPORT :: c_cobyla
#elif __GNUC__
        !GCC$ ATTRIBUTES DLLEXPORT :: c_cobyla
#endif
#if PRIMA_real_PRECISION == 0
    procedure(c_objcon_64) :: calcfc
#elif PRIMA_real_PRECISION == 32
    procedure(c_objcon_32) :: calcfc
#elif PRIMA_real_PRECISION == 64
    procedure(c_objcon_64) :: calcfc
#elif PRIMA_real_PRECISION == 128
    procedure(c_objcon_128) :: calcfc
#else
    procedure(c_objcon_64) :: calcfc
#endif
    integer(CIK), intent(in) :: m
    real(CRP), intent(inout) :: x(:)
    real(CRP), intent(out) :: f
    real(CRP), intent(out), optional :: cstrv
    real(CRP), intent(out), allocatable, optional :: constr(:)
    real(CRP), intent(in), optional :: f0
    real(CRP), intent(in), optional:: constr0(:)
    integer(CIK), intent(out), optional :: nf
    real(CRP), intent(in), optional :: rhobeg
    real(CRP), intent(in), optional :: rhoend
    real(CRP), intent(in), optional :: ftarget
    real(CRP), intent(in), optional :: ctol
    real(CRP), intent(in), optional :: cweight
    integer(CIK), intent(in), optional :: maxfun
    integer(CIK), intent(in), optional :: iprint
    real(CRP), intent(in), optional :: eta1
    real(CRP), intent(in), optional :: eta2
    real(CRP), intent(in), optional :: gamma1
    real(CRP), intent(in), optional :: gamma2
    real(CRP), intent(out), allocatable, optional :: xhist(:, :)
    real(CRP), intent(out), allocatable, optional :: fhist(:)
    real(CRP), intent(out), allocatable, optional :: chist(:)
    real(CRP), intent(out), allocatable, optional :: conhist(:, :)
    integer(CIK), intent(in), optional :: maxhist
    integer(CIK), intent(in), optional :: maxfilt
    integer(CIK), intent(out), optional :: info

    call cobyla(calcfc, m, x, f, cstrv=cstrv, constr=constr, f0=f0, &
    & constr0=constr0, nf=nf, rhobeg=rhobeg, rhoend=rhoend, &
    & ftarget=ftarget, ctol=ctol, cweight=cweight, maxfun=maxfun, &
    & iprint=iprint, eta1=eta1, eta2=eta2, gamma1=gamma1, &
    & gamma2=gamma2, xhist=xhist, fhist=fhist, chist=chist, &
    & conhist=conhist, maxhist=maxhist, maxfilt=maxfilt, &
    & info=info)
    end subroutine
end module prima_wrapper

This would enable to build a python API for his package based on https://numpy.org/doc/stable/reference/routines.ctypeslib.html that can be easily imported in SciPy just as importing Numpy.
A useful link on python callback in Fortran here: https://stackoverflow.com/questions/57816519/python-callback-in-fortran-via-ctypes
I have shared this code with @zaikunzhang after I successfully compiled an "fpm"ized version of his package and thought that any inputs from the community would be very much welcome.
Cheers,

This would enable to build a python API for his package based on https://numpy.org/doc/stable/reference/routines.ctypeslib.html that can be easily imported in SciPy just as importing Numpy.

Normally we compile Fortran as a static library and then fold it into a Python extension module with f2py. Here we'd compile it as a shared library. Given that users and downstream packagers may compile with either of the two Fortran ABIs and any of a host of compilers, what now takes care of the Fortran name mangling rules so the correct name can be used to call the shared library?

Doing a quick search it seems that NAG and nvfortran (from its pgi heritage) also support the "!DEC$ ATTRIBUTES DLLEXPORT ::" decorators. Haven't check if LFortran has something like this, @certik ?

Okay, educated myself a bit on bind(c), that works fine to remove name mangling.

Correct me if I'm wrong, but from what I understand about f2py it automagically creates the shared object without the developer seen that step (https://foreign-fortran.readthedocs.io/en/latest/python/f2py.html) ... So at the end it should be the same thing but removing the middle-man. No ? And since nothing has been done regarding the wrapper module except a rough proposal the naming convention can be adapted :)

That wrapper code looks pretty reasonable and is a lot easier to understand than f2py imho. I think we could give that a try. Please do avoid use of c_float128, we don't want any new long double code. float/double routines are fine.

but from what I understand about f2py it automagically creates the shared object without the developer seen that step

Yes you are right. My question was what not using f2py would mean - but bind(c) takes care of all that, and is way better than the unreadable generated code that f2py emits to deal with name mangling.

One question about iso_c_binding is copying. Does it handle C/Fortran ordered inputs automatically, does it avoid copying if the ordering is correct etc.

Please do avoid use of c_float128,

That should not be a problem

Does it handle C/Fortran ordered inputs automatically

Could you elaborate on this? Just in case:
in the example I linked before, you can pass an array (on the python side) by:
ctypes.byref(np.ctypeslib.as_ctypes( X ))
this will pass the pointer by its reference to Fortran, no memory is copied.
You can also pass an array allocated on the Fortran side of things to Python in the same manner. The key advantage I see with this approach is that iso_c_binding+ctypes justs gives a "data types" bridge, without imposing any kind of restrictions on who holds the data.

Handling here means do we have to transpose the data and convert to contiguous buffer to make it work?

Oh ok, I see your point. So actually I build this example precisely because of a discussion regarding that topic. So I would say that in principle you do not have to transpose the data. When you allocate the memory with numpy, lets say a numpy.array, it is contiguous in memory in a row-major alignment given by the C convention. When you recover the momery on the Fortran side, it will still see the same memory region. Simply that the Fortran code will handing the multidimensional array indexing from left to right:
Easier with an example:
If on the python side I have this array:
X = np.array([     [1.0, 2.0],     [3.0, 4.0],     [5.0, 6.0], ])
On the Fortran side I would access X[0,0:1] by x(1:2,1) (which access the same contiguous layout, is the indexing that changes)
So, the Fortran code would, if you do nothing, treat the rows as columns. And I think it is preferable to just let it do so.

All arrays are flat memory buffers anyways. But we need to send the correct array that Fortran side expects. Otherwise it would be working on the wrong data. f2py handles this by being the intermediate doing the conversions (costing performance but reducing headaches). For nd arrays, contiguousness won't work as a C-layout array of (i, j, k, k) have contiguous kxk arrays but on the fortran side (i, j, k, k) they are not.
If there is any Lapack-like call where the array data is expected to be contiguous, this would cause problems. The index swapping or transposing is not going to solve the ndarrays.
But also in the 2D sliced arrays say I have a 500x1000 array a and I send the a(500, :500) part to fortran layer, it will segfault or return the wrong answer still accessing 500x1000 addresses.

A typical example we always do on Cython/LAPACK/BLAS calling is you have
import numpy as np
import scipy.linalg as la
rng = np.random.default_rng()
A = rng.random([10, 10])
x = rng.random(10)
y = la.blas.dgemv(1.,A, x)
np.testing.assert_allclose(y, A@x)

Here the internal memory layout swapping of A is done by the f2py wrapper noticing that A is C-layout. It doesn't copy this if I start with an F-array to start with. And many other details similar to this is handled by f2py.
If we are bypassing f2py then we should be wary of this and implement lots of guards in place to not to send wrong structure to fortran because it doesn't error but segfaults which is a no-no for us.

Thanks for the example!! I'll take a look at this!!

it doesn't error but segfaults which is a no-no for us.

This is more than understandable!!

Yes, iso_c_binding takes care of name mangling, you select exactly what C name you want as well as other details of the C ABI, and every Fortran compiler then obeys it.
The issue of y = la.blas.dgemv(1.,A[:10, :10], x) that @ilayn mentioned is the main issue to resolve. What I have done in the past is to use Cython that does the check for me that the array is contiguous and gives an exception if it isn't. As a user that's actually what I want, since I don't want any temporary copies, which is bad for performance. If a copy is needed, I rather do it by hand. However, for SciPy I can see that you might want to do this automatically. In which case we need a layer to take care of this.

Then comes the question of how much we are reinventing f2py.

Then comes the question of how much we are reinventing f2py.

I'd say a very big chunk.

What I have done in the past is to use Cython that does the check for me that the array is contiguous and gives an exception if it isn't.

Instead we manually copy the data. That's what we are doing internally for all Cython LAPACK/BLAS calls. For example https://github.com/scipy/scipy/blob/main/scipy/linalg/_cythonized_array_utils.pyx#L68-L97 swaps the memory layout[^1].
[^1]: Nevermind the recursion, we are flattening this routine to a flat double loop. I wrote it when I was thinking highly of myself.
In the PRIMA case, it is possible to hand craft the wrappers so it is not a blocker if we are very careful but we will need some safeguards in place. Hence the reason I mentioned this in my list in #18367 that forces us to f2py and makes fortran very tedious to use. I know fortran folks are not used to this issue but for us a substantial rite of passage.

I'm not sure it's comparable, I'd like to see it worked out. If you look at what's involved in the f2py wrapper, it's:

cobyla.pyf, 40 LoC (importantly, not checkable by a compiler, which is the source of a ton of problems)
_cobylamodule.c, 1100 LoC of more or less unreadable generated C code
numpy/f2py/src/fortranobject.c|h, ~1600 LoC of messy C

Here we seem to be getting:

the wrapper above, ~100 LoC of reasonably readable Fortran code
a few dozen lines of Cython code using fused types (assuming we indeed need/want both float32 and float64 routines) to check for C/Fortran order and contiguity, and copy as needed
ctypes/cffi/cython wrapper code (to be written, and preferably in Cython it looks to me) to expose c_cobyla to Python.

So it would seem that something more generic like:
#if __GNUC__
        !GCC$ ATTRIBUTES DLLEXPORT :: c_cobyla
#else
        !DEC$ ATTRIBUTES DLLEXPORT :: c_cobyla
#endif

could work

This isn't enough it looks like, it gives this warning when not on Windows:
Warning: 'dllexport' attribute directive ignored [-Wattributes]

Don't the public/private keywords help for this? Having to use a bunch of pre-processor boilerplate for every wrapped function/subroutine to get portable and warning-free compilation into a shared library seems messy.

I would not use any !GCC$ ATTRIBUTES DLLEXPORT :: c_cobyla, but just use iso_c_binding and then build a regular Python extension module via Cython (say).

@rgommers so I did a test and it seems, at least on Linux, that GNU compilers don't need the decorators, the simple use of bind(c) + the flag "-fpic" when creating the static library enables creating the shared object. Also, if one were to let the decorator the warning can be bypassed with "-Wno-attributes". So I think it could be even less verbose than the first proposal... have yet to try different combinations OS/compiler.

a regular Python extension module via Cython (say).

Any chance you have a worked example @certik ? I imagine the last mile here is rather long, and it would be great to have a canonic example which is known to work across a range of OSes and compilers.

I'm not sure that using Cython would actually simplify things.
Looking at this example workflow https://stackoverflow.com/questions/22404060/fortran-cython-workflow it seems that one would need: The Fortran Library + Fortran wrapper using iso_c_binding + a .h header file + a .pyx file > integrate in the final python package. All of this to do exactly the same thing at the end: compile a shared object from a static library that will be called from the python module.
Using the ctypes library included in numpy the workflow could be the following:
The Fortran Library + Fortran wrapper using iso_c_binding + its build system to generate the shared object(s) (a binary compiled library per targeted OS). The Python package includes as .py which calls the library with "numpy.ctypeslib.load_library" within properly arranged functions. So no need for the .h and .pyx files.
If I retake the toy example published in the fortran-discourse and update it with the previous discussions, this would look like:
(for simplicity everything is in a single folder path)
fortran2python.f90
module fortran2python
  use iso_c_binding
  implicit none
  contains

  subroutine say_something(something_cptr,len_cptr) bind(c)
#if _WIN32 & __GNUC__
    !GCC$ ATTRIBUTES DLLEXPORT :: say_something
#elif _WIN32
    !DEC$ ATTRIBUTES DLLEXPORT :: say_something
#endif
    type(c_ptr), value :: something_cptr
    integer(c_int), value :: len_cptr
    character(len=len_cptr,kind=c_char), pointer :: something
    !-------------------------------
    call c_f_pointer(something_cptr, something)
    print *, something

  end subroutine

  subroutine change_array_value(X,dim,num_points) bind(c)
#if _WIN32 & __GNUC__
    !GCC$ ATTRIBUTES DLLEXPORT :: change_array_value
#elif _WIN32
    !DEC$ ATTRIBUTES DLLEXPORT :: change_array_value
#endif
    integer(c_int), intent(in) :: num_points
    integer(c_int), intent(in) :: dim
    real(c_double), intent(inout) :: X(dim,num_points) !! array of coordinates
    integer :: i
    !-------------------------------
    print *, 'hello from fortran'

    print *, 'Whole array: ',X(:,:)

    do i = 1, num_points
        print *, 'by data point', x(:,i)
    end do

    x(1,2) = 2*x(1,2) ! modify this value corresponding to

  end subroutine

end module

test.py
import os
import numpy as np
import ctypes

folderpath = ''
if os.name == 'nt': #windows
    DLLname = 'fortran2python.dll'
elif os.name == 'posix': #Linux/Mac
    DLLname = 'libfortran2python.so'
api = np.ctypeslib.load_library(DLLname,folderpath)

something = 'Something'
api.say_something( ctypes.c_char_p(something.encode("utf-8")) , len(something) )

X = np.array([
    [1.0, 2.0],
    [3.0, 4.0],
    [5.0, 6.0],
])

print('X shape:',X.shape)
print(X[0,0],X[0,1])
print(X[1,0],X[1,1])
print(X[2,0],X[2,1])

api.change_array_value(  ctypes.byref(np.ctypeslib.as_ctypes( X )) ,
                ctypes.byref(ctypes.c_int(X.shape[1])) ,
                ctypes.byref(ctypes.c_int(X.shape[0]))
             )

print('back in python')
print(X[0,0],X[0,1])
print(X[1,0],X[1,1])
print(X[2,0],X[2,1])

Succesfully compiled with

Linux + gfortran 12.2

gfortran -cpp -O3 -march=native -fpic -c fortran2python.f90
gfortran -shared -o libfortran2python.so fortran2python.o

Linux + ifort 19

ifort -fpp -O3 -march=native -fpic -c fortran2python.f90
ifort -shared -o libfortran2python.so fortran2python.o

Windows + ifort 2021

ifort -fpp -O3 -c fortran2python.f90
ifort -dll -exe:fortran2python.dll fortran2python.obj

Runing "python test.py" gives:
 Something
X shape: (3, 2)
1.0 2.0
3.0 4.0
5.0 6.0
 hello from fortran
 Whole array:    1.0000000000000000        2.0000000000000000        3.0000000000000000        4.0000000000000000        5.0000000000000000        6.0000000000000000
 by data point   1.0000000000000000        2.0000000000000000
 by data point   3.0000000000000000        4.0000000000000000
 by data point   5.0000000000000000        6.0000000000000000
back in python
1.0 2.0
6.0 4.0
5.0 6.0

A build system as meson or CMake can take care of generating the binary for the shared library and on the python side it would just be needed a function definition calling the binded function with its signature in/outs. At the end the maintenance (per function) boils down to the wrapper and the python caller.

We also need Windows + gfortran through MinGW64 leg unfortunately, as we can't enforce ifort also we need to show an example that actually links to OpenBLAS or MKL no matter how trivial it is, say, simple call to ?axpy. Because as you are experiencing first hand, it is getting more and more sophisticated as we add more OS and compilers into the picture.

Oh wait nevermind the blas thing, prima handcodes qr stuff. I'm starting confuse things.

I compiled his library quite easily as he doesn't have dependencies, I was impressed and relieved :)

What should be the DLL syntax for gfortran? Because I got missing dependencies error already :)

All of this to do exactly the same thing at the end: compile a shared object from a static library that will be called from the python module.

Not quite. There is a distinct difference between the Python extension module shared object produced by Cython+C-compiler and a plain old shared object produced by Fortran+iso_c_binding. Not least, all of the declarations needed to export the shared symbols handled by the build system in the Cython case, regardless of the Fortran compiler and OS. It seems that we still need to work out the right idiom for that in the Fortan+iso_c_binding case to handle the variety of Fortran compilers we target.
One very important thing that f2py does for us is handling the reverse wrapping of the Python callback. I assume, but don't know for sure that iso_c_binding turns those Fortran callbacks into C function pointers in a predictable way. A minimal example in either Cython or ctypes needs to demonstrate that as well. That's probably a point in favor of ctypes, as I see the way we can make ad hoc callback function pointers from Python functions in thread-safe ways, but Cython doesn't provide much out of box for that.

What should be the DLL syntax for gfortran? Because I got missing dependencies error already :)

Most likely it is expecting some other runtimes that should be visible in the path. haven't worked with the mingw64 installation but I'll take a look at that. In the mean time, you can try checking the dependencies of the library and check if you have them in your system, after that, you can locally add the path in the python test with
if sys.version_info[1]<=7: # for python 3.7
os.environ['PATH'] = '\path\to\run_times'
else: # from python 3.8
os.add_dll_directory('\path\to\run_times')

Yes, but then we get into DLL hell discussion. So let's not go into that part, @rkern's comment is more important, we'll figure something out when that part settles

Yes, here is a worked out example: https://www.fortran90.org/src/best-practices.html#interfacing-with-c.
You can do bind(c, name="xxx"), is that the same thing as !GCC$ ATTRIBUTES DLLEXPORT :: xxx ?

@ilayn I installed mingw64 and compiled with
gfortran -shared -O3 -fPIC -o fortran2python.dll fortran2python.f90

had to manually import 'libgfortran-5.dll' though
No need for the decorators in this case, managed to remove them and it works.

You can do bind(c, name="xxx"), is that the same thing as !GCC$ ATTRIBUTES DLLEXPORT :: xxx ?
In principle they should serve they same purpose. What I'm getting at after these tests is that just bind(c) should work on Linux for gfortran and ifort, on windows with gfortran mingw64; for windows+ifort one still needs the decorator '!DEC$ ...'

@jalvesz, you may add -static:
gfortran -shared -static -O3 -fPIC -o fortran2python.dll fortran2python.f90 to get rid of the libgfortran-5.dll dependency.

Indeed!! that worked like a charm!!

So I use both -shared and -static to make a shared library but without the gfortran meddling and then we are the bad guys when we complain about compilers... that's bananas.

All this is the reason I want to bring mingwpy back to life - to make the life easier on windows. If only i could purchase some time...

That's a GCC convention, not a gfortran one, FWIW.

Not quite. There is a distinct difference between the Python extension module shared object produced by Cython+C-compiler and a plain old shared object produced by Fortran+iso_c_binding

@rkern I wanted to ask something, after reading this I went to look at the generated package and saw that the libraries are (for windows) .pyd files. Reading here https://docs.python.org/3.9/faq/windows.html#is-a-pyd-file-the-same-as-a-dll I see the following:

Is a *.pyd file the same as a DLL?¶
Yes, .pyd files are dll’s, but there are a few differences. If you have a DLL named foo.pyd, then it must have a function PyInit_foo(). You can then write Python “import foo”, and Python will search for foo.pyd (as well as foo.py, foo.pyc) and if it finds it, will attempt to call PyInit_foo() to initialize it. You do not link your .exe with foo.lib, as that would cause Windows to require the DLL to be present. (...)

So, not exactly a dll, but almost. Saying that one manage to generate a fully independent shared library as just seen thanks to @carlkl, what would you say that could be a remaining blocking aspect of the ctypes workflow (embedding a dll/so)?

Well, no dice
C:\Users\ilhan\Documents\pydump\fortranctype> gfortran -shared -static -O3 -fPIC -o fortran2python.dll fortran2python.f90
fortran2python.f90:7:2:

    7 | #if _WIN32 & __GNUC__
      |  1
Warning: Illegal preprocessor directive
fortran2python.f90:9:2:

    9 | #elif _WIN32
      |  1
Warning: Illegal preprocessor directive
fortran2python.f90:11:2:

   11 | #endif
      |  1
Warning: Illegal preprocessor directive
fortran2python.f90:22:2:

   22 | #if _WIN32 & __GNUC__
      |  1
Warning: Illegal preprocessor directive
fortran2python.f90:24:2:

   24 | #elif _WIN32
      |  1
Warning: Illegal preprocessor directive
fortran2python.f90:26:2:

   26 | #endif
      |  1
Warning: Illegal preprocessor directive
C:\Users\ilhan\Documents\pydump\fortranctype>

That's a GCC convention,

That makes it even more bananas. I'm scared to ask whether the order of the -shared -static matters

Did you forget -cpp? in the command line shown before it was assumed that they were no longer there.
Saying the macros are kept for the ifort+windows case, you should use the preprocessors -cpp for gfortran and -fpp for ifort

Why cpp?
Nevermind, I don't have enough patience to go into this hell again. Let's fix the happy weather route, if the wrappers etc are working, then we can fight with windows later.

Why cpp?

If for any reason there are macros "#if #else #define etc etc" in C/C++/Fortran source codes you need the preprocessors... If there are non, you can still safely use them, anyhow those would be in the final build_system (be it meson/CMake/a setup.py procedure)
Revisited minimum working example:
module fortran2python
  use iso_c_binding
  implicit none
  contains

  subroutine say_something(something_cptr,len_cptr) bind(c)
#if _WIN32 & __INTEL_COMPILER
    !DEC$ ATTRIBUTES DLLEXPORT :: say_something
#endif
    type(c_ptr), value :: something_cptr
    integer(c_int), value :: len_cptr
    character(len=len_cptr,kind=c_char), pointer :: something
    !-------------------------------
    call c_f_pointer(something_cptr, something)
    print *, something

  end subroutine

  subroutine change_array_value(X,dim,num_points) bind(c)
#if _WIN32 & __INTEL_COMPILER
    !DEC$ ATTRIBUTES DLLEXPORT :: change_array_value
#endif
    integer(c_int), intent(in) :: num_points
    integer(c_int), intent(in) :: dim
    real(c_double), intent(inout) :: X(dim,num_points) !! array of coordinates
    integer :: i
    !-------------------------------
    print *, 'hello from fortran'

    print *, 'Whole array: ',X(:,:)

    do i = 1, num_points
        print *, 'by data point', x(:,i)
    end do

    x(1,2) = 2*x(1,2) ! modify this value corresponding to

  end subroutine
end module

import os
import numpy as np
import ctypes

folderpath = ''
if os.name == 'nt': #windows
    DLLname = 'fortran2python.dll'
elif os.name == 'posix': #Linux/Mac
    DLLname = 'libfortran2python.so'
api = np.ctypeslib.load_library(DLLname,folderpath)

something = 'Something'
api.say_something( ctypes.c_char_p(something.encode("utf-8")) , len(something) )

X = np.array([
    [1.0, 2.0],
    [3.0, 4.0],
    [5.0, 6.0],
])

print('X shape:',X.shape)
print(X[0,0],X[0,1])
print(X[1,0],X[1,1])
print(X[2,0],X[2,1])

api.change_array_value(  ctypes.byref(np.ctypeslib.as_ctypes( X )) ,
                ctypes.byref(ctypes.c_int(X.shape[1])) ,
                ctypes.byref(ctypes.c_int(X.shape[0]))
             )

print('back in python')
print(X[0,0],X[0,1])
print(X[1,0],X[1,1])
print(X[2,0],X[2,1])

To build with:
Linux + gfortran
gfortran -cpp -O2 -fpic -c fortran2python.f90
gfortran -shared -o libfortran2python.so fortran2python.o

Windows + mingw64-gfortran
gfortran -cpp -O2 -fpic -c fortran2python.f90
gfortran -shared -static -o fortran2python.dll fortran2python.o

Linux + ifort
ifort -fpp -O3 -fpic -c fortran2python.f90
ifort -shared -o libfortran2python.so fortran2python.o

Windows + ifort
ifort -fpp -O3 -c fortran2python.f90
ifort -dll -exe:fortran2python.dll fortran2python.obj

Saying that one manage to generate a fully independent shared library as just seen thanks to @carlkl, what would you say that could be a remaining blocking aspect of the ctypes workflow (embedding a dll/so)?

The minimal example needs to demonstrate a callback.
Integrating that build recipe into our meson build system (hopefully, you can avoid the legacy setup.py build system). We don't currently use ctypes for anything in scipy. I'm not sure if there is a meson recipe for building and placing ctypes-usable DLLs/SOs, so you might have to work that out and contribute it upstream. I'm betting there isn't one worked out using each of the Fortran toolchains to do the linking.

Integrating that build recipe into our meson build system (hopefully, you can avoid the legacy setup.py build system). We don't currently use ctypes for anything in scipy. I'm not sure if there is a meson recipe for building and placing ctypes-usable DLLs/SOs, so you might have to work that out and contribute it upstream. I'm betting there isn't one worked out using each of the Fortran toolchains to do the linking.

This should need no upstream changes, it's a standard shared library. To install it to the correct location, use install_dir: py3.get_install_dir() / 'scipy/optimize'. I can help with this part if there's anything more to it. And indeed, don't worry about setup.py.

I'm adding a C interface to prima: https://github.com/libprima/prima/pull/28
From there a Python interface should be easier to implement, I know SWIG well, but suggestions are welcome

Well... that is @zaikunzhang decision to make, but this would imply a "3-languges" issue, which just adds unnecessary boiler-plate. SWIG would require making the whole code compliant with it, which is less fun than it would seem from its automagical promise (which is the same issue of f2py). An iso_c_binding-pure-Fortran module wrapper is all one needs to make the library accessible from C/C++/Python.

Have you tried using the numpy version of LoadLibrary?  https://numpy.org/doc/stable/reference/routines.ctypeslib.html
import numpy as np
...
_handle = np.ctypeslib.load_library(_lib_name, _relative_path )

It wrapps cdll.LoadLibrary, adding cross-platform considerations to facilitate things afterwards.
Everything else would be exactly the same thing ;)

that's what we would want to do, right ?

Yes, I think so ... let's say that the python wrapper file ends within scipy/optimize and that the generated shared library from COBYLA in scipy/optimize/binaries (this one doesn't exist but I'm just proposing something parallel to 'cython_optimize')
Within your python file you could invoke the library as
import os
here = os.path.abspath(os.path.dirname(__file__))
_path = os.path.join(here,'binaries')
_handle = np.ctypeslib.load_library( _lib_name, _path )

That way, it will be transparent whether you are debuging scipy from a local version or using the installed version in site-packages

Could we please make this thing a bit more tangible as you know "should be able to" is very different than "works" when it comes to fortran. If you need some examples, you can find them here
https://github.com/scipy/scipy/blob/main/scipy/optimize/tests/test_cobyla.py
Also reverse callback needs to be tested with the non-f2py way of working, that is pass a function from python to fortran.

@jalvesz I updated my code so as to simplify the fortran calls if you were to use that is scipy; now the C symbols are only still there to pass the arguments by value, which is more practical

Hello all,
Sorry for being silent for a long time.
Julien @jschueller is working on the Python binding of PRIMA. I hope it will be used to integrate PRIMA solvers into SciPy to replace the buggy F77 version as soon as possible. Therefore, the opinions from the SciPy community are critical.
In addition, Python is totally beyond my knowledge. I cannot give many comments except for some rough ones when it comes to techniques. Hence, I hope the SciPy community could actively provide advice in the development of the Python binding.
Thank @jschueller very much for the initiative. Many thanks to all!
Best regards,
Zaikun

I quickly glanced over the Python binding code, there's pretty much no comments on it, so it'll harder to go through.

New solvers being added to scipy should ideally use the optimize.NonlinearConstraint/optimize.LinearConstraint syntax for specifying constraints, and use optimize.Bounds for lower/upper bounds.
the build system for scipy uses meson, not cmake.
if possible the minimiser should provide a callback ability to update the user on the progress of the minimisation.
the minimiser should have the ability to add extra user defined args, f(x, *args).
bindings should follow existing approaches, i.e. f2py/cython/..., no SWIG or handwritten C extension modules.
all fortran code should be re-entrant.

Thank you @andyfaff for the comments. I have copied them to the pull request under libprima/prima. You are also very welcome to make comments over there directly.
I can respond to the following comments regarding the Fortran code, while those concerning Python / building may be addressed by @jschueller .

*if possible the minimiser should provide a callback ability to update the user on the progress of the minimisation.

As long as we can decide the signature of the callback function (e.g., callback(x, f, constr), f and constr being the objective and constraint values at x), then this is not difficult to implement.

*all fortran code should be re-entrant.

The PRIMA code is re-entrant and thread-safe.
Thank you.

Hello, SciPy maintainers,
PRIMA is considering adding a callback function to the minimizers. Before that, we have to decide the signature of the callback. What is the standard and preference of SciPy? You are invited to give your comments at
https://github.com/libprima/prima/issues/65
Thank you very much!
Best regards,
Zaikun

Hello,
Sorry for bugging you. Julien @jschueller is working to make PRIMA solvers accept callback functions (see https://github.com/libprima/prima/issues/65 and https://github.com/libprima/prima/pull/69).
We do need comments from the SciPy community (e.g., what is the ideal signature of this function?) It would be great if you could provide your opinions here or at
https://github.com/libprima/prima/issues/65
Thank you for your attention.
Best regards,
Zaikun

Thank you for your hard work, but please be patient. It is the weekend, and people only have so much bandwidth.
The design of the callback is made easier if the Fortran code can be stepped through as an iterator. For example, the core of differential_evolution is something akin to the following:
def solve(self):
    for i in range(numiters):
        if converged:
            # return final result
        else:
            result = next(self)

def __next__(self):
    # call the external code to take one solve step, fortran returns interesting things.
    x, fun, etc = make_fortran_step()

    # make intermediate result
    im_result = OptimizeResult(x=x, fun=fun)

    # call callback
    callback(intermediate_result=im_result)
    return im_result

Modern callbacks are being designed to return intermediate OptimizeResult. See the docstring for minimize. The intermediate result should be filled out with as much information that makes sense to offer to the user.

Thank you for these useful comments. I have copied it to https://github.com/libprima/prima/issues/65.

Hello all,
I am working with @zaikunzhang on PRIMA, specifically the Python/Fortran integration. I've read through this thread and related threads on the PRIMA side and I'd like to make some suggestions addressing some of the issues raised and also propose a path forward.
First on the issues raised:

Also reverse callback and non-Fortran noncontiguous array cases need to be tested with the non-f2py way of working, that is pass a function from python to fortran.

Here is my update to @jalvesz's (very useful) minimal example from earlier using iso_c_bindings to demonstrate these two pieces of functionality. I'm a novice with both Fortran and ctypes so if there are cleaner ways of doing this I'm happy for the suggestions.
The Fortran code was modified to include an abstract interface that can be used for the callback, as well as adding a function that accepts and call such a callback (and also I changed dim/numpoints to rows/cols since that made more sense):
module fortran2python
  use iso_c_binding
  implicit none

  abstract interface
    subroutine callback_interface(x, rows, cols, f) bind(c)
      use iso_c_binding
      implicit none
      real(c_double), intent(in) :: x(:, :)
      integer(c_int), value :: rows
      integer(c_int), value :: cols
      real(c_double), value :: f
    end subroutine callback_interface
  end interface

  contains

  subroutine say_something(something_cptr,len_cptr) bind(c)
#if _WIN32 & __INTEL_COMPILER
    !DEC$ ATTRIBUTES DLLEXPORT :: say_something
#endif
    type(c_ptr), value :: something_cptr
    integer(c_int), value :: len_cptr
    character(len=len_cptr,kind=c_char), pointer :: something
    !-------------------------------
    call c_f_pointer(something_cptr, something)
    print *, something

  end subroutine

  subroutine change_array_value(X,rows,cols) bind(c)
#if _WIN32 & __INTEL_COMPILER
    !DEC$ ATTRIBUTES DLLEXPORT :: change_array_value
#endif
    integer(c_int), intent(in) :: rows
    integer(c_int), intent(in) :: cols
    real(c_double), intent(inout) :: X(rows,cols) !! array of coordinates
    integer :: i
    !-------------------------------
    print *, 'hello from fortran'

    print *, 'Whole array: ',X(:,:)

    do i = 1, rows
        print *, 'by data point', x(i,:)
    end do

    x(1,2) = 2*x(1,2)

  end subroutine

  subroutine call_me_back(X,rows,cols,callback) bind(c)
#if _WIN32 & __INTEL_COMPILER
    !DEC$ ATTRIBUTES DLLEXPORT :: call_me_back
#endif
    integer(c_int), intent(in) :: cols
    integer(c_int), intent(in) :: rows
    real(c_double), intent(inout) :: X(rows,cols) !! array of coordinates
    procedure(callback_interface) :: callback

    x(1,2) = 2*x(1,2)

    call callback(X, rows, cols, 2.23_8)

  end subroutine
end module

The Python side of things has been modified to use np.asfortranarray to demonstrate how to use this with non-Fortran non-contiguous data, as well as demonstrating the reverse callback functionality
import os
import numpy as np
import ctypes

folderpath = ''
if os.name == 'nt': #windows
    DLLname = 'fortran2python.dll'
elif os.name == 'posix': #Linux/Mac
    DLLname = 'libfortran2python.so'
api = np.ctypeslib.load_library(DLLname,folderpath)

something = 'Something'
api.say_something( ctypes.c_char_p(something.encode("utf-8")) , len(something) )

X = np.array([
    [1.0, 2.0, 3.0],
    [4.0, 5.0, 6.0],
    [7.0, 8.0, 9.0],
], dtype=np.float64)

# Demonstrate non-Fortran non-contiguous array
X = np.asfortranarray(X[0:, 1:])

print('X shape:',X.shape)
print(X)

api.change_array_value(X.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
                ctypes.byref(ctypes.c_int(X.shape[0])),
                ctypes.byref(ctypes.c_int(X.shape[1]))
             )

print('back in python')
print(X)

def callback(x, rows, cols, f):
    # We reverse rows/cols and then transpose to get the right memory layout
    x = np.ctypeslib.as_array(x.contents, shape=(cols, rows)).T
    print(f'{x=}')
    print(f'{f=}')

callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(ctypes.POINTER(ctypes.c_double)), ctypes.c_int, ctypes.c_int, ctypes.c_double)

api.call_me_back(X.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
                ctypes.byref(ctypes.c_int(X.shape[0])),
                ctypes.byref(ctypes.c_int(X.shape[1])),
                callback_type(callback))

I think this covers the main issues that have been under discussion most recently in the thread. I think @andyfaff's suggestion makes a certain amount of sense, but would require a pretty significant refactor in PRIMA. Whether or not that's a good idea and/or suitable for PRIMA is outside the scope of this discussion, which I believe should remain focused on bringing PRIMA intro Scipy.
On that note, I've discussed this with @zaikunzhang and we think the path forward that makes the most sense is for PRIMA itself to take in all these suggestions and for us to publish a Python library that SciPy can then depend on. This option removes the need for SciPy to have any Fortran code (at least as far as COBLYA and the other algorithms in PRIMA are concerned) and shifts the burden for compiling for multiple platforms onto PRIMA, which is acceptable to us.
There are 3 alternatives that I can see:

SciPy includes PRIMA as a submodule and builds it. While this made sense with COBYLA since it was just a couple Fortran files, with PRIMA this leads to a big duplication of effort, since PRIMA already has a need to build libraries, so we might as well build them on our end and provide them.
SciPy includes the pre-built PRIMA libraries and maintains the iso_c_bindings and ctypes code
SciPy includes the pre-built PRIMA libraries that incorporate the iso_c_bindings and maintains the ctypes code.

Options 2 and 3 are basically the same as what I've proposed, just moving around where the iso_c_bindings and ctypes code lives. SciPy will need to maintain a thin wrapper in any of these scenarios to do things like unwrap optimize.LinearConstraint and wrap results into OptimizeResult, etc. (which I would be happy to contribute as part of this effort). Options 2 and 3 require more maintenance work on the SciPy end which is why the proposed path forward brings that into PRIMA (call it Option 4 if you like).
Sorry for the long post, I hope it's comprehensive. Does this approach work for SciPy and does it sound like something we can move forward on? Happy for any other comments folks might have.

Hi @nbelakovski, thanks for the update, good to see you pushing this forward!

Whether or not that's a good idea and/or suitable for PRIMA is outside the scope of this discussion, which I believe should remain focused on bringing PRIMA intro Scipy.

agreed

On that note, I've discussed this with @zaikunzhang and we think the path forward that makes the most sense is for PRIMA itself to take in all these suggestions and for us to publish a Python library that SciPy can then depend on
...
There are 3 alternatives that I can see:

We cannot take a new runtime dependency (we have exactly one, numpy, and that won't change) or rely on a pre-built static library. And an external dynamic library is quite problematic with Python packaging. The only suitable option here is option 1, with a tweak:

SciPy includes PRIMA as a submodule and builds it. While this made sense with COBYLA since it was just a couple Fortran files, with PRIMA this leads to a big duplication of effort, since PRIMA already has a need to build libraries, so we might as well build them on our end and provide them

This does not have to lead to duplication. What we can do is:

include PRIMA as a git submodule, in a top-level subprojects/ directory (so outside of the scipy/ source tree) - we will also do this with HiGHS
if we need a fix for a build config in PRIMA, we contribute it upstream to your repository first and then update our git submodule to point to a new commit on PRIMA's main branch that includes that fix
we ensure there is Meson build support in your repository. that will include a prima_dep dependency of a library, which we can build statically
when we build SciPy, we build PRIMA as part of the process

It does require having support for building with Meson in https://github.com/libprima/prima, but given that your CMakelists.txt's are <300 lines in total, that shouldn't be a large overhead I'd think. And we can help set that up and maintain it - easier all around than us maintaining the build definitions that are needed anyway separately.

Thanks for the reply and the feedback @rgommers.

we ensure there is Meson build support in your repository. that will include a prima_dep dependency of a library, which we can build statically

Two questions, perhaps related:

What does Meson build support in prima repo look like? Are we talking about Meson build files that parallel the CMake/Make files currently in the repo?
I'm not sure what you mean by a prima_dep dependency of a library?

I'm fine with setting up the Meson. I haven't used this particular build system, but I've used SCons, Waf, CMake so I'm at least familiar with the basic concepts.
If I take a stab at answering my own question #2, it sounds like something along the lines of building a static library called prima_dep with all the optimization algorithms, then a shared library "on top" of that one with the iso_c_bindings, and then finally some "glue code" in the form of a Python script with the relevant ctypes code, is that right?

What does Meson build support in prima repo look like? Are we talking about Meson build files that parallel the CMake/Make files currently in the repo?

indeed

If I take a stab at answering my own question #2, it sounds like something along the lines of building a static library called prima_dep with all the optimization algorithms, then a shared library "on top" of that one with the iso_c_bindings, and then finally some "glue code" in the form of a Python script with the relevant ctypes code, is that right?

Effectively, yes. I am still not 100% sure about ctypes usage, because that's usually more fragile than generating a Python extension module - I still have to explore if we cannot get that folded into a Python extension module. But a "dependency" is basically a one-line wrapper around a static or shared library, so we need the equivalent of add_library (primaf) in your CMakelists.txt, and then wrap that. Something like:
prima_f_sources = [
  'common/linalg.f90',
  ...
]
prima_f = static_library('prima_f',
  prima_f_sources.
  include_directories: ['primaf', 'common'],
  fortran_args: [],  # add compile flags/defines
)

prima_with_bind_c = shared_library('prima_with_bind_c',
  'bind_c_wrapper.f90',
  link_with: prima_f,
  install: true,
)

prima_dep = declare_dependency(
  include_directories: 'primaf',
  link_with: prima_with_bind_c
)

Cc @HaoZeke, who has time and interest to help with this. Our discussion today came to the tentative conclusion that writing a Python extension module in C, using the PRIMA library and headers, may make sense. f2py doesn't yet support the needed features here, but @HaoZeke could implement that at the same time, which may make life easier in the future.

Sorry that I missed the discussions. Nickolai @nbelakovski is visiting me in Hong Kong, and we have detailed discussions on this project. Everything sounds great! I have told him my opinions. Nickolai will continue to work on it. Thank @rgommers and @HaoZeke and others for the precious help and contributions!


HW2-jp4013

I cannot find why it won't automatically merge yet can still let me rebase yet.

It let me squash the changes on rebase but it doesn't show up here. I'm confused.


Delete invoke message

I would love the feature that would allow me to use a 'delete_invoke = True' attribute that deletes the message that invoked the help command.

Will be added to V2, thanks for the request.


Speech to text doesn't work with popup - false (android)

Hello, so I have an issue on Unity 2020.3.20f, as in title the whole recognition works, but only with pop-up set to true, if I switch it to false it doesn't work. (Tested on my own script and on test scene from package)

What helped me resolve the problem is described here, I had the same issues in both 2020.1.2f1 and 2021.3.5f1:
https://github.com/j1mmyto9/speech-and-text-unity-ios-android/issues/74

I am having the same problem. Did you manage to solve this?

I am having the same problem


Add percy test for categories panel and pills

I'm a bit confused about this since I can't find information on how to run percy tests locally. I've tried setting the percy token to the one in percy.io and followed the instructions on https://docs.percy.io/docs/local-development then ran

npm run percy
npm run test:e2e:percy

I've asked a question on slack and I'll just move this back to the backlog for now since I might not get a response today, and I'm taking leave starting tomorrow.

https://xwp.slack.com/archives/CCCAUAH6F/p1595899002201200 link to question.

First you'll haqve to clarify whether you want to run Percy for the Karma tests or the E2E tests. I assume the former.
In your Karma tests, call await fixture.snapshot(); to take a snapshot of the current state. The function takes an optional parameter for the snapshot name.
Then, run npm run test:karma:edit-story -- --headless --snapshots --viewport=1600:1000 to run the test suite as usual.
snapshots makes sure that snapshots are taken (otherwise they're ignored).
The viewport option makes sure that we're using the same viewport consistently everywhere.
Now you can run PERCY_TOKEN=<yourtoken> npm run percy to upload the snapshots to percy.

@swissspidy Where are the snapshots stored locally when npm run test:karma:edit-story -- --headless --snapshots --viewport=1600:1000 is run?

Snapshots are stored in .test_artifacts/karma_snapshots. Not sure about the location on percy when running locally. I think it just uses your current Git branch, but would need to consult the docs.


Navigating to other application due to universal/app links

Thanks so much for this plugin
I have one feature request and any solution on this would be really appreciated
Feature request
When I open my login url into inappbrowser, after login it redirected to other application due to universal/app link present as redirection url.
I want to prevent the inappbrowser to redirect to other application so that I can control the login flow and navigate back to my application.
Any suggestions?
Thank you in advance.

The WKWebView component of this plugin has now been merged into cordova-plugin-inappbrowser via this PR therefore you should raise any feature requests or bugs against cordova-plugin-inappbrowser


Improve tests according to Humbug log

In this moment humbug is showing ~70% of MSI, so it means not all tests are highest quality.
This issue is about increase this number to 90% at least (aiming for 100%)
70 mutations were generated:
      50 mutants were killed
       9 mutants were not covered by tests
      11 covered mutants were not detected
       0 fatal errors were encountered
       0 time outs were encountered

Metrics:
    Mutation Score Indicator (MSI): 71%
    Mutation Code Coverage: 87%
    Covered Code MSI: 82%

Log after fixes:
66 mutations were generated:
      59 mutants were killed
       7 mutants were not covered by tests
       0 covered mutants were not detected
       0 fatal errors were encountered
       0 time outs were encountered

Metrics:
    Mutation Score Indicator (MSI): 89%
    Mutation Code Coverage: 89%
    Covered Code MSI: 100%


Update photos.js

Add platform photo for Hauptbahnhof and label photo for Alexanderplatz U8.

Yeah the Hauptbahnhof picture is really mediocre. But better than nothing, thanks!


Native shared libraries are not packaged

Servo's APK should include the libmozjs.so library from the mozjs-sys package, but this library is no longer included in the APK since our latest Rust/Cargo upgrade.  This is because this library is no longer saved to the native directory that apk-builder searches in find_native_libs.  I think this is related to rust-lang/cargo#1195 or other recent changes to Cargo.
I'm trying out a patch here to search the -L paths for *.so files instead, but this results in false positives, since these paths contain many shared libraries that don't need to be packaged (for example, packages that build both static and dynamic libraries, but we only use the static version):  https://github.com/mbrubeck/android-rs-glue/tree/library-path
Alternately, we could change either Cargo or our custom build scripts to put libmozjs.so back in native or some other special location that apk-builder can find.

CC @acrichto @brson

The -lmozjs parameter is passed to the apk-builder, right? So it should be possible to restrict the list of .so files to the libraries that were passed with -l.

The -lmozjs parameter is passed to the apk-builder, right? So it should be possible to restrict the list of .so files to the libraries that were passed with -l.

This looks like it might work, actually.  The static libraries we use are already linked into libmain.so by this point, so they aren't passed as -l parameters to apk-builder.


chore: Webpack to Vite

I've swapped out Webpack for Vite to have a better future support & faster builds.
Updated the example to also use Vite.
Removed some unneeded boilerplate from the react example
Bumped some other dependencies

@xxmuaddib do have some time to check the PR's I've opened for your repo?


Typing "function" in a *.js file with atom-jshint enabled causes Atom to freeze

I'm running Atom 0.187.0, on both Linux Mint 17.1 Rebecca x64 and Windows 7 x64. When I have Joezo/atom-jshint enabled, even if all my other non-standard packages are disabled, typing the word function at any point in a *.js file will cause Atom to instantly hang and I must force quit. No errors show up on the dev-tools js console or when I run atom -f in the console. It simply freezes.
Only "function" seems to cause this problem, I can type anything else without a problem. (!??)
If there is another way that Atom might be telling me what the error is, please let me know and I'll provide more info. (I haven't been able to find if atom logs errors to a file somewhere.) The bug is very replicable, on 2 different operating systems on 2 different computers.
This error also happens when I use the other jshint, sindresorhus/atom-jshint, with all other custom packages disabled.
What is going on??? Thanks.

Sounds like something that was specific to a bad release of jshint itself, as it was affecting both atom packages. I'm going to close this in the hope that now we're on version 0.201.0 of atom and jshint has been updated, this problem will have solved itself. Please let me know if it hasn't though. (i published 2.0.0 of this package earlier today)


Update README to mention Windows support.

Thank you @hawkinsp  for the windows wheels release,
I have been following the windows release thread for some time.
Glad that it is finally released.
I am a contributor at https://github.com/OpenMined/PySyft
This would greatly help us.


Create Administrator Role

Create an Administrator role object with read write access to anything.

The role is created via the parse dashboard. Then we need to set up all classes with their appropriate ACL's


IMPLEMENT PRIORITY QUEUE USING LINKED LIST in C language (GSSoC'21)

🚀 Feature
Have you read the Contributing Guidelines on Pull Requests?
Yes !
AIM :: To make the bubble sort more optimize and adaptive by improved time complexities.
Motivation
As i didn't found IMPLEMENTATION OF PRIORITY QUEUE USING LINKED LIST in C language , So i am willing to contribute it...
Pitch
Hi I am GSSoC'21 participant,
"I am looking to work on the implementation of Priority queue using Linked List in C language
I will use C.
I will write the clean and optimized code , by keeping best case time and space complexity in mind and
with  brief commenting and algorithm.
So i would to like @bhav09 , @HarshCasper sir to please assign this issue to me , so that i can do PR ,
as i have already completed with coding :)
/assign

/assign


Add support for Reactive and RxJava2 interfaces for RemoteService object

Done
https://github.com/redisson/redisson/wiki/9.-distributed-services#913-remote-service-asynchronous-reactive-and-rxjava2-calls
https://github.com/redisson/redisson/wiki/9.-distributed-services#914-remote-service-asynchronous-reactive-and-rxjava2-call-cancellation