Improve startup time by pushing some work off the main thread to thread pool

PR Summary
Comparing with v7.1.0, the changes in this PR improves the startup of a stable version PowerShell by about 19% on a 8 logical core machine, by about 10% on a 4 logical core machine.
Stable version of PowerShell means: experimental features are disabled in the build by default, just like a GA version.
Major changes:

Push the call to InitialSessionState.CreateDefault2() to a worker thread.
Push the call to GetConfigurationNameFromGroupPolicy() to a worker thread.
The method Utils.GetPolicySetting<T>(...) is pretty expensive at startup time, including loading system.collections.concurrent.dll, system.io.filesytem.dll, newtonsoft.json.dll, the overhead of using ConcurrentDictionary, and the overhead of reading settings from the configuration file.
Push the call to SendPSCoreStartupTelemetry to a worker thread.
This is also an expensive call at startup time, including loading microsoft.applicationinsights.dll, intializing the telemetry client and actually write out telemetry.

By pushing them to worker threads, the assembly loading and type initalization are all taken place on the worker thread, so as to have the main thread move forward until it has to block for any of the tasks to finish.
Implementation-wise,

ConsoleHost.DefaultInitialSessionState is changed to make its getter able to either return a previously set InitialSessionState value, or waiting for a task result and return the result.
Similarly, CommandLineParameterParser.ConfigurationName is changed to have its getter wait for the 'GetConfigurationNameFromGroupPolicy' task to finish.
Telemetry doesn't block anything, but we need to make sure the startup telemetry gets sent before the process exits. (we ignore the cases where the process is forcefully terminated, in which case we cannot guarantee the telemetry to be sent anyways)
Remove CreateRunspace(object runspaceCreationArgs) and RunspaceCreationEventArgs. They add an intermediate layer of method call, which is not needed at all.

Startup time comparison
The following results are 3 continuous run of the benchmarck program, which compares v7.1.0, v7.2-wip (this pr), and Windows PS 5.1.
Notes:

since we are comparing with a stable version v7.1.0, v7.2-wip is built using Start-PSBuild -Clean -ReleaseTag v7.2.0 -CrossGen -Configuration Release to diable all experimental features (-ReleaseTag v7.2.0 means building for a stable version).
the module analysis cache file is already created for all those 3 versions of PowerShell.

Benchmark Run
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
Intel Core i7-1065G7 CPU 1.30GHz, 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100
  [Host]     : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
384.8 ms
7.50 ms
10.99 ms
1.00
0.00
711 B

V72WipStartupTime
313.6 ms
5.26 ms
4.66 ms
0.81
0.03
218 B

WinPSStartupTime
211.2 ms
3.48 ms
2.91 ms
0.55
0.02
711 B

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
391.4 ms
7.81 ms
10.16 ms
1.00
0.00
711 B

V72WipStartupTime
317.2 ms
6.10 ms
6.78 ms
0.81
0.03
218 B

WinPSStartupTime
215.3 ms
4.21 ms
7.48 ms
0.55
0.02
60 B

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
394.4 ms
7.57 ms
9.84 ms
1.00
0.00
60 B

V72WipStartupTime
316.2 ms
6.25 ms
6.14 ms
0.81
0.02
218 B

WinPSStartupTime
217.8 ms
3.87 ms
5.29 ms
0.55
0.02
711 B

Benchmark program
using System.Diagnostics;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

namespace benchmark
{
    class Program
    {
        static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<Benchmark_pwsh_startup>();
        }
    }

    [DisassemblyDiagnoser(printSource: true)]
    public class Benchmark_pwsh_startup
    {
        private ProcessStartInfo v710, v72wip, winps;

        [GlobalSetup]
        public void Setup()
        {
            v710 = new ProcessStartInfo(@"C:\Program Files\PowerShell\7\pwsh.exe", "-noprofile -c echo 1");
            v72wip = new ProcessStartInfo(@"C:\arena\repos\PowerShell\src\powershell-win-core\bin\Release\net5.0\win7-x64\publish\pwsh.exe", "-noprofile -c echo 1");
            winps = new ProcessStartInfo(@"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe", "-noprofile -c echo 1");
        }

        [Benchmark(Baseline = true)]
        public void V710StartupTime()
        {
            Process.Start(v710).WaitForExit();
        }

        [Benchmark]
        public void V72WipStartupTime()
        {
            Process.Start(v72wip).WaitForExit();
        }

        [Benchmark]
        public void WinPSStartupTime()
        {
            Process.Start(winps).WaitForExit();
        }
    }
}

PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[ ] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[ ] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[x] None
OR
[ ] Experimental feature(s) needed

[ ] Experimental feature name(s):

User-facing changes

[x] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[x] N/A or can only be tested interactively
OR
[ ] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[x] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and enumerated concerns in the summary. This may include:

Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode (which runs in a different PS Host).
Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

Eh, I've always said that MSFT developers shouldn't be given the latest hardware :-)
When I started # 14268 my first desire was to follow the EarlyStartup.Init() pattern and move slow operations to background threads but after testing on my old laptop, I did not see any performance gain.
From this I concluded that this approach is not well applicable if we want to be fast on high-load systems and  low-power equipment (IoT, Wasm and so on).
(We already create many CPU-bound threads and adding new ones only makes sense for hardware with many cores - this is definitely not my laptop :-) )
So I did some PRs with direct performance improvements. @daxian-dbw Could you please consider them first? Even if we decide to move some from them to background threads, it will make them also lighter and faster.
I especially have doubts about GetPolicySetting(). It is slow totally because of newtonsoft.json. It would be more promising for us to migrate to .Net Json and see the result. It will definitely be better. If we see problems we can ask the .Net team, which continues to actively work on it.
(Perhaps we need take new look on EarlyStartup.Init().)

@iSazonov Thanks for raising your concerns. I run the benchmark on a 2017-year 4 logical core machine (so not a latest hardware 😄), and I still observe over 10% improvements in the startup.
Below are some of the results:
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
Intel Core i5-7300U CPU 2.60GHz (Kaby Lake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.100
  [Host]     : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
429.1 ms
5.67 ms
5.30 ms
1.00
0.00
218 B

V72WipStartupTime
372.6 ms
5.68 ms
5.03 ms
0.87
0.02
218 B

WinPSStartupTime
254.2 ms
3.12 ms
2.77 ms
0.59
0.01
40 B

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
427.8 ms
6.32 ms
5.60 ms
1.00
0.00
218 B

V72WipStartupTime
377.5 ms
5.18 ms
4.33 ms
0.88
0.02
40 B

WinPSStartupTime
252.5 ms
2.90 ms
2.57 ms
0.59
0.01
40 B

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
431.0 ms
6.08 ms
5.69 ms
1.00
0.00
40 B

V72WipStartupTime
373.3 ms
5.20 ms
4.61 ms
0.87
0.01
160 B

WinPSStartupTime
269.0 ms
5.00 ms
7.18 ms
0.62
0.02
699 B

@iSazonov Thanks for raising your concerns. I run the benchmark on a 2017-year 4 logical core machine (so not a latest hardware 😄), and I still observe over 10% improvements in the startup.

2017? :-) How would you classify my 2011 laptop? :-) Note that it is significantly newer than Windows 7 which still runs PowerShell 7. :-)
Seriously, my suggestion is to first make direct corrections, open cases in .Net if necessary (they already shared plans to speed up runtime initialization in 6.0 milestone), and only before next RC to evaluate how much we need to do additional threads.

I agree that leveraging worker thread for parallel processing won't result in much difference in high-load system or low-power/low-rank equipment. However, that's not a reason to not use it, because it's obvious that many more users will benefit from it.
Parallel processing is not the answer to everything (Amdahl's law), but it surely is a good tool as long as you use it cautiously -- be careful about the number of threads you spin up and be careful about what work loads to process in parallel, so as to avoid contention/blocking as much as possible.
Also, it doesn't contradict with the work you have been doing or the improvement that .NET 6 will bring to us, does it?

2017? :-) How would you classify my 2011 laptop? :-) Note that it is significantly newer than Windows 7 which still runs PowerShell 7.

I don't think the OS version matters too much. It's more about how many logical cores the machine has, and of course, how much work load the machine is having at the moment.
But can you please run the benchmark on your 2011 machine share the results? I would like to see if there is any penalty from context switching on a low-power machine. Please make sure you build my branch with Start-PSBuild -Clean -ReleaseTag v7.2.0 -CrossGen -Configuration Release, and then make sure its module cache is populated before running the benchmark. That can be done by 1) start the generated pwsh.exe, 2) run a few commands, 3) wait for a minute or two, for safety.

But can you please run the benchmark on your 2011 machine and share the results?

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.1198 (1909/November2018Update/19H2)
Intel Core i5-2410M CPU 2.30GHz (Sandy Bridge), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.100
  [Host]     : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR <IP_ADDRESS>904, CoreFX <IP_ADDRESS>904), X64 RyuJIT

Method
Mean
Error
StdDev
Ratio
RatioSD
Code Size

V710StartupTime
686.8 ms
7.93 ms
6.19 ms
1.00
0.00
218 B

V72WipStartupTime
682.2 ms
10.90 ms
9.66 ms
0.99
0.02
218 B

WinPSStartupTime
365.0 ms
4.85 ms
4.30 ms
0.53
0.01
218 B

It is stable result in some runs.
This seem to confirm that moving works in background threads is not so good for startup scenario.
As I said in my PR, it's much better to ask the ApplicationInsights owners to improve this scenario and do not create additional threads.

This seem to confirm that moving works in background threads is not so good for startup scenario.

Nah, this is actually not bad at all, as it shows no penalty (context switching and etc.) on your 2011 laptop.
Again, leveraging parallel processing doesn't contradict with the work that improves individual code paths. If it's handled correctly, they amplify the final result.

it shows no penalty (context switching and etc.)

I say not about a penalty. From my experience Windows works well on 2 cores (including applications) and I guess we will see the same result for our test on more slow hardware.
I say about (1) roadmap and (2) how many benefits we will get from parallelization in the startup scenario.
It is well known that replacing a bad algorithm with a better one can have the greatest effect. Also, direct fixes are generally more effective than workarounds. For example, in our case, we can get a benefit of 20% from direct fixes, and from the subsequent parallelization only 1%, even on fast equipment because the code is already fast.
So my suggestion for a roadmap is to work on direct fixes here and in other repositories before RC and then investigate parallelization.
This will simplify the final performance checks before 7.2 release.

Don't think parallelization as a workaround, think it as a tool. There is no good or bad about a tool, but only right way or wrong way to use it. As I said above, the right way is "be careful about the number of threads you spin up and be careful about what work loads to process in parallel, so as to avoid contention/blocking as much as possible."
An example as to the work load aspect:

Today, DefaultInitialSessionState = CreateDefault2() is called in ManagedEntrance.Start, but the created InitialSessionState instance is not used until in DoCreateRunspace. I did some measurement on my "latest" machine, CreateDefault2() taske ~36 ms, and then it takes ~40 ms for the code flow to reach the first use of DefaultInitialSessionState in DoCreateRunspace.

Meanwhile, the CreateDefault2() call is pretty selfcontained, no synchronization is needed when having it run on a worker thread.

Together, they make CreateDefault2() a good candidate work-load to run as a task, as of today.

For example, in our case, we can get a benefit of 20% from direct fixes, and from the subsequent parallelization only 1%, even on fast equipment because the code is already fast.

This sounds like a pretty extragerated hypothetical example :) But, if we cannot find the right workload for parallelization, then that just means we should not use that tool.
I want to emphsize again, parallelization doesn't contradict with the work that improves individual code paths. If it's handled correctly, they amplify the final result.

So my suggestion for a roadmap is to work on direct fixes here and in other repositories before RC and then investigate parallelization.

Yeah, I agree to not rush in this PR. As I mentioned in our discussion at https://github.com/PowerShell/PowerShell/pull/14281#issuecomment-738449342, more work and testing is needed to "fine-tune the decisions like how many tasks to use, and what work load shall be done in tasks." And decisions will change as more perf work on our side and .NET side get in.
But I believe we cannot wait till the RC timeframe, as changes in this PR (or similar PRs) need sufficient bake time.

But I believe we cannot wait till the RC timeframe, as changes in this PR (or similar PRs) need sufficient bake time.

I mean last Preview before RC is good time to check, investigate and fix performance issues.

Today my test show regression up to 20% because ... slow DNS request from ApplicationInsights! I saw this previously. So any network issue delays PowerShell startup due to ApplicationInsights - good argument to move it in a different thread.

@iSazonov Are you sure it's due to ApplicationInsights? Did you try disabling the telemetry to prove that's the cause? As we discussed in https://github.com/PowerShell/PowerShell/pull/14281#issuecomment-738449342, ApplicationInsight doesn't send telemetry packets synchronously.
What you saw looks to me more likely a variant of #10983 -- DNS issue may cause the certificate security check to block.

@iSazonov Are you sure it's due to ApplicationInsights? Did you try disabling the telemetry to prove that's the cause? As we discussed in #14281 (comment), ApplicationInsight doesn't send telemetry packets synchronously.

I saw DnsResolver in ApplicationInsights constructor.
It would be easier to investigate if the code was open source. :-)

I'm pretty sure ApplicationInsight sends telemetries in an asynchronous way. We are using the default configuraiton with InMemoryChannel, whose behavior is (see "more telemetry channels" for details):

InMemory channel Microsoft.ApplicationInsights.Channel.InMemoryChannel is a lightweight loosy telemetry channel that is used by default. It will batch data and initiate sending every SendingInterval (default is 30 seconds) or when number of items exceeded MaxTelemetryBufferCapacity (default is 500. It also will not retry if it failed to send telemetry.

When using in-memory channel, the Flush call is synchronous. It forces data to be sent immediately.

Sources https://github.com/microsoft/ApplicationInsights-dotnet :-)


[NavigationAdvancedSample] Even top level destinations show up/back arrow

With update to navigation 2.4.0-alpha01, even top level destinations show the up arrow.
Tested on Samsung Galaxy S8. Android 9.

Seems the kotlin version of ActionBarOnDestinationChangedListener always sets the actionBar.setDisplayHomeAsUpEnabled to true.
Below diff is from https://android.googlesource.com/platform/frameworks/support/+/5b19fddb91b129168429b3cbf77323414d638b03^!/#F1
-
-    @Override
-    protected void setNavigationIcon(Drawable icon,
-            @StringRes int contentDescription) {
-        ActionBar actionBar = mActivity.getSupportActionBar();
-        if (icon == null) {
-            actionBar.setDisplayHomeAsUpEnabled(false);
-        } else {
-            actionBar.setDisplayHomeAsUpEnabled(true);
-            ActionBarDrawerToggle.Delegate delegate = mActivity.getDrawerToggleDelegate();
-            delegate.setActionBarUpIndicator(icon, contentDescription);
-        }
-    }
-
+    override fun setNavigationIcon(icon: Drawable, @StringRes contentDescription: Int) {
+        val actionBar = activity.supportActionBar
+        require(actionBar != null) {
+            "Activity $activity does not have an ActionBar set via setSupportActionBar()"
+        }
+        actionBar.setDisplayHomeAsUpEnabled(true)
+        val delegate = activity.drawerToggleDelegate
+        require(delegate != null) {
+            "Activity $activity does not have an DrawerToggleDelegate set"
+        }
+        delegate.setActionBarUpIndicator(icon, contentDescription)
+    }

Seems the parameter icon was set to non-null initially and then later made nullable in https://android.googlesource.com/platform/frameworks/support/+/d5f4ba078465516da538bcf96042a33e0cb6edd8/navigation/navigation-ui/src/main/java/androidx/navigation/ui/ActionBarOnDestinationChangedListener.kt. But the null check was not added back.

@ammargitham Guess there are some unexpected bugs in "alpha" version. waiting for "beta"

Update the navigation version to 2.4.0-alpha03 (latest as of 6/23/21)


chore(deps): update common-dev-assets digest to 3e33725

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
da6272e -> 3e33725

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Snapshot wiki for each release

There is an issue with names with ? in them on windows so they will need to be renamed and their references changed.

Wiki points to online pages which each release may change. Will snapshot wiki each major release instead,


Upgrade controller-runtime, ginkgo, gomega.

This didn't update any go files?

nope, because this repo doesn't keep a vendor/ dir.


Error when executed from Command prompt

Hi Muthu,
Thanks for  the php code.
When running from command prompt, I have tried PHP version 5.6.32-nts-Win32-VC11-x86 as well as 7.1.11-Win32-VC14-x86 and both give me an error "Call to undefined function curl_init()".
If I place the same PHP file on my host and run it from a browser it works fine.
Any ideas why it does not work from Command prompt.

Thanks
Hil

Hi, Looks like curl not installed or enabled for cli. You have enable curl on command line in order to use it.
Thanks.

Thanks Muthu.
I managed to enable it in php.ini with extension=php_curl.dll but this time I get "Use of undefined constant DEBUG_LENDING_API".

Thanks

Make sure you didn't commented following line on index.php file
/*** set false to receive only datas ***/
define("DEBUG_LENDING_API", false);
If you want to enable debugging, set value from false into true.

Hi Muthu,
You are right. I did not have the line. I was probably using an earlier version of the script. I created the PHP file again and this time the error is gone.
Although when I try to get anything  (For example get_Balance or  get_notes) it returns nothing.
I have attached a copy of the php.ini and  index.php I am using.
index.php
php.ini
Again I am executing PHP from a windows Command line (Elevated as well as non-elevated).
OS = Windows 10
I have tried both versions of PHP (PHP 7.1.11 and PHP 5.6.32)
Also if it will help I set define("DEBUG_LENDING_API", enable); and here is the output:
log.txt
Thanks

define("DEBUG_LENDING_API", true);
Put the above line and make sure your lending API credentials are right.
Still if you can't get output, send me credentials of your server, I will
debug the script. Because on my environment I will not get the same error.
Keep coding... never give up.
On 12/11/2017 9:19 PM, "hdsouza"<EMAIL_ADDRESS>wrote:

Hi Muthu,
You are right. I did not have the line. I was probably using an earlier
version of the script. I created the PHP file again and this time the error
is gone.
Although when I try to get anything (For example get_Balance or get_notes)
it returns nothing.
I have attached a copy of the php.ini and index.php I am using.
index.php
https://github.com/muthu-kc/lendingclub-php-script/files/1464778/index.txt
php.ini
https://github.com/muthu-kc/lendingclub-php-script/files/1464779/php.txt
Again I am executing PHP from a windows Command line (Elevated as well as
non-elevated).
OS = Windows 10
I have tried both versions of PHP (PHP 7.1.11 and PHP 5.6.32)
Also if it will help I set define("DEBUG_LENDING_API", enable); and here
is the output:
log.txt
https://github.com/muthu-kc/lendingclub-php-script/files/1464826/log.txt
Thanks
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/muthu-kc/lendingclub-php-script/issues/1#issuecomment-343746319,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHjnGMg4e4VzNnNoBSsmB8Q-CmqobRUzks5s1xOigaJpZM4QUIm-
.

Hi Muthu,
i added define("DEBUG_LENDING_API", true); and here is the output
log.txt
Also I am not running the  script from a server. As mentioned  earlier I am running it from Windows 10 system via a command prompt.
If you run your PHP file from a windows command prompt you will see what i mean.
Could it be that JSON needs to be enabled/configured in PHP?
Thanks for your help
Hil

Sorry, I don't have windows on my end. You have to debug it on your end.
the log doesn't have much data. Make sure your API credentials and account
numbers are correct.
On Sun, Nov 12, 2017 at 11:22 PM, hdsouza<EMAIL_ADDRESS>wrote:

Hi Muthu,
i added define("DEBUG_LENDING_API", true); and here is the output
log.txt
https://github.com/muthu-kc/lendingclub-php-script/files/1464929/log.txt
Also I am not running the script from a server. As mentioned earlier I am
running it from Windows 10 system via a command prompt.
If you run your PHP file from a windows command prompt you will see what i
mean.
Could it be that JSON needs to be enabled/configured in PHP?
Thanks for your help
Hil
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/muthu-kc/lendingclub-php-script/issues/1#issuecomment-343754429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHjnGGm-HItRqZjmz7qqVlIqdLPDb9_mks5s1zBDgaJpZM4QUIm-
.

Hi Muthu,
I got it to work by placing the PHP file on my HOST.
It should the account balance as well as the notes in my account.
But If I try to buy a note I get "Internal Server Error<PHONE_NUMBER>610"
Does the script work on the  Secondary Market?
For example lets say I were to buy this note from the secondary market.
https://www.lendingclub.com/foliofn/browseNotesLoanPerf.action?showfoliofn=true&loan_id=79154140&order_id=115222867&note_id=129717636
Here is the PHP file I am using.
lc_buynotes.php
Thanks
Hil

You have to figure out this on your end. Thank you.

All I was asking was is it working for the secondary market.
If it does not work anymore, just say so.

Hi, I don't know much about lending api. I just created this script for one
of my client before one year. If its working, use it. If it doesn't you may
need some other script.
Thank you.
On 15/11/2017 4:09 AM, "hdsouza"<EMAIL_ADDRESS>wrote:

All I was asking was is it working for the secondary market.
If it does not work anymore, just say so.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/muthu-kc/lendingclub-php-script/issues/1#issuecomment-344423586,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHjnGPo_BlJDVWUUZ6gqyjQ11rd75y93ks5s2hacgaJpZM4QUIm-
.

Thats cool. Would you be interested in doing it as a project.
If interested you could email (hdsouza @ yahoo) me cost or post here.


trying to index attributes which are not meant to be indexed?

Hi folks,
We use include Elasticsearch::Model in order to index a rails model into an elasticsearch index (indice) and I've come across a problem which I'm not sure should be raised as a bug. Before I do so, I'd like to confirm my understanding, as well as validate that my troubleshooting is correct.
Seamless integration with models
When include Elasticsearch::Model is used in a model, the as_indexed_json method is called in order to automatically send/index a model into elasticsearch when the model .save method is called. The model settings do... mappings do... allow to define WHICH attributes are actually indexed/how they are analysed, etc.
Problem •1
Although of a lesser importance, but I seem to have confirmed that if an index/indice is not created before hand, the .save method will automatically create the index, ignoring the settings defined (namely dynamic: 'false' that I'm using) and as a result ES automatically indexes all model attributes. Manually calling the model.__elasticsearch__.create_index! first works around this.
This also only seems to be a problem when a model is created without the index present. if I use model.import without the index present, settings are observed. Or am I crazy?
Configuration I'm trying to implement;

dynamic: 'false';
_source: false;

Problem •2
The main problem I have here is that when I do model.save and only a single attribute was changed, the gem is 'smart enough' to only send the attribute changed to the existing document in ES, which in turns throws up a tantrum because it has no clue about that attribute (thanks to my config#2 -- source:false, WITH source, it doesn't index, but doesn't whinge). The gem however is ignoring the settings configuration in this case, not unlike problem •1.
When calling model.__elasticsearch__.index_document: the whole document is sent (-d {"id":"c479a116-d78c-4957-8591-c791415d486f"})
curl -X PUT -H 'User-Agent: elasticsearch-ruby/8.16.0; elastic-transport-ruby/8.3.5; RUBY_VERSION: 3.3.4; linux aarch64, x-elastic-client-meta: es=8.16.0,rb=3.3.4,t=8.3.5,fd=1.10.3,ty=1.4.1, Content-Type: application/json' 'http://localhost:9200/companies/_doc/c479a116-d78c-4957-8591-c791415d486f?pretty' -d '{"id":"c479a116-d78c-4957-8591-c791415d486f","legal_name":......

When calling model.save with only the .packaging attribute changed: a partial document is sent (-d '{"doc":{"packaging":)
curl -X POST -H 'User-Agent: elasticsearch-ruby/8.16.0; elastic-transport-ruby/8.3.5; RUBY_VERSION: 3.3.4; linux aarch64, x-elastic-client-meta: es=8.16.0,rb=3.3.4,t=8.3.5,fd=1.10.3,ty=1.4.1, Content-Type: application/json' 'http://localhost:9200/companies/_update/c479a116-d78c-4957-8591-c791415d486f?pretty' -d '{"doc":{"packaging":[{"id":"056efb26-ad3a-4e43-a7b9-242dba99ed1b"......

Is my understanding correct? And if so, can I get the gem to observe the model settings before sending? or always send the whole document?.

Ok, doing more investigation I seem to have answered Problem •1;
I was not crazy 😅 and just discovered this is an elastic behaviour to auto create index when missing, a setting activated by default. Nothing to do with the gem. As documented.
When turning this off, the model creation (when the underlying index doesn't exist) correctly returns a index_not_found_exception, with the genius message 😍 no such index [companies] and [action.auto_create_index] is [false].
Now onto the Problem •2. I have confirmed that with/without _source enabled, the server 'never' likes receiving attributes which are not indexed/mapped. For all intents and purposes, this renders the documented mappings dynamic: 'false' do unusable given any attribute is sent to ES when changed/updated?
Any guidance/help in confirming my thinking and pointing me in the right direction would be appreciated 👍


Fix log message

Not actually an error message, just printing out the determined mode.

Ok this should hopefully be good now 😛

NM something is up with the build


Matrix exponential

From @chjackson on June 8, 2014 13:47
Feature request for a matrix exponential function, as raised recently on the stan-users list.
Copied from original issue: stan-dev/stan#680

from Michael Betancourt on stan-users, 06/07/2014:

The problem is indeed in the derivatives — unless the matrix is symmetric the derivatives
of the matrix exponential are either ill-defined or very hard to compute.  The first order
derivatives have actually been computed for Riemannian HMC but they are a mess, and
likely won’t be incorporated into a matrix_exp function anytime soon.
If you add this as a feature request at https://github.com/stan-dev/stan/issues then it
won’t get lost in the mailing list, but no promises on when it might ever be implemented.

from Chris Jackson on stan-users, 06/07/2014:

I think it'd be lovely to have matrix exponentials in Stan.   I would use this for continuous-time Markov models.  I know it's notoriously hard to calculate, but JAGS has a module for it (using Pade approximation) and the "expm" R package has a variety of algorithms, with a C API.   Not sure about calculating the derivatives, I suspect it would be difficult if the matrix is not diagonalizable - though in the class of models I'd use it for, it generally would be.

from Sean O'Riordain on stan-users, 06/07/2014:

In case the developers are not already aware of it... there is a nice paper on calculating matrix exponentials...
"Nineteen Dubious Ways to Compute the Exponential of a Matrix, Twenty-Five Years Later∗", SIAM Review Vol 45, No.1, pp3-000
Seán

From @wds15 on May 27, 2015 11:16
The matrix exponential would be extremely relevant for analytic solvable ODE systems when the solution is then exp(A*t). Here A is the coefficient matrix of the ODE system at hand. The catch is that the matrix must be evaluated repeatedly. The ODE systems which matter for PK systems fall into the class of "almost non-negative", i.e. only the diagonal entries are negative and all others are positive. These matrices apparently have some nice properties with respect to the solution of exp(At) which are documented here:
http://www.mscf.uky.edu/~qye/reports/expENM.pdf
I hope this helps for an implementation in Stan.

From @cdriveraus on May 11, 2015 16:51
I've been working on an R package for estimating continuous time dynamic models, it would be great to be able to include an option to estimate using STAN. As far as I can see the only thing missing is the matrix exponential...

From @jpritikin on March 7, 2015 13:14
The most recent method is due to Higham,
http://epubs.siam.org/doi/abs/10.1137/090768539
It's already implemented in Eigen/unsupported

Often it's the action of the exponential on a vector or which is desired, e.g. case of exponential schemes for ODEs, SDEs where the stepping scheme has the form exp(tA) v, [1] . Higham also has a recent paper on approximating this through a series of matrix-vector multiplications, which may be more friendly to autodiff than other approximations, manuscript is [here] (http://eprints.ma.man.ac.uk/1591/01/covered/MIMS_ep2010_30.pdf) and the author's MATLAB implementation here.
[1] evaluating the action directly is especially beneficial where A can be sparse, as is often the case for an ODE system Jacobian, and its exponential will be dense.

@maedoc Thanks for the ref.  Looks like something we were discussion before on the mailing list.  I don't know much about this, but if someone can get this working in the math library, it'll be easy to add to the language.

Hi,
as mentioned above one use case is for ODEs where we have exp(t A) and we need the derivative w.r.t time. This is calculated by
d/dt exp(t A) = exp(t A) A
Since the matrix exponential is implemented in Eigen, as stated above, would it not be easy to implement this here?
I am new to stan::math, so I would be glad about a pointer how to achieve this. (define a new function matexp(t,A) function with its gradient w.r.t. t).

It's an involved process:
https://github.com/stan-dev/stan/wiki/Contributing-New-Functions-to-Stan
Note that the matrix exponential is in Eigen-unsupported, not in
the supported part of Eigen.  We currently use the Eigen-unsupported
FFT, so that's not a dealbreaker, just a warning that it will require
more testing.

Bob

On Aug 10, 2016, at 1:54 PM, Georg Martius<EMAIL_ADDRESS>wrote:
Hi,
as mentioned above one use case is for ODEs where we have exp(t A) and we need the derivative w.r.t time. This is calculated by
d/dt exp(t A) = exp(t A) A
Since the matrix exponential is implemented in Eigen, as stated above, would it not be easy to implement this here?
I am new to stan::math, so I would be glad about a pointer how to achieve this. (define a new function matexp(t,A) function with its gradient w.r.t. t).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Charles Margossian and I (but mainly Charles) are already working on adapting the Eigen/unsupported code. Since the gradient w.r.t. the matrix A is pretty messy in the general case, the plan is to revise the code to autodiff the algorithm.
Bill

Neat.  What do you need to do to revise the algorithm
for autodiff?  Someone was offering to pay someone a year
or so ago to do this as part of a grant, but that offer has
probably expired.
Several people have pointed us to this paper:
A NEW SCALING AND SQUARING ALGORITHM FOR THE MATRIX EXPONENTIAL∗
AWAD H. AL-MOHY† AND NICHOLAS J. HIGHAM†
SIAM J. MATRIX ANAL. APPL.
I'm not sure if the Eigen implementation is related.

Bob

On Aug 11, 2016, at 2:46 PM, Bill Gillespie<EMAIL_ADDRESS>wrote:
Charles Margossian and I (but mainly Charles) are already working on adapting the Eigen/unsupported code. Since the gradient w.r.t. the matrix A is pretty messy in the general case, the plan is to revise the code to autodiff the algorithm.
Bill
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

The Eigen implementation cites Nicholas J. Higham, "The scaling and squaring method for the matrix exponential revisited," SIAM J. Matrix Anal. Applic., 26:1179-1193, 2005, an earlier article by the same author. So the article you mentioned is related to the Eigen version but might reflect some improvements in the algorithm. We'll look into that...
Bill

I posted this in a thread on stan-users a while ago, maybe it helps someone:
I've written a couple of matrix exponential variants within STAN. I guess it's slower than an ideal solution would be, but still works... here is the code for the functions. expmt is the taylor expansion approach (currently set to 50 iterations) and relies on the other posted functions, while expmp is self contained and uses pade approximants, it is based on the R function expm.Higham08. The pade approach is more efficient and stable, but I couldn't think of a lazy way to incorporate the necessary matrix / vector of values so just put it in as data using my wrapper function for continuous time models.
functions{
real factorial(int m);
real factorial(int m) {
if (m == 0)
return 1;
else
return (m * factorial(m-1));
}
matrix matrix_pow(matrix a, real n);
matrix matrix_pow(matrix a, real n) {
if (n == 0){
return diag_matrix(rep_vector(1, rows(a)));
}
else
return a * matrix_pow(a, n - 1);
}
matrix expmt(matrix mat, real t){
matrix[rows(mat),rows(mat)] out;
out<-diag_matrix(rep_vector(1.0, rows(mat)));
for(i in 1:50){
out<-out+ matrix_pow(mat,i) * t^i /factorial(i);
}
return out;
}
matrix expmp(matrix A, matrix padeC, vector padeCbig){
int n;
real nA;
real colsum;
int l;
matrix[4,10] C;
vector[4] t;
matrix[rows(A),rows(A)] I;
matrix[rows(A),rows(A)] P;
matrix[rows(A),rows(A)] U;
matrix[rows(A),rows(A)] V;
matrix[rows(A),rows(A)] X;
vector[14] Cbig;
real s;
real si;
matrix[rows(A),rows(A)] B;
matrix[rows(A),rows(A)] B2;
matrix[rows(A),rows(A)] B4;
matrix[rows(A),rows(A)] B6;
matrix[rows(A),rows(A)] A2;
si <- 0;
C <- padeC;
Cbig <- padeCbig;
n <-rows(A);
if(n != cols(A)) print("expmp: Matrix not square!")
if (n <= 1) X <- exp(A);
else{
// nA <- Matrix::norm(A, "1")
nA <- 0;
for(coli in 1:n){
colsum<-0;
for(rowi in 1:n){
colsum<-colsum+fabs(A[rowi,coli]);
}
if(colsum > nA) nA <- colsum;
}
I <- diag_matrix(rep_vector(1,n));
if (nA <= 2.1) {
t[1] <- 0.015; t[2]<- 0.25; t[3]<- 0.95; t[4]<- 2.1;
//l <- which.max(nA <= t)
for(ti in 1:4){
if(l==0){
if(nA <= t[ti]) l <- ti;
}
}
A2 <- A * A;
P <- I;
U <- C[l, 2] * I;
V <- C[l, 1] * I;
for (k in 1:l) {
P <- P * A2;
U <- U + C[l, (2 * k) + 2] * P;
V <- V + C[l, (2 * k) + 1] * P;
}
U <- A * U;
X <- inverse(V - U) * (V + U);
}
else {
s <- log2(nA/5.4);
B <- A;
if (s > 0) {
s <- ceil(s);
B <- B/(2^s);
}
B2 <- B * B;
B4 <- B2 * B2;
B6 <- B2B4;
U <- B(B6*(Cbig[14] * B6 + Cbig[12] * B4 + Cbig[10] * B2) + Cbig[8] * B6 + Cbig[6] * B4 + Cbig[4] * B2 + Cbig[2] * I);
V <- B6*(Cbig[13] * B6 + Cbig[11] * B4 + Cbig[9] * B2) + Cbig[7] * B6 + Cbig[5] * B4 + Cbig[3] * B2 + Cbig[1] * I;
X <- inverse(V - U) * (V + U);
if (s > 0) {
while (si < s){
si <- si + 1;
X <- X * X;
}
}
}
}
return X;
}
matrix kron_prod(matrix mata, matrix matb){
int m;
int p;
int n;
int q;
matrix[rows(mata)rows(matb),cols(mata)cols(matb)] C;
m<-rows(mata);
p<-rows(matb);
n<-cols(mata);
q<-cols(matb);
for (i in 1:m){
for (j in 1:n){
for (k in 1:p){
for (l in 1:q){
C[p(i-1)+k,q(j-1)+l] <- mata[i,j]*matb[k,l];
}
}
}
}
return C;
}
}
  data {

matrix[4,10] padeC;
vector[14] padeCbig;
}
and this goes in the stan data object:
padeC=rbind(c(120, 60, 12, 1, 0, 0, 0, 0, 0, 0), c(30240,
15120, 3360, 420, 30, 1, 0, 0, 0, 0), c(17297280,
8648640, 1995840, 277200, 25200, 1512, 56, 1, 0,
0), c(17643225600,<PHONE_NUMBER>,<PHONE_NUMBER>, 302702400,
30270240, 2162160, 110880, 3960, 90, 1)),
padeCbig= c(64764752532480000,<PHONE_NUMBER>6240000,<PHONE_NUMBER>897600,
1187353796428800,<PHONE_NUMBER>64000,<PHONE_NUMBER>1600,
670442572800,<PHONE_NUMBER>0,<PHONE_NUMBER>, 40840800,
960960, 16380, 182, 1)

@cdriveraus: Thank you for sharing your code!
@bob-carpenter:  should we move this thread to the dev-list?
Eigen-unsupported doesn't compute the action of the Matrix Exponential, so it looks like we'll need to supplement it with/ use another algorithm. Possibly the matlab code @maedoc referenced, or Expokit.
To revise the Eigen-unsupported algorithm for AD, my first attempt is to template as much of the code as possible. I'm trying to understand how to do this for functions in <cmath>. For one, I cannot find the source code on my local machine. A bit of research suggests this is because the function is implemented in the hardware... which is a new concept to me.
Looking at how Stan handles sin() , I'm guessing I need to write the gradient of <cmath> functions "by hand". The function causing me trouble is frexp(), and since one of its outputs is in an integer, the gradient is not defined. I'm trying to figure out whether we need that gradient at all, or ways to skip this step gradient-wise, but in any case, frexp() needs to be overloaded for AD variables.

Bob

On Aug 11, 2016, at 10:30 PM, Charles Margossian<EMAIL_ADDRESS>wrote:
@cdriveraus: Thank you for sharing your code!
@bob-carpenter: should we move this thread to the dev-list?

We're trying to have general discussions on the mailing
lists, summarize designs on the Wiki, and then only move
to issues when things are concrete enough to implement
and then restrict discussions to implementations.

Eigen-unsupported doesn't compute the action of the Matrix Exponential, so it looks like we'll need to supplement it with/ use another algorithm. Possibly the matlab code @maedoc referenced, or Expokit.
To revise the Eigen-unsupported algorithm for AD, my first attempt is to template as much of the code as possible. I'm trying to understand how to do this for functions in . For one, I cannot find the source code on my local machine. A bit of research suggests this is because the function is implemented in the hardware... which is a new concept to me.

All of the cmath functions are coded with full templating in
Stan with gradients directly.  They're in stan-dev/math in
directory stan/math/rev/scal/fun (with forward mode autodiff
in path replacing "rev" with "fwd").

Looking at how Stan handles sin() , I'm guessing I need to write the gradient of  functions "by hand".

Already done.  See above.

The function causing me trouble is frexp(), and since one of its outputs is in an integer, the gradient is not defined.

That's why it's not in stan/math!  There's no gradient and it
breaks continuous differentiability in the general case.  But if
it's used in an algorithm, it's possible the entire algorithm's
result is differentiable.

I'm trying to figure out whether we need that gradient at all, or ways to skip this step gradient-wise, but in any case, frexp() needs to be overloaded for AD variables.

That's easy to do---look at how the step functions like round()
are implemented (in retrospect, we probably shouldn't have included
those, either, because they're dangerous --- there's a big warning
on all these non-continuously-differentiable functions in the manual).

@bob-carpenter: Thank you for the guidance.
After writing frexp() for rev and fwd variables, I got a working prototype. I tried it on a few matrices and it returned the right output. I tested derivatives with diagonal matrices (I picked an easy case so that I could check the calculations by hand, but obviously more difficult tests will have to be passed) -- at least we know frexp doesn't prevent AD!
I'll write unit tests, and upload the code. The end result should be a function which, given a matrix A, returns  exp(A). The function will call <unsupported/Eigen/MatrixFunctions> and a template version of computeUV (the original function is defined in MatrixFunctions). In addition, we'll need to add the rev and fwd version of frexp (in rev, fwd, and prim) to stan/math.
Once this is done, I'll focus on the action of the matrix exponential on a vector.

That sounds great.  I've been told matrix exponentials are
very numerically sensitive, so any help in the doc on where
it'll work and where it won't will be appreciated.    We'd
rather throw an exception than return the wrong answer.

There are examples in Moler & Van Loan, 2003 ("Nineteen Dubious Ways to Compute the Exponential of a Matrix, ...") designed to make non-robust algorithms fail.
The bigger concern is testing derivatives. I could compare with what other programs return (MatLab seems the place to go to).
I think I should be able to work out some cases by hand. My idea is to write down a diagonal matrix and then change the basis to make it non-diagonal. That way I can deal with a simple derivable form, while Stan is tasked with a more challenging and general form. Ok, sounds good theory, but we'll see how the implementation goes.
I'll also compare the derivatives from @cdriveraus 's code to those I obtain. Lastly, I'll try using matrix exponentials to solve the simple harmonic oscillator DE.

We typically evaluate derivatives by comparing to finite
differences.  External validation is a bonus.
If you can include some of the robustness tests from
that paper, it'd be great.  We can edit out the ones
that don't pass and make them future to-do items.

We typically evaluate derivatives by comparing to finite
differences.

Right, makes sense.
I'm having the following issue with computeUV. The function is declared as a private member of the MatrixExponential class in eigen/unsupported. In the prototype I have, I forward declare computeUV for stan::math::var inside the class definition, and define computeUV in main.cpp. Trying to write the function in a header file causes compiling errors:

variable has incomplete type
namespace math does not enclose namespace matrixExponential (this occurs when I define ComputeUV in stan::math).

I see two solutions:

Hack in the code of eigen/Unsupported and overload computeUV there. The function will not be in stan::math + we would have to edit an "exterior" library. On the other hand, this should be little work.
Rewrite a new matrixExponential class, or rewrite the functions in matrixExponential, and include them in stan::math. This is what's been done for Eigen::Matrix, right? Specifically, I am looking at the example of block.hpp which has been turned from a class function into a function that returns a matrix.

Is (1) acceptable? Is there are more elegant approach than (2)?

On Aug 15, 2016, at 11:01 PM, Charles Margossian<EMAIL_ADDRESS>wrote:

...

I'm having the following issue with computeUV. The function is declared as a private member of the MatrixExponential class in eigen/unsupported. In the prototype I have, I forward declare computeUV for stan::math::var inside the class definition, and define computeUV in main.cpp.

You shouldn't edit the Eigen code directly.  You need
to put the var version inside stan::math in order for argument-dependent
lookup (ADL) to work properly.  See:
https://en.wikipedia.org/wiki/Argument-dependent_name_lookup

Trying to write the function in a header file causes compiling errors:
• variable has incomplete type

That comes from trying to use a class before it's fully defined.
See, e.g.,
http://stackoverflow.com/questions/6349822/incomplete-type-in-class

• namespace math does not enclose namespace matrixExponential (this occurs when I define ComputeUV in stan::math).

Right, should be standalone.

I see two solutions:

Hack in the code of eigen/Unsupported and overload computeUV there. The function will not be in stan::math + we would have to edit an "exterior" library. On the other hand, this should be little work.

This won't work because of the ADL issue.

Rewrite a new matrixExponential class, or rewrite the functions in matrixExponential, and include them in stan::math. This is what's been done for Eigen::Matrix, right? Specifically, I am looking at the example of block.hpp which has been turned from a class function into a function that returns a matrix.

Don't write a new class, write two new functions in the
the stan::math namespace:
stan/math/prim/mat/fun/matrix_exp.hpp:
double matrix_exp(const MatrixXd& x);
stan/math/rev/mat/fun/matrix_exp.hpp:
var matrix_exp(const Matrix<var, Dynamic, Dynamic>& x);
stan/math/fwd/mat/fun/matrix_exp.hpp:
template
fvar matrix_exp(const Matrix<T, Dynamic, Dynamic>& x);
Just have them include and call whatever they need inside of Eigen.

Bob

I should have mentioned that the functions I need to overload act on private members of the MatrixExponential class. As far as I know, I cannot access private members  without hacking. Writing an alternate version of the class doesn't seem that easy either; a first attempt suggests I may have to rewrite certain files in supported Eigen. But this is still work in progress.
Has there been a precedent for such a situation in Stan?

Early on, we used to patch Boost with some bug fixes. But we have moved
away from that years ago. It can be done, but in most cases, it's a poor
design choice or there's a different way to implement it.
Do you have access to push to branches directly? If so, could you put the
work on a branch and I can take a look to see if that's really necessary.
If not, I can give you permissions.
On Wed, Aug 17, 2016 at 3:53 PM, Charles Margossian <
<EMAIL_ADDRESS>wrote:

I should have mentioned that the functions I need to overload act on
private members of the MatrixExponential class. As far as I know, I cannot
access private members without hacking. Writing an alternate version of the
class doesn't seem that easy either; a first attempt suggests I may have to
rewrite certain files in supported Eigen. But this is still work in
progress.
Has there been a precedent for such a situation in Stan?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/issues/32#issuecomment-240527297, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_FzRrzjLjd91YXgKCyi-Gja6Q40K4ks5qg2asgaJpZM4FTDl2
.

I'm still not sure what you mean by "act on private members".
Do you mean private member variables (non-static variables declared
in the class) or non-static class functions?  If it's
member variables, are they things you pass into the class to
start with or things that are easy to recreate?
Private parts of classes aren't part of the interface and
aren't guaranteed to be stable in future releases---that's why
we don't like to hack them.
I'm also not sure what you want to overload here.   The
only function that will be overloaded in the Stan math
API are multiple versions of matrix_exp() for double, const var&, and
a templated version for const fvar&.
It's usually much easier to look at code than to guess.

@syclik: Yes, I have permission to push. GitHub gymnastic is a little new to me, so I had to set up a few things (following the guidelines here: https://github.com/stan-dev/stan/wiki/Dev:-Git-Process). Sorry for the delay is responding!
Let me start by sharing the non-ideal working prototype, where the exp function works for rev AD (this is easily extendable to fwd AD), and where I hacked into Eigen/unsupported.
Created branch: feature/issue-32-matrix-exponential
Changes from dev:
	modified:   lib/eigen_3.2.8/unsupported/Eigen/src/MatrixFunctions/MatrixExponential.h
	modified:   make/libraries
	modified:   makefile
	modified:   stan/math/rev/mat.hpp
	new file:   stan/math/rev/mat/fun/matrix_exp.hpp
	modified:   stan/math/rev/scal.hpp
	new file:   stan/math/rev/scal/fun/frexp.hpp

When I tried to run A.exp(), where A is a matrix of var, there were two functions that did not recognize stan variables: frexp and computeUV. These are the two functions I overloaded. Frexp was from cmath, so this was easy to do without changing Eigen/unsupported. ComputeUV was a class function from MatrixExponential, so I changed it in the class. Subsequently, exp() worked on matrices of var.

I'm still not sure what you mean by "act on private members".
Do you mean private member variables (non-static variables declared
in the class) or non-static class functions?  If it's
member variables, are they things you pass into the class to
start with or things that are easy to recreate?

I mean private member variables. They are passed (in an initialization list) into the class, but they are easy to recreate, seeing they correspond to various properties of the input matrix. My plan is to write a function that creates these variables, and then applies various function members of the MatrixExponential class. Which is almost the same as rewriting the class, and should do the job. What I was looking for was a more elegant solution that took advantage of the MatrixExponential class.
Here's the C++ code I used to do some quick checks on the prototype I posted. In this example, the exp function does not have the intended format, namely exp(A), rather A.exp().
#include <unsupported/Eigen/MatrixFunctions>
#include <stan/math.hpp>

int main() {

    // Look at simple of a 1 x 1 matrix
    stan::math::var mu = 2.0, lp;
    Eigen::Matrix<stan::math::var, 1, 1> mu_matrix;
    Eigen::Matrix<stan::math::var, 1, 1> mu_exp;

    mu_matrix << 2*mu;
    mu_exp = mu_matrix.exp();

    lp = mu_exp(0);
    std::cout << "mu = " << mu.val() << "\n";
    std::cout << "exp(2*mu) = " << lp.val() << "\n";
    lp.grad();
    std::cout << "d.f / d.mu = " << mu.adj() << "\n\n";

    // Expected result: d.f / d.mu = 2*exp(4) ~ 110

    std::cout << "-------------------------------------------- \n\n";

    // Look at non-diaginal case (example from Moler & Van Loan, 2003)
    Eigen::Matrix<stan::math::var, 2, 2> E, E_diag, E_b, E_b_1;
    stan::math::var E11=-1, E22=-17, lp11;

    E_diag << E11, 0,
              0, E22;

    E_b << 1, 3,
           2, 4;

    E_b_1 << -2, 1.5,
             1, -.5;

    E = E_b*E_diag*E_b_1;

    std::cout << "The Matrix E is:\n" << E << "\n\n"
              << "The matrix exponential of E is:\n" << E.exp() << "\n\n";

    Eigen::Matrix<stan::math::var, 2, 2> expE =  E.exp();
    lp11 = expE(1,1);
    lp11.grad();
    std::cout << "d.exp(E)[2,2]/dx = " << E11.adj() << "\n\n";
    // Expected Result (worked out by hand) for d.exp(E)/dx:
    //
    //      dE/dx = -.736   .552
    //              -1.472  1.104
    //
    // Note: derivative is identical to E.

    return 0;

}

Giles (section 2.3.5) asserts that the fwd and rev mode autodiff of the matrix exponential (via scaling and squaring) can be constructed from what he already demonstrated
http://eprints.maths.ox.ac.uk/1079/1/NA-08-01.pdf
Actually doing so is left as an exercise for the reader.

@bgoodri Thank you, I'll take a look into it!
I actually just finished implementing a working prototype with eigen 3.2.9, forking code from 3.3-beta2, and I will post it on gitHub after some polishing and finishing the unit tests. Subject to revision after reading Giles.

Theano also has an implementation of the derivatives of the matrix exponential
https://github.com/Theano/Theano/blob/master/theano/tensor/slinalg.py#L458

I think Theano's method is better explained (among others) in
http://dsp.vscht.cz/konference_matlab/MATLAB08/prispevky/017_brancik.pdf

Can you include a specialization for the expm of a 2x2 matrix?

@bgoodri Thanks for the articles. Let me try the scaling and squaring method (2.3.5 in Giles and 7 in Brancik).
What would the specialization of a 2x2 matrix entail? The 2x2 case seems simple enough to be efficiently handled by the general case.

The end of http://mathworld.wolfram.com/MatrixExponential.html has a general formula for the 2x2 case involving trig functions. It seems like that would be faster than going through the scaling and squaring.

Nice!
There are a few matrix properties we could exploit. matrix_exp could check for these properties and then pick an algorithm. For example, a symmetric matrix can have its matrix exponential efficiently constructed using eigen decomposition.
I think we could make matrix_exp pay attention to the following properties:

2x2
symmetric
nilpotent
more I am not thinking of right now

(diagonal, 1x1, and zero seem trivial enough to be handled by the user).
Should these features be included in matrix_exp, or should the user call a separate function, such as matrix_exp_2x2 ? I prefer the former approach (less functions, seems more elegant, and I'm not sure about how helpful the added flexibility of multiple functions would be). I do wonder how much checking for properties would slow down the code.

I would say the user should call expm() and expm() should branch whenever
it makes sense. Even if we had expm_symmetric() and whatnot exposed, Stan
conventions would dictate that we check symmetry inside expm_symmetric(),
so that wouldn't save any time relative to checking for symmetry within
expm(). Of course, if someone knows that they have a symmetric matrix, they
can do the eigendecomposition approach themselves.
On Thu, Sep 1, 2016 at 12:17 PM, Charles Margossian <
<EMAIL_ADDRESS>wrote:

Nice!
There are a few matrix properties we could exploit. matrix_exp could check
for these properties and then pick an algorithm. For example, a symmetric
matrix can have its matrix exponential efficiently constructed using eigen
decomposition.
I think we could make matrix_exp pay attention to the following
properties:

2x2
symmetric
nilpotent
more I am not thinking of right now

(diagonal, 1x1, and zero seem trivial enough to be handled by the user).
Should these features be included in matrix_exp, or should the user call a
separate function, such as matrix_exp_2x2 ? I prefer the former approach
(less functions, seems more elegant, and I'm not sure about how helpful the
added flexibility of multiple functions would be). I do wonder how much
checking for properties would slow down the code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/issues/32#issuecomment-244131681, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADOrqnEAea7y7aORiy3USb0JJCeNTnozks5qlvqggaJpZM4FTDl2
.

On Sep 1, 2016, at 9:04 PM, bgoodri<EMAIL_ADDRESS>wrote:
I would say the user should call expm() and expm() should branch whenever
it makes sense. Even if we had expm_symmetric() and whatnot exposed, Stan
conventions would dictate that we check symmetry inside expm_symmetric(),
so that wouldn't save any time relative to checking for symmetry within
expm().

That's a very good point.  But checking for symmetry would
slow down the usual matrix exponential calc, but probably not
much as it's only a quadratic operation.
2 x 2 is a no-brainer as that's a simple constant-time check
for size.

Of course, if someone knows that they have a symmetric matrix, they
can do the eigendecomposition approach themselves.

Only if they're as good as Ben at linear algebra.

Bob

The end of http://mathworld.wolfram.com/MatrixExponential.html has a general formula for the 2x2 case involving trig functions. It seems like that would be faster than going through the scaling and squaring.

@bgoodri The algorithm for matlab doesn't work for all 2x2 matrices. One other condition needs to be met, because of the delta term.
For A(2, 2) << a, b, c, d   , we need (a - d)^2 + 4bc > 0.
I adjusted the matrix_exp function to use the 2x2 algorithm if the matrix has the right dimensions AND the above condition is met.

@charlesm93 implemented the matrix_exp function! Closing this issue.

@bob-carpenter mentioned Al-Mohy's paper. Is it on someone's todo list? If not I'm gonna assign it to myself.

Hi @charlesm93 @syclik, the matrix_exp function passes matrix A by value, not by reference. Is this intended?

Wow, that was cryptic. Can you either send the file name + line number? Or find it directly in GitHub online, click the line, then send that link?

Sorry. I was referring this
https://github.com/stan-dev/math/blob/ad01b2cc20f1158eac34d875f7253a42a3fab940/stan/math/prim/mat/fun/matrix_exp.hpp#L21

Thanks. No, I don't think that was intentional. Mind creating a new issue for it?

On Feb 16, 2018, at 6:11 PM, Yi Zhang<EMAIL_ADDRESS>wrote:
Sorry. I was referring this
https://github.com/stan-dev/math/blob/ad01b2cc20f1158eac34d875f7253a42a3fab940/stan/math/prim/mat/fun/matrix_exp.hpp#L21
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I think it's a mistake, as the matrix is only ever passed as const reference.
I recall seeing a pass-by-value that was intentional for a copy, but I think we should avoid doing that and I don't think that's what's going on here.

The not passing by reference looks like a mistake. I can create an issue and a fix.

Issue #769 and pull request #770 .
@yizhang-cae There are several ways to optimize the current implementation of the matrix exponential. I can discuss here or create a new issue (on optimizing matrix exp function -- or is that not specific enough)?

Let's discuss in the new issue thread. Thanks.
Regards,
Yi Zhang
metrumrg.com
On Sat, Feb 17, 2018 at 4:38 PM, Charles Margossian <
<EMAIL_ADDRESS>wrote:

Issue #769 https://github.com/stan-dev/math/issues/769 and pull request
#770 https://github.com/stan-dev/math/pull/770 .
@yizhang-cae https://github.com/yizhang-cae There are several ways to
optimize the current implementation of the matrix exponential. I can
discuss here or create a new issue (on optimizing matrix exp function -- or
is that not specific enough)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/issues/32#issuecomment-366473337, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AfkhsEAamUyl8HIGGqOzfcxCb_8_Ip-Wks5tV0a4gaJpZM4FTDl2
.


ios-deploy: Unable to locate DeviceSupport directory

Steps to Reproduce
I was attempting to flutter run with xcode 7.2.something and I would see
Starting debug of iPhone 5s (GSM) 'drewwarren’s iPhone' (4127c6ffa53f153512916da5fe2a3afac8d23798) connected through USB...
2016-06-02 11:17:59.787 ios-deploy[4022:2958268] [ !! ] Unable to locate DeviceSupport directory. This probably means you don't have Xcode installed, you will need to launch the app manually and logging output will not be shown!
Could not install ios/.generated/build/Release-iphoneos/Runner.app on 4127c6ffa53f153512916da5fe2a3afac8d23798.
Error running application on drewwarren’s iPhone.

This says I might not have Xcode installed, but I did. flutter doctor made everything look okay for Xcode.
When I was in this state if I attempted to open FlutterApplication.xcodeproj in Xcode and click the play button I would see Could not find Developer Disk Image.

I get the same issue, but I'm using the latest version of XCode. When I try to run it in XCode, I get a white screen on my device and this in XCode.

@smrtsmrf can you check whether you're using Xcode 8.3.3? If so, it appears Xcode has changed the format it uses for the iOS DeviceSupport directory which is causing ios-deploy (one of our dependencies for launching iOS apps on devices) to get confused.
Upstream ios-deploy issue is: phonegap/ios-deploy#292
We can see what's in the iOS device support dir via:
ls "$(xcrun --sdk iphoneos --show-sdk-platform-path)/DeviceSupport"

With Xcode 8.3.3, I see:
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:27 10.0/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:29 10.1/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:26 10.2/
drwxrwxr-x   4 cbracken  eng  136 26 Apr 19:26 10.3.1 (14E8301)/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:26 8.0/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:28 8.1/
drwxrwxr-x   4 cbracken  eng  136 10 May 22:21 8.2/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:27 8.3/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:28 8.4/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:27 9.0/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:26 9.1/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:28 9.2/
drwxrwxr-x   4 cbracken  eng  136  2 Feb 07:28 9.3/

Looks like a quick hacky fix is:
cd "$(xcrun --sdk iphoneos --show-sdk-platform-path)/DeviceSupport"
ln -s 10.3.1\ \(14E8301\) 10.3

If you installed Xcode via the App Store (in which case, these files will all be owned by root), you may get a permission denied error and may need to run ln as root:
sudo ln -s 10.3.1\ \(14E8301\) 10.3

@cbracken thanks, that did the trick. 👍

Thanks for confirming! Will keep this issue open for reference until a fix goes into ios-deploy.

Thanks @cbracken your solution fixed my 10.3.3 even I added particular subversion. There is more talk about this on ios-deploy repo https://github.com/phonegap/ios-deploy/issues/292
dp

We might want to bump up the priority of this fix. The workaround has 12 thumbs up, and the workaround is still required. Maybe our tool can just make the symlink? Or, print out a link to this bug?

@sethladd We can't autogen the symlink since it requires root privileges but we can definitely output the one-line command to do it. I'll land a doctor check (and run it at build time as well) that outputs this.

ios-deploy 1.9.2 was released today and includes a fix for this issue. I've added a doctor check to bump to that version.

I am still having this issue in 1.9.2 with Xcode 9.0.1.
I have also tried using the solution to add the symlinks but doesn't seem to work for me.
------ Debug phase ------
Starting debug of 38813d6fae7464bb3f099582ea78ce53955e8632 (N66mAP, iPhone 6s Plus, iphoneos, arm64) a.k.a. 'iPhone' connected through USB...
[  0%] Looking up developer disk image
[ 95%] Developer disk image mounted successfully
2017-10-30 16:25:46.621 ios-deploy[44705:5930783] [ !! ] Unable to locate DeviceSupport directory with suffix 'Symbols'. This probably means you don't have Xcode installed, you will need to launch the app manually and logging output will not be shown!```

Just as a follow up, if you are having this issue, the files are probably missing from this path:
~/Library/Developer/Xcode/iOS\ DeviceSupport/

If that is the case, which fixed things for me, make sure your phone is attached to your mac and open xcode. The files will automatically appear and things should work.

I'm using XCode 11.3.1 and I'm still getting this issue. I checked this path:
/Library/Developer/Xcode/iOS\ DeviceSupport/

But the files were there. What else could be causing this issue?

I'm using XCode 11.3.1 and I'm still getting this issue. I checked this path:
/Library/Developer/Xcode/iOS\ DeviceSupport/

But the files were there. What else could be causing this issue?

Same XCode, also having the same issue. Files are also there.
How does one fix this issue?

I

I'm using XCode 11.3.1 and I'm still getting this issue. I checked this path:
/Library/Developer/Xcode/iOS\ DeviceSupport/

But the files were there. What else could be causing this issue?

Same XCode, also having the same issue. Files are also there.
How does one fix this issue?

I encountered the same issue and managed to fix it by updating to the latest version of ios-deploy - https://github.com/ios-control/ios-deploy


scala-java8-compat 0.9.1

Contributing to Twilio

All third-party contributors acknowledge that any contributions they provide will be made under the same open-source license that the open-source project is provided under.

[x] I acknowledge that all my contributions will be made under the project's license.

Codecov Report

Merging #661 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #661   +/-   ##
=======================================
  Coverage   81.13%   81.13%
=======================================
  Files          78       78
  Lines        5163     5163
  Branches      130      130
=======================================
  Hits         4189     4189
  Misses        974      974

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 623fc43...fec350a. Read the comment docs.


Update the doc issue template

Addresses #31144
Thanks @Porkechebure! ... I'm updating the issue template here. Rick will get back to you on your routing issue shortly.

Internal previews

📄 File
🔗 Preview link

.github/ISSUE_TEMPLATE/doc-issue.md
Before you open an issue

Should we add

I'll go ahead with it. Tom will patch it if there's something better to do with the text.


Lateral water hazard has incomplete tags warning.

I'm getting warnings that golf=lateral_water_hazard is missing tags because there is not natural=water tag on the way.    There is a stream centerline running through the center of the hazard polygon.
Here is the relevant recommendation from the wiki: "Usually defined by red stakes or lines (Rule 26) - Always tag natural=water for the water areas if they exist."

The issue you posted is unrelated to this repo. Please move your issue to the project that handles tagging: https://github.com/openstreetmap/id-tagging-schema/issues


[FEA] Support project expression rewrite for the case stringinstr(str_col, substr) > 0 to contains(str_col, substr)

Is your feature request related to a problem? Please describe.
In the case of stringinstr(str, substr) > 0, currently it's interpreted as GreaterThan with stringinstr as its child expression.  And in this case, we don't care about the string location, thus, we can rewrite this condition with contains at the physical plan level within Spark Rapids Plugin, avoiding an extra binary operation. And binary_operation is more expensive than string contains.

This is going to require a small cudf change to let us get access to a cudf::contains that uses a column of values instead of a scalar.
https://github.com/rapidsai/cudf/blob/64d651dc1a1dc309f08fee2502198f6a2e0e13b0/cpp/include/cudf/strings/find.hpp#L136

Never mind, that is a nice to have, but we don't support anything bust a scalar for substr.


opt:dictionary and opt:thesaurus

Howdy,
This just so happens to fill an exact need I have, but had a quick design question. I have my setup configured to add all dictionaries/wordlists to opt:dictionary as well as do something similar with opt:thesaurus (see here). I was curious if it is possible to just pull from those sources instead of having to mark them in the setup for this plugin?

I currently use lua to load dictionaries, but are you asking if we can use neovim's standard feature of completion for this?
I thought it was not possible because I don't know of any function that can get candidate completions. If you can, I would like to know.

I'm not sure what the purpose of this issue is, although I suppose it would work just as well if you registered the dictionary in dic['*'] of the setup as well.

I'm not sure what the purpose of this issue is, although I suppose it would work just as well if you registered the dictionary in dic['*'] of the setup as well.

Aha, so I got that working with this but for some reason I don't get completion when I type ;-;

Do you use lazy loading?
 Check help.
https://github.com/uga-rosa/cmp-dictionary/blob/0ba3df56258b48a5a6a0130d31ae3cf4908c9567/doc/cmp-dictionary.txt#L250

I don't use a plugin manager, but instead use git submodule to manage my plugins. I tried running ´CmpDictionaryUpdate´, but it still does not provide any suggestions sadly ;-;

Cannot be reproduced. Please create a minimum setup and procedure to reproduce the issue.

Cannot be reproduced. Please create a minimum setup and procedure to reproduce the issue.

Install the latest version of neovim.
Allow vim to load all plugins in init.lua
´´´
-- The load.lua file contains all lua related to loading plugins. When this file is loaded the below
-- lua will do the following:
--    1. Configure settings for loading plugins.
local env = vim.env
local opt = vim.opt
local cmd = vim.cmd
-- Add the 'XDG_CONFIG_HOME/.config/nvim/' directory to the plugin path.
local config_directory = env.XDG_CONFIG_HOME
opt.packpath:append(config_directory .. '/nvim/pack')
-- Load all plugins found in the plugin path.
cmd([[ packloadall ]])
-- Load filetype plugins.
cmd([[ filetype plugin on ]])
cmd([[ filetype indent on ]])
-- Enable file type detection.
cmd([[ filetype on ]])
´´´
Clone needed plugins
´´´
mkdir -p ~/.config/nvim/pack/start
git submodule add https://github.com/hrsh7th/nvim-cmp.git
git submodule add https://github.com/uga-rosa/cmp-dictionary.git
´´´
Create needed plugin settings
´´´´
mkdir -p ~/.config/nvim/plugin
-- add to nvim-cmp.lua:
local cmp = require('cmp')
cmp.setup {
sources = {
-- Provide completion from the current buffer, file system paths, and
-- language server protocol.
{ name = "buffer",   keyword_length = 1 },,
},
completion = {

]],
},
window = {
	-- Wrap the completion menu window in a border.
	completion = cmp.config.window.bordered(),

	-- Wrap the completion menu's documentation window in a border.
	documentation = cmp.config.window.bordered(),
},

}
-- add to cmp-dictionary.lua
local env = vim.env
local fn  = vim.fn
local cmp = require("cmp")
local cmp_dictionary = require("cmp_dictionary")
-- Load all files from 'HOME/.usr/share/dict' and '/usr/share/dict/' to be used as dictionary files.
local dictionary_directories = env.HOME .. '/.usr/share/dict/' .. ',' .. '/usr/share/dict/'
local dictionary_files = fn.globpath(dictionary_directories, '*', 0, 1)
cmp.setup({
sources = {
{
-- An dictionaries to the Cmp sources list.
name = "dictionary",
keyword_length = 1,
},
}
})
require("cmp_dictionary").setup({
dic = {
["*"] = dictionary_files,
},
-- Only show candidates with an exact prefix match.
exact = -1,

-- Do not ignore the case of the first character.
first_case_insensitive = false,

-- Do not ctivate document using external command.
document = false,

-- Use wn(1) for the above document tool.
document_command = "wn %s -over",

-- Perform the initialization in a separate thread.
async = true,

-- Cache at least 10 dictionaries.
capacity = 10,

-- Do not output debug messages.
debug = false,

})
´´´´
Then open nvim and run ´CmpDictionaryUpdate´. Start typing and you will see that no dictionary completion is offered, but if you type the word Hello on line one and then type He you will get Hello as a completion offer (therefore I know nvim-cmp is working).

local env = vim.env
local opt = vim.opt
local cmd = vim.cmd
local fn  = vim.fn

local config_directory = env.XDG_CONFIG_HOME
opt.packpath:append(config_directory .. '/nvim/pack')

cmd([[
packloadall
filetype plugin indent on
]])

local cmp = require('cmp')

cmp.setup {
	sources = {
		{
			name = "dictionary",
			keyword_length = 1,
		},
	},
}

local cmp_dictionary = require("cmp_dictionary")

local dictionary_directories = env.HOME .. '/.usr/share/dict/' .. ',' .. '/usr/share/dict/'
local dictionary_files = fn.globpath(dictionary_directories, '*', 0, 1)

cmp_dictionary.setup({
	dic = {
		["*"] = dictionary_files,
	},
	exact = -1,
	first_case_insensitive = false,
	document = false,
	document_command = "wn %s -over",
	async = true,
	capacity = 10,
	debug = false,
})

This is the very init.lua I used and it seems to work fine. Can you try it?

Yep! The solution was to merge the cmp-dictionary.lua and nvim-cmp.lua files into one file. Thank you so much!


Security Problem: Screenshot Function

Original issue 402 created by TriplexAccount.P.N on 2014-07-08T10:27:41.000Z:
Copied from https://github.com/google/google-authenticator/issues/401 because it applies to Android app (as well?).
What steps will reproduce the problem?

Call Screenshot Function from the specific smartphone

What version of the product are you using? On what operating system?
Google Authentificator: 2.49
Please provide any additional information below.
In general, it is not possible to take a screenshot of any inner user interface from a banking app for security reason. The Google Authentificator can be comprimised by using a trojan with screenshot function. Please disable the ability to take a screenshot from the main interface from the Google Authentificator.

GA should probably use "FLAG_SECURE":: https://developer.android.com/reference/android/view/WindowManager.LayoutParams.html#FLAG_SECURE

Something reported on 2014 gets fixed 6 years later. Great job Google.

No, FLAG SECURE does not protect you from a11y malware (and Google couldn't have protected 2FA tokens that easily)


evaluate whether LeafletLayer should implement InteractiveLayer

https://leafletjs.com/reference-1.7.1.html#interactive-layer

as part of #112 the answer here is no; we need to cut the scope of this project to be a drop-in replacement for raster tiles, not a full system with selectable features, since those are better served by using MapLibre.


Add converted NNAPI CTS tests for reduceMax / reduceMin / reduceProduct / reduceSum ops.

This PR is to add 24 converted reduce tests.
Besides, I also updated previous NNAPI to WebNN Op mapping table to align with latest WebNN API Spec.
@huningxin PTAL, thanks.

@BruceDai , could you please check the failure of macos bot?

I reset the first commit, then made GitHub Actions trigger again, CI pass this time.
@huningxin PTAL, thanks.


Revert "Spanner grpc config"

Reverts GoogleCloudPlatform/google-cloud-php#1236
We ran into a small issue while running the tests on OSX, going to revert this out until we have a fix ready.
/cc @WeiranFang @ZhouyihaiDing

confirm


compile error with iot-hub-device-update 1.0.1: 'IOTHUB_CLIENT_CONNECTION_QUOTA_EXCEEDED' not handled in switch

Hi,
during integration the provided release 1.0.1, i get an compile error in the new communication manager.
I use a  gcc compiler version 11.
Compiler Log Message:
/src/communication_managers/iothub_communication_manager/src/iothub_communication_manager.c:685:9: error: enumeration value 'IOTHUB_CLIENT_CONNECTION_QUOTA_EXCEEDED' not handled in switch [-Werror=switch] |   685 |         switch (g_connection_status_reason) |       |         ^~~~~~ | cc1: all warnings being treated as errors
I have provided a possible solution in this PR: https://github.com/Azure/iot-hub-device-update/pull/345

Please see comment therein: https://github.com/Azure/iot-hub-device-update/pull/345#issuecomment-1387602218
Agent is pinned at an older release of azure-iot-sdk-c and that enum value was added in a later release and that enum value was introduced in a later release of azure-iot-sdk-c:  LTS_07_2022_Ref02
1.9.1
You should be able to fix it by installing the older version--one way is via ./scripts/install-deps.sh --install-azure-iot-sdk
that uses this older version:
azure_sdk_ref=LTS_07_2021_Ref01
as seen here https://github.com/Azure/iot-hub-device-update/blob/main/scripts/install-deps.sh#L46
Do let us know if you have particular needs for using the newer version of azure-iot-sdk-c as it may help prioritize moving the device udpate agent to a newer version, which is planned.

yes it's right. We using latest LTS sdk revision LTS_07_2022_Ref02 for building iot-hub-device-update.  I will change my PR based on proposal from https://github.com/Azure/iot-hub-device-update/pull/345#discussion_r1080851828

closed, fixed in PR https://github.com/Azure/iot-hub-device-update/pull/345


ARM64 Docker Support (Also other arm architectures)

Hey
I tried ackee on Vercel and would like to host it on my own Cloud VM (using Oracle Free Cloud tier), but unfortunately the official Docker Image doesn't support ARM yet.
I read about about the same in #286 but still can't see the ARM tagged images for Ackee.
Would really appreciate if there would be an official ARM image for Ackee.

Hey @Divkix, thanks for letting me know. Is there anything I need to add to the Dockerfile to support ARM64 (and other architectures)?

Hey @electerious, the Dockerfile seems fine, you'll need to use docker buildkit in order to build images for multiple architectures.
Can tou let me know how do tou build the images before pushing them to Docker Hub?

I've just publish a new develop tag to Docker with support for other platforms. Please let me know if everything works as expected so I can create a new release 😊

The release works great on linux/arm64, I haven't been able to try on linux/arm/v7 because I don't have anything which runs on it, but I think that it should work.
Everything works as expected for arm64 :)

Perfect, thanks for letting me know! v3.3.0 is now live: https://github.com/electerious/Ackee/releases/tag/v3.3.0


Improving the UI Of the menu section

Current Behaviour:
https://github.com/user-attachments/assets/6e3c6c9f-22c8-41d7-9e8a-12d33ddb4aea
Expected Behavior:
You need to replicate the hover effect where two lines appear below each navigation item upon hovering. This effect will enhance the visual appeal and interactivity of the navigation menu.
Hover Effect: On hover, two lines should animate below each nav item.
The first line should appear, followed by the second line.
Ensure the lines are styled to complement the overall design of the website.
Expected Outcome:
This hover effect will provide users with clear feedback, improving navigation and user engagement on the site.
Expected Hover Effect
https://github.com/user-attachments/assets/c82cf51a-84a3-49b3-aa86-587baceae741

I have fixed this issue.

I have fixed this issue


Fixed #24813 -- Documented {% include %} debug behavior variance

Added doc note about difference in handling of missing template by
include tag based on DEBUG setting.

merged in fc3409b0932959f2aca42715e9147113d0e40f79, thanks!


Updates to trackers and global variable handling

[x] updated catalog to use global vars from utils and tracker
[x] fixed logging levels in mriqc tracker
[x] refactored bids_tracker - Closes #93
[x] updated sample_run_nipoppy to set log-level and prototypical mriqc run with tracker

@michellewang

I'm a bit concerned with how this PR will integrate with PR https://github.com/neurodatascience/nipoppy/pull/162, which makes kind of big changes (mostly restructuring) to run_tracker.py. Might be tricky to merge...

My changes to run_tracker.py are minor. If you prefer, I can wait for your PR to be merged.

@michellewang - the absolute bids_dir path binds work!


Understanding the Progress Bar

I train on MNIST with data loaders defined below (full train / test sets with batch_size=128).
'val_check_interval': 0.1, so per training epoch, I have 10 validation runs.
Now:

10000 (test) images / 128 (batch_size) = 78.125, so steps such as 54/79 do make sense.
60000 (train) images / 128 (batch_size) = 468.75, so I'd expect something like 120/469.

What is the "1259" representing in the progress bar? I can observe in tensorboard, that the epoch number goes up at exactly 459.
Validating:  68%|██████▊   | 54/79 [00:08<00:03,  6.57it/s]
Epoch 4:  78%|███████▊  | 976/1259 [04:01<01:09,  4.05it/s, loss=19279.273, v_num=0]

Code
Data Loaders
    def train_dataloader(self) -> DataLoader:
        """Pytorch-lightning function."""
        transform = torchvision.transforms.Compose([torchvision.transforms.Resize((32, 32)),
                                                    torchvision.transforms.ToTensor()])
        train_set = torchvision.datasets.MNIST(root=DATA_DIR_PATH / 'mnist_data',
                                               train=True,
                                               download=True,
                                               transform=transform)
        return DataLoader(train_set,
                          batch_size=128,
                          shuffle=True,
                          num_workers=0)

    def val_dataloader(self) -> DataLoader:
        """Pytorch-lightning function."""
        transform = torchvision.transforms.Compose([torchvision.transforms.Resize((32, 32)),
                                                    torchvision.transforms.ToTensor()])
        val_set = torchvision.datasets.MNIST(root=DATA_DIR_PATH / 'mnist_data',
                                             train=False,
                                             download=True,
                                             transform=transform)
        return DataLoader(val_set,
                          batch_size=128,
                          shuffle=False,
                          num_workers=0)

What's your environment?

OS: Ubuntu 20.04
Packaging: pipenv
Lightning Version: 0.8.5

It's 79*10 (10 times validation per epoch) + 469(train batches) = 1259. The progress bar contains both train and validation steps.

I see, thanks @rohitgr7 :+1:


Should association with uniqueIndex work?

GORM Playground Link
https://github.com/go-gorm/playground/pull/690
Description
Ran into a similar issue and reproduced it in the playground, should this work or no?
If this should work, then I guess this is a bug?

Its mistake is
main_test.go:50 UNIQUE constraint failed: profiles.ssn

Obviously, the SSN of the two Profile you created are both empty strings, it cannot satisfy unique index constraints

I understand it does not work now. It seems it should be allowed to have a uniqueIndex field, even in the case of a "has many" relationship. But due to the way the association is implemented, this happens, which seems to be a bug to me.


Some game data files may have been corrupted

After deploying example.yaml on my cluster I can see in the pod server logs:
**************************************************
WARNING: baseq3/pak0.pk3 is present but its checksum<PHONE_NUMBER>)
is not correct. Please re-copy pak0.pk3 from your
legitimate Q3 CDROM.
**************************************************

Not sure if this a known non-issue or something that needs to be re-copied into the container image.

I may be misunderstanding the need to use the standalone game build.  I haven't tried to use one of the custom quake games yet, but wanted to ensure that it could be done in the future, however, it seems like just setting the basegame should be all that is necessary for that.  I think some more investigation is necessary here, my biggest concerns would be that it somehow violates the license (although looking at the q3 demo license it doesn't appear to).

I think this should be addressed with #16 but I welcome any follow-up discussion (we can reopen this or start a new issue). Thanks again!


Add support for updating memory used by the VM

This is the hyperkit-side of https://github.com/code-ready/crc/issues/1522

Fixed by https://github.com/code-ready/machine-driver-hyperkit/pull/19


manifest.json Error When Following These Instructions

A colleague of mine and I both followed these instructions and were presented with an error regarding the manifest.json.  The error detail was:
Error while reading manifest.json: "staticTabs[0].name" is required
"staticTabs[0].contentUrl" is required
"staticTabs[1].name" is required
"staticTabs[1].contentUrl" is required
It appears that the App Studio generates two tabs automatically, and the automatically-generated manifest.json does not have these properties present.  If we attempt to edit the properties for these tabs, that is not possible in App Studio (screenshot below).

Furthermore, the error page that's presented when the app fails to install is designed in such a way that the most prominent control for the user jumps to the app store and loses the context without informing the user that it saved the progress thus far.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ce60ea6f-d01c-9714-f3f7-db3a11806a34
Version Independent ID: 435cfd9f-8b1d-d810-a58c-655ca6780286
Content: Add a Power Virtual Agents bot to Microsoft Teams - Power Virtual Agents
Content Source: power-virtual-agents/publication-add-bot-to-microsoft-teams.md
Service: dynamics-365-ai
GitHub Login: @iaanw
Microsoft Alias: iawilt

I'm investigating this issue and will hope to have an answer for you. I'm not sure if this is something that can be fixed with documentation, so you might want to post this on the PVA forum https://powerusers.microsoft.com/t5/Power-Virtual-Agents-Community/ct-p/PVACommunity

Please repost this on the community forum as it's not a doc issue unfortunately, thanks.


Smart Playlist Without Tracks Creates Empty Playlist and Shows Error

The Problem

When creating a smart playlist, if no songs are found, a playlist is still created with no songs in it and an error page is shown.
Expected Behavior

Informing the user that there are no songs found with their smart playlist rules and limits would be preferable to creating an empty playlist (junk) and showing a generic error (confusing to users).
Steps to Reproduce the Problem

Go to Create Smart Playlist page
Name the smart playlist "Test"
Add a rule where "Artist Name is 'tubsjwlvhxcvuiqer;'" (or some other nonsensical string that will have no matches)
Click Create Smart Playlist button
Observe in Spotify, an empty playlist named "JAMMS: Test" is created
Observe in browser a redirect to the JAMMS error page

Additional Information

This may also be solved by separating the "gathering songs" piece from the "creating playlist" piece of "create smart playlist" logic.
Previewing what songs are gathered by the rules and sending them to the browser for the user to see allows this to become a transparent problem.  If that were the case, and users could only create smart playlists that had songs in them, this problem would be transparent and resolved.

This half solves the problem - https://github.com/iansantagata/jamms/pull/25


fix: rollup plugin add necessary deps

Background
@stylexjs/rollup-plugin loose necessary dependencies, So this pull request add them.

You also need to run npm i in the root dir to update the package-lock.json file.

@nmn thx


belindas-closet-nextjs_8_remove-unrelated-fields_product-detail-page

Resolves #462, #464
Related #454
note: this is from sprint 7 but I'm doing this for sprint 8 (that's why two labels are on this)
This PR is for updating the product detail page by displaying correct fields for certain products, removing unrelated ones from displaying.
Pants waist and inseam only appear on pants products:

Shoes only display shoe size:

Shirts and other products display size and description

This is all I could find for incorrect fields
Let me know if this is correct or if changes are needed, I can make them
future suggestion issue: maybe adding more distinct fields for certain items
Update: Shoes only display shoe size, removed size field (Ignore previous image of shoes)

I removed the Product Size field for Shoe products.

oh thanks @heosman, ill make these changes

@Asfand00 No worries, I already pushed the changes.


expliot: relax dependency constraints

Description of changes
Fix build (https://hydra.nixos.org/build/168588954)

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 163894 run on x86_64-linux 1

1 package built:

expliot


[DO NOT MERGE] KB TOC A

Opening (and then immediately closing) this PR so that the change history is preserved after the branch is deleted (and branch content can be restored later if needed).

Docs Build status updates of commit c779edc:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error] Cannot sync git repo to specified state. Please make sure the pull request is mergeable. Details: Git merge results in conflict, repository remote url: https://github.com/OfficeDev/office-js-docs-pr, working directory: W:\xl4l-s, target branch: master, target branch commit: 43735e90ba7a99a4654a19fe56fa4db1cc89e3e2, source commit: c779edce239305e4073085741b6e964795b2e77b.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


bug(gatsby-plugin-fastify): all reverse proxy redirects end up as wildcards

Describe the bug
When configuring Reverse Proxy redirects, they will be configured as a wildcard reverse proxy whether or not you include a * in the redirect fromPath.
To Reproduce
Bug is reproduced in the repos test site at gatsby-uc\plugins\integration-tests\plugin-fastify
  createRedirect({
    fromPath: "/example-proxy",
    toPath: "http://example.com",
    statusCode: 200,
  });

  createRedirect({
    fromPath: "/example-proxy-star/*",
    toPath: "http://example.com/*",
    statusCode: 200,
  });

Expected behavior
All expected behaviour below can be demonstrated on Gatsby cloud site, ie: https://pluginsfeatfastifydefaultcusto.gatsbyjs.io/example-proxy-star/ or configuring above redirects and pushing to gatsby cloud. Actual behaviour can be tested in localhost using npm start/gserve with the above configured redirects in place.

/example-proxy-star, /example-proxy-star/

should respond 200 and serve example.com page
ACTUAL BEHAVIOUR: responds correctly ✔

/example-proxy-star/anything-else

should respond with 404 from example.com and serve the example.com page
ACTUAL BEHAVIOUR: responds correctly ✔

/example-proxy, /example-proxy/

should respond 200 and serve example.com
ACTUAL BEHAVIOUR: responds correctly ✔

/example-proxy-anything-else

should respond with 404 from gatsby site and serve gatsby site 404 page
ACTUAL BEHAVIOUR: responds correctly ✔

/example-proxy/anything-else

should respond with 404 from gatsby site and serve gatsby site 404 page
ACTUAL BEHAVIOUR: responds 404 from example.com and serves example.com page ❌

Additional Info
I discovered this working on #392 and will have the fix committed there, just wanted to track it as a separate issue as well.

thanks! If we can put it in a separate PR that'd be great.

thanks! If we can put it in a separate PR that'd be great.

will extract it Monday morning

thanks! If we can put it in a separate PR that'd be great.

https://github.com/gatsby-uc/plugins/pull/401

With the announcement of the Gatsby Adapters spec, we'll be discontinuing any work on this plugin. Should the need for a performant node server still be needed, this work can continue under the official spec as gatsby-adapter-fastify.


09/10/2018 17:44 : Create Pull Request for branch prTesting_2018091017 -> master

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[Known][Linux] File Drag and Drop doesn't work

Will be eventually resolved when upstream issue ticket is fixed, or if that takes too long, I might implement it myself.

MacOS drag and drop does seem to work after further testing.

Drag and drop (or + sign) don't work in Mac OS, either. Please amend the title to reflect this. I'll have to use it in a Windows VM. Also, two years and no progress? This is useless if it doesn't work.


Brew GraphiQL App not working

My GraphiQL brew app is not working.  It just sits and clocks.  I am getting the following error:

I tried updating brew and reinstalling and still get the same errors.
Have you guys seen this before?

Waiting for this to be merged: https://github.com/caskroom/homebrew-cask/pull/35648
It should work at that point.

Go ahead and give the latest Cask a try @LucasRettig :)

This works now.  Thanks @skevy !


Remove-np-ninf

All Submissions:

[x] Contributions should target the dev branch. Did you create your branch from dev?
[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

[x] Does your submission pass tests?
[x] Have you installed pre-commit with pip3 install pre-commit and set up hooks with pre-commit install?

Changes to Core Features:

[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your core changes, as applicable?
[x] Have you successfully ran tests with your changes locally?

Numpy 2 dropped support for the np.INF and NINF and instead it recomends np.inf and - np.inf
WIP: I'm still looking for the contribution guide, inparticular the how to run the tests locally bit, as I'm not familiar with poetry. Tests pass in the runners but I'm pretty sure the current test suite doesn't cover the changed code.

I can show that if you upgrade numpy to 2.x it fails the current test suite, but with the change, the client doesn't fail the tests. This requires the client without fastembed, as that is pinned to below 2 for numpy:

Thank you for addressing this


add ointment and bruise packs to medfab

About the PR

yohoho
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl:

add: Added bruise packs and ointment to medical techfab.

I'm sorry for being out of line as a simple tester, but is it really fine moving obviously organic healing items to techfab to be crafted with plastic, glass and steel?


Fixing ConcurrentModificationException for gobblin-config-store

Making loadRawTopologyFromFallBack() synchronized , so that it will be loaded truly once.
Returning new ArrayList() so that we won't get the ConcurrentModificationException

@aditya1105 I am working on this as well...

@aditya1105 can this be closed?

Closing as duplicate. @aditya1105 please re-open if necessary.