Additional pigpiod command line option "-m" to disable the alerts

I use the pigpio library only to transmit 433MHz codes of a remote control.
To optimize CPU usage I added a new command line option "-m" for the pigpiod daemon.
This option disables the alerts thread and reduces CPU usage from 6-7% to 1.5% on my RPI 2.
It would be great when you could merge this to master.

I agree. I just went through and added this option myself before I even saw that this pull request existed. I am using it for PWM control of a 24V RGBW light strip through Mosfets.
I had switched over to Pi-Blaster because of this extra CPU usage but I wasn't happy with the PWM frequency of Pi-Blaster. I could not see the LEDs flickering with my naked eye but I could see it when recorded on video. I have since fixed the flickering issue in Pi-Blaster and decided to come back over to pigpio and see what I could do about the unneeded extra CPU usage.
I did everything pretty much the same as you except I added two more PI_DISABLE_ALERT checks:
https://github.com/TheRaspyDev/pigpio/blob/master/pigpio.c#L6295
if (!(gpioCfg.ifFlags & PI_DISABLE_ALERT))
{
   pthAlertRunning = PI_THREAD_RUNNING;
}

https://github.com/TheRaspyDev/pigpio/blob/master/pigpio.c#L8477
if (!(gpioCfg.ifFlags & PI_DISABLE_ALERT))
{
   while (pthAlertRunning != PI_THREAD_RUNNING) myGpioDelay(1000);
}


add a flag to data generation scripts, so, that it only creates .ly/.txt files and does not render images

Not happening anymore


Use pkg-config to locate libusb-1.0

This PR resolves https://github.com/raspberrypi/usbboot/issues/81, and closes https://github.com/raspberrypi/usbboot/pull/25.
This is a very simple change that uses pkg-config to locate libusb-1.0 rather than assuming its path, allowing users who have installed libusb-1.0 with homebrew to successfully build with make.

Thanks to @khorben (PR: #25) and @Parnoud (Issue: https://github.com/raspberrypi/usbboot/issues/81#issuecomment-851637615) for laying out the solution.

@pelwell, @peterharperuk, or @timg236 I would really appreciate your quick look at this, as macOS users can't sucessfully build without this small change.

Although initially concerned about it being a more heavyweight solution, even CygWin has pkg-config, so most users should be able to cope.


Move artwork creation logic from artworks/post.php to Artwork.php

Per discussion, this PR moves construction of Artwork objects to the class itself.
The only functional changes are that tags/artists are no longer created if form validation fails. Also, failing to move/create image/thumbnail files will result in orphaned database objects (and a log message).
@acabal I've included you on this PR as it relates to your feedback.

Right on, Job, thanks for following up on previous tasks! I tested it locally, and creating new artwork works as expected.
My feedback on the change: Artwork::Build() is effectively a constructor with a different name. Even though we moved away from using a big constructor previously, it seems appropriate now and moves logic out of www/artworks/post.php ("Fat Model, Skinny Controller"). LGTM, but wait for Alex.

Right, it is just a big constructor. I do think a constructor is appropriate here though, as it needs to interpret some arguments and the alternative would be creating class properties for use only during construction. Also, with PHP 8.0's named arguments, I'd say it's just as readable as assigning properties individually.


2.0.1

Coverage decreased (-2.5%) to 93.909% when pulling c3187282f3e90fb185237468c2f1af0085ce88b1 on 2.0.1 into cde664af680f9714f341f9354d2199a7e8283156 on master.


build: use webpack for umd module export

:tada: This PR is included in version 2.0.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Explore compatibility with Python 3

HARK is written in Python 2, and does not run on Python 3; this was the takeaway from issue #80, which this issue is meant to replace.  Some of the differences between 2 and 3 are superficial and we can find language that works for both.  Other differences are very deep down and probably can't be reconciled.
Issue for discussion and future work.

This has come up again in #123 .  Someone should go into each one of our files and add near the top, right after the imports:
major_version = sys.version_info[0]
if major_version != 2:
....raise Exception('HARK code is written for Python 2, but you are running it on Python 3.  You should                               expect HARK to crash quite quickly.')
(But without the dots, which I added to force indentation)

I used futurize to make HARK Python 3 compatible. There are still issues but in general HARK works with Python 3. I successfully ran the FashionVictim model on my Windows 10 machine. The HARKcore.py file and others output Sorry, HARKcore doesn't actually do anything on its own. To see some examples of its frameworks in action, try running a model module. Several interesting model modules can be found in /ConsumptionSavingModel. For an extraordinarily simple model that demonstrates the "microeconomic" and "macroeconomic" frameworks, see /FashionVictim/FashionVictimModel. using Python 3.
The primary error I get when running code in the other folders is TypeError: _warning() takes from 1 to 4 positional arguments but 6 were given. I have not looked into the problem as of now.
HARK branch Python 3 compatible

This is very interesting to know, thank you.  Looking through the changes on the first new commit on your branch, it looks like the changes are fairly minor, and not particularly extensive.
To deal with the warning issue, look at the top of HARKutilities.  We import the warnings package, then overwrite the format of _warning.  This is a bit of legacy code that has been there since before HARK was HARK, written by Nathan Palmer.  I think you can just comment those lines out and use the built in warnings, which offer more verbose feedback (about the line number and file where the warning is originating).

You are welcome. I agree that the changes are fairly minor because futurize makes Python 2 code  compatible with Python 2 and 3.
I'll take a look at the warning issues later today and get back to you.

Jackie Kazil is actively working on this.  We're trying to clear out the backlog of PRs, then we'll go full steam on making HARK Python 3 compatible.  And hopefully specify a plan for when Python 2 won't be supported and we fully switch.

ganong123 asked in #150 :
"I am considering upgrading from Python 2.7 to Python 3.6.
The README suggests installing Python 2.7.
Is there any part of HARK which will not work with Python 3.6? Thanks!"
Roughly: Nothing in HARK will work with Python 3, in the sense that basically every module will crash immediately when run.  This is not due to some humongous incompatibility with Python 3, merely a bunch of boring line formatting issues, a handful of old-style calls to print (not written by me!), along with a few calls to old packages (e.g. pyplot vs matplotlib).
In the fork that johnrieth linked above, he used the futurize package to try to make HARK compatible with Python 3.  I haven't tested it personally.
Jackie Kazil is actively working on this, and (I believe) has HARK working in Python 3.  That work was done in a private repo that uses most of the main HARK files, so that doesn't help you much.  BUT: The plan is to make the master branch of HARK compatible with both Python 2 and 3 in the very near future, then announce a plan for our eventual migration to only Python 3.

@jasonaowen Check out https://github.com/econ-ark/HARK/issues/102#issuecomment-365290806 as you're estimating how long the migration will take.

@johnrieth It looks like your fork, and thus your branch are no longer on GitHub. Is your futurize-related branch still available for us to look at?
@jackiekazil could I have a look at that private repo?
I'm checking out http://python3statement.org/ , http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html , and https://docs.python.org/3/howto/pyporting.html .

@brainwane (forgive me if I repeat something that was said. I haven't read the whole convo) there was no py3 of hark, but there was this repo: https://github.com/llorracc/cAndCwithStickyE-Latest/. Which has a copy of Hark and code for that model scenario written in with it. That was the repo that I updated.
I don't have admin rights to it. It is under @llorracc and he was the one who gave me access to it.

An update from another direction -- I forked HARK and made a single example fully Py2 & Py3 compatible, just to see what it would take to do it. I've just pushed that commit to my branch here: https://github.com/npalmer-professional/HARK-1/commit/84be05470aa89ac32d4909ca199f999a88027129
The commit description lays out the details of what was required (plus a bug or two that were revealed during the process, but I'll file an appropriate bug report for those elsewhere). It wasn't as bad as I thought, and I can verify that it runs under both Py2 and Py3 now. I will continue to chug through all executable examples under the current HARK version, in my fork.

Great.  So, we are on track for making a draft update in a couple of days?
(A week before CEF?)
On Tue, Jun 12, 2018 at 12:08 AM, npalmer-professional <
<EMAIL_ADDRESS>wrote:

An update from another direction -- I forked HARK and made a single
example fully Py2 & Py3 compatible, just to see what it would take to do
it. I've just pushed that commit to my branch here:
npalmer-professional/HARK-1@84be054
https://github.com/npalmer-professional/HARK-1/commit/84be05470aa89ac32d4909ca199f999a88027129
The commit description lays out the details of what was required (plus a
bug or two that were revealed during the process, but I'll file an
appropriate bug report for those elsewhere). It wasn't as bad as I thought,
and I can verify that it runs under both Py2 and Py3 now. I will continue
to chug through all executable examples under the current HARK version, in
my fork.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/econ-ark/HARK/issues/102#issuecomment-396402989, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABQlfwPOvy6T0Lc2SepItSIUmGBfdA7Yks5t7upPgaJpZM4Li8qQ
.

--

Chris Carroll

@llorracc I think that is within the realm of possibility, but it is not for certain. I'm going to ping you and @mnwhite for a priority-listing of items to make compatible.

@lorracc and @mnwhite -- OK, this commit to HARK on my fork is now: https://github.com/npalmer-professional/HARK-1/commit/05e22da3bfa80908b0f399c4962fec98f702da9a
...is highly Py2 & Py3 compatible, with the exception of  cstwMPC and  cAndCwithStickyE (which were at the lowest-priority spots in Matt's useful prioritization list).
As expected the vast majority of isses by count were print/exceptions, but those were addressed pretty quickly and easily. There were probably a good solid 4 or 5 issues which were due to more "in the weeds" things with how Py2 changed to Py3. I solved all of them except one directly. The one major remaining Py2->Py3 error that I couldn't solve I've instead solved indirectly. Specifically, something with how Py3 now works with joblib and dill/pickle breaks one of the examples from FashionVictimModel (where the exact same code still runs fine under Py2).
The indirect solution was easy enough. It turns out HARKcore.Market.solveAgents() only uses "multiThreadCommands," so this effectively broke that FashionVictimModel example. (I had to double-check to see if I'd somehow gotten an old version, since I thought we'd already fixed this long ago -- but nope.)
BUT the super-easy solution was to just add a try/catch to HARKcore.Market.solveAgents() so if joblib failed, it would call the "multiThreadCommandsFake" using the exact same syntax. Easy-peasy -- just slower.
I did try to spend some time fixing the actual underlaying problem, but couldn't solve it and it appears this particular breaking example will take a bit more effort. So I left it as it currently is. (Interestingly enough none of the other examples break in this way.)
There are a lot of detailed notes in the commit, along with a list of all the files that run under Py2 and Py3 -- see "Appendix A" in the commit notes for the short version, and Appendix B for the long detailed version. It's almost everything.
Finally, a good chunk of time was independently running the files after a change was made. I basically modeled what I did on how unit testing would have run the whole process. Doing it once is much quicker than writing all those units (since that's almost certainly a multi-day project), but in the longer-run, we will almost certainly want to write out the unit tests to make a future refactoring faster / more assuredly correct.
Ah, I also tagged this commit as "postfuturize."

Note: #168 starts to address this issue.

Now that #168 is merged, I'd like to test the current version of HARK to check how well it works in an all-Python 3 environment.


Clean up builders defined in devicelab_staging_config.star

We have limited Mac_ios bots in staging, and we need these bots to validate flaky targets.
We should clean up as many builders in staging as possible to leave capacity for bringup: true targets.

https://flutter-review.googlesource.com/c/infra/+/32840


Feature Request: Nested links & network graphs

This is really great project. I have few suggestions for additional features.

Support for nested links as in Markdown Mindmap.

- A
  - B
  - C
    - D
      - [E](test.md)

2. It would be great if it would work on anchor links as well.
- A
  - B
  - C
    - D
      - [E](test.md#c)

I don't know if it could be placed in the scope of this project, but I would like to have the option to create networks graphs, instead of mindmaps. Mindmaps are great, but sometimes it would be good to visualize markdown as network (graph with cycles), so instead of creating infinite mindmap, visualization could create edge to node that already exist.

- A
  - B
  - C
    - D
      - E
        - [C](test.md#c)

@nf-core would also be intersted in something along the lines of this. It's maybe not directly relevent for for the primary usecase,  but the sort of 'london underground/metro' design that markmap offers, is very similar to what is quite popular in various nf-core bioinformatics pipeline diagrams  (https://nf-co.re/developers/design_guidelines#examples). So we would be really happy to see graph-like linking implemented, as we are currently doing everything manually in inkscape :grimacing:


Allow absolute urls (url schema != component nesting structure)

Nested components in a Router create a nested URL structure, but sometimes I'd like my nested component to use an absolute URL:
function UrlRouter() {
  return (
    <Router>
      <AppFrame
        path="/app">
        <MainPage
          path="main"
          comment="relative path"/>
        <SignInPage
          path="/site-root/please/sign-in/"
          comment="absolute path"/>
      </AppFrame>
    </Router>
  );
}

I guess you don't want to just use  tag here, for the reason of client-side only reload page. I'd love to here more what's the best way to programmatically navigate with  tag ?

@revskill10 This is only related to creating the URL structure in the Router, it's independent from navigation that structure.

good idea

Just to understand more, is your use case such that knowing the environment has many site roots and can't be hardcoded in?

Hi @mtliendo, no, site-roots are unrelated (in my use-case, I just have one). This is just about using a url that does not reflect the html nesting structure.

We supported that in React Router v1-3, but there were composition tradeoffs we quit making in v4 (and also here in @reach/router).
Looks like AppFrame wraps all of your routes, so it doesn't need to be a route component at all:
function UrlRouter() {
  return (
    <AppFrame>
      <Router>
        <MainPage path="app/main" />
        <SignInPage path="site-root/please/sign-in" />
      </Router>
    </AppFrame>
  )
}

If it doesn't wrap all routes, then you can do all sorts of things to get the UI and the URLs that you want:
function SignInPage() {
  return (
    <AppFrame>

stuff

    </AppFrame>
  )
}

function Register() {
  return

}

function UrlRouter() {
  return (
    <Router>
      <AppFrame path="app">
        <MainPage path="main" />
        {/* these are wrapped w/ the route + layout coupling */}
      </AppFrame>

      {/* other nested paths can also get it automatically*/}
      <AppFrame path="account">
        <Profile path="profile" />
      </AppFrame>

      {/* this can use it in its own render logic (see SignInPage) */}
      <SignInPage path="site-root/please/sign-in" />

      {/* and this doesn't use it at all */}
      <Register path="not/using/app/layout" />
    </Router>
  )
}


Fix HeliosSoloDeployment.undeployLeftoverJobs()

See if there are jobs running on any helios agent. If we are using
TemporaryJobs, that class should've undeployed them at this point.
Any jobs still running at this point have only been partially
cleaned up. We look for jobs via hostStatus() because the job
may have been deleted from the master, but the agent may still not
have had enough time to undeploy the job from itself.

@negz @mattnworb

Current coverage is 48.08%

Merging #887 into master will increase coverage by +0.01% as of f6fd045

@@            master    #887   diff @@
======================================
  Files          271     271
  Stmts        12850   12850
  Branches      1649    1649
  Methods          0       0
======================================
+ Hit           6177    6179     +2
+ Partial        468     466     -2
  Missed        6205    6205

Review entire Coverage Diff as of f6fd045

Powered by Codecov. Updated on successful CI builds.

In the case that the job has been deleted from the master, but the agent is in the process of undeploying it, does it really make sense to undeploy it again? Maybe it is simpler to just have HSD wait a little bit longer?
I think maybe we might need to customize the cleanup logic that HSD goes through, where somehow shutting down the container causes everything that helios-solo-container has ever deployed to be removed/undeployed - which means that helios-solo would have to keep track of what it has deployed somehow.
I think trying to do cleanup on the java side, in HeliosSoloDeployment.close(), is bound to always miss stuff.

When you say "trying to do cleanup on the java side, in HeliosSoloDeployment.close(), is bound to always miss stuff," do you mean have logic in start.sh instead?

@gimaker

@mattnworb I explored how to optionally set HSD in TemporaryJobs so that when a job is deployed, TemporaryJobs sends a reference of the docker container ID to HSD. But there are many more levels of indirection than I first thought: TemporaryJobs -> TemporaryJobsBuilder -> Deployer -> DefaultDeployer -> TemporaryJob.
I'm leaning towards just merging this. Let me know what you think.

Merging for now. I'm open to better solutions as we think of them.


hotfix: fixed the unterminated interval for registry syncing

Current dependencies on/for this PR:

main

PR #842   👈

This stack of pull requests is managed by Graphite.


Expose un-healthy nodes in info

docker info should expose un-healthy nodes along with an error message.
Pseudo-Example:
Nodes: 1
 dev: <IP_ADDRESS>:4242
  └ Status: Down
  └ Error: <IP_ADDRESS>: No route to host

Thoughts?

:+1: Good idea as it helps troubleshoot connectivity issues to Swarm nodes. Most of the time we would think it's a Swarm bug (in discovery for example) while in reality it might just be a misconfigured firewall or an unexposed port.

It's a great step towards cluster management APIs that you have mentioned earlier.

When using a token with dozens of registered machines, what if you just want to show the active nodes instead of every entry? When using a backend such as Etcd, inactive nodes are automatically removed after a set TTL so how would this work with that?

:+1: More easy to find what's wrong.

Merged into the node management proposal (#1486)


Move exercises into subdirectories.

Requires an updated configlet script which came from xerlang git repo.

Ah, I thought we had taken out the configlet scripts from the repo; must have missed a spot!


standard unit list might be pulling from EML 2.2 list instead of 2.1.1

This editor creates EML 2.1.1
However, it looks to me like the standard unit list it is using is the one from EML 2.2.
Evidence: The drop down includes milligramPerLiter which is not in the 2.1.1 unit list (it was added for EML 2.2)
EML 2.1 has milligramsPerLiter (note the 's') which is valid 2.2, but deprecated in favor of the singular spelling.

We're doing EML 2.2, so this is ok.


[SYSTEMML-769] [WIP] Adding support for native BLAS

Based on the discussion with @frreiss and @bertholdreinwald , I am proposing to switch our default BLAS from Java-based BLAS to native BLAS. We will recommend using Intel MKL and provide optional support other BLAS such as OpenBLAS (and possibly Accelerate) etc. Also, if no BLAS is installed, we will switch to Java-based BLAS. This future-proofs SystemML from hardware improvement made in other BLAS and also simplifies testing.
The proposed solution in this jar will work:

In distributed setting with no additional dependency other than BLAS.
On hybrid cluster with different types of BLAS.
With Parfor.
Without an additional artifact: systemml-accelerator.jar (like https://github.com/apache/incubator-systemml/pull/307).

Since we are not including external dependency (such as BLAS), this PR adds no additional overhead on the release process. Also when we add support to TSMM, SYSTEMML-1166 will be resolved and will again future proof SystemML on related issues.
The initial performance numbers are same as that of https://github.com/apache/incubator-systemml/pull/307
@mboehm7  @dusenberrymw  @nakul02 @lresende @deroneriksson @asurve  @fschueler

Will we produce different jar releases for different platforms? If so, it might help with JCuda as well.

No, I would recommend packaging .so, .dll, etc in single jar similar to JCuda approach. Since the JNI API is extremely lightweight, the overhead would be minimal.

Thanks @niketanpansare  for sharing your proposal.
Can you share benefits of using native based BLAS over Java-based BLAS, other than two below?
- One you mentioned about possible hardware improvements made in other BLAS can benefit SystemML.

Second you mentioned, Simplifies testing

How much performance improvement from native BLAS over Java based BLAS?
In my opinion, we have to write CPP interface for every BLAS for every needed functionality on top of JNI code to interact with generic SystemML Matrix CPP which will burden SystemML in development and increase test effort. If there is significant performance improvement then its worth.
Other thing to consider if there is a benefit from hardware improvements in other BLAS, either those could be available in Java based BLAS or we can add them in Java based BLAS (assuming those are open source library).
Why can't Java based BLAS supports different type of BLAS? If its not supporting, shouldn't it be the place to have such code.
I would be cautious to understand whole proposal before we can take this into consideration.

Please see PR 307 for the performance number. The difference is about 2x to 5x for matrix multiplication and about 3x for conv2d.
If the concern is about development overhead, there are two solutions:

Only support and test with the community version of Intel MKL.
The only code that varies with different BLAS is threading logic to enforce sequential BLAS. If necessary we can factor that out but it would still require as many shared libraries as types of BLAS.

Let me re-emphasize that development overhead of maintaining the JNI overhead is far less than continuously supporting Java based BLAS.
If you can concerned about testing, in fact adding native BLAS simplifies it as Intel would have already done performance testing on different shapes n sizes on different CPUs. Think about testing interfaces vs rigorous performance testing for mat multiplication, tsmm, etc.
I don't understand few of your questions:
Why can Java based blas support different type of blas ?
Adding hardware improvements to Java based blas ? (May be we are not on same page. I would encourage you to look at LibMatrixMult code and how people support avx, fma, numa awareness, etc in C++. I actually think our Java based BLAS is much better than any other Java based BLAS out there (especially in cache friendliness), but there is a cost of maintaining it with evolving hardware improvement.)

I see that you've put a lot of effort in here but I'm still not convinced by the experiments conducted so far.
Instead of trying many combinations of small row/column dimensions (that have limited impact on real workloads), could we please select four representative scenarios and thoroughly evaluate them (incl local and distributed operations, different BLAS libraries, different sparse representations, etc)? Let's use the following scenarios (where (1) and (2) are supposed to be memory-bandwidth bound, whereas (3)-(5) are compute-bound; (1) and (2) are representative for regression and binomial classification, (3) and (4) are representative for multinomial classification, and (5) represents deep learning):
(1) Matrix-vector/vector-matrix 1M x 1K, dense (~8GB)
(2) Matrix-vector/vector-matrix 1M x 10K, sparsity=0.1 (~12GB)
(3) Matrix-matrix/matrix-Matrix 1M x 1K x 20, dense (40GFLOPs)
(4) Matrix-matrix/matrix-Matrix 1M x 10K (sparsity=0.1) x 20, (40 GFLOPs)
(5) Squared matrix 3K x 3K, dense (54 GFLOPs)

Let's run all five scenarios (with left/right exchange, and local/distributed operations) - if we would add support for BLAS libraries then anyway for both dense and sparse. Also let's run at least with MKL and OpenBLAS as it would ensure that the integration is sufficiently general. Personally, I would be fine with adding BLAS support (enabled by default) if it consistently achieves speedups by more than 2-3x.  If the results are a mixed bag, we might still consider it as an optional feature (disabled by default) but it would certainly not yield simplifications as we would still need to maintain over own java-based library.

Thanks, that clarifies a lot. 2x to 3x seems reasonable to decide whether to keep it as optional or default. I will work on the experiments today. Since this PR does not change sparse matmult, would you still want me to test with sparsity=0.1 ?

right now, we're talking only about micro benchmarks - I would recommend to integrate the calls to sparse BLAS operations over a CSR representation just to be able to run these experiments. This should be pretty straightforward and not much effort, right?

OK, I don't see any BLAS specific interface. I am assuming these BLAS libraries have common interfaces. So my concern over maintenance of code specific to BLAS has resolved,'
Based on code change, you have checked in generated library files (.so).
I would not prefer to check in generated output (.so) in the repository, rather there should be build process to generate these library on a required platform.
If someone does not have access to target platform to build such library they can get release version of library from Maven repository.
Again, this is not to be worried at this point until you finalize the approach.
But then first step would be to provide instructions and/or scripts to build such libraries and next step  to automate running script as a part of build process.
Thanks
Arvind

@mboehm7 I would prefer to first get PR with dense BLAS in first before starting to work on sparse BLAS and/or add other operations (such as solve, dsyrk, etc). Please see below the numbers:
For 10 operations:

Matrix-vector/vector-matrix: 1M x 1K, dense:
Java BLAS: 2.252 sec, MKL: 5.403 sec and OpenBLAS: 23.477 sec

Matrix-matrix/matrix-Matrix 1M x 1K x 20, dense:
Java BLAS: 12.204 sec, MKL: 18.927 sec and OpenBLAS: 33.840 sec

Matrix-matrix/matrix-Matrix 1M x 1K x 100, dense (just out of curiousity):
Java BLAS: 53.218 sec, MKL: 17.090 sec and OpenBLAS: 102.779 sec

Matrix-matrix/matrix-Matrix 3K x 3K x 3K, dense:
Java BLAS: 12.948 sec, MKL: 2.612 sec and OpenBLAS: 2.828 sec

thanks for the initial results. Couple of comments:

Could you please specify what kind of HW you ran these experiments?
Please don't call it Java BLAS as we're not implementing the BLAS interface
Let's add the sparse operations - otherwise we cannot make an informed decision whether or not we want to integrate BLAS at all.

Additional considerations are:

for high dimensional matrices that are compute intensive, the benefits of using MKL/OpenBLAS seem to be there. However for smaller matrices/vectors (m <=20 in your posting), they are slower. What will be the heuristic for switching?
it would be good to see single threaded experiments as well to decide whether there could be any benefits of using MKL in our distributed operations.
I am a little concerned that MKL operations will use memory that is not reflected in our memory estimates, and hence may lead to OOMs.
-I am ok with initially doing it for dense matrices only as sparse ops have much more variety and will require more effort for which we may not have the bandwidth right now.

while thinking about potential "heuristics for switching", we should also ask ourselves if these compute-intensive cases are not already covered by the in-progress GPU backend.

@mboehm7 - we should keep in mind that a GPU may not be as ubiquitous or easy to find as an installed BLAS library (or MKL)

yes, but with more and more workloads being migrated to the could this becomes less of an issue as we can actually request preferred node types.

For simplifying the performance testing, I am OK with paying the penalty for few cases where we are better optimized than Intel MKL. At best, we can push matrix-vector operation to Java matmult and rest to Intel MKL.

Since we officially support compute-intensive operators on CP (eg: deep learning builtin functions, matrix multiplication in case of autoencoder, etc), we should definitely try to optimize these cases (either by improving java matmult or by adding support for Intel MKL).

Regarding OOMs, I don't think MKL makes a copy (atleast the specification doesn't say it does). Whether a certain JDK makes a copy or not for JNI call, is a fair concern. The specification for JDK (as well as IBM JDK) specifies that they don't create a copy; if a certain JDK disregards that specification and if because of that we get OOM, then its a JDK bug.

Where did you get that the JNI call would not create a copy? Of course it does (and the numbers reflect that). Since the JVM is free to compose a logical array from multiple physical fragments, the JVM must do so; otherwise, it could not provide a contiguous array. Even if the array is not fragmented, it is likely always copied, because the asynchronous garbage collector is free to re-arrange the array at any time.

@mboehm7 - Please take a look at this page.
There is a note there which clearly says:
As of JDK/JRE 1.1, programmers can use Get/Release<primitivetype>ArrayElements functions to obtain a pointer to primitive array elements. If the VM supports pinning, the pointer to the original data is returned; otherwise, a copy is made.

New functions introduced as of JDK/JRE 1.3 allow native code to obtain a direct pointer to array elements even if the VM does not support pinning.

So how many JVMs do you think actually support pinning of large arrays? I tend to believe that all major JVMs actually store arrays in multiple fragments. Take a look at http://www.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.aix.70.doc/diag/understanding/jni_copypin.html and https://www.ibm.com/developerworks/library/j-jni/ - usually only small arrays are returned as a direct pointer due to the issues mentioned above. How else would you explain the 2.5x slowdown for matrix-vector, which is a trivial operation?

@bertholdreinwald Please see below for the results of single-threaded implementation:
For 10 operations (single-threaded):
Matrix-vector/vector-matrix: 1M x 1K, dense:
LibMatrixMult: 75.753 sec, MKL: 71.533 sec and OpenBLAS: 62.354 sec
Matrix-matrix/matrix-Matrix 1M x 1K x 20, dense:
LibMatrixMult: 209.818 sec, MKL: 93.709 sec and OpenBLAS: 91.460 sec
Matrix-matrix/matrix-Matrix 1M x 1K x 100, dense (just out of curiousity):
LibMatrixMult: 648.379 sec, MKL: 146.276 sec and OpenBLAS: 161.662 sec
Matrix-matrix/matrix-Matrix 3K x 3K x 3K, dense:
LibMatrixMult: 138.946 sec, MKL: 15.938 sec and OpenBLAS: 16.843 sec

As I said before, please specify the HW; otherwise we can't really interpret these numbers.

Please see below the hardware specification:
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
CPU(s):                24
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K

Memory allocated in the above tests: 20G

Thanks @niketanpansare  that helps. However, it also shows measurement errors: how do you explain for Java matrix-vector a speed-up of 34 given 24 virtual cores?
Furthermore, and just to avoid that people draw wrong conclusions here: distributed operations will look very similar to the multi-threaded (not single-threaded!) operations because both become memory-bandwidth bound. In any case, we need to run the distributed operations as well - there are no shortcuts.

Thanks @niketanpansare that helps. However, it also shows measurement errors: how do you explain for Java matrix-vector a speed-up of 34 given 24 virtual cores?

The numbers reported here are execution time for b+* from our Statistics and are fairly easy to reproduce. All I did was redirect matrixMult(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, int k) to matrixMult(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret) ... Since there is fairly complicated logic (for example: recomputeNonZeros in parallel vs sequential, interleaving of memory access), I would not be suspicious of the speedup.

Furthermore, and just to avoid that people draw wrong conclusions here: distributed operations will look very similar to the multi-threaded (not single-threaded!) operations because both become memory-bandwidth bound. In any case, we need to run the distributed operations as well - there are no shortcuts.

That is a fair point, one cannot directly correlate single-threaded performance with distributed performance. However, as I said earlier, "For simplifying the performance testing, I am OK with paying the penalty for few cases where we are better optimized than Intel MKL."

well I'm not fine with it because a 2-3x slowdown on scenario (1) and (2) directly affects end-to-end performance of common algorithms such as LinregCG, GLM, L2SVM, MSVM, MLogreg, Kmeans, and PageRank.

Then, please feel free to update the memory bound logic when the PR is merged. Though slightly tricky, Statistics will still help us with performance debugging. I hope we both agree on the fact that Intel MKL is much-better optimized than our LibMatrixMult.

Also, seeing a speedup of 34 for matrix-vector on 24 virtual cores should raise red flags. Usually, we only see very moderate speedups of 3-5x for matrix-vector because again, at one point this operation becomes memory-bandwidth bound.
I suspect that garbage collection, or just-in-time compilation to interfere with the measurement here. Hence, I would recommend to setup a proper micro-benchmark, with isolated block operations, a number of warm-up runs for just-in-time compilation, and verbose GC flag to exclude runs that overlapped with GC.

That could be the case. Again, I don't dispute that LibMatrixMult is better option for memory-bound cases and identifying those cases should not stop us from integrating Intel MKL.

Absolutely, for scenarios like (5), we should definitely consider it. However, modifying a core operation like matrix multiplication requires a thorough experimental analysis of individual operations and their integration. Right now, I would be very uncomfortable merging this PR in its current form into SystemML master.

Could we also run a larger matrix-matrix test?  How about n rows x 196,608 columns %% 196,608 x 512, where 196,608 is the number of pixels in a single 2562563 HW*C image.  Given recent results from the breast cancer project, it would be good to test scenarios that aren't the traditional skinny matrices.

@mboehm7  I am worried we have spent too much time on this without making any progress. Would you be comfortable with the following proposal:

We start by keeping BLAS support optional. That is, one has to explicitly enable it by setting the environment variable "SYSTEMML_BLAS=mkl".
Then we create a manageable list of experiments along with a precise specification beforehand and we distribute the experiments among the committers.
Then, depending on the results of the experiments we decide to enable it by default.

@dusenberrymw The results of 10 matrix multiplication for n rows x 196,608 columns %*% 196,608 x 512 are as follows:
Single-threaded performance:
n = 64 => LibMatrixMult: 36.783 sec, MKL: 6.196 sec, OpenBLAS: 6.900 sec
n = 512 => LibMatrixMult: 284.322  sec, MKL: 33.440 sec, OpenBLAS: 35.167 sec

Multi-threaded performance:
n = 64 => LibMatrixMult: 5.487 sec, MKL: 1.142  sec, OpenBLAS: 6.771 sec
n = 512 => LibMatrixMult: 24.260 sec, MKL: 4.780 sec, OpenBLAS: 7.482 sec
n = 1024 => LibMatrixMult: 47.501 sec, MKL: 10.037 sec, OpenBLAS: 13.068 sec

@Mike sure let's use this as a sixth scenario. I've seen both squared and rectangular matrix multiplications as common operations across different applications of text, image and speech.
@niketanpansare sorry no; at a bare minimum we should remove the integration from LibMatrixMult because these block operations apply implicitly to all backends. Having not tested distributed operations, sparse operations and the break-even points of performance, would only leave us the option to integrate it into AggregateBinaryCPInstruction, for dense inputs, and with very conservative assumptions. However, given the status of many experimental features and recently integrated workarounds (like sparse and caching flags), I'm really concerned that such an unfinished integration would stay for long time. Instead of spending our time arguing about it, why could we not simply create a prototype and test this out?

@bertholdreinwald  @mboehm7 Please see below points:

Right now, we are working in performance testing of rc1. I can perform experiments on distributed operations once the cluster is free.
I would not prefer to implement sparse operations in this PR as I am spread too thin. But I am not against supporting sparse operations, so if someone has free cycles, I would really appreciate the help.
Regarding the break-even points, I am OK with keeping it at matrix-vector operations.
I ran 1M X 1K %*% 1K X 1K matrix multiplication with MKL with 20GB JVM and didnot get an OOM. I cannot conclusively answer whether JNI makes a copy or not.

To summarize, I have completed following experiments:
(1) Matrix-vector/vector-matrix 1M x 1K, dense (~8GB)
(3) Matrix-matrix/matrix-Matrix 1M x 1K x 20, dense (40GFLOPs)
(5) Squared matrix 3K x 3K, dense (54 GFLOPs)
(6) n rows x 196,608 columns %*% 196,608 x 512

Following experiments are remaining:
(2) Matrix-vector/vector-matrix 1M x 10K, sparsity=0.1 (~12GB)
(4) Matrix-matrix/matrix-Matrix 1M x 10K (sparsity=0.1) x 20, (40 GFLOPs)
(7) Distributed Experiments ==> where we increase the number of rows such that RHS matrix in above cases doesnot fit in the driver.

(2) Well, for me sparse operations are a must-have.
(3) We should come up with something better than matrix-vector/vector-matrix as a heuristic because otherwise common algorithms like Kmeans or Mlogreg will get a slowdown for common scenarios of number of clusters or number of classes <=20.
(4) Regarding OOMs: of course you will not see an OOM from Java; the risk here is swapping because you allocate native memory. Our TensorFlow comparison last year  (which copied data from TF data structures out to Eigen) showed exactly this problem once you're in a space where your matrix does not fit twice into physical memory. In our case, it would mean that we always have to leave headroom (above JVM heap size) for the largest matrix.
Finally, what's the purpose of scenario (7), once you increase the RHS while increasing the LHS you end up with shuffle-based implementations that dominate everything, because any broadcast-based operator would need to hold the broadcast data at the driver too. Please increase the number of rows in the LHS for scenarios (1)-(4) while keeping all other parameters constant. For scenario (5) and (6) you might want to use PMAPMM; otherwise you're again bound by shuffle.

(3) We should come up with something better than matrix-vector/vector-matrix as a heuristic because otherwise common algorithms like Kmeans or Mlogreg will get a slowdown for common scenarios of number of clusters or number of classes <=20.

If you are suggesting that the break-even point should be 20, I am OK with that. The above numbers also support this hypothesis.

(4) Regarding OOMs

This was just for sanity check so that this PR doesnot break the existing script.

Finally, what's the purpose of scenario (7), once you increase the RHS

Sorry, there is a type: it should have been LHS :)

several arguments got misinterpreted in this long thread. Will set up a phone conversation in the next couple of days to resolve this and define the next steps.

Here are the notes from the discussion with Berthold and Matthias:
Native BLAS provide benefits as exploit SIMD on newer CPUs. JVMs don't compile packed instructions.
JVMs make copy of data communicating to native BLAS.

Single Node (CP).

A. Microbenchmarks.
Measure purely the BLAS performance leaving out all DML overhead, i.e. 5 warmup runs to have JIT, 20 runs and remove outliers.
Bypass DML and use direct lib matrix invocation driver.
Experiments:

Dense Data, i.e. sparsity > 0.4
a. matrix x vector: 1M x 1K x 1
b. matrix x small matrix: 1M x 1K x 20
c. square matrices: 3K x 3K x 3K
d. short-wide matrix: 100 x 196K x 512 <-- compare this nbr to GPU performance.
e. XtX 1M x 1K

Sparse Data. MKL has 6 different representations; for a quick experiment choose 1 rep, e.g. CSR.
Purpose of experiment is to find out if/how much benefit native BLAS will provide for sparse data.
If feasible, try same micro benchmarks as for dense, otherwise try matrix x small matrix case.

B. End-to-End

use perf suite to run
a. 1 of following: LinregDS, PCA
b. One of following: linregCG, GLM, L2SVM, Mlogreg (binomial)
c. one of following: kmeans, mlogreg (multinomial)
d. 1 convnet.
on sparse and dense data and data characteristics currently used
on single node

Distributed
Purpose of this experiment is to find out the "right" integration strategy, i.e. "switch" in matrix lib, or new instructions.
set MKL #threads to 1.
run 1 compute intensive case where we should see biggest benefit, i.e. scale up squared matrix multiply. Want to enable PMAPMM as it stress most compute. A CPMM plan would not be good as is shuffle-bound.

Staging: initially AggBinary only; subsequently look at decomposition/solve/cholesky kernels.

CP - dense
Distributed - dense
Decomposition kernels, etc.
Sparse? (staging of sparse depends on experiments 1.A for sparse data)

thanks for the notes @niketanpansare ; some smaller additions:

The micro-benchmarks map to end-to-end scenarios as follows: A.a -> B.b, A.b -> B.c, A.cd -> B.d, A.e -> B.a.
The purpose of (2) is to see how the differences in (1) carry through to end-to-end performance and if there are side effects in distributed settings which require additional experiments (e.g., GC effects, etc)
If CP is staged independently, the block-level integration needs to be moved to instruction level

Since we are anyways planning to perform the distributed experiments before merging this PR and this PR already has distributed dense integration, let's make a decision (after the experiments ofcourse) whether to even support distributed BLAS. If yes, then we merge this PR as is and if not, then as @mboehm7 mentioned "the block-level integration needs to be moved to instruction level". So the revised staging will be as follows:

Either CP instruction-level dense integration or CP+Distributed block level dense integration.
Decomposition kernels, TSMM, etc.
Sparse? (staging of sparse depends on experiments 1.A for sparse data)

I have created a experimental setup for microbenchmarks as per @mboehm7 suggestions. Here is how you can reproduce them:
git clone https://github.com/niketanpansare/incubator-systemml.git
git checkout switch_blas_experiment
mvn package -P standalone-jar
cp target/systemml*standalone.jar SystemML.jar
for i in 1 2 ...
do
        java -Xmx40G -server -cp SystemML.jar:commons-io-2.4.jar org.apache.sysml.api.DMLScript -exp $i
done

Regarding the sparse experiments, I implemented initial prototype for sparse matrix multiply, but for some reason the output is all zeros:
https://github.com/niketanpansare/incubator-systemml/blob/switch_blas_experiment/src/main/cpp/libmatrixmult.cpp#L80 ... @bertholdreinwald @mboehm7 @nakul02 Since you have more experience with our sparse matrix multiplication, please let me know if you have suggestions to fix the bug. Once the bug is fixed, I can run the sparse experiments.
The end-to-end experiments are waiting for cluster availability.

@niketanpansare I'm encouraged by the work in this thread so far, but unfortunately I need to bring up one item. I don't believe we will be able to include the .so and .dll shared library binaries into the source code, per Apache rules.  When we started, we had to remove existing .dll files since they are considered opaque "binary" files. It would be good to double check on this though.

@dusenberrymw @niketanpansare
I believe .so and .dll files are fine but that they can't be in the 2 src distributions (tgz and zip). I believe the rules with the src distributions are 1) you must be able to build the project and 2) you must be able to run the tests and the tests pass. I believe they are fine in the other artifacts assuming that all Apache license guidelines are followed.
@lresende Is this your understanding?

@deroneriksson Does that imply that our src release should have the ability to target the GPU backend once it is built?

Based on @deroneriksson, we will have to cleanup the resources before compiling the src distribution.

@dusenberrymw
Apache release policy for source releases (http://www.apache.org/legal/release-policy.html#source-packages) says:
"Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools."
At same time, these source artifacts can't contain binaries such as dll files, which we experienced with hadoop.dll and winutils.exe.
It seems to me that the main restriction is that the source releases can't contain binaries but the binary releases can contain binaries. I think that is the restriction and you can do most things as long as you work within the Apache guidelines.

It's good to verify this with our mentors @lresende @hsaputra

@niketanpansare src/assembly/source.xml contains an example of excluding hadoop_bin_windows from the source artifacts:
L53: <exclude>./src/test/config/hadoop_bin_windows/**</exclude>

That is correct. No binary files, including so or dll should be in source artifact release.
Need to be able to build in runtime or have config to use the native BLAS option

A. Microbenchmarks.
git clone https://github.com/niketanpansare/incubator-systemml.git
git checkout switch_blas_experiment
mvn package -P standalone-jar
cp target/systemml*standalone.jar SystemML.jar
# Dense experiments
for i in 1 2 5 7 8 9
do
        java -Xmx40G -server -cp SystemML.jar:commons-io-2.4.jar org.apache.sysml.api.DMLScript -exp $i
done

Setup 1: java -Xmx40G ...
Setup 2: java -Xmx20G ...
Both setup 1 and 2 are run of the following machine:
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
CPU(s):                24
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K

We ignore first 5 iterations and then run 20 iterations. We drop the best and the worst run and average the time of the remaining 8 runs.
The time is reported in milliseconds.

Setup
LibMatrix (single)
MKL (single)
LibMatrix (multi)
MKL (multi)

Matrix-vector 1M x 1K, dense
1
953.1612
552.7734
188.5189
466.4393

Matrix-matrix 1M x 1K x 20, dense
1
15131.62
2392.395
1231.446
1052.258

Squared matrix 3K x 3K, dense
1
11682.76
1389.884
1062.401
218.4674

64 rows x 196,608 columns %*% 196,608 x 512
1
3047.32
484.3707
498.5716
149.428

1024 rows x 196,608 columns %*% 196,608 x 512
1
49041.85
5820.965
4006.641
903.6708

Sparse Data: Waiting for help from @mboehm7 or @nakul02 for bug described above
B. End-to-End

MKL Total (dist)
LibMatrixMult (dist)

Squared matmult: 50000 dense
3067.811 (cpmm)
3023.844 (cpmm)

Squared matmult: 80000 dense
failed with OOM
failed with OOM

The setup is exactly same as our VLDB paper except the Spark version used is 2.0.1

MKL Total (1Mx1K)
MKL Total (10Mx1K)
LibMatrixMult (1Mx1K)
LibMatrixMult (10Mx1K)

L2SVM
31.93
190.050
29.2576
147.763

MultiLogReg
147.381
1241.498
129.23
945.18

MSVM
85.289
439.477
68.845
426.852

Naive Bayes
23.603
155.74
22.554
173.986

Linear Regression Direct Solve
65.127
197.672
67.82
202.265

Linear Regression CG
22.903
161.807
20.05
157.191

GLM
52.152
324.045
53.568
283.870

Thanks, Niketan. Very good. Couple of questions/clarifications:

For the Microbenchmarks, does dense mean 100% ?

For the End-to-End, I assume that you had a block size of 1k x 1k, and

in that case, in the executors, did mmult run single or multithreaded?
Either way, MKL should be 5-6 faster. But as it does not show any improvement, it says that it is not the bottleneck, and the bottleneck is somewhere else. Just to verify, good you take a look at the stats?

Also, choosing a larger blocksize would shift the bottlenecks, and MKL could show some benefits.

For the end-to-end algs, I assume that those were single node runs for both data sizes.

Given the micro benchmarks, I'd suggest to rerun the numbers, and not use MKL for matrix-vector.

And may be add 1 DL alg to see numbers with matrix-matrix mult in the loop.

for distributed squared multiply, please use pmapmm instead of cpmm; otherwise you're bound by shuffling.

For the Microbenchmarks, does dense mean 100% ?

No sparsity = 0.9

For the End-to-End, I assume that you had a block size of 1k x 1k, and in that case, in the executors, did mmult run single or multithreaded?

Single-threaded calls to MKL by many tasks in parallel.

Either way, MKL should be 5-6 faster. But as it does not show any improvement, it says that it is not the bottleneck, and the bottleneck is somewhere else. Just to verify, good you take a look at the stats?
Given the micro benchmarks, I'd suggest to rerun the numbers, and not use MKL for matrix-vector.

Please see below chart that shows the performance of MKL without matrix-vector vs LibMatrixMult

MKL Total (1Mx1K)
MKL Total (10Mx1K)
LibMatrixMult (1Mx1K)
LibMatrixMult (10Mx1K)

L2SVM
27.22
159.753
29.2576
147.763

MultiLogReg
142.31
1317.388
129.23
945.18

MSVM
78.333
286.65
68.845
426.852

Naive Bayes
22.779
154.291
22.554
173.986

Linear Regression Direct Solve
64.602
198.84
67.82
202.265

Linear Regression CG
29.02
198.936
20.05
157.191

GLM
55.998
293.102
53.568
283.870

Also, choosing a larger blocksize would shift the bottlenecks, and MKL could show some benefits.

Possible but that might introduce unintended side-effects. So, for now, we should not change the block size.

For the end-to-end algs, I assume that those were single node runs for both data sizes.

The 1Mx1K refers to the bar pointed by the black arrow and 10Mx1K refers to the bar pointed to by the red arrow. The cases when SystemML cp+spark backend does not create a SparkContext are annotated with "X". The cases when the input data fits into aggregated cluster memory but not a single node and where perform automatic checkpoint injection are annotated with "Y".

And may be add 1 DL alg to see numbers with matrix-matrix mult in the loop.

For Lenet with MNIST dataset,
Statistics for training 800 iterations without MKL
Total elapsed time:             317.491 sec.
Total compilation time:         2.160 sec.
Total execution time:           315.331 sec.
Number of compiled Spark inst:  147.
Number of executed Spark inst:  4.
Native  calls (Mult/DNN/Fails): 0/0/0.
Cache hits (Mem, WB, FS, HDFS): 109700/0/0/2.
Cache writes (WB, FS, HDFS):    60440/15/8.
Cache times (ACQr/m, RLS, EXP): 3.726/0.046/37.985/1.265 sec.
HOP DAGs recompiled (PRED, SB): 0/15493.
HOP DAGs recompile time:        3.330 sec.
Functions recompiled:           1.
Functions recompile time:       0.026 sec.
Spark ctx create time (lazy):   1.004 sec.
Spark trans counts (par,bc,col):0/0/2.
Spark trans times (par,bc,col): 0.000/0.000/3.689 secs.
Total JIT compile time:         35.829 sec.
Total JVM GC count:             802.

Number of executed Spark inst:  4.
Native mkl calls (Mult/DNN/Fails):      2598/6898/0.
Cache hits (Mem, WB, FS, HDFS): 109700/0/0/2.
Cache writes (WB, FS, HDFS):    60440/15/8.
Cache times (ACQr/m, RLS, EXP): 3.517/0.041/39.695/1.245 sec.
HOP DAGs recompiled (PRED, SB): 0/15493.
HOP DAGs recompile time:        3.369 sec.
Functions recompiled:           1.
Functions recompile time:       0.028 sec.
Spark ctx create time (lazy):   0.985 sec.
Spark trans counts (par,bc,col):0/0/2.
Spark trans times (par,bc,col): 0.000/0.000/3.481 secs.
Total JIT compile time:         35.505 sec.
Total JVM GC count:             517.

Number of executed Spark inst:  74.
Native mkl calls (Mult/DNN/Fails):      0/0/0.
Cache hits (Mem, WB, FS, HDFS): 1900/0/0/43.
Cache writes (WB, FS, HDFS):    672/2/1.
Cache times (ACQr/m, RLS, EXP): 307.222/0.095/1.285/0.164 sec.
HOP DAGs recompiled (PRED, SB): 1/181.
HOP DAGs recompile time:        0.411 sec.
Spark ctx create time (lazy):   31.268 sec.
Spark trans counts (par,bc,col):0/68/42.
Spark trans times (par,bc,col): 0.000/17.052/306.578 secs.
ParFor loops optimized:         1.
ParFor optimize time:           0.549 sec.
ParFor initialize time:         0.033 sec.
ParFor result merge time:       0.008 sec.
ParFor total update in-place:   0/26/29
Total JIT compile time:         64.062 sec.
Total JVM GC count:             142.
Total JVM GC time:              20.393 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sp_mapmm        216.800 sec     68
-- 2)   +*      194.874 sec     140
-- 3)   *       142.629 sec     348
-- 4)   sp_chkpoint     31.906 sec      6
-- 5)   sel+    28.835 sec      98
-- 6)   >       21.710 sec      77
-- 7)   1-*     16.732 sec      98
-- 8)   tak+*   11.350 sec      154
-- 9)   >=      6.235 sec       42
-- 10)  rmvar   3.890 sec       3107

With LibMatrixMult:
Total elapsed time:             424.994 sec.
Total compilation time:         2.015 sec.
Total execution time:           422.979 sec.
Number of compiled Spark inst:  16.
Number of executed Spark inst:  74.
Cache hits (Mem, WB, FS, HDFS): 1900/0/0/43.
Cache writes (WB, FS, HDFS):    672/1/1.
Cache times (ACQr/m, RLS, EXP): 471.759/0.006/0.268/0.194 sec.
HOP DAGs recompiled (PRED, SB): 1/181.
HOP DAGs recompile time:        0.362 sec.
Spark ctx create time (lazy):   31.668 sec.
Spark trans counts (par,bc,col):0/68/42.
Spark trans times (par,bc,col): 0.000/14.155/471.240 secs.
ParFor loops optimized:         1.
ParFor optimize time:           0.541 sec.
ParFor initialize time:         0.034 sec.
ParFor result merge time:       0.007 sec.
ParFor total update in-place:   0/26/29
Total JIT compile time:         55.369 sec.
Total JVM GC count:             136.
Total JVM GC time:              19.416 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sp_mapmm        561.430 sec     68
-- 2)   +*      263.128 sec     140
-- 3)   *       235.146 sec     348
-- 4)   sp_chkpoint     32.300 sec      6
-- 5)   sel+    25.936 sec      98
-- 6)   >       18.506 sec      77
-- 7)   1-*     10.323 sec      98
-- 8)   tak+*   7.302 sec       154
-- 9)   rmvar   6.078 sec       3107
-- 10)  >=      3.896 sec       42

To get below results, I forced the matrix multiplication operator to pmapmm and used the below given DML script:
X = rand(rows=$1, cols=$2)
Y = rand(rows=$2, cols=$3)
Z = X %*% Y
if(1 == 1) {
        print(sum(Z))
}

With $1 = $2 = $3 = 50K,
Statistics with MKL enabled (no error):
17/02/10 14:37:04 INFO api.DMLScript: SystemML Statistics:
Total elapsed time:             1076.101 sec.
Total compilation time:         0.599 sec.
Total execution time:           1075.502 sec.
Number of compiled Spark inst:  4.
Number of executed Spark inst:  4.
Native mkl calls (Mult/DNN/Fails):      0/0/0.
Cache hits (Mem, WB, FS, HDFS): 1/0/0/0.
Cache writes (WB, FS, HDFS):    0/0/0.
Cache times (ACQr/m, RLS, EXP): 0.000/0.001/0.000/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/2.
HOP DAGs recompile time:        0.007 sec.
Spark ctx create time (lazy):   32.085 sec.
Spark trans counts (par,bc,col):0/0/13.
Spark trans times (par,bc,col): 0.000/0.000/122.305 secs.
Total JIT compile time:         51.14 sec.
Total JVM GC count:             213.
Total JVM GC time:              74.344 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sp_pmapmm       1037.506 sec    1
-- 2)   sp_rand         32.421 sec      2
-- 3)   sp_uak+         0.648 sec       1
-- 4)   rmvar   0.044 sec       7
-- 5)   print   0.000 sec       1
-- 6)   createvar       0.000 sec       4
-- 7)   castdts         0.000 sec       1
-- 8)   cpvar   0.000 sec       1

With LibMatrix, I get following error:
17/02/10 15:52:56 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Executor for container container_1485300997081_0427_01_000005 exited because of a YARN event (e.g., pre-emption) and not because of an error in the running job.
17/02/10 15:52:56 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Error requesting driver to remove executor 4 for reason Executor for container container_1485300997081_0427_01_000005 exited because of a YARN event (e.g., pre-emption) and not because of an error in the running job.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
        at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154)
        at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:129)
        at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)
        at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)
        at org.apache.spark.rpc.RpcEndpointRef.ask(RpcEndpointRef.scala:63)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receive$1.applyOrElse(YarnSchedulerBackend.scala:263)
        at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
17/02/10 15:52:57 INFO memory.MemoryStore: MemoryStore cleared
17/02/10 15:52:57 INFO storage.BlockManager: BlockManager stopped
17/02/10 15:52:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/02/10 15:52:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/02/10 15:52:57 INFO spark.SparkContext: Successfully stopped SparkContext
17/02/10 15:52:57 INFO api.DMLScript: SystemML Statistics:
Total elapsed time:             4546.056 sec.
Total compilation time:         0.577 sec.
Total execution time:           4545.478 sec.
Number of compiled Spark inst:  4.
Number of executed Spark inst:  4.
Cache hits (Mem, WB, FS, HDFS): 1/0/0/0.
Cache writes (WB, FS, HDFS):    0/0/0.
Cache times (ACQr/m, RLS, EXP): 0.000/0.001/0.000/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/2.
HOP DAGs recompile time:        0.006 sec.
Spark ctx create time (lazy):   36.146 sec.
Spark trans counts (par,bc,col):0/0/13.
Spark trans times (par,bc,col): 0.000/0.000/117.186 secs.
Total JIT compile time:         48.357 sec.
Total JVM GC count:             210.
Total JVM GC time:              64.583 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sp_pmapmm       4500.858 sec    1
-- 2)   sp_rand         36.491 sec      2
-- 3)   sp_uak+         2.842 sec       1
-- 4)   rmvar   0.061 sec       7
-- 5)   print   0.000 sec       1
-- 6)   createvar       0.000 sec       4
-- 7)   castdts         0.000 sec       1
-- 8)   cpvar   0.000 sec       1

@mboehm7 @nakul02 @bertholdreinwald @deroneriksson @asurve @dusenberrymw  I plan to merge this PR once 0.14 RC is approved to give us enough time to perform testing under various setups before 1.0. If you have time, please try below commands on your laptop/cluster:
$ git clone https://github.com/niketanpansare/incubator-systemml.git
$ cd incubator-systemml
$ git checkout switch_blas
$ mvn package
$ cd target
# Create a file with following content:
$ cat matmult.dml
M = 4000
N = 3000
X = matrix(seq(1, M*N), rows=M, cols=N)
Y = matrix(seq(1, N*N), rows=N, cols=N)
Z = matrix(0, rows=M, cols=N)
v = 1
for(i in 1:100) {
        Z = Z + (X %*% Y)
}
print(sum(Z))
$ ~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit SystemML.jar -f matmult.dml -stats

In general, merging this into 1.0 is fine once the following remaining issues are addressed:

Remove the integration from LibMatrixMult and call it either from MatrixBlock or even better from individual instructions. I'd like to keep LibMatrixMult a pure java library - remember that we want to isolate the entire matrix block library as an individual jar in the near future.

Improve the conditions for selecting native BLAS libraries. I find the condition of dispatching all dense matrix-matrix multiplications too optimistic.

Please also cleanup the minor code issues as highlighted below.

@mboehm7 I will incorporate your suggestions today.
All tests passed.

Absolutely. I want to merge this earlier (as soon as we get go-ahead from current RC) to give us about 2-3 months to test performance/robustness on different hardware, OS, BLAS, configurations, networks, workloads, etc. :)

@asurve This is a good reason to have a 0.14 branch. Cutting all the 0.14 RCs from master is causing delays, which means our next release will need to be postponed or won't be tested as thoroughly as it otherwise could be.

I am not clear why this change is waiting for 1.0 based (or major) release.
If you are adding new functionality as a experimental feature until its fully ready, you can add it on regular interval, as far as it does not breaks supported functionality. You don't need to wait for major release.
Are we breaking any backward compatibility? if yes, then it justifies for separate branch for major release.
Are we changing BLAS from Java-based BLAS to native BLAS? Does it causes break in the backward compatibility for users?
Are there any other changes which may cause potential issues in currently supported functionality?
@niketanpansare Please provide details on your concerns for not adding to current master branch.

@asurve Right now we can't commit major updates to master because the RCs are being cut directly from master rather than from a dedicated 0.14 branch. Introducing a major commit such as this PR going from one 0.14 RC to the next 0.14 RC would not be advisable since all the RCs are being cut from the master branch. I don't understand this change of not having a 0.14 branch. For 0.13, we had a 0.13 branch and cherry-picked RC updates on to the 0.13 branch (see https://github.com/apache/incubator-systemml/commit/83f56f30d521152a68e83e698819121624fd97b7).

Are we breaking any backward compatibility?
No.

Are we changing BLAS from Java-based BLAS to native BLAS?
Yes.

Please provide details on your concerns for not adding to current master branch.
This feature depends on BLAS, which in turn depends on the OS and other configuration. Hence, before we add it to a release, we have to test this feature on different setup.

@asurve The master branch is always reserved for the bleeding edge of the project.  Any releases should be placed on separate release branches.  We can't merge in large features that could introduce bugs elsewhere in the system randomly in the middle of two release candidates.

@niketanpansare thanks for clarification.
We need to have another RC sooner, will have RC and branch as well. That should work for you, right?

@nakul02 @bertholdreinwald Here is the CMake file that Nakul created: https://github.com/niketanpansare/systemml-accelerator/blob/master/src/main/cpp/CMakeLists.txt
@nakul02 @bertholdreinwald @asurve @frreiss @aloknsingh  @dusenberrymw  There are three options for this PR, please suggest your preference:

Current approach: Build .so/.dll/.dylib into the jar. Pros - This simplifies deployment. Cons: We are packaging system-dependent files into our jar.
Similar to our approach with JCuda: Depend on separate jar: I had created systemml-accelerator package for this option, but other committers didn't like it.
Ask users to build .so files from source: This complicates deployment.

All tests passed on jenkins. Unless anyone has feedback or suggestions, I would like to merge this PR.

I am still struggling with my local environment to see it working.
Also, prior to deciding between preference (1) and (2), I'd like to understand all the dependencies first required for a successful deployment.

I like approach 2. Its much like JCuda, which we are already using.
(I didn't like the name systemml-accelerator, it would imply that backends other than a pure Java implementation would live in that project).

Definitely either (1) or (2).  Option (3) will definitely complicate deployment.  I think I like option 2, perhaps paired with some optional platform-specific builds that do include the extra jar within our uber jar.  That way, we could still build & release just the plain jar that relies on the user including the appropriate blas jar on the classpath, but also have the option to build and release platform-specific jars by including the appropriate native blas jar (and appropriate jcuda jar) into our uber jar.  The platform-specific jars would just be for user convenience.

Please note that option 2 can be further divided into:

Option 2a: Platform-independent jar but in SystemML project - something along the lines of systemml-1.0.0-blas.jar. Pros: Simpler to support in pip. Cons: Few more KBs added to the jar.
Option 2b: Platform-dependent jar but in SystemML project - Cons: Simpler to support in pip. For example: we donot support GPU via pip.
Option 2c: As a separate project - Cons: Release process more difficult.

I recommend that we either go with option 1 or option 2a. If we come up with a solution for supporting GPU via pip, we can then go with option 2b instead of option 2a.

There is a PR against this branch - https://github.com/niketanpansare/incubator-systemml/pull/12
In a previous PR, I had added some documentation. For some reason, this documentation was removed and a truncated version was introduced along with a shell script.
In my opinion, having a script to do builds in this situation is an incorrect approach.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1376/

Build failed, see build log for details

Build successful

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1379/

Build successful

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1380/

Build successful

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1389/

Build successful

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1397/

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1402/

Build successful

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1410/

Build successful

Here are the updated results on Lenet for 2000 iterations:
With MKL on Linux:
Iter:2000.0, training loss:0.01354500195846287, training accuracy:100.0
Iter:2000.0, validation loss:211.49319894899025, validation accuracy:97.07991803278688
SystemML Statistics:
Total elapsed time:             480.432 sec.
Total compilation time:         0.000 sec.
Total execution time:           480.432 sec.
Number of compiled Spark inst:  79.
Number of executed Spark inst:  2.
Native mkl calls (LibMatrixMult/LibMatrixDNN):  4270/10553.
Cache hits (Mem, WB, FS, HDFS): 281999/0/0/0.
Cache writes (WB, FS, HDFS):    147642/0/0.
Cache times (ACQr/m, RLS, EXP): 0.103/0.074/2.183/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/11305.
HOP DAGs recompile time:        6.609 sec.
Spark ctx create time (lazy):   0.022 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
Total JIT compile time:         36.785 sec.
Total JVM GC count:             4566.
Total JVM GC time:              72.493 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sel+    56.005 sec      6283
-- 2)   conv2d_backward_filter  51.247 sec      4026
-- 3)   conv2d_bias_add         48.540 sec      4514
-- 4)   -*      35.711 sec      16104
-- 5)   +       34.503 sec      30566
-- 6)   +*      34.500 sec      8052
-- 7)   maxpooling_backward     34.057 sec      4026
-- 8)   ba+*    31.741 sec      12566
-- 9)   conv2d_backward_data    28.271 sec      2013
-- 10)  relu_backward   26.899 sec      6039

With Java on Linux:
Iter:2000.0, training loss:0.0059023118210415025, training accuracy:100.0
Iter:2000.0, validation loss:151.31859200647978, validation accuracy:97.46413934426229
SystemML Statistics:
Total elapsed time:             654.523 sec.
Total compilation time:         0.000 sec.
Total execution time:           654.523 sec.
Number of compiled Spark inst:  79.
Number of executed Spark inst:  2.
Cache hits (Mem, WB, FS, HDFS): 281999/0/0/0.
Cache writes (WB, FS, HDFS):    147642/0/0.
Cache times (ACQr/m, RLS, EXP): 0.097/0.073/2.021/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/11305.
HOP DAGs recompile time:        6.575 sec.
Spark ctx create time (lazy):   0.024 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
Total JIT compile time:         39.636 sec.
Total JVM GC count:             7094.
Total JVM GC time:              98.133 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         149.524 sec     4514
-- 2)   conv2d_backward_filter  113.605 sec     4026
-- 3)   sel+    54.545 sec      6283
-- 4)   conv2d_backward_data    50.524 sec      2013
-- 5)   ba+*    41.264 sec      12566
-- 6)   -*      36.036 sec      16104
-- 7)   +*      33.817 sec      8052
-- 8)   +       32.274 sec      30566
-- 9)   *       26.529 sec      35275
-- 10)  maxpooling_backward     25.734 sec      4026

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1419/

Current this PR is up to:
Conversation 102 | Commits 26 | Files changed 51
It has been open since January. We should consider merging or closing before the 1.0.0 release with significant time for testing if it is merged.
@bertholdreinwald @frreiss @mboehm7 @nakul02

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1420/

@deroneriksson I absolutely agree - something like this needs to go in at least a month or two before a release. Let's resolve all remaining build and deploy issues and then bring it in (along with a testsuite that checks for existing libraries and only runs if they are available; of course we should install it on our build server).

@deroneriksson @mboehm7  I think this PR has addressed all the deploy/build issues on Linux. Remaining issues can be addressed later as bugfixes in subsequent PR.
@bertholdreinwald @nakul02  After spending significant amount of time in trial and error for Mac (and Windows), I think we should only support BLAS on Linux machines for following reasons:

GNU OpenMP is not enabled by default on Mac, and it requires either downloading GCC or using experimental clang-openmp.
GNU OpenMP is not supported by Intel MKL on Mac. Hence we see potential degradation due to switching between GNU OpenMP and Intel OpenMP for operations such as conv2d_backward_data.
To compile with only Intel OpenMP, we need to buy an Intel Compiler. GCC or Clang doesnot allow us to compile with Intel OpenMP.
Even if we buy Intel OpenMP, the problem remains with OpenBLAS+OpenMP support.

Since we have already have cmake setup in place, we can always support BLAS on Mac and Windows in future when OpenMP related issues are resolved.

For your reference, please see below for instructions for compiling BLAS-enabled SystemML on Mac and Windows:
64-bit x86 Windows

Install MKL or Download the OpenBlas Binary
Install Visual Studio Community Edition (tested on VS 2017)
Use the CMake GUI, select the source directory, the output directory
Press the configure button, set the generator and use default native compilers option
Set the CMAKE_BUILD_TYPE to Release, this sets the appropriate optimization flags
By default, USE_INTEL_MKL is selected, if you wanted to use OpenBLAS, unselect the USE_INTEL_MKL, select the USE_OPEN_BLAS.
You might run into errors a couple of times, select the appropriate library and include files/directories (For MKL or OpenBLAS) a couple of times, and all the errors should go away.
Then press generate. This will generate Visual Studio project files, which you can open in VS2017 to compile the libraries.
The current set of dependencies are as follows:
MKL: mkl_rt.dll (for functions: mkl_set_num_threads and cblas_dgemm).
OpenBLAS: libopenblas.dll (for functions: openblas_set_num_threads and cblas_dgemm).
Visual C OpenMP: vcomp140.dll (for function: omp_get_thread_num).
Visual C Runtime: vcruntime140.dll (for functions: memcpy and memset).
API-MS-WIN-CRT-ENVIRONMENT-L1-1-0.DLL, API-MS-WIN-CRT-RUNTIME-L1-1-0.DLL and API-MS-WIN-CRT-HEAP-L1-1-0.DLL (for functions: malloc and free).
KERNEL32.dll
If you get an error Error LNK1181 cannot open input file 'C:/Program.obj',
you may have to use quotation marks around the path in Visual Studio project properties.
Property Pages > C/C++ > General > Additional Include Directories
Property Pages > Linker > Command Line
If you get an error CMake Error at cmake/FindOpenBLAS.cmake:71 (MESSAGE): Could not find OpenBLAS,
please set the environment variable OpenBLAS_HOME or edit the variables OpenBLAS_INCLUDE_DIR and OpenBLAS_LIB
to point to the include directory and libopenblas.dll.a respectively.
If you get an error install Library TARGETS given no DESTINATION!, you can comment install(TARGETS systemml preload LIBRARY DESTINATION lib)
in CMakeLists.txt and manually place the compiled dll in the src/main/cpp/lib directory.

64-bit x86 Mac
The version of clang that ships with Mac does not come with OpenMP. brew install either llvm or g++. The instructions that follow are for llvm:

Intel MKL - CMake should detect the MKL installation path, otherwise it can specified by the environment variable MKLROOT. To use (with clang):

mkdir INTEL && cd INTEL
CXX=/usr/local/opt/llvm/bin/clang++ CC=/usr/local/opt/llvm/bin/clang LDFLAGS=-L/usr/local/opt/llvm/lib CPPFLAGS=-I/usr/local/opt/llvm/include cmake  -DUSE_INTEL_MKL=ON -DCMAKE_BUILD_TYPE=Release ..
make install

(with gcc-6):
mkdir INTEL && cd INTEL
CXX=g++-6 CC=gcc-6 cmake  -DUSE_INTEL_MKL=ON -DCMAKE_BUILD_TYPE=Release ..
make install

OpenBLAS - CMake should be able to detect the path of OpenBLAS. If it can't, set the OpenBLAS environment variable. If using brew to install OpenBLAS, set the OpenBLAS_HOME environment variable to /usr/local/opt/openblas/. To use (with clang):

export OpenBLAS_HOME=/usr/local/opt/openblas/
mkdir OPENBLAS && cd OPENBLAS
CXX=/usr/local/opt/llvm/bin/clang++ CC=/usr/local/opt/llvm/bin/clang LDFLAGS=-L/usr/local/opt/llvm/lib CPPFLAGS=-I/usr/local/opt/llvm/include cmake  -DUSE_OPEN_BLAS=ON -DCMAKE_BUILD_TYPE=Release ..
make install

(with gcc-6):
export OpenBLAS_HOME=/usr/local/opt/openblas/
mkdir OPENBLAS && cd OPENBLAS
CXX=g++-6 CC=gcc-6 -DUSE_OPEN_BLAS=ON -DCMAKE_BUILD_TYPE=Release ..
make install

I agree with your assessment, only supporting the native backend for Linux is fine. This PR looks good to me :+1:
You can address my comments in this PR or in a subsequent PR.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1423/

With OpenBLAS on Lenet on the same machine
Iter:2000.0, training loss:0.009108877504719245, training accuracy:100.0
Iter:2000.0, validation loss:166.14787969186656, validation accuracy:97.87397540983606
SystemML Statistics:
Total elapsed time:             476.380 sec.
Total compilation time:         0.000 sec.
Total execution time:           476.380 sec.
Number of compiled Spark inst:  79.
Number of executed Spark inst:  2.
Native openblas calls (LibMatrixMult/LibMatrixDNN):     4270/10553.
Cache hits (Mem, WB, FS, HDFS): 281999/0/0/0.
Cache writes (WB, FS, HDFS):    147642/0/0.
Cache times (ACQr/m, RLS, EXP): 0.111/0.068/2.028/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/11305.
HOP DAGs recompile time:        6.428 sec.
Spark ctx create time (lazy):   0.023 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
Total JIT compile time:         40.452 sec.
Total JVM GC count:             4557.
Total JVM GC time:              71.705 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sel+    55.054 sec      6283
-- 2)   conv2d_bias_add         49.798 sec      4514
-- 3)   ba+*    46.230 sec      12566
-- 4)   conv2d_backward_filter  41.418 sec      4026
-- 5)   -*      36.461 sec      16104
-- 6)   +*      32.389 sec      8052
-- 7)   maxpooling_backward     32.378 sec      4026
-- 8)   +       32.204 sec      30566
-- 9)   relu_backward   26.499 sec      6039
-- 10)  conv2d_backward_data    26.011 sec      2013

Here is another run with MKL to double-check if the performance is reproducible
Iter:2000.0, training loss:0.013034947944848269, training accuracy:100.0
Iter:2000.0, validation loss:274.1293912979084, validation accuracy:96.28586065573771
SystemML Statistics:
Total elapsed time:             482.600 sec.
Total compilation time:         0.000 sec.
Total execution time:           482.600 sec.
Number of compiled Spark inst:  79.
Number of executed Spark inst:  2.
Native mkl calls (LibMatrixMult/LibMatrixDNN):  4270/10553.
Cache hits (Mem, WB, FS, HDFS): 281999/0/0/0.
Cache writes (WB, FS, HDFS):    147642/0/0.
Cache times (ACQr/m, RLS, EXP): 0.149/0.085/2.201/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/11305.
HOP DAGs recompile time:        6.666 sec.
Spark ctx create time (lazy):   0.022 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
Total JIT compile time:         39.626 sec.
Total JVM GC count:             4503.
Total JVM GC time:              71.892 sec.
Heavy hitter instructions (name, time, count):
-- 1)   sel+    55.166 sec      6283
-- 2)   conv2d_backward_filter  51.424 sec      4026
-- 3)   conv2d_bias_add         47.443 sec      4514
-- 4)   -*      38.041 sec      16104
-- 5)   +*      35.513 sec      8052
-- 6)   +       34.988 sec      30566
-- 7)   maxpooling_backward     34.836 sec      4026
-- 8)   ba+*    31.071 sec      12566
-- 9)   conv2d_backward_data    28.552 sec      2013
-- 10)  relu_backward   27.039 sec      6039

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1425/

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1426/


[BUG] Error log seen when logging in as azure AD Page Iterator received incorrect value of type *models.DirectoryRole in rancher logs

Rancher Server Setup

Rancher version: v2.9-head 3c4ccdc5bc
Installation option (Docker install/Helm Chart): Docker install

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

Information about the Cluster

Kubernetes version: Local
Cluster Type (Local/Downstream): v1.30.2+k3s2

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

If custom, define the set of permissions: Admin

Describe the bug
Enabling and logging in as azure AD auth user returns an error log in rancher logs.
To Reproduce

Enable azure ad auth on rancher
Logout and login as the admin user for which the auth is enabled.

Result
Notice the error logged in the rancher logs
2024/07/15 21:36:52 [ERROR] [AZUREAD_PROVIDER] Page Iterator received incorrect value of type *models.DirectoryRole

Expected Result
Expected no errors in rancher logs.
Additional context

Does not happen with all users only a few users.
Does not happen on v2.8.5.

I have tried a few installs of rancher with AzureAD and I am unable to reproduce this problem. I occasionally see this error the very first time I enable AzureAD, but it goes away quickly.
2024/08/01 13:50:25 [ERROR] [AZUREAD_PROVIDER_CACHE] Failed to read the access token from Kubernetes: error getting secret cattle-global-data:azuread-access-token: secrets "azuread-access-token" not found


Carousel Component

Wondering when a Carousel component will be included.

Soon... very soon... I'll leave this open until we've got it in so you'll be aware!

Awesome thanks Eric!  At one point I'll start contributing on the repo as well.
Cheers,
Oscar

From: Eric Thompson<EMAIL_ADDRESS>Sent: Wednesday, November 25, 2015 8:14:43 AM
To: OfficeDev/Office-UI-Fabric
Cc: Oscar Medina
Subject: Re: [Office-UI-Fabric] Carousel Component? (#220)
Soon... very soon... I'll leave this open until we've got it in so you'll be aware!

Reply to this email directly or view it on GitHubhttps://github.com/OfficeDev/Office-UI-Fabric/issues/220#issuecomment-159658968.

Was it @philworthington who worked on a Carousel component? How much work is involved in getting into Fabric as a component?

@battletoilet worked on this one and we're still working towards design approval before pulling in.

After discussing with the team, we decided that this component doesn't make sense right now. We can revisit the Carousel if there's a future need.

Can we re-open this issue? A carousel component would be useful.

@gunjandatta: Are you using Fabric React? I can add a component request to our Trello board.

Yes, I'm using the Fabric React Framework, and if you could add a request to the Trello board I would really appreciate it.

Added here: https://trello.com/c/lfvOTWeE/16-carousel

What's the status on the carousel?  There doesn't seem to be any activity on the trello board or fabric react side.  Does this need to be added to the fabric first before becoming a react component?

It's probably best to open an issue in the Fabric React repo for this feature request.


[LICENSE] need proper licensing of Preconditions class?

org.apache.fury.util.Preconditions seems to be basically derived from Guava's class.
If so. we need

to replace the license header with the license header from the original Guava source file
ideally add a comment with a link to the Guava source file
update the LICENSE file to acknowledge the Guava license and copyright

I think we need a very good look at https://github.com/apache/incubator-fury/pull/1244 and https://github.com/apache/incubator-fury/pull/1114 to see if there are other source files where we have borrowed from Guava.

Hi @pjfanning , thanks for bring this license issue up, I'll take a look later

I'll close this for now. Updating the project LICENSE/NOTICE can be done later.


Problem scrolling in 1.8.0

VSCode Version: Code 1.8.0 (38746938a4ab94f2f57d9e1309c51fd6fb37553d, 2016-12-13T17:38:28.425Z)
OS Version: Darwin x64 15.6.0
Extensions:

Extension
Author
Version

jsx
TwentyChung
0.0.1

vscode-css-formatter
aeschli
0.0.2

vscode-svgviewer
cssho
1.2.0

jshint
dbaeumer
0.10.15

vscode-eslint
dbaeumer
1.1.0

vscode-npm-source
dkundel
1.0.0

githistory
donjayamanne
0.1.4

vscode-semistandard
flet
1.1.2

Theme-FlatlandMonokai
gerane
0.0.4

vscode-semistandard-format
homerjam
0.0.9

Theme-TomorrowKit
ms-vscode
0.1.3

theme-oceanicnext
naumovs
0.0.4

view-in-browser
qinjia
0.0.3

sass-helper
ramyaraoa
0.0.2

PostCSS
ricard
1.0.1

ava
samverschueren
0.4.0

stylelint
shinnn
0.20.4

code-spell-checker
streetsidesoftware
0.12.2

gitblame
waderyan
1.3.0

JavaScriptSnippets
xabikos
1.2.0

ReactSnippets
xabikos
1.2.0

TabSpacer
yuichinukiyama
1.0.2

Material-theme
zhuangtongfa
1.0.3

Steps to Reproduce:

Have a massive minified css tag in an html file.

Scrolling is a little broken on version 1.8.0. See gif, can't scroll past huge block of css. (don't judge me its a google amp thing I swear)

Here's a gist of the code that can be used to reproduce it
https://gist.githubusercontent.com/orrybaram/f791d0fedd916ce320497130d7cf3d31/raw/214c07003184f777c5ccd84f59b2c67064d0ef7b/vscode-scrolling-issue.html

@orrybaram Thanks. I confirmed this issue.
I believe this error message is related:
t o (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:26565)
at e._getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:29561)
at e.getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:28657)
at t.getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:62:7835)
at e.getLinesViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:43:804)
at t.getLinesViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:60:7316)
at t._actualRender (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:6641)
at file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:5079
at B (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:76:17614)
at t._renderNow (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:5057)e.onUnexpectedError @ shell.ts:541
shell.ts:541 Cannot read property 'length' of undefined: TypeError: Cannot read property 'length' of undefined
at o (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:26565)
at e._getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:29561)
at e.getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:61:28657)
at t.getDecorationsViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:62:7835)
at e.getLinesViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:43:804)
at t.getLinesViewportData (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:60:7316)
at t._actualRender (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:6641)
at file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:5079
at B (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:76:17614)
at t._renderNow (file:////Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/electron-browser/workbench.main.js:77:5057)e.onUnexpectedError @ shell.ts:541(anonymous function) @ shell.ts:457e.onUnexpectedError @ errors.ts:68o @ errors.ts:88B @ viewImpl.ts:959t._renderNow @ viewImpl.ts:834t._flushAccumulatedAndRenderNow @ viewImpl.ts:297getPositionFromDOMInfo @ viewImpl.ts:382e._actualDoHitTestWithCaretRangeFromPoint @ mouseTarget.ts:463e._doHitTestWithCaretRangeFromPoint @ mouseTarget.ts:431e._doHitTest @ mouseTarget.ts:582e._unsafeCreateMouseTarget @ mouseTarget.ts:328e.createMouseTarget @ mouseTarget.ts:224t._createMouseTarget @ mouseHandler.ts:229t._onMouseMove @ mouseHandler.ts:256(anonymous function) @ mouseHandler.ts:168t._handle @ mouseHandler.ts:68handler @ mouseHandler.ts:56l @ dom.ts:470(anonymous function) @ dom.ts:481_wrapHandler @ dom.ts:196
Investigating now

Great thanks for the quick response!

@alexandrudima Can you please take a look at this? I see you did some work on decorations recently.
If the root cause is not clear, can we just add a null check in insert as a quick fix before investigating why this is being hit in the first place.

To workaround this issue:
"editor.wrappingColumn": -1

Reproduces when rendering a wrapped model line containing before/after decorations below the viewport.
In other words, most likely to happen in minified (wrapped) CSS (color boxes are before decorations).

There is something still wrong with scrolling in the 1.8.1 build.  There are still long pauses.  You can see this just scrolling over the 1.8.1 release notes:
VS Code - 1.8.1
OS - Windows 10
Hardware - Surface Pro 3

Can confirm that it hangs a bit when scrolling past gifs in the release notes, but I haven't noticed any issues in day to day use.

@seltzer1717 If you are scrolling with the trackpad, there is a known issue with Chromium (#13612) where it does not process trackpad gestures correctly. One workaround which I use successfully is to "restore" and then maximize back VS Code.

I have this issue, but it does not slow down, just freeze. Need to force close and open again. Happen with every JS file minified. (I know this should not be open, but the editor should not hang if I do it).