Sparse Matrices

Sparse matrix support in the stan language and backends.
Rendered Version
Summary
There has been much discussion about sparse matrices in Stan. This design doc brings together the discusions on how to implement them at the language, IO, and math levels. The below gives a TL;DR for each section.
Language
There will be a new sparse_matrix type with the non-zero (NZ) sparsity structure defined as bounds.*
sparse_matrix<nz_rows=nz_row_ind, nz_cols=nz_col_ind>[N, M] A;

bounds make specifying the sparsity optional so that the sparsity pattern can be deduced under the hood for algebra etc. at the stan math level.
sparse_matix[N, N] B = A * A';

I/O
Sparse matrices come in lists of lists from the json or rdump
Stan math
We can either do a big refactoring to simplify the codebase or include specializations for the functions that take sparse matrices.
* I personally prefer the attribute style mentioned in the alternatives section, but Dan and Aki have both expressed interest in the <> style and did not like the attributed style. While it's only an N of 2 I like to think the user is right in what is aesthetically pleasing to them.

I had this open on a separate draft Pr and am going to bring over some of the discussion

On Aug 28, 2019, at 3:05 PM, seantalts<EMAIL_ADDRESS>wrote:
I think the doc could use a little discussion on whether sparse matrices deserve their own type in Stan.

If they don't have their own type, how does the decision to represent a matrix get handled?  At the very least, we'll need a new I/O format so that we don't need quadratic memory in and out.
What happens when we print or save parameters to a CSV file during sampling?  Does it do one thing for sparse and one thing for dense?  How will the user be able to figure out what's going to happen?  What's going to ensure we have the same parameters every iteration if there's no way to type the sparsity pattern?
What about row vectors?  That's a third way to represent a sequence of real values.
Then at higher orders, we get the same issue with array of vectors (type of vectorized outcomes for multinormal), array of row vectors, a matrix, and a 2D array?    Now we have four Stan types for the same abstract 2D structure.
Is this at all like integer vs. double?  We can assign integer to double (perhaps with loss of precision, and vice-versa with rounding), but mathematically we tend to think of them differently.   That difference doesn't make much difference in stats other than that there are naturally integer disributions like Poisson or Bernoulli.
How do you think about integer vs. double?  That's a common distinction in programming languages that languages like R decided to sweep under the rug.  It's there in the guts of R implicitly, as well as a distinction from bool; I imagine something similar is being suggested for Stan, where the decision is implicit on the framework's part (in R, you can modify and inspect, but I think the proposal for Stan is to literally collapse the types from the user's perspective so that there's no way to tell if a matrix is sparse or dense).

Some starter points off the top of my head:
pros:
• can add functions dealing with sparsity in an adhoc fashion to the math library slowly over time.
• can prevent users from doing inefficient things with sparse matrices, like using it in a function that would just convert it to dense first.

It's easy enough to add converters all over the place that convert sparse to dense when necessary.  But that won't give us the right result.  We can't just remove zero values, as we may need them for autodiff (this is a problem with current Eigen implementations). In particular, we need to be sure that we get the same sparsity in output for parameters at each iteration.  If we just willy-nilly promote to dense, that won't be the case or might not be predictable.

cons:
• Users don't necessarily care that their matrices are stored in sparse or dense form; they want efficient inference. I think there's some wisdom in trying to keep a 1-to-1 correspondence between a Stan model and the mathematical representation of it, if we can keep computational issues behind the silk screen.

They will very much care during I/O.
What mathematical representation is supposed to be one-one with a Stan program?  The mapping from programs to posterior densities is many to one.

• I think everyone universally regrets adding both real[] and vector to Stan, which is a pretty similar use-case in that both are representing the same mathematical object just with slightly different computational properties. Keeping users from doing linear algebra operations on real[] doesn't seem to have served any real purpose.

I don't regret it!  It's a natural consequence of having primitive types  int, real, vector, row_vector and matrix and closing under taking arrays.   If you don't allow it, it complicates every definition and lots of code where it has to be excluded from the natural definition.  I'm not saying it's impossible, just that it's going to add its own set of complications.
I don't think of arrays and vectors as the same mathematical object any more than I think of a complex number as a pair of real numbers, despite their being the same abstract data type in terms of data.
I made the same initial distinction as is made in Eigen, namely that linear algebra operations apply to matrix types.  It's also clunky in Eigen.  But it is worth keeping in mind that the major C++ matrix lib did find it useful to make that distinction.  They're even stricter, I think, in limiting elementwise operations to arrays.
What about row vectors?  That's a third way of representing a sequence of real numbers.  The reason it's nice to distinguish these types is that multiplication signatures are real(row_vector, vector) and matrix(vector, row_vector).
I think the argument is that real arrays are redundant because there's nothing you can do with an array you can't do with a vector or row_vector instead.  That may be true now so that we wouldn't lose any functionality getting rid of them.  I don't think it would fail anywhere.

The "what do users want" question difficult to test because anyone who is already aware of sparse matrices will probably consider them necessary as first-class objects, but there's a whole class of Stan user who would just like their model to run faster and wouldn't care if we switched to sparse under the hood. I wonder if @lauren can offer advice here?

Lauren is @lauken13, but I'm answering via email, so my @ won't work.
Dan Simpson probably has the most experience with this on the project, though I'd suggest asking Andrew, too.

If it’s not a type it might as well not be done. It would be extremely
difficult to use. Akin to not having a dense matrix type and making users
deal with flattened representations.
Not sure why any function would cast a sparse matrix to a different type
automatically rather than just throw a “not defined for type sparse”
warning. Struggle to think of a use case for a sparse-to-dense cast. What’s
your use case.
No earthly idea how to “pretend” sparse matrices are just matrices that are
different in hidden under the hood ways. Neither matlab or R does this.
On Wed, 28 Aug 2019 at 15:05, seantalts<EMAIL_ADDRESS>wrote:

I think the doc could use a little discussion on whether sparse matrices
deserve their own type in Stan. Some starter points off the top of my head:
pros:

can add functions dealing with sparsity in an adhoc fashion to the
math library slowly over time.
can prevent users from doing inefficient things with sparse
matrices, like using it in a function that would just convert it to dense
first.

cons:

Users don't necessarily care that their matrices are stored in
sparse or dense form; they want efficient inference. I think there's some
wisdom in trying to keep a 1-to-1 correspondence between a Stan model and
the mathematical representation of it, if we can keep computational issues
behind the silk screen.
I think everyone universally regrets adding both real[] and vector
to Stan, which is a pretty similar use-case in that both are representing
the same mathematical object just with slightly different computational
properties. Keeping users from doing linear algebra operations on
real[] doesn't seem to have served any real purpose.

The "what do users want" question difficult to test because anyone who is
already aware of sparse matrices will probably consider them necessary as
first-class objects, but there's a whole class of Stan user who would just
like their model to run faster and wouldn't care if we switched to sparse
under the hood. I wonder if @lauren https://github.com/lauren can offer
advice here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBWJGDIWECK6LGTGTJDQG3EBFA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MEZRQ#issuecomment-525880518,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBRHGWMVFTWJ3MXPWHLQG3EBFANCNFSM4IPODLXA
.

I don't think of arrays and vectors as the same mathematical object any more than I think of a complex number as a pair of real numbers, despite their being the same abstract data type in terms of data.

I agree — arrays are a general container for any type and vectors and
row vectors are linear algebraic objects with well-defined operations
one can apply on them.  If one defines a type by its admissable
operations then the similarity between  real[] and vector/row_vector, not
to mention matrix[1, N] and matrix[N, 1]!, is only a superficial one.
I think lots of the user confusion comes from people accustomed to R
and its dynamic/weak type system.  They expect the structural similarity
to carry over to functional similarity and get frustrated when it doesn’t.
At the very least this is something that should be emphasized very strongly
in introductory material so that users are at least aware for the issue.

Not sure why any function would cast a sparse matrix to a different type
automatically rather than just throw a “not defined for type sparse”
warning. Struggle to think of a use case for a sparse-to-dense cast. What’s
your use case.

One that tops to my mind is incomplete implementation of sparse
functionality.  For example if I wanted to compute a matrix exponential
but no one has yet implemented a sparse matrix exponential then can
I no longer proceed or can I cast my sparse matrix to a dense one and
proceed, albeit with the more expensive calculation.

No earthly idea how to “pretend” sparse matrices are just matrices that are
different in hidden under the hood ways. Neither matlab or R does this.

There is potential for some expression template methods that would
lazily evaluate expressions containing dense and sparse matrices
and figure out how to propagate the sparsity only at that final evaluation,
but that would be a pain to implement.
I think that the only useful “hidden” abstraction would be hiding sparse
matrices entirely and just have functions that take in sparse inputs and
return dense or low-dimensional outputs, like quadratic forms, determinants,
or densities.

On Wed, Aug 28, 2019 at 16:09 Bob Carpenter<EMAIL_ADDRESS>wrote:

Not sure why any function would cast a sparse matrix to a different type
automatically rather than just throw a “not defined for type sparse”
warning.
The motivation would be to make sure user programs with the right math
don't break because they got the data types wrong. We see that all the time
now because our users aren't programmers.
Of course, if we have a function that takes a sparse matrix in and then
outputs a dense matrix, that's going to be considered broken by most people
who use sparse matrices.

This would be extremely bad design. It should throw and informative error.
Otherwise we’ll spend our whole time explaining to people why they added
one line that had an implicit cast and their code no longer worked because
it ran out of memory.
No earthly idea how to “pretend” sparse matrices are just matrices that are

different in hidden under the hood ways. Neither matlab or R does this.
No pretending necessary. Mathematically, a sparse matrix is just a dense
matrix with a lot of zeros (no surprise to anyone there, I hope). So the
user just treats everything as a matrix (except, perhaps, during I/O) and
leaves the system to figure out where things should be dense or sparse
under the hood. There's a question of feasibility, but we're not violating
any laws of computation here.
R doesn't have a built-in sparse matrix type (though there is a
contributed Matrix package with sparsity).
MATLAB does have built-in sparsity and uses a separate type system.
https://www.mathworks.com/help/matlab/sparse-matrices.html
Under the hood, it just represents dense matrix types as an array and
there's no specific vector or row-vector types, just matrix types.
https://www.mathworks.com/help/matlab/matrices-and-arrays.html
Maybe we should've done that. I really wanted to have row vector times
vector to be a scalar and really wanted matrix times vector to be a vector.
In MATLAB, everything's a matrix and they just use vector constructors. In
MATLAB, [1 2 3] is a 1 x 3 matrix (i.e., a row vector) whereas [1; 2; 3]
is a 3 x 1 matrix and [1 2; 3 4] is a 2 x 2 matrix.
Consider what R does for integer vs. real vs. boolean distinctions:
is.integer(1)
[1] FALSE
is.integer(as.integer(1))
[1] TRUE
I find this kind of behavior in R very confusing. Same with transposing a
vector, which turns it into a matrix.
c(1, 2, 3)
[1] 1 2 3
t(c(1, 2, 3))
[,1] [,2] [,3]
[1,] 1 2 3
It gets really fun trying to predict the result of mixed operations
t(t(c(1, 2, 3))) == c(1, 2, 3)
[,1]
[1,] TRUE
[2,] TRUE
[3,] TRUE
I really don't like that t(t(x)) != x.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBQAWMUIKLRCGWOLR5TQG3LRBA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MKH4A#issuecomment-525902832,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBVOM3AMAFQXUZU5GFDQG3LRBANCNFSM4IPODLXA
.

There's no way to compute a sparse matrix exponential and get a sparse matrix out (because maths). So that would be easy to specialize

Sure, the question is what is the user experience be if it hadn’t
been specialized.
I feel like at some point users will have some corner applications
where they need to finish a calculation using dense methods.
Given the potential memory blowout from naively going from a
sparse matrix to a giant dense matrix it seems prudent to not do
this implicitly but an explicit casting function that maps sparse to
dense (and potentially even throws with an informative error if
the dimensions are too big) might be a good compromise initially.

I can't disagree more.
These aren't incomplete or unimplemented features, these are features that
should not exist. For scale, it's like when people say that we should have
a Gibbs sampler "just in case".
For any example where you should use sparse matrices, a to_dense will break
everything. A model with sparse matrices that needs a matrix exponential
(or an eigendecomposition or anything that is naturally dense) is just not
a realistic case for using a sparse matrix and should throw an error.
Either it's a different type or it will basically not work for 90% of
people who use it (because then you need to know all of the internals to
work out how to use it well rather than just having to accept "I can't do
xxxx with a sparse matrix").
On Wed, 28 Aug 2019 at 16:46, Michael Betancourt<EMAIL_ADDRESS>wrote:

There's no way to compute a sparse matrix exponential and get a sparse
matrix out (because maths). So that would be easy to specialize

Sure, the question is what is the user experience be if it hadn’t
been specialized.
I feel like at some point users will have some corner applications
where they need to finish a calculation using dense methods.
Given the potential memory blowout from naively going from a
sparse matrix to a giant dense matrix it seems prudent to not do
this implicitly but an explicit casting function that maps sparse to
dense (and potentially even throws with an informative error if
the dimensions are too big) might be a good compromise initially.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBV7TPPYHNZANOR5X3DQG3PZHA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MNL6I#issuecomment-525915641,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBXPNDSPRW55Y54MRILQG3PZHANCNFSM4IPODLXA
.

Very cool, thank you all for the discussion! My one request for this PR would be to capture a summary of this in the "Rationale and Alternatives" heading.
Another possible point for discussion - I think adding GPU matrix support to the Stan language is almost exactly the same, modulo some degree of difference on the break-even matrix size where converting to a dense [cpu] matrix is so inefficient as to be impossible and should be outlawed. Meaning that in GPU calculation land I think it will actually be somewhat common to want to go back and forth as we flesh out the GPU-aware implementations. So I might propose that we try to treat those the same way, meaning if we go with the full new-types approach we'd have all new cl(_cholesky_factor_corr | _cov | cholesky_factor_cov | ... )_matrix types as well (luckily someone said you'd never want to put a sparse matrix on a GPU or we'd have that combinatoric explosion as well).

The challenge then is designing a spare matrix type, and the
accompanying documentation, that clearly communicates those
limitations.
At best I think that most users will interpret sparse matrices as
“special kinds of matrices that sometimes admit much faster
calculations” and not “special kinds of matrices that do not play
with the standard matrices”.  I understand the technical utility
of keeping a sparse type completely encapsulated in its own
world, but those not as familiar with sparse linear algebra as
yourself will need something to guide them beyond the
interpretational baggage they bring with them, for example
very careful compiler error messages, extensive documentation,
etc.

On Aug 28, 2019, at 4:53 PM, dpsimpson<EMAIL_ADDRESS>wrote:
I can't disagree more.
These aren't incomplete or unimplemented features, these are features that
should not exist. For scale, it's like when people say that we should have
a Gibbs sampler "just in case".
For any example where you should use sparse matrices, a to_dense will break
everything. A model with sparse matrices that needs a matrix exponential
(or an eigendecomposition or anything that is naturally dense) is just not
a realistic case for using a sparse matrix and should throw an error.
Either it's a different type or it will basically not work for 90% of
people who use it (because then you need to know all of the internals to
work out how to use it well rather than just having to accept "I can't do
xxxx with a sparse matrix").
On Wed, 28 Aug 2019 at 16:46, Michael Betancourt<EMAIL_ADDRESS>wrote:

There's no way to compute a sparse matrix exponential and get a sparse
matrix out (because maths). So that would be easy to specialize

Sure, the question is what is the user experience be if it hadn’t
been specialized.
I feel like at some point users will have some corner applications
where they need to finish a calculation using dense methods.
Given the potential memory blowout from naively going from a
sparse matrix to a giant dense matrix it seems prudent to not do
this implicitly but an explicit casting function that maps sparse to
dense (and potentially even throws with an informative error if
the dimensions are too big) might be a good compromise initially.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBV7TPPYHNZANOR5X3DQG3PZHA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MNL6I#issuecomment-525915641,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBXPNDSPRW55Y54MRILQG3PZHANCNFSM4IPODLXA
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=AALU3FTSHLWCDSTEE6W35X3QG3QVRA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MOBNI#issuecomment-525918389, or mute the thread https://github.com/notifications/unsubscribe-auth/AALU3FUD3Z2XPJMOFQ4HOH3QG3QVRANCNFSM4IPODLXA.

Also curious about the ICAR model - is that a good representative example? It seems to just want to store an adjacency matrix efficiently, but otherwise to not use any sparse computation at all...? I thought someone at StanCon had said we would want to completely disallow element-wise indexing into a sparse matrix, which would appear to disallow this use-case... I am assuming I'm confused here and misheard probably multiple parts of this so please correct me with exuberance.

I understand the technical utility
of keeping a sparse type completely encapsulated in its own
world, but those not as familiar with sparse linear algebra as
yourself will need something to guide them beyond the
interpretational baggage they bring with them, for example
very careful compiler error messages, extensive documentation,
etc.

100% agreed - I still don't really get why you would disallow to_dense in the type system instead of just throwing a runtime warning like we do for deepcopies and other inefficient stuff. But I'm happy to defer to folks who actually write models that need sparse computation. I still dream of a world in which folks don't even need to learn about sparse matrix computations in order to experience their benefit, but perhaps that's a Stan 3 (or 4) thing.

On Aug 28, 2019, at 5:23 PM, seantalts<EMAIL_ADDRESS>wrote:
Very cool, thank you all for the discussion! My one request for this PR would be to capture a summary of this in the "Rationale and Alternatives" heading.
Another possible point for discussion - I think adding GPU matrix support to the Stan language is almost exactly the same, modulo some degree of difference on the break-even matrix size where converting to a dense [cpu] matrix is so inefficient as to be impossible and should be outlawed. Meaning that in GPU calculation land I think it will actually be somewhat common to want to go back and forth as we flesh out the GPU-aware implementations. So I might propose that we try to treat those the same way, meaning if we go with the full new-types approach we'd have all new cl(_cholesky_factor_corr | _cov | cholesky_factor_cov | ... )_matrix types as well (luckily someone said you'd never want to put a sparse matrix on a GPU or we'd have that combinatoric explosion as well).

You mean something like a sparse_cholesky_factor_cov type?  We'll need that, too.  A problem that'll arise is that it may be possible to write down impossible constraints, like a structural row of zeroes (we can already do that with <lower = 1, upper =  -1> on real values).
We might want to think about getting rid of or deprecating the dense correlation and covariance matrices.  It's almost always a bad idea to use those.  The only place I think they're really necessary is with Wishart priors, which we don't have defined for Cholesky factors.  Maybe in multivariate student T, but that should be fixed if we only have full matrix implementations.

On Aug 28, 2019, at 5:30 PM, seantalts<EMAIL_ADDRESS>wrote:
Also curious about the ICAR model - is that a good representative example?

The adjacency matrix is sparse, but it's boolean, not continuous, and not something we want to do linear algebra on.

It seems to just want to store an adjacency matrix efficiently, but otherwise to not use any sparse computation at all...?

Right.

I thought someone at StanCon had said we would want to completely disallow element-wise indexing into a sparse matrix, which would appear to disallow this use-case...

The tricky part is allowing indexing into positions that are structural zeros.  Those shouldn't be settable.  Structural non-zeros can be settable, but I don't know if that'd ever be useful. Everything should be gettable.

I am assuming I'm confused here and misheard probably multiple parts of this so please correct me with exuberance.
I understand the technical utility
of keeping a sparse type completely encapsulated in its own
world, but those not as familiar with sparse linear algebra as
yourself will need something to guide them beyond the
interpretational baggage they bring with them, for example
very careful compiler error messages, extensive documentation,
etc.
100% agreed - I still don't really get why you would disallow to_dense in the type system instead of just throwing a runtime warning like we do for deepcopies and other inefficient stuff.

I think it's because Dan can't imagine a case where that'd be useful, and it's better not to have something than to have something that never works.  I think Dan's got the most experience with these things on the project, so I'd listen to him!

You mean something like a sparse_cholesky_factor_cov type?  We'll need that, too.  A problem that'll arise is that it may be possible to write down impossible constraints, like a structural row of zeroes (we can already do that with <lower = 1, upper =  -1> on real values).

Yeah, I was writing it out for opencl types assuming a cl_ prefix, but yeah it seems like we'd need a bunch of those (all of them?) for sparse types as well. And I was just saying it's good we don't ever want to combine these computational types together, because then we'd really have a ton of types - in that world, a matrix could either be sparse or dense; it could live on the CPU, GPU, or some other hardware; and it can be some more structured form of matrix that has nice computational or statistical properties (cholesky, covariance, both cholesky AND covariance, etc).

I think it's because Dan can't imagine a case where that'd be useful, and it's better not to have something than to have something that never works.  I think Dan's got the most experience with these things on the project, so I'd listen to him!

That's what I proposed doing in the next line: "But I'm happy to defer to folks who actually write models that need sparse computation. " :)

Setting aside sparse matrices for a moment, I just wanted to say that, yes, I understand the conceptual difference between vectors and arrays.  But when working in practice with Stan, I'm having to clutter my code with to_vector and to_array (or whatever it's called), because some parts of Stan only accept vectors and other parts of Stan only accept real arrays.  So I do think that it would make sense for any operation that accepts 1-d real arrays to also accept vectors, and for any operation that accepts 2-d real arrays to also accept matrices, and vice-versa.

On Aug 28, 2019, at 4:09 PM, Michael Betancourt<EMAIL_ADDRESS>wrote:

I don't think of arrays and vectors as the same mathematical object any more than I think of a complex number as a pair of real numbers, despite their being the same abstract data type in terms of data.

I agree — arrays are a general container for any type and vectors and
row vectors are linear algebraic objects with well-defined operations
one can apply on them. If one defines a type by its admissable
operations then the similarity between real[] and vector/row_vector, not
to mention matrix[1, N] and matrix[N, 1]!, is only a superficial one.
I think lots of the user confusion comes from people accustomed to R
and its dynamic/weak type system. They expect the structural similarity
to carry over to functional similarity and get frustrated when it doesn’t.
At the very least this is something that should be emphasized very strongly
in introductory material so that users are at least aware for the issue.

I understood the type multiplication thing, just missed that cl_ was related to OpenCL.
It's going to get worse with specialized structured sparsity like banded or Toeplitz matrices.  Or ones where we definitely need some kind of expression template magic like Kronecker products.  We could also conceivably have general triangular matrix types, but I don't know what the use would be for that.
We always get a Cholesky factor with a covariance matrix parameter as it's the unconstrained representation.  Are you thinking about deeper connections like caching decompositions?

On Aug 28, 2019, at 5:56 PM, seantalts<EMAIL_ADDRESS>wrote:
You mean something like a sparse_cholesky_factor_cov type? We'll need that, too. A problem that'll arise is that it may be possible to write down impossible constraints, like a structural row of zeroes (we can already do that with <lower = 1, upper = -1> on real values).
Yeah, I was writing it out for opencl types assuming a cl_ prefix, but yeah it seems like we'd need a bunch of those (all of them?) for sparse types as well. And I was just saying it's good we don't ever want to combine these computational types together, because then we'd really have a ton of types - in that world, a matrix could either be sparse or dense; it could live on the CPU, GPU, or some other hardware; and it can be some more structured form of matrix that has nice computational or statistical properties (cholesky, covariance, both cholesky AND covariance, etc).
I think it's because Dan can't imagine a case where that'd be useful, and it's better not to have something than to have something that never works. I think Dan's got the most experience with these things on the project, so I'd listen to him!
That's what I proposed doing in the next line: "But I'm happy to defer to folks who actually write models that need sparse computation. " :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Is there something you can do to a real array you can't do to a vector?  The usual problem is that only vector and matrix types support linear algebra.
This is probably not the right location for this as it's the sparse matrix design docs, but while we're here...
The problems for me are more at the level of converting matrix to vector so you can use it on the lhs of a univariate sampling statement, e.g.,
matrix[10, 15] a;
a ~ normal(0, 1);             // BAD
to_vector(a) ~ normal(0, 1);  // OK
We didn't want to vectorize over matrices because it gets confusing how to match that to arguments in normal.  If the locations are matrices, you wind up needing two to_vector calls.
Or in cases where you want to have an array of vectors for multivariate normal,
matrix[5, 10] b;
vector[10] a[5];
a ~ multi_normal(...);  // OK
b ~ multi_normal(...);  // BAD
We decided we didn't want to support that b form of multinormal because of row vs. column confusion.
I also run into problems when I want multivariate priors.  The right way to do it
for the prior is the following for varying slopes and varying intercepts:
vector[2] ab[50];
ab ~ multi_normal_cholesky(mu, L_Sigma);
But then it's really clunky to use if you need to pull out a vector of slopes
or intercepts, which requires to_vector(ab[, 1]) and to_vector(ab[, 2]).

On Aug 28, 2019, at 6:00 PM, Andrew Gelman<EMAIL_ADDRESS>wrote:
Setting aside sparse matrices for a moment, I just wanted to say that, yes, I understand the conceptual difference between vectors and arrays. But when working in practice with Stan, I'm having to clutter my code with to_vector and to_array (or whatever it's called), because some parts of Stan only accept vectors and other parts of Stan only accept real arrays. So I do think that it would make sense for any operation that accepts 1-d real arrays to also accept vectors, and for any operation that accepts 2-d real arrays to also accept matrices, and vice-versa.

On Aug 28, 2019, at 4:09 PM, Michael Betancourt<EMAIL_ADDRESS>wrote:

I don't think of arrays and vectors as the same mathematical object any more than I think of a complex number as a pair of real numbers, despite their being the same abstract data type in terms of data.

I agree — arrays are a general container for any type and vectors and
row vectors are linear algebraic objects with well-defined operations
one can apply on them. If one defines a type by its admissable
operations then the similarity between real[] and vector/row_vector, not
to mention matrix[1, N] and matrix[N, 1]!, is only a superficial one.
I think lots of the user confusion comes from people accustomed to R
and its dynamic/weak type system. They expect the structural similarity
to carry over to functional similarity and get frustrated when it doesn’t.
At the very least this is something that should be emphasized very strongly
in introductory material so that users are at least aware for the issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Things I can do with a vector or matrix but not a 1-d or 2-d real array:

vector and matrix operations (multiplication etc)
various functions that are vectorized.  For example, I think you can do theta ~ normal(0,1); if theta is a vector but not if it's a 1-d array.  I might be wrong about that particular example, but there are other examples like that.
various functions for subsetting, concatenation, etc., work for vectors and matrices but not arrays.

The main things I can do with a real array but not a vector or matrix:

pass it into an ode function
easily change code from linear to logistic regression.  (It's natural for me to write my linear regression data as vector[N] y; but then if I want to switch to logistic regression it becomes int y[N]; and then various other aspects of the code have to be changed cos I'm switching from vector to array.

On Aug 28, 2019, at 9:13 PM, Bob Carpenter<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
Is there something you can do to a real array you can't do to a vector? The usual problem is that only vector and matrix types support linear algebra.
This is probably not the right location for this as it's the sparse matrix design docs, but while we're here...
The problems for me are more at the level of converting matrix to vector so you can use it on the lhs of a univariate sampling statement, e.g.,
matrix[10, 15] a;
a ~ normal(0, 1); // BAD
to_vector(a) ~ normal(0, 1); // OK
We didn't want to vectorize over matrices because it gets confusing how to match that to arguments in normal. If the locations are matrices, you wind up needing two to_vector calls.
Or in cases where you want to have an array of vectors for multivariate normal,
matrix[5, 10] b;
vector[10] a[5];
a ~ multi_normal(...); // OK
b ~ multi_normal(...); // BAD
We decided we didn't want to support that b form of multinormal because of row vs. column confusion.
I also run into problems when I want multivariate priors. The right way to do it
for the prior is the following for varying slopes and varying intercepts:
vector[2] ab[50];
ab ~ multi_normal_cholesky(mu, L_Sigma);
But then it's really clunky to use if you need to pull out a vector of slopes
or intercepts, which requires to_vector(ab[, 1]) and to_vector(ab[, 2]).

On Aug 28, 2019, at 6:00 PM, Andrew Gelman<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
Setting aside sparse matrices for a moment, I just wanted to say that, yes, I understand the conceptual difference between vectors and arrays. But when working in practice with Stan, I'm having to clutter my code with to_vector and to_array (or whatever it's called), because some parts of Stan only accept vectors and other parts of Stan only accept real arrays. So I do think that it would make sense for any operation that accepts 1-d real arrays to also accept vectors, and for any operation that accepts 2-d real arrays to also accept matrices, and vice-versa.

On Aug 28, 2019, at 4:09 PM, Michael Betancourt<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:

I don't think of arrays and vectors as the same mathematical object any more than I think of a complex number as a pair of real numbers, despite their being the same abstract data type in terms of data.

I agree — arrays are a general container for any type and vectors and
row vectors are linear algebraic objects with well-defined operations
one can apply on them. If one defines a type by its admissable
operations then the similarity between real[] and vector/row_vector, not
to mention matrix[1, N] and matrix[N, 1]!, is only a superficial one.
I think lots of the user confusion comes from people accustomed to R
and its dynamic/weak type system. They expect the structural similarity
to carry over to functional similarity and get frustrated when it doesn’t.
At the very least this is something that should be emphasized very strongly
in introductory material so that users are at least aware for the issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=AAZYCUMZCVU2E46U4TFJK7LQG4PB3A5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5M5HXQ#issuecomment-525980638, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZYCUMGLDQB2HWTZ3BCWPTQG4PB3ANCNFSM4IPODLXA.

I don’t think we should do this. Huge memory waste limits it to very small
matrices. R/Python have easy to access sparse matrix support
On Thu, Aug 29, 2019 at 03:54 Rok Češnovar<EMAIL_ADDRESS>wrote:

@rok-cesnovar commented on this pull request.
In designs/0004-sparse-matrices.md
https://github.com/stan-dev/design-docs/pull/8#discussion_r318931714:

+## Helper Functions

+We can also include helper functions to extract the sparsity pattern's row and column information.
+
+stan +int K = num_nz_elements(x); +// Extract effectively a tuple representation of the sparse matrix. +matrix[K, 3] = get_nz_elements(x); +
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+## I/O
+
+Data can be read in from json via a list of lists and the Rdump from the equivalent.
Should we also support that the input is in a dense matrix form in the
input file but then the parser & data handler automatically constructs the
sparse matrix, ignoring zeroes? We would need to somehow inform the parser
to ignore zeroes during parsing, otherwise we would first create the dense
matrix and then transform which defeats the purpose of introducing
sparsity..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBWSAUHGVDG6BSMBYSLQG56C7A5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCDB7KCI#pullrequestreview-281277705,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBVDW2CB2RINAAZPNTLQG56C7ANCNFSM4IPODLXA
.

On Aug 28, 2019, at 9:56 PM, Andrew Gelman<EMAIL_ADDRESS>wrote:
Things I can do with a vector or matrix but not a 1-d or 2-d real array:

The question was whether the problem exists the other way around.  Is there something you can do with a real[] that you can't do with a vector or row_vector?

vector and matrix operations (multiplication etc)

That's intentional.  It doesn't make sense to do matrix operations on arrays.  You have to guess about orientation like R and it's a mess.

various functions that are vectorized. For example, I think you can do theta ~ normal(0,1); if theta is a vector but not if it's a 1-d array.

Yes, that works if it's a 1D array.  But theta ~ multi_normal(mu, Sigma) doesn't work if mu or theta are real[] types---they have to be vectors.  I did it that way because they're considered as a whole in multi_normal.  It wouldn't kill me to loosen this.

I might be wrong about that particular example, but there are other examples like that.

If you can ping this thread (or just me) when one comes up, that'd be great.

various functions for subsetting, concatenation, etc., work for vectors and matrices but not arrays.

I'm not sure what you mean here.  All the slicing works the same way.
If what you mean is that you can't concatenate a real[] and vector, that's by design.  What would the result be?

The main things I can do with a real array but not a vector or matrix:

pass it into an ode function

This is a real limitation.  But this is the simplest possible to_array or to_vector conversion, right?  We could generalize this, but I think we'll be going with an even better solution based on closures.
These higher-order functions are special in that they're implemented very differently.  We could generalize the signatures without much effort if you really find to_vector to be that onerous.  But it'd have to wait until stanc3 gets rolled out as work on the old parser is frozen pending that release.

easily change code from linear to logistic regression.

That doesn't seem like it should be so easy.  The likelihood changes, you need to add a link function, and there's no more scale parameter.  The constraints on y also change to be lower = 0 and upper = 1 from unconstrained.

(It's natural for me to write my linear regression data as vector[N] y; but then if I want to switch to logistic regression it becomes int y[N]; and then various other aspects of the code have to be changed cos I'm switching from vector to array.

This is because Stan distinguishes integer and real types (again, so it doesn't have to guess like R).  I'm strongly opposed to having integer vector types because we're not doing number theory.
As a programmer, I like that the types reflect the underlying data type rather than being implicit.

Okay, to try to corral some of the conversation back, it seems like folks basically want to go with what our potential sparse matrix users would like to see in Stan's type system here, and so far the only person like that we have commenting is @dpsimpson. Dan's proposal is that we should have a separate type for every kind of matrix that could also be sparse and that we would only allow efficient operations to be allowed on them in the type system. I think we want to add some clarification for which types and operations those actually are concretely. Is it enough to add a single sparse_matrix type to Stan with the operations listed here? That's what's in this proposal now, just double checking that's still kosher.
If so, I think we need to change the design doc to add a different model as an exemplar model; the ICAR model would not benefit from sparse matrices as described in this proposal as we wouldn't be allowed to do element-wise access. Does anyone have another model that would benefit from the design in this proposal they can contribute?

Or I may be wrong about ICAR being a bad fit - if there's a more efficient version possible once we have sparse matrices, relevant lines from that would be good to include in this design doc to show the before-and-after.

No, the ICAR isn't a good fit.   Sparse GPs would be where I think the plan's to add sparse log determinant and sparse Cholesky factorization (on precision matrices, I think---you'd need to ask Dan!).
You an also think smaller in terms of a big regression problem where the data matrix is sparse and the coefficient vector is dense.  So we definitely need some mixed operations.

On Aug 31, 2019, at 7:40 PM, seantalts<EMAIL_ADDRESS>wrote:
Or I may be wrong about ICAR being a bad fit - if there's a more efficient version possible once we have sparse matrices, relevant lines from that would be good to include in this design doc to show the before-and-after.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Sparse GPs are a different thing that don’t involve sparse matrices. (I
know!) So they aren’t a use case.
Multivariate Gaussians with sparse precision matrices is the main use case.
On Sat, Aug 31, 2019 at 20:28 Bob Carpenter<EMAIL_ADDRESS>wrote:

No, the ICAR isn't a good fit. Sparse GPs would be where I think the
plan's to add sparse log determinant and sparse Cholesky factorization (on
precision matrices, I think---you'd need to ask Dan!).
You an also think smaller in terms of a big regression problem where the
data matrix is sparse and the coefficient vector is dense. So we definitely
need some mixed operations.

On Aug 31, 2019, at 7:40 PM, seantalts<EMAIL_ADDRESS>wrote:
Or I may be wrong about ICAR being a bad fit - if there's a more
efficient version possible once we have sparse matrices, relevant lines
from that would be good to include in this design doc to show the
before-and-after.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBX6I3WMD4VEN77NQCTQHMEEJA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5TXWVI#issuecomment-526875477,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBUSTOWPUCA5TN4IE5LQHMEEJANCNFSM4IPODLXA
.

A good use case along the lines of the ICAR that needs full linear algebra
is this model
https://mc-stan.org/users/documentation/case-studies/mbjoseph-CARStan.html
(Note the case study doesn’t use the sparsity to do anything, so it’s just
the model that’s relevant.). In this case you need the log determinant if a
matrix with var values.  You probably need to use a non-centred
parameterization.
On Sat, Aug 31, 2019 at 20:30 Daniel Simpson<EMAIL_ADDRESS>wrote:

Sparse GPs are a different thing that don’t involve sparse matrices. (I
know!) So they aren’t a use case.
Multivariate Gaussians with sparse precision matrices is the main use
case.
On Sat, Aug 31, 2019 at 20:28 Bob Carpenter<EMAIL_ADDRESS>wrote:

No, the ICAR isn't a good fit. Sparse GPs would be where I think the
plan's to add sparse log determinant and sparse Cholesky factorization (on
precision matrices, I think---you'd need to ask Dan!).
You an also think smaller in terms of a big regression problem where the
data matrix is sparse and the coefficient vector is dense. So we definitely
need some mixed operations.

On Aug 31, 2019, at 7:40 PM, seantalts<EMAIL_ADDRESS>wrote:
Or I may be wrong about ICAR being a bad fit - if there's a more
efficient version possible once we have sparse matrices, relevant lines
from that would be good to include in this design doc to show the
before-and-after.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBX6I3WMD4VEN77NQCTQHMEEJA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5TXWVI#issuecomment-526875477,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBUSTOWPUCA5TN4IE5LQHMEEJANCNFSM4IPODLXA
.

What's the application of multivariate Gaussians with sparse precision matrices?  I had wrongly supposed it was GPs.
Setting aside generic sparsity, aren't there some GPs where the covariance matrix is structured like banded or something like that?

I thought the full CAR model was a GP.  That's the kind of model where I thought we'd need the sparse log determinant.
On Aug 31, 2019, at 9:01 PM, dpsimpson<EMAIL_ADDRESS>wrote:

A good use case along the lines of the ICAR that needs full linear algebra
is this model
https://mc-stan.org/users/documentation/case-studies/mbjoseph-CARStan.html

Gaussian processes specify a Gaussian density through a structured
covariance matrix which is inverted, while CAR models specify a Gaussian
density through sparse precision matrix which does not need to be
inverted.  In the latter case the precision matrix is ill-posed and cannot
be inverted into a precision matrix and the log determinant term isn’t
an exact log determinant but something like a log determinant with a
singularity subtracted away.
The kind of random sparsity that sparse linear algebra is most effective
for arises is natural when modeling precision matrices (which encode
conditional independencies, like the adjacency graph of a CAR model)
where as Gaussian processes yield covariances that aren’t sparse
but rather dense and structured (Kronecker, Toeplitz, etc).
At least that is my understanding.

On Sep 1, 2019, at 4:00 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:
I thought the full CAR model was a GP. That's the kind of model where I thought we'd need the sparse log determinant.
On Aug 31, 2019, at 9:01 PM, dpsimpson<EMAIL_ADDRESS>wrote:

A good use case along the lines of the ICAR that needs full linear algebra
is this model
https://mc-stan.org/users/documentation/case-studies/mbjoseph-CARStan.html

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=AALU3FWWMA3IFIO4IGNYF5DQHPDJLA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UC67Q#issuecomment-526921598, or mute the thread https://github.com/notifications/unsubscribe-auth/AALU3FULGMLDDPCKZLVZWXDQHPDJLANCNFSM4IPODLXA.

The point of Gaussian model with a sparse precision matrix is efficiency - we can fit models that we couldn’t if the “model” would be dense. Think millions of parameters. This is why Dan was so adamant about avoiding casting sparse to dense.
The critical point is conditional independence, which gives sparse structure. Examples are time-series (banded precision), some spatial (CAR with neighbourhood structure, SPDE with some meshing magic), phylogeny (banded precision), pedigree (expanded band) and some other models.

Thanks all, and especially @gregorgorjanc for joining the conversation and explaining more of the background for me :)
So we will need to allow element-wise reads after all, is that right? I'm thinking that because that is the most obvious way to code up the ICAR example, but it's also possible we would still want to disallow that and only let users use some new distributions functions we'd be creating (which I guess would be a bunch of our multivariate functions but with sparse_ prepended to the name, to follow the convention; e.g. "sparse_multi_normal`).
I want to also think ahead and think about how we can define more of the Math library in Stan itself (if we can do that efficiently), so if we can think ahead and not put in any constraints that prevent a user from coding their own sparse multivariate distributions in Stan itself that's probably a good forward-looking move in my opinion.

For completeness of discussion, BUGS “language” specifies Gaussian distribution with a precision parameter. I believe this was due to the natural/canonical parametrisation of the Gaussian and the conditional (graph) model structure embedded implicitly in the “language”.

Sean: No. you do it build the matrix with sums and scalar products. Density
implemented with matvecs. Linear algebra is your friend.
Gregor: Hi!! Yes! And thanks!
Michael: ugh. Kinda? “Random” is a weird word there. CAR models have
invertible precisions (the ICAR That Mitzi used is a boundary case. We’ve
not looked at others because we don’t have the linear algebra). GPs are a
much broader class of things than you talk about there.
Not sure a design doc is the place to learn about statistical uses of
sparse matrices. I recommend Rue and Held’s book. Or that we move this less
relevant portion of the discussion to the discourse.
On Sun, Sep 1, 2019 at 12:35 Gregor Gorjanc<EMAIL_ADDRESS>wrote:

For completeness of discussion, BUGS “language” specifies Gaussian
distribution with a precision parameter. I believe this was due to the
natural/canonical parametrisation of the Gaussian and the conditional
(graph) model structure embedded implicitly in the “language”.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBSAURYMUSV7EG2VGNDQHPVN5A5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UGDDI#issuecomment-526934413,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBQDTQZT2F4DLU463JTQHPVN5ANCNFSM4IPODLXA
.

Ok cool! If people can build their own I'm happy with the design. Would someone mind adding what the ICAR model would look like with sparse matrices to the design doc as an example?

(Or post it as a comment in this thread and I can add it to the PR for the doc)

On Sep 1, 2019, at 12:04 PM, seantalts<EMAIL_ADDRESS>wrote:
Thanks all, and especially @gregorgorjanc for joining the conversation and explaining more of the background for me :)
So we will need to allow element-wise reads after all, is that right? I'm thinking that because that is the most obvious way to code up the ICAR example,

It's the CAR example that has the full structure here.  The ICAR is easy to compute as is.

but it's also possible we would still want to disallow that and only let users use some new distributions functions we'd be creating (which I guess would be a bunch of our multivariate functions but with sparse_ prepended to the name, to follow the convention; e.g. "sparse_multi_normal`).

I'm not sure we need to change the name here for sparse cases, but we could.  I just worry about crossing that with _precision, etc.

I want to also think ahead and think about how we can define more of the Math library in Stan itself (if we can do that efficiently), so if we can think ahead and not put in any constraints that prevent a user from coding their own sparse multivariate distributions in Stan itself that's probably a good forward-looking move in my opinion.

Right now, we can't code the density functions in Stan because we don't have type traits and we don't have the vector views.

So I think this is just waiting on @SteveBronder for:

A before-and-after example illustrating what the new syntax will provide, and
confirmation from Aki that we will need to represent the sparsity structure of a matrix in the Stan program (because parameters will otherwise have no way of knowing how to construct themselves).

I think this is ready to go as-is after that.

I’m in Helsinki next week so maybe we should finish it off then
On Mon, 16 Sep 2019 at 23:34, seantalts<EMAIL_ADDRESS>wrote:

So I think this is just waiting on @SteveBronder
https://github.com/SteveBronder for:

A before-and-after example illustrating what the new syntax will
provide, and
confirmation from Aki that we will need to represent the sparsity
structure of a matrix in the Stan program (because parameters will
otherwise have no way of knowing how to construct themselves).

I think this is ready to go as-is after that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/design-docs/pull/8?email_source=notifications&email_token=ADRICBW5VSASJYO3KGIVUNLQKACYRA5CNFSM4IPODLXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD62W7LQ#issuecomment-531984302,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADRICBVQYYDDLPLEXGTBSFLQKACYRANCNFSM4IPODLXA
.

Are you guys all together there? Want to finish this design doc up? Just need that before-and-after example assuming we're not changing the syntax.

Are you guys all together there? Want to finish this design doc up? Just need that before-and-after example assuming we're not changing the syntax.

Apologies! In Paris until Monday. I'll update this on Tuesday, ask Aki about (2) and get on the phone with Dan

// Define real a,b,c;
// Define sparse_matix A,B,C,D;
// We need
sparse_matrix E = a*A + b*B + C*D;

Sorry for my radio silence, I've been pretty deep in the compiiler. I just updated this with @seantalts notes above and a recent conversation with Aki and Dan. You can see those changes here and the notes from the meeting in the commit message of a938b45

Overall
This doc should be self-contained and include any necessary material from earlier proposals rather than linking.  Reference links at the end are really nice.
Summary
the matrices -> a matrix's
The exclusion of sparse vectors from the initial spec should be mentioned up front.
Motivation
ICAR does not use sparse matrices.  A lot of big data regressions in fields like natural lanaguage processing are sparse.
fan-dangling -> finagling (?), index fiddling (?)
Can we really support all existing methods for matrix types?  Does that mean all the mixed type operations, too such as sparse matrix times dense matrix?  Is there a useful way to roll this out more incrementally?
Does all existing methods extend to types?  That is, will we have a sparse simplex or sparse positive definite matrix (unfortunately called "covariance" because I was young and didn't know better---it should be called "spd" for symmetric-positive-definite).
I think the list here should just be things that wouldn't be included under "all existing methods".
Given the spec to support all existing functions (methods are technically non-static class functions), this list should just be for new things not already in the library.
Is return always sparse unless one of the arguments is dense?  If so, we can just say that.
And there needs to be more description for the new operations:

fill-reducing reordering
required elements of the inverse
operations to move from sparse to dense and vice versa
specializations for some log pdfs and pmfs

likelihood -> density (our lpdf and lpmf can also be used in other contexts than just likelihoods, such as in priors)
Notes on implementation (like the reverse-mode derivative of Cholesky) can go in an implementation notes section below the spec.  I think this spec should be more functional in terms of what things will look like for the user.
Guide-level
Move citation of previous specs to the references section at the end.
What does no explicit indexing within a sparse matrix mean?  If a is a sparse matrix, does that mean a[i, j] isn't legal as either an rvalue or lvalue?
Sparse matrices dimensions -> Sparse matrix size specifications
Note: Stan separates dimension an size specifications in that function arguments are dimensioned but not sized; try to use "rows" and "columns" or "size" rather than "dimension" because "dimension" is ambiguous in the context of a spec (a vector is a one-dimensional data structure, but a vector of size K is used to represent a point in K-dimensional space).
Could you elaborate on what it means to order a square sparse matrix after initial construction.  Is that a new function?  Something done automatically under the hood?
The enumeration of blocks needs to include function arguments.  In Stan, the function arguments are not sized, whereas block and local arguments are.  Similarly, there are constrained types which only show up as block variables.
I'd be extra explicit in the example that nz_row_alt is different than nz_row, because it should be legal if both sparsity patterns are the same even if they're not specified by the same variable or expression.
Data
The overall description of coordinate list notation should be pulled up front, with the example.  The col/row label is reversed.
This data section can then stick to how sparse matrices are used in the data block.
What is proper IO?  There needs to be a spec for the data formats in both JSON and rdump at this point.
If vectors are not in the initial implementation, please pull them out into a section of things to do later or otherwise indicate how things are going to be staged.
We've thought about having sizes also be read from the data structure directly, but haven't done that anywhere.  That's not to say we shouldn't do it here.
The example should have
int nz_row_ind[K] = { 2, 3, ...., 5 };

to be well-formed Stan code.  But the example needs to move down to the transformed data section where it's relevant because no definitions are allowed in the data block.
What type are the values that are set?  Are those in the form of a dense vector or an array of reals?
Transformed data
The Stan code for actually building up nz_row_ind, etc. should be moved down here.
The to_sparse_matrix function should be specified directly.  You don't need to say much as it should be clear to everyone what it does.
Linear Algebra -> linear algebra (capitalize names, not concepts)
Does the linear algebra example imply that we can use sparse_matrix in the data block with no sizing whatsoever.  Or should that be sparse_matrix[N, N]?
There's nothing in the bounds, just <>?
The sense in which the sparsity pattern is a matter for the math library is that the sparsity pattern checks have to be run time, not compile time.  That still leaves them in-scope for this spec as it has to cover both the math lib part and the languiage.
Parameters, ...
Break these three into their own sections.
Parameters must specify sizes in their declarations---there's no way to deduce them.  They must have th same sparsity pattern in each iteration, too.  Mention that an alternative parameter declaration is just the values, with the sparse matrix being created elsewhere (model or transformed parameters).
Transformed parameters and generated quantities should presumably work the same way as transformed data, with the caveat noted that the size/sparsity-pattern not change iteration to iteration.  The way to say that is that they have to be data-only variables, which is already a requirement for sizes in these (and the parameters) block.
If you want to add historical notes, put them later.
There are no temporary scopes in the parameters block.  I don't see why we couldn't use them as parameters given your spec.  If they are not going to be allowed as parameters, that needs to be clearly indicated.  And I think it would cause a lot of user confusion.
Helper functions
These should be laid out explicitly.  Propose the ones we'll start with and then we can add more later.  The point is that this spec should be as specific as possible.
Reference-level explanation
The JSON I/O is user level.
It would be much more compact to use three parallel arays in JSON rather than breaking it down by value this way.  That also matches the way it's declared, so that's probably not such a hardship for users.  Same in Rdump.  So how about JSON like
{
...
  "X" : {
    "row" : { 20, 3, ... },
    "col" : { 1, 11, ... },
    "val" : { 1.85, 2.71, ... }
  }
...
}

where the row and col values are integers and the val floats (this needs to be in the spec even though JSON does not distinguish numerical types).  Numbers will be represented as elsewhere (there's some complication around NaN and inf, but that's not this spec's responsibility).
And then in Rdump:
...
X <- list(row = c(20, 3, ...),
          col = c(1, 11, ...),
	  val = c(1.85, 2.71, ...))
...

"Something like what Ben did [here]" needs to be unfolded so the spec is self-contained and precise about what's being proposed.
The implementation details aren't so important and can be saved for later.  If they're included, outerIndex and innerIndices and nnz variables need to be explained.
This spec doesn't need implementation details down to the for-loop level.
Stan Math
Templating should be described.
In the hard way, The "coefficient access error Dan saw" needs to be described.  Is the proposal to use A.coeff() everywhere in the code to make this work?
The simple way should be at same level of heading.
Those specializations for spare matrices need to be careful to use Eigen's mixed types properly.  I hadn't realized we were doing that and could use it for addition.
These implementations should all move for reverse mode into the adjoint-partial approach which will dramatically cut down on stack funtions and virtual function calls over the way the examples are coded here by delegating to operator+ in Eigen.
Keeping Permutation Matrix
What does "this adds a bit of state" mean?  That sounds scary as the entire math library is assumed to be stateless as is the generated Stan model code.
Drawbacks
I'd remove "code smell" and just list the drawback---everything's going to get more complicated on the template and code duplication front as well as in the parser as this spec has us overloading more heavily.
Rationale
Why did this get consensus?  Is it the close connection to Eigen? The simpler long-form inputs?  Both are positives here, I think.  We definitely don't want to be innovating in the sparse matrix space!
I strongly prefer the type name to attributes.  Once you have sparsity, you are normally having to deal with it both computationally and statistically.
Prior art
The prior art I'd look for would be in R, Python (NumPy/pandas), and MATLAB.  Any idea what those do?
Unresolved questions
That first bullet point should be removed---it's instructions, not an unresolved question.
Language design is not an unresolved question.  What needs to be described more fully is how type inference is going to work and how assignment is going to work. Can we assign sparse matrices to dense matrices, for example?  Is any indexing allowed?  These absolutely must be answered and not left unresolved before going forward with this.
The new Stan compiler is just the implementation of this.  I don't think we need a lot of details on that in this spec unless there's some potential pitfall that's not obvious.
The whole theory of MCMC changes when you allow different numbers of parameters per iteration.  For example, what's it mean to converge?  We can leave that can of worms closed for now.
Cost not being worth it is the overall question for the spec, not an unresolved question.
You can leave it to reviewers to chime in on prior art.
Eigen
So our constructors are going to require conversion back and forth to the inner representation of Eigen.  That seems like it should be listed under drawbacks.


Use ivy

The Swiper project contains Ivy, Counsel, and Swiper, each of which seems potentially very useful.
See also this article.

Resolved in a5ad6b21b2999aa69f960b54b16cb7e45850b344.


fix: separate public and internal types for all instrumentations

Which problem is this PR solving?

fixes #1120 See detailed explanation in the issue

Short description of the changes

Added GUIDELINES.md file to the repo, which documents the applied conventions for future instrumentation implementors. This file can be extended in the future with more guidelines.

For all instrumentations:

types.ts file now contains only public api types (instrumentation config + transitive types)
all types which are required for the instrumentation and should not be publicly exported are moved to internal-types.ts file. They are only imported inside the instrumentation files.
export * from 'types' in index.ts. many instrumentations had types that were not exported, making it difficult to use. Now that types.ts contains only public types, it's simpler and less error prone to just export them all instead of picking a subset of the relevant ones.
I also exported ./enums/AttributeNames for custom attribute names. They are now exported everywhere, making them available for users to consume.
Added the type for relevant imports from the instrumented packaged. e.g. import type * as foo from 'foo';. This guarantee that we only use types. Since the instrumented package is not a dependency for the instrumentation, types is all we are allowed to import

@Flarna You investigated the instrumentation types issues quite a lot.
If you have some free cycles to review this change, I would highly appreciate it 🙏

I haven't looked into each and every instrumentation but the pattern itself looks good and the files I took a look were fine.

Thank you for the review 🙏

Great Job!


How to connect servers?

I have two paper 1.21 servers with IOSync  on it, I set custom path in storage-path, and when you connect to other server inventory of the player doesn't saves and loads on another server, can you help me maybe I set up something incorrect

Let's say you have two servers, in directories called server-a and server-b. You need a common path to put profiles in, I call mine players and place it right next to the servers. In this case, your directory structure will look like this:

players/
server-a/
server-b/

From there, here's the minimal content of config.yml to make it work:
storage-path: ../players/

And that's it! As players start joining and leaving your servers, the content of the players directory will gradually fill out with their shared inventories.

can we go to discord so I can explain what it looks like to me?

I suppose the problem is that I have these servers on hosting and they are not in the same partition, but for each of them a different partition is created and I do not understand where to store the data.

Ah, Minecraft hosting. The two could be installed on the same hardware, but in different accounts, so they don't see each other. In this case, it won't work as-is.
You can always try to ask your host if they could make a directory from account-a accessible to account-b. Let's say you create a directory named players/ inside of account-a. If its full path is in /home/account-a/players/, you ask your host to make this read-writable for both accounts. Then you configure both plugins to point to that same address, and it could work.
If both servers are on different hardware entirely, this isn't supported at the moment and won't work for you.

Thanks for your help! I will try to solve the issue with hosting


Make a readme

##I want help
i open new account
make me read me

Sorry @Nobody-ai-eng
I had to block you and now you cannot interact with my repos and profile.
Reason: Spamming


Icon Request: DreamFactory, Figma, H&R Block, NameSilo, TaxAct

Hi there! 👋
Love this set of icons and really appreciate your work here. Recently switched to Aegis and these are the only services I came across that didn't have an icon. Not sure if multiple requests in the same issue are okay; if not, I'm happy to make separate ones. Thank you!
DreamFactory

URL of the site or app (required):: https://www.dreamfactory.com/

Figma

URL of the site or app (required): https://www.figma.com

H&R Block

URL of the site or app (required): https://www.hrblock.com

NameSilo

URL of the site or app (required): https://www.namesilo.com

TaxAct

URL of the site or app (required): https://www.taxact.com

Thanks! Yeah, it's OK to make multiple requests.
Here's the icons:
https://github.com/krisu5/aegis-icons/blob/master/PNG/DreamFactory.png
https://github.com/krisu5/aegis-icons/blob/master/PNG/Figma.png
https://github.com/krisu5/aegis-icons/blob/master/PNG/H%26R Block.png
https://github.com/krisu5/aegis-icons/blob/master/PNG/NameSilo.png
https://github.com/krisu5/aegis-icons/blob/master/PNG/TaxAct.png

So awesome, thank you!


Are rotate_left/right/swap_bytes methods possible?

I'm curious what it would take to get functionality like integer primitives' methods.

I'm also curious about swap_bytes and others.  How many of the missing primitive methods are possible to implement here?

Sine one can easily perform these using shuffles, I think the question is rather whether it is worth it to implement these. Is there an ISA that supports them?

To be clear, I want to rotate each lane. Not shuffle the lanes around.
Sent from my mobile device

On Apr 9, 2018, at 23:07, gnzlbg<EMAIL_ADDRESS>wrote:
Sine one can easily perform these using shuffles, I think the question is rather whether it is worth it to implement these. Is there an ISA that supports them?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

To be clear, I want to rotate each lane

I see. In the meantime you can manually implement rotates using bitwise operators, shifts, and splat:
#![feature(stdsimd)]
use std::simd::*;
fn rotate_left(x: u32x4, n: u32) -> u32x4 {
    let n = n % 32;
    (x << n) | (x >> (32 - n) % 32)
}

(note that vector << n currently means vector << splat(n), but this behavior will be deprecated soon and one will need to use splat in the near future)

Not shuffle the lanes around.

The problem is that "rotate left" is both an operation on the primitive type (e.g. u32 above), and an operation on a sequence of element types (e.g. &mut [u32]). Both operations are equally reasonable things to want to do on vector types so we can't really give them the same names. Until now the naming schemes used try to reduce the friction of vertical operations, so I think using rotate_{left,right} for vertical operations and something else for the horizontal operations makes sense (e.g. rotate_elements_{left,right}).
For swap_bytes I don't know which names would be good.

I'm curious what it would take to get functionality like integer primitives' methods.

Somebody sending a PR with a tested implementation might just be enough. From there to having these land on stable Rust, a mini-FCP might be enough as well.

pull request here https://github.com/rust-lang-nursery/stdsimd/pull/496

We need swap_bytes_horizontally for rand, should I make a PR here or implement it in rand? https://github.com/pitdicker/rand/pull/1

Oh I was thinking to use shuffles. Would llvm.bswap be better?

Indeed, shuffles should definitely work. I don't know if llvm.bswap would be better (maybe @rkruppe knows). For the portable vector types, you can define a trait in the codegen module and implementing for each type using shuffles, and then the macro just calls the trait (e.g. check how it works for sqrt). We can then choose to use shuffles or bswap for each type independently.

I have little context but ISTM that element-wise bswap is quite a different operation from "interpret the vector as a huge integer and bswap that". But even if I'm mistaken about that, huge integer types like i256 and i512 (and even i128 to a lesser degree) have very spotty support in most backends even functionality-wise, let alone in terms of performance optimizations. Shuffles OTOH sound great. Except:

Just keep in mind that llvm.bswap only works for integers, not vectors, so you are going to have to cast the 256-bit and 512-bit vectors to i256 and i512 somehow..

Where'd you get that idea? https://godbolt.org/g/NG6Gxu

I got it from the LLVM LangRef :/
On Fri 29. Jun 2018 at 21:06, Robin Kruppe<EMAIL_ADDRESS>wrote:

I have little context but ISTM that element-wise bswap is quite a
different operation from "interpret the vector as a huge integer and bswap
that". But even if I'm mistaken about that, huge integer types like i256
and i512 (and even i128 to a lesser degree) have very spotty support in
most backends even functionality-wise, let alone in terms of performance
optimizations. Shuffles OTOH sound great. Except:
Just keep in mind that llvm.bswap only works for integers, not vectors, so
you are going to have to cast the 256-bit and 512-bit vectors to i256 and
i512 somehow..
Where'd you get that idea? https://godbolt.org/g/NG6Gxu
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang-nursery/stdsimd/issues/419#issuecomment-401446306,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA3Npk1Ouepr-TqH7FgWjg-MSusQI3nzks5uBnqXgaJpZM4TNoRb
.

Swap bytes PR https://github.com/rust-lang-nursery/stdsimd/issues/509

@rkruppe https://llvm.org/docs/LangRef.html#llvm-bswap-intrinsics doesn't mention vectors, only integers :/

That's curious, spot checking. But I'm inclined to say it's a docs bug (that someone could report to get clarity) since most similar intrinsics can be applied to vectors "by default" and vector-bswap even has target-independent tests.

I've reported the bug: https://bugs.llvm.org/show_bug.cgi?id=38012

This is fixed in https://github.com/gnzlbg/packed_simd


Is there a way to 'trick' the window width?

I am trying to create an image of a web page for export to PDF, but It will be relatively useless unless I can get it to fit the dimensions of a standard PDF page, 8 1/2 x 11.
I have tried the width option, but it seems that it just cleaves an image of the full window width.
Sincere thanks for any help or suggestions.

1.0.0 has the option to define window width/height


refactor(accordion): organize code inside component

Another more.
If there is a constructor, always the first method (after inputs / outputs).
Input goes in alphabetical order, doesn't matter if they have setters or not. Hopefully we will get rid of setters anytime soon.

I must say that I find it harder to read (that is, field declaration mixed with methods).
For now it is -1 from me, but open to be convinced

So, should we do:
class Foo {

  @Input() a;
  @Input() h;

  @Input()
  set b(...) { ... }

  get b() { ... }

  @Input()
  set r(...) { ... }

  get r() { ... }

AKA, all simple inputs first and then the setter / getters one. What about Outputs? before Input setters or after?

I think I agree with Pawel - having the simple declarations all together is easier to read since it doesn't require an extra line in between to separate it from the getters/setters.

Updated it.

Now it looks like an extra line is added - this LGTM, although you should just merge it in if it's this trivial IMO.

Yeah, I merge trivial for sure, but never realized that. I moved code around, made the PR, moved code around again and pushed back, but never looked into the diff.


Slicing mx.nd.array with postive start and neg end alone same axis

Current mx.nd.array does not support slicing with positive start end negative end alone same axis while numpy can do it.
import os
import sys
import numpy as np
import mxnet as mx

A = np.asarray([i for i in range(16)])
A = A.reshape((4,4))
B = mx.nd.array(A)
print A[1:-1] #ok
print B[1,:-1] #ok
print B[1:-1] #wrong
A[1:-1] = 0
print A #ok
B[1:-1] = 0
print B #wrong

fixed in #7609.


feat: add historical candles

Adds historical candles to shrimpy client
Relates to: https://github.com/trakx/market-data/issues/509
Fixes retry with same nonce

@monsieurleberre
Codacy erroring on code gen'd code? Can we ignore this?

@monsieurleberre Codacy erroring on code gen'd code? Can we ignore this?
Preferably we should use the extension .g.cs for generated code and ignore these files on codacy

Yes I will do that, thanks for raising


looks like the tool does not do anything

Hello,
When I execute this tool this is the output that I get:
Sub0ver -v -l domains.txt

It looks like the tool isn't working.
The content of the file domains.txt is a list of domains, for example:
www.example.com
new.domain.com
other.domain.com

@rhalyc,
You will any result if you sub domain is vulnerable for the subdomain takeover, otherwise this is expected output, any way you can use verbose mode to see what's happening.

@rhalyc You can try using -v flag to enable verbosity. @madaratech Thanks for explaining. The tool will report takeovers only when it's found some.

Thank you for the responses,
In my example I used the flag "-v" but the output is the same using or not that flag. That is why I think that maybe is something not working properly?

Try to update your version of the tool. If it doesn't work, use the -a flag to test all hosts.
AFAIK updating the tool will fix it.

Perfect. It is working.
Thank you!


new error

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.

Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Required by:
project :app > project :flutter_braintree > com.braintreepayments.api:drop-in:5.2.0 > com.braintreepayments.api:three-d-secure:3.15.0
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Could not get resource 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'.
Could not GET 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'. Received status code 403 from server: Forbidden

getting the same error.
FAILURE: Build failed with an exception.

What went wrong:
Could not determine the dependencies of task ':app:lintVitalRelease'.

Could not resolve all artifacts for configuration ':app:debugRuntimeClasspath'.
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Required by:
project :app > project :flutter_braintree > com.braintreepayments.api:drop-in:5.2.0 > com.braintreepayments.api:three-d-secure:3.15.0
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Could not get resource 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'.
Could not GET 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'. Received status code 403 from server: Forbidden

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.

Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Required by:
project :app > project :flutter_braintree > com.braintreepayments.api:drop-in:5.2.0 > com.braintreepayments.api:three-d-secure:3.15.0
> Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
> Could not get resource 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'.
> Could not GET 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'. Received status code 403 from server: Forbidden

Same Error When Run App
FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.
Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Required by:
project :app > project :flutter_braintree > com.braintreepayments.api:drop-in:5.2.0 > com.braintreepayments.api:three-d-secure:3.15.0

Could not resolve org.jfrog.cardinalcommerce.gradle:cardinalmobilesdk:2.2.4-1.
Could not get resource 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'.
Could not GET 'https://cardinalcommerce.bintray.com/android/org/jfrog/cardinalcommerce/gradle/cardinalmobilesdk/2.2.4-1/cardinalmobilesdk-2.2.4-1.pom'. Received status code 403 from server: Forbidden

i am also facing the same issue...
did anyone solved it?

i am also facing the same issue...
did anyone solved it?

https://github.com/pikaju/flutter-braintree/issues/74#issuecomment-882661183

i am also facing the same issue...
did anyone solved it?

Please try out the new version 2.0.0 and let me know if this fixes your issue. Thank you for your patience.


M07-Lab01-Ex3 Changes to Alert Automation

Contact Details
<EMAIL_ADDRESS>What happened?
Exercise : 03
Task: 01
Step: 24
Description of issue:
Alert automation (classic) is no longer available, we must create another Automation Rule.
Lab
Lab 07 Exercise 03 Create a Scheduled Query
Relevant screenshots

Do you want to help us? 👏

[ ] Create a Pull Requestt by following this project's Contribution Guide

Used the same rule but added new steps.

Merged my PR with these fixes, or something similar.


[1pt] Switch badges for Missions - Conditional components

Missions that the user has joined already should show a badge "Active Member" instead of the default "NOT A MEMBER" and a button "Leave Mission" instead of the "Join Mission" button (as per design).
Use the React conditional rendering syntax:
{rocket.reserved && (
    // render Cancel Rocket button
)}

https://github.com/inspecta/react-group-project-1/pull/34/


Tool name update

Tool name updated. Minor edits
@blackmist Please check and sign-off or comment.

@cjgronlund : Thanks for your contribution to the Azure documentation! The author, @Blackmist, has been notified to review your proposed change.

Thanks @cjgronlund ! #sign-off


UTF-8 encoded fields

Dear contributors, please consider if the attached changes are useful in the base repository. Test sets of course may be extended. Thanks for the great product!

Thanks for the contrib!
I have a naive question. Specification wise, shouldn't UTF-8 chars be quoted? Shouldn't mime messages be using US-ASCII charset?
Do you have at hand:

Specification for mime message in UTF-8 ?
Behaviour of other product / libs passing UTF-8 directly without encoding it?

Regarding directly UTF-8 encoding.

I have E-Mail samples from customers which are correctly processed in other clients. And they want to see them in BGERP as well.
Found that partially (only for fields) such encoding supported in currently used JavaMail with activation mail.mime.allowutf8 parameter.
https://javaee.github.io/javamail/docs/api/index.html?javax/mail/internet/package-summary.html
Unfortunately have no glue is that specified somewhere or not.

Q/Base64 encoding in fields as far as I know is completely standard, supported in JavaMail and often seen in sources.
https://github.com/apache/james-mime4j/pull/55/files#diff-038c4f63b17523939c4c5cae5e32c1dc8d35a5cecb9c2093941382984e4891afR121
https://github.com/apache/james-mime4j/pull/55/files#diff-110316b99e33390efddd1d52a7b30e9fc098b2e93fa45e84a1ad07743151a9c7R442

Q/Base64 encoding in fields as far as I know is completely standard, supported in JavaMail and often seen in sources.
There is methods to decode encode Q in Mime4j though I admit the API to be unfriendly and hard to discover.

Can you open an issue there https://issues.apache.org/jira/projects/MIME4J/issues/MIME4J-250?filter=allopenissues ?

An issue per case I guess?

Ok, you basically want Q encoding decoded everywhere by default.

I would separate the UTF-8 topic from the Q encoding topic.

Did you evaluate the performance impact of your changes (/benchmark for instance?)

Did you evaluate the performance impact of your changes (/benchmark for instance?)
Not yet. Will do that after separating them.

Any update on the benchmark front?

Any update on the benchmark front?

Not yet. Sorry, missed your question. For now I'm solving all the findings for our project in a separated fork's branch. But going to return everything finally to the upstream.

Running mvn test in /benchmark directory.
Branch master.
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.534 s
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.646 s

Branch utf8-encoded-fields
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.350 s
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.535 s

So, there is no noticeable performance impact so far.

Do you think we can claim support for https://datatracker.ietf.org/doc/html/rfc6532 Internationalized Email Headers ?
I read the RFC, it seems to be it!

Looks quite similar. But the support was already in code, I am just fixing some special cases, met by our customers.

My pleasure! What should be more done with the PR from my side in order to finish it and accept the changes?

Honestly I almost merged it myself.
But I ain't sure if @chibenwa intends of having a mention somewhere that we officially claim support the corresponding RFC?

Somewhere being https://github.com/apache/james-mime4j/blob/master/src/site/xdoc/start/index.xml
We also have that on James side: https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/architecture/implemented-standards.adoc#message-formats


AB#7154 Send pointOfSale correctly when calling Personalize flows and web experiences

… of pos

Nice work Jeff! Fixed.


Bumped into error in ade-0.1.1f zip file which failed to compile opencv-python in python:3.10-+alpine image today which usually success previously

Expected behaviour
Successfully install opencv-python as usual in python:3.10-alpine image as usual
Actual behaviour
Compilation failed
Steps to reproduce
pip install opencv-python in python:3.10 alpine image
opencv-python = opencv-python==<IP_ADDRESS>
62.48 ninja: job failed: /usr/bin/c++ -I/tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ippicv/ippicv_lnx/icv/include -I/tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ippicv/ippicv_lnx/iw/include -I/tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include -isystem /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG -std=c++11 -fPIC -MD -MT CMakeFiles/ade.dir/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp.o -MF CMakeFiles/ade.dir/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp.o.d -o CMakeFiles/ade.dir/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp.o -c /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp
62.48 In file included from /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/passes/topological_sort.hpp:17,
62.48 from /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp:7:
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:101:10: error: 'uintptr_t' in namespace 'std' does not name a type
62.48 101 | std::uintptr_t m_srcGraph;
62.48 | ^~~~~~~~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:22:1: note: 'std::uintptr_t' is defined in header ''; did you forget to '#include '?
62.48 21 | #include "typed_metadata.hpp"
62.48 +++ |+#include
62.48 22 |
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp: In member function 'const ade::Graph& ade::ConstTypedGraph::getCGraph() const':
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:106:53: error: 'm_srcGraph' was not declared in this scope
62.48 106 | return reinterpret_cast(m_srcGraph);
62.48 | ^~~~~~~~~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp: In constructor 'ade::ConstTypedGraph::ConstTypedGraph(const ade::Graph&)':
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:122:9: error: class 'ade::ConstTypedGraph' does not have any field named 'm_srcGraph'
62.48 122 | m_srcGraph(reinterpret_caststd::uintptr_t(&graph))
62.48 | ^~~~~~~~~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:122:42: error: 'uintptr_t' in namespace 'std' does not name a type
62.48 122 | m_srcGraph(reinterpret_caststd::uintptr_t(&graph))
62.48 | ^~~~~~~~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:122:37: note: 'std::uintptr_t' is defined in header ''; did you forget to '#include '?
62.48 122 | m_srcGraph(reinterpret_caststd::uintptr_t(&graph))
62.48 | ^~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp: In constructor 'ade::ConstTypedGraph::ConstTypedGraph(const ade::ConstTypedGraph<OtherTypes ...>&)':
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:130:9: error: class 'ade::ConstTypedGraph' does not have any field named 'm_srcGraph'
62.48 130 | m_srcGraph(other.m_srcGraph)
62.48 | ^~~~~~~~~~
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp: In instantiation of 'ade::Graph& ade::TypedGraph::getGraph() const [with Types = {ade::passes::TopologicalSortData}]':
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:238:16: required from 'ade::Graph::NodesListRange ade::TypedGraph::nodes() [with Types = {ade::passes::TopologicalSortData}; ade::Graph::NodesListRange = ade::util::Range::MapRange<ade::util::Range::IterRange<__gnu_cxx::__normal_iterator<std::shared_ptrade::Node*, std::vector<std::shared_ptrade::Node > > >, ade::Graph::HandleMapper>]'
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/source/topological_sort.cpp:39:40: required from here
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:167:53: error: 'const class ade::TypedGraphade::passes::TopologicalSortData' has no member named 'm_srcGraph'
62.48 167 | return reinterpret_castade::Graph(this->m_srcGraph);
62.48 | ^~~~
62.48 Note: Class cv::Feature2D has more than 1 base class (not supported by Python C extensions)
62.48 Bases: cv::Algorithm, cv::class, cv::Feature2D, cv::Algorithm
62.48 Only the first base class will be used
62.48 Note: Class cv::detail::GraphCutSeamFinder has more than 1 base class (not supported by Python C extensions)
62.48 Bases: cv::detail::GraphCutSeamFinderBase, cv::detail::SeamFinder
62.48 Only the first base class will be used
62.48 ninja: subcommand failed
62.48 Traceback (most recent call last):
62.48 File "/tmp/pip-build-env-z3sx64lf/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
62.48 cmkr.make(make_args, install_target=cmake_install_target, env=env)
62.48 File "/tmp/pip-build-env-z3sx64lf/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 697, in make
62.48 self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
62.48 File "/tmp/pip-build-env-z3sx64lf/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 742, in make_impl
62.48 raise SKBuildError(msg)
62.48
62.48 An error occurred while building with CMake.
62.48 Command:
62.48 /tmp/pip-build-env-z3sx64lf/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake --build . --target install --config Release --
62.48 Install target:
62.48 install
62.48 Source directory:
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a
62.48 Working directory:
62.48 /tmp/pip-install-ikpi2bfw/opencv-python_cd41126da3fb445da2c1bce7c2f69c2a/_skbuild/linux-x86_64-3.10/cmake-build
62.48 Please check the install target is valid and see CMake's output for more information.
62.48
62.48 [end of output]
62.48
62.48 note: This error originates from a subprocess, and is likely not a problem with pip.
62.48 ERROR: Failed building wheel for opencv-python
62.48 Successfully built unstructured-inference
62.48 Failed to build opencv-python
62.48 ERROR: Could not build wheels for opencv-python, which is required to install pyproject.toml-based projects
Is it something newly added in the ade source that cause something broken?
tmp/pip-install-tltrnbg7/opencv-python_218517a01db34d7aa0d9c09d5f167498/_skbuild/linux-x86_64-3.10/cmake-build/3rdparty/ade/ade-0.1.1f/sources/ade/include/ade/typed_graph.hpp:22:1: note: 'std::uintptr_t' is defined in header ''; did you forget to '#include '?
412.4 21 | #include "typed_metadata.hpp"
412.4 +++ |+#include
412.4 22 |

opencv-python==<IP_ADDRESS>

There is no way to amend already released code (>1 year ago).
Try to upgrade and verify the latest release.

note: 'std::uintptr_t' is defined in header ''; did you forget to '#include '?

Looks like compiler is broken in your build environment.
Correct message should look like:
note: ‘uintptr_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?

@opencv-alalek  erm this one i tried in python 3.11-alpine based image.. it failed because of imcompatible of alpine version. Hence i tried to downgrade to python3.10-alpine image. Then it was working all the way until recently bumped into issue. I tried to install the unstructured[pdf] python module and underlying it install opencv. I do saw it download the latest version but due to pip runtime backtracing issue it will still downloading the old version to find the most suitable to match across all the dependencies. Hence i bumped to this problem and stuck.
For environment, i think im using the same system env which previously working. and since im working on docker images and i think those things will be independent as stated in Dockerfile.