Feature/issue 2682 add 7 parameter ddm pdf

Summary
We provide an implementation of the LPDF drift diffusion model with up to 7 parameters. There will be two new functions available in the stan::math namespace: wiener_full_lpdf and wiener_full_prec_lpdf. The only difference between both functions is the control of the precision in the computation of the partial derivatives. We did not deal with the CDF functions. With the new funtion it is possible to compute the 4-, 5-, 6-, 7- parameter drift diffusion models. Therefore, it is also an alternative to the wiener_lpdf by @kendalfoster.
The PR consists of the following new files:
Main file:
-stan/math/prim/prob/wiener_full_lpdf.hpp.
Helper files:
-stan/math/prim/fun/wiener5_lpdf.hpp: log density and partial derivatives for 5 parameter model
-stan/math/prim/fun/wiener7_lpdf.hpp: log density and partial derivatives for 7 parameter model
-stan/math/prim/functor/hcubature.hpp: new multidimensional integrator, adapted  from Steven Johnson
-test/unit/math/prim/prob/wiener_full_test.cpp: own unit tests
-test/unit/math/prim/prob/wiener_full_prec_test.cpp: own unit tests for function with precision control
The following files are adapted in order to be able to implement a function with 8 parameters:
-stan/math/prim/functor/operands_and_partials.hpp
-stan/math/rev/functor/operands_and_partials.hpp
Discourse threads:
Testing for 7-parameter function with known partials -> Therefore, we only let scalar input be accepted and no vectorization.
Is there a multidimensional integrate-function? -> Therefore, we implemented hcubature.
How to document correctly?/How to add a new function’s documentation to the Stan Function Reference? -> Up to now, we have the doxygen documentation.
Adding a function with LARGE known partial derivatives -> We introduce 3 helper files.
HMC: Does precision of derivatives affect validity of results?
Issue:
Extension operands_and_partials to 8 edges #2698
Tests
stan/math/test/unit/math/prim/prob/wiener_full_test.cpp and wiener_full_prec_test.cpp
In this discourse thread we asked on how to test the new function. Since we have more parameters than all other functions, we wrote own unit tests to test all exceptions for invalid input, and tested  whether valid scalar types give the correct result. Results are compared with values from the R-package WienR.
Side Effects
We had to adapt the operands_and_partials routine such that it accepts 8 parameters.
We had to implement a new, multi-dimensional integrator: hcubature.
Release notes
Two new functions in the stan::math namespace availabe: wiener_full_lpdf and wiener_full_prec_lpdf.
Along with some helper functions in stan::math::internal.
New integrator: hcubature.
Adaptation of operands_and_partials to 8 parameters.
Checklist

[x] Math issue #2682

[x] Copyright holder:
- Franziska Henrich<EMAIL_ADDRESS>- Valentin Pratz<EMAIL_ADDRESS>- Christoph Klauer<EMAIL_ADDRESS>- Raphael Hartmann<EMAIL_ADDRESS>The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[x] the basic tests are passing

unit tests pass (to run, use: ./runTests.py test/unit)
header checks pass, (make test-headers)
dependencies checks pass, (make test-math-dependencies)
docs build, (make doxygen)
code passes the built in C++ standards checks (make cpplint)

[x] the code is written in idiomatic C++ and changes are documented in the doxygen

[x] the new changes are tested

I'm not an expert in these but I'm curious how the 5 parameter one different than the wiener_lpdf already in stan? It's 4 parameters but then we separate in a model whether the outcome hit the lower or upper bound.

The 5 parameter model incorporates the variability in the drift rate parameter. This is not possible in wiener_lpdf. You can set this variability to 0, then you obtain the 4 parameter model which has the same output as wiener_lpdf, but since we have all partial derivatives implemented, the whole procedure iscomputationally much more efficient. We also need the wiener5 functions for the calculation of the wiener7 functions.

I believe you'll want @SteveBronder, @andrjohns, or @rok-cesnovar to have a closer look. I think we'll want to discuss what's the best name for these.
I'm not an expert in these but I'm curious how the 5 parameter one different than the wiener_lpdf already in stan? It's 4 parameters but then we separate in a model whether the outcome hit the lower or upper bound.

I can review this tomorrow

The 5 parameter model incorporates the variability in the drift rate parameter. This is not possible in wiener_lpdf. You can set this variability to 0, then you obtain the 4 parameter model which has the same output as wiener_lpdf, but since we have all partial derivatives implemented, the whole procedure is computationally much more efficient. We also need the wiener5 functions for the calculation of the wiener7 functions.

Very cool! When we expose this to Stan we can just use wiener_lpdf and have different signatures for each type. Do we even want to keep the current wiener_lpdf if this one is superior?
If you get up the energy to add the lcdfs and the rng functions, I'd love to have those in. I have a model that needs the lcdf and lccdf I've been wanting to test. I did code up the 4 parameter _rng function (https://github.com/stan-dev/math/issues/2801) in Stan but haven't had the time to put it in.

add the lcdfs and the rng functions,

Yeah, thanks @spinkney! That's true. When setting all variabilities to 0 and calling the function with wiener_full_lpdf(a,v,w,t0,0,0,0) the four parameter model will be computed. Therefore, the wiener_lpdf() gives no advantages anymore.
We have not yet implemented the lcdf- and rng-functions. We have to check how much effort it is to bring them to Stan. But we will check this and may contribute them later in a separate PR. First, we wanted to add the lpdf-functions.

We have not yet implemented the lcdf- and rng-functions. We have to check how much effort it is to bring them to Stan. But we will check this and may contribute them later in a separate PR. First, we wanted to add the lpdf-functions.

Yes, I agree! It's best to do these in separate prs.

@Franzi2114 from my understanding this PR comprises three main contributions:

Extending operands_and_partials to 8 parameters
Introducing hcubature integration routine
Introducing wiener_full_lpdf and wiener_full_prec_lpdf

Can you split this into three PRs rather than all together? That will make it much easier to ensure that each component is fully tested, and also gives a bit of a cleaner development history for the repo. Thanks!

Can you split this into three PRs rather than all together?

Hi @andrjohns, thanks for your input! Yes, we can split our PR into three PRs. I will refer to the same issue in the branchnames (Feature/issue 2682 operands and partials 8, and Feature/issue2682 hcubature).
Up to now, we don't have separate tests for hcubature. The integration is checked in the tests for wiener_full_lpdf(). So we will add separate tests and do a third PR.

Hey @andrjohns, I am confused. I only merged the develop branch (where the new operands_and_partials_8 routine is integrated now) into this branch and the continuous-integration test fails at a point I do not understand. Did I do anything wrong?

@serban-nicusor-toptal would you mind restarting the Threading tests for this Jenkins run?
Is it possible to give my account permissions to restart runs as well? So you don't have to get bothered with pings everytime

@WardBrian would the Distribution test failures here be resolved by #2869, or is this a different problem?

#2869 has been merged into develop, feel free to merge develop into this branch.

Hello together,
now the two split-PRs (#2833, #2838) are merged into develop and into this branch and all tests run successfully. Who would review this PR?
@andrjohns, @spinkney, @SteveBronder, @rok-cesnovar, @charlesm93, someone else?
We already conducted a simulation recovery study and a simulation-based calibration study. Everything looks fine with the new implementation. Recovery is satisfying and SBC is successful. The paper is still in the review process.

I'll take the review for this one

I'll take the review for this one

Hey @andrjohns, if I can give you any information or background regarding this implementation just let me know ;) Do you have an idea when you can start with the PR?

Hey @andrjohns, I just saw that the next release is being planned and that the feature freeze will be on 4th of April, in three weeks. Do you think this PR can make it into the next release? That would be nice, since I am planning to implement the cdf of this function as well and I would like to know which programming style is going to be accepted to make all of us less work.

Was slightly derailed by rstan and rstanarm CRAN prep, I'll have this updated by tomorrow

I'll have this updated by tomorrow

Hi @andrjohns, is there any news on this PR? Will it be part of this release as @spinkney suggested?

I believe it is too late for this distribution to be included in the 2.32 release

Hello, I plan to also implement the CDF functions. It would be nice to have the PDF functions in the develop-branch before I start, so that I know which style will be accepted. Could we continue with this PR?

@Franzi2114 this is just about good to go, but there's a test hang in rev/prob/wiener_full_test that I can't replicate locally in MacOS or linux. Would you mind checking out the latest updates and seeing if the test hangs for you locally as well?

@Franzi2114 this is just about good to go, but there's a test hang in rev/prob/wiener_full_test that I can't replicate locally in MacOS or linux. Would you mind checking out the latest updates and seeing if the test hangs for you locally as well?

False alarm sorry, found it

@Franzi2114 That's the last of my changes. To summarise, I've:

Updated hcubature to take a tuple of parameters instead of casting structs. There's an additional simplification available by having the function take a parameter pack instead, but that can be changed in the future
Removed wiener_full_prec overload in favour of a default argument for the precision for wiener_full
Consolidated the separate density and derivative functions for wiener5 into a single function to reduce repetition
Added a utility function for handling the process of estimating the density/gradient, checking against an error value, and re-estimating if failed
Added two utility functions to simplify the process of calculating the wiener7 quantities, either directly from the respective wiener5 function or by integrating over it

Overall that reduced the amount of code by a little over half for each of the density functions.
Can you have a look over the changes and let me know if there's anything that doesn't make sense or you'd like changed? Thanks!

Can you have a look over the changes

Thanks a lot for all your effort! I'll have a look at this and let you know when I'm through everything.

@Franzi2114 Ah hold on, there's something a bit off with your construction of the wiener_full likelihood here:
  for (size_t i = 0; i < N; i++) {
    // Calculate 4-parameter model without inter-trial variabilities (if
    // sv_vec[i] == 0) or 5-parameter model with inter-trial variability in
    // drift rate (if sv_vec[i] != 0)
    if (sw_vec[i] == 0 && st0_vec[i] == 0) {
      return wiener5_lpdf<propto>(y, a, t0, w, v, sv, precision);
      // Calculate 6-, or 7-parameter model
    } else {
      return wiener7_lpdf<propto>(y, a, t0, w, v, sv, sw, st0, precision);
    }
  }

This loop will only test the firstsw and st0 values, and then depending on those will pass all inputs to either wiener5_lpdf or wiener7_lpdf to evaluate.
To simplify this, is the separate wiener_full distribution necessary? We could instead change the wiener7_lpdf to test these values and delegate to wiener5 where appropriate. Let me know if that's ok with you and I'll make the update

Makes sense! I'll update that now

Hey @andrjohns , in your wiener5 implementation there are some spots to be revised. I will mark them later. But now @SteveBronder has some comments on the new structure. Maybe we should first agree on whether to have all functions as separate  functions as it was before, or all functions combined in one big function with many if statements, as it is now.

@SteveBronder I've refactored the wiener5 implementation out to separate functions, can you confirm whether this structure works for you (variable and function names aside)? Also noting that we generally don't want to double-review and this PR has been open for quite some time, so I've marked a few of the review comments as out of scope but will open a separate issue for them to be addressed later.

@Franzi2114 Can you update the variable and function names (e.g., ans0, lg1, kss, etc.) to be a bit more descriptive of quantities they represent and then ping @SteveBronder when it's ready for another look?

Can you update the variable and function names

Hi @andrjohns, I'll go over the files and make the name-changes. I'll ping both of you when I'm done.

Can you update the variable and function names

Hi @andrjohns, I'll go over the files and make the name-changes. I'll ping both of you when I'm done.

Great! Also feel free to message me if there are any other changes/questions for the functions, you mentioned some after the first refactor. You can either comment on the code here and ping me or PM me via discourse or the Stan slack, whichever's easiest

Hey @SteveBronder, are there any news? When do you think you have time to look at this PR?

Hi sorry I will look at this today

2. Remove a lot of the lambdas

Hey @SteveBronder, thanks for your comments. I changed the things in hcubature. Is it better now?
What do you mean with 'lambdas'? I think I don't get the term. I only know lambda as the greek letter for variable names. Could you explain a bit more how you think this call_wiener7_function could be made smaller?

I went through some of this but theres a lot here. See the differences here
https://github.com/Franzi2114/math/compare/feature/issue-2682-Add-7-parameter-DDM-PDF...stan-dev:math:fix/7-param-ddm
Mostly I would look hard at anywhere you have an in/out parameter and try to rephrase it so that there is no in/out parameters. For instance with functions that have an err that is an in/out you can just return an std::pair<>

See the differences here

I merged your suggestions and changed some names and docs.
But I still have no good idea how to get rid of the wiener7_integrand_impl function in wiener7_integrate.

Yeah I think given you need to do the conditional stuff that may be the best way to do it for now. If I can think of another way I'll lyk.
I made a few other changes below
https://github.com/Franzi2114/math/compare/feature/issue-2682-Add-7-parameter-DDM-PDF...stan-dev:math:fix/weiner_lpdf_full2

@SteveBronder, do you know where this MPI-test error comes from that makes the continuous integration test fail?

I just restarted that test, that is a very weird error. If it shows up again we can ask @serban-nicusor-toptal to have a look. My guess (hope) is that it was a one off error

one off error

Hm, unfortunately it failed again ... due to unstableness. @SteveBronder do you have an idea, or @serban-nicusor-toptal?

@serban-nicusor-toptal when you get a moment could you check out this jenkins error? It looks like linking issue in the downstream tests?
https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FStan/detail/downstream_tests/306/pipeline

Hey @SteveBronder the downstream tests are finishing with UNSTABLE and it fails the upstream. This gate fails in Stan, somehow this warning is reported 200-ish times.
Stan does have a filter to exclude anything under /lib so it's weird that it failed with lib/stan_math/stan/math/prim/prob/wiener5_lpdf.hpp , I restarted the downstream tests and will watch them closely. ( I fear that the filter is wrong )

@Franzi2114 Can I please ask you to merge develop in? I've merged a fix for the reporting plugin, thanks!

Also I have some changes below to help set this up to be vectorized
https://github.com/Franzi2114/math/compare/feature/issue-2682-Add-7-parameter-DDM-PDF...stan-dev:math:fix/wiener_full_fix3

everything to work over vectors

We chatted about the vectorization topic in the beginning with @bob-carpenter and @WardBrian in this discourse thread. The main argument was that we would have too many signatures if we allowed vectorization for all 8 input-arguments. This would blow up the testing framework. Therefore, we allowed only scalar input.

Reading that thread I think Bob is saying we still want to use the expect_ad() framework, but fixing some parameters to doubles so they don't get tested.
imo I think we should make the C++ work for vectors, @WardBrian and I are talking about rewriting the function lookup in the compiler to handle larger signature sizes like this and I'd rather we the C++ vectorized now so that we can then use in the the language when the compiler is able to handle it

vectorized now so that we can then use in the the language when the compiler is able to handle it

So you mean, this 7-parameter diffusion model density will rely on your extension of the signature lookup function and this function must be finished before the 7-parameter function can work?

I think we would initially expose just the non-vectorized signature, but it would be nice if the vectorized ones were already implemented for said future improvements to the compiler

Ok.
Then I'll probably undo the last merge, where I merged Steves changes for the vectorization part from the wiener_full_fix3 branch. Correct? The suggestions should still be available in this latter branch, shouldn't they?
I already tried do implement some mix tests, but was not successful yet. I used normal_test.cpp as a prototype but this did not work out in any way. Can you give me a hint how to implement them or which test could serve as a prototype?

I think it’s a good idea to have the fullest version of the C++ implemented, even if using all of it might take some time, but I’ll defer to @SteveBronder

We should write the c++ to support vectorization, but we don't need to wait for anything from the compiler to merge this.
What I meant to say is that we merge this supporting vectorization so that once the compiler can support large function signatures like this we can just flip on the code to allow the vectorized version to work in the stan language

Ok.
So now, the code works for vectors. Is the code fine now?
Then, only the mix tests are missing, correct? Can you give me a hint how to construct them?

For the mix tests I'd look into the test files like ./test/unit/math/mix/prob/bernoulli_logit_glm_lpmf_test.cpp. There we have an extra parameter we want to add and pass it in to a lambda that then generates the lambda for testing. So if you have 7 parameters the max that can go into the expect_ad framework is 3. So you can do something like
auto f = [](const auto x1&, const auto& x2, const auto& x3, const auto& x4) {
  return [&x1, &x2, &x3, &x4](const auto& p1, const auto& p2, const auto& p3) {
    return weiner_full_lpdf(x1, x2, x3, x4, p1, p2, p3);
  }
};

And test vector / scalar combinations with that for the first 3. Idt you need to do every possible combination, I think as long as each parameter is shown to be able to take in a vector/scalar version of itself that should be fine.
Also can we call weiner_full_lpdf something like weiner7_lpdf? idt full is a great descriptor to users

I adopted the scheme from the ./test/unit/math/mix/prob/bernoulli_logit_glm_lpmf_test.cpp. Then the following error occurs:
cannot convert ‘stan::math::fvar<stan::math::fvar<double> >::Scalar’ {aka ‘stan::math::fvar<double>’} to ‘const double’ in initialization
in wiener_full_lpdf in line 432 |     const double sv_val = sv_vec.val(i);  . This error occurs for all variables, not only for sv. Do we have to change something in the test or in the code? Do we have to give sv another type? I already tried some things but then more errors pop up.
I have not yet added my mix test to this repo since it ends with errors. If that's no problem I can add and commit the file.

It doesn't look like there are any tests in that branch. Can you add them so I can just run the below and see the error?
./runTests.py ./test/unit/math/mix/prob/weiner_full_lpdf_test.cpp

It doesn't look like there are any tests in that branch. Can you add them so I can just run the below and see the error?

Hey @SteveBronder, do you have any suggestions how to get rid of these errors?

Hello @SteveBronder, @andrjohns, @WardBrian, @spinkney,
is there a possibility that the not-vectorised version of the model can go into the next release? There are already some people that would like to use the seven-parameter diffusion model for their projects. Also the paper now is published here. There is an explanation included how to use it not vectorised.
Next, can anybody help me to get the mix test running? I'm stuck with errors I do not know how to deal with. The mix test is included in this branch issue-2682-auto-types.

I'm also not great at reading the compiler messages generated by this kind of error, but it seems like some of the templates you're using are expanding to double rather than the autodiff type that is expected of them. For example,  xmax declared on line 448 of wiener_full_lpdf seems to be an Eigen array of double in all cases, which causes the assignment on the next couple lines to fail

is there a possibility that the not-vectorised version of the model can go into the next release?

The next release is already locked in so this would have to be next release
I'll look at that branch this week to see what the error is

Eigen array of double

What is the correct type declaration for auto diff vectors?

What is the correct type declaration for auto diff vectors?

It matters if you are working with forward mode autodiff (fvar<double>) or reverse mode autodiff (var). For forward mode it can be Eigen::Matrix<fvar<double>, -1, 1> for gradients or it can be Eigen::Matrix<fvar<var>, -1, 1> for tangents. For reverse move it's (Eigen::Matrix<var, -1, 1>)..
When working with autodiff scalars (var and fvar<T>) we need to make sure, when mixing them with doubles, that the result type is also autodiffable. So if  we call a function f with f(var, double) or f(fvar<var>, double) then the result should be var for the first and fvar<var> for the second.
So for the code you have here. If your inputs are mixes of var and double then the var type you are working with is var and the Eigen vector type would be Eigen::Matrix<var, -1, 1>.
Our functions in prob need to work for all of our autodiff types var, fvar<double>, fvar<var>, fvar<fvar<var>> and doubles. Though note we do not support mixing autodiff types so you will never need to worry about a signature like f(var, fvar<var>, double, fvar<double>). Where you see partials_return_t, that type trait looks through the types and finds the least upper bound of the input autodiff types's inner type that it uses to store values. So the inner type of fvar<double> is a double and the inner type for a fvar<var> is a var. For plain var the inner type is double since that is what we store the values and adjoints as. For double it's just double since there is no inner type
So for partials_return_t<fvar<var>, double, fvar<var>, double> the "inner types" are var, double, var, double so the return type would be var. For partials_type_t<var, double, double> the return type would be double.
The functions here need to be written so that, for the combinations of valid var, fvar, and double signatures, they compile and are able to take the gradient. Our tests check that var, fvar<double>, fvar<var> work in combination with double types.
What I would do here is start small. Write a test that just uses fvar<double> and  double in the signature. Then write one that uses var and double in the signature. At this point just checking that it can compile. You might want to start with hcubature as it looks like wiener_full sends the partials type through it and that is a source of a lot of the problems I'm seeing in your branch you pushed above.

@Franzi2114 Is there any update on the progress for this? I would love to use the 7-parameter DDM in Stan.

update on the progress for this

Hi @singmann, I'm still struggling with the mix tests for the model. I'm in personal contact with @SteveBronder and hope that the code will be in the next release. Meanwhile you can use the cmdstan-tarbal from my Github page. Here is an instruction on how to get started with the 7-parameter diffusion model with this tarball.

@Franzi2114 Thanks for the update. Looking forward to seeing this in Stan.

Hey @SteveBronder, @andrjohns, now all tests are successful and all mix tests. Would it be possible to include this in the next release for January 8th?

Hi @SteveBronder, hi @andrjohns, as @WardBrian wrote in the planning of the release 2.34, he could include this PR in the next release when it is approved befor 6th of January (see here).
As all test are successful now, could you approve the PR this week?

I started a review for this today and should be done tommorow. There's a lot of code and this is a pretty complicated PR I really don't want to rush this to try to get it into the next release.

I'm still working on the below so don't merge it to your branch, but I have a branch where I'm going through and making changes. A lot of the changes are just easier to write myself instead of put into a code review
https://github.com/Franzi2114/math/compare/feature/issue-2682-Add-7-parameter-DDM-PDF...stan-dev:math:fix3/wiener5

Hey @SteveBronder, your changes are integrated, the docs are changed and the mix tests are completed. All tests pass now.

Agh looking at the error it looks like boost's gauss kronrod only works with literal types like double, float, etc. I made an issue with boost about it. I actually wonder if we can just use Stan's integrate function? Let me play around a bit and if I can't get that today then we will revert back to the gauss kronrod impl we had before
https://github.com/boostorg/math/issues/1077

Looking at this more I think to use the boost version we would need to have integrate_1d_gauss_kronrod like what is suggested in #3000. I'm going to revert it back to using the current impl and put that issue on my plate at some point in the future. I think implementing that is outside the scope of this PR

Hi Steve, could you look at the code once more? The errors are now because I commented the two #include <stan/math/prim/prob/wiener_lpdf.hpp> lines out. How to proceed now?

You should uncomment those two includes. Those are still needed for the 4 parameter model

Those are still needed for the 4 parameter model

This old implementation is not anymore needed. The new implementation is also able to compute the estimates for the four parameter model (when setting sv to 0). It is also more stable and robust than the old one and works in combination with a mixed model for outliers (the old one does not). I added another few lines to call our model with four parameters:
template <bool propto = false, typename T_y, typename T_a, typename T_t0,
          typename T_w, typename T_v>
inline auto wiener_lpdf(const T_y& y, const T_a& a, const T_t0& t0,
                        const T_w& w, const T_v& v,
                        const double& precision_derivatives = 1e-4) {
  return wiener_lpdf(y, a, t0, w, v, 0, precision_derivatives);
}  // end wiener_lpdf

Then, this command calls our four parameter model: wiener_lpdf(y, a, t0, w, v);, but only when the old one is deleted as they look the same after you renamed wiener_full_lpdf to wiener_lpdf. This was the reason that we named it wiener_full_lpdf to have the other one coexisting. But this is now not possible. Therefore, we should delete the old files.

We don't want to get rid of the old code. To make sure I'm understanding are you saying the compiler cannot tell the difference between the old wiener distribution and the new one? These tests were passing previously so I'm not following why the old code needs deleted?

cannot tell the difference between the old wiener distribution and the new one

This is now only the case for the 4-parameter model. Our new implementation can model a 7, 6, 5, and 4 parameter model (and even less parameters when setting more parameters to 0). And for the 7, 5, and 4 parameter model there are distinct function calls:
4 parameters: wiener_lpdf(y, a, t0, w, v);
5 parameters: wiener_lpdf(y, a, t0, w, v, sv);
7 parameters wiener_lpdf(y, a, t0, w, v, sv, sw, st0);
But the call for the 4 parameter model is now identical to the call for the old wiener implementation. So this has nothing to do with the tests. They are still passing as  before.
Nevertheless, it would be possible to model the 4 parameter model (in our implementation) with the call
4 parameters: wiener_lpdf(y, a, t0, w, v, 0);
But more convenient would be the call above, and then ending in our implementation as it works better than the existing one.

@SteveBronder, could you look at the error message? I can't make sense of the threading error. When I run the test (python3 runTests.py -j4 test/unit -f thread) on my PC there does not occur a Segmentation Fault error.

Thanks! I'll look at this today and Monday

changes to the jenkinsfile

Which changes to which jenkinsfile? I think I didn't change anything ... at least not on purpose.

@Franzi2114 feel free to click the merge button at your convenience :)
Your contribution is very appreciated! When you have a moment would you mind writing the Stan docs for the new 5 and 7 parameter wiener distributions in the page below?
https://github.com/stan-dev/docs/blob/master/src/functions-reference/positive_lower-bounded_distributions.qmd#L173

@Franzi2114 feel free to click the merge button at your convenience :)

Great! Wow, I am so happy that the PR is eventually completed! ^^
However, I cannot press the merge button as I am not authorized to do this. So, could one of you merge the PR?

I think we can add @Franzi2114 to let him do the honors himself

Oh! I'm terribly sorry! idk when we changed that? I think it's nice for contributors to click merge on their own PRs. Let me see tmrw how to change that

@Franzi2114 I just made you a member of the Stan math repo :) You may have to check your email to turn it on. But you should have merge permissions now!

Hm ... I accepted the invitation and are now member of the repo. I reloaded this page and signed out and in again. But still the message is 'You are not authorized to merge this pull request.' Seems more tricky to do the last step than expected ^^

@Franzi2114 sorry try one more time. Thought I gave you the right access but it was one less than you needed

Now there popped up another box with a racket saying 'This branch has not been deployed', but I still cannot merge the PR.

My apologies the whole thing is confusing. I just sent you a member request for the Stan dev team which should give you sufficient privileges (check your email). If after clicking the email you still cannot merge the PR then I'll merge it.
I find this really weird though and want to figure out why this is happening. I really think it's important and satisfying to let collaborators merge their own pull requests (first PR is a big deal!)

@serban-nicusor-toptal do you know how to make it so outside collaborators can still click the merge button on their own PRs?

Good morning @SteveBronder @Franzi2114, I think everything is setup correctly. In order for Franzi to get proper access, he will need to accept the invitation to join the Math org team. Once that's done, the team will grant you the required permissions.

YEEEEEEEEEEYYYYYYYYYY!!!!!!!!!! It worked!!!!!!!!!!!!!!!!!!!

Wuhu! Thanks =D
Wow, that was really a long journey, but I think I learned a lot.
Thank you so much for all your proof reading, input, corrections and all the time you spent on this PR!
I already prepared the CDF functions for the 7-parameter diffusion model. I will make the PR in the next days.

Congrats on the huge effort!
We can expose the basic scalar version in the compiler very soon (most likely next week!). A more advanced typechecking which lets us expose the vectorized version is something we’ve started discussing as well

Congrats! Looking forward to testing this out and seeing the cdfs. Are you planning to implement the rng functions too?

rng functions too?

First, I do the CDF and CCDF functions. But compared to the CDF and the PDF, the rng should be quickly done. So I'll probably implement this as well.

In terms of use I think the rngs would be really nice so folks can do prediction etc.


Move AKS from Ember to Vue

In support of moving all public cloud provisioning off Ember (https://github.com/rancher/dashboard/issues/8083 and in a way, tied to work for finding all pages off Ember in https://github.com/rancher/dashboard/issues/8718), this ticket is to move the AKS pages to being in Vue.
Currently, to provision AKS, it's iFramed from Ember into Vue.
Original embedding work: https://github.com/rancher/dashboard/issues/2738

We should start this in Q3 but it will likely land in Q4. I need to link to various other AKS tickets that should be handled as part of this work.

@nwmac this ticket looks like it skipped QA and stayed close. However, QA has been testing parts of this as evidenced by linked issues. Should we put this back To Test?

Yes, we should

I am not sure if there is sure shot way of testing this. We have already filed a few UI bugs, after running an entire test suite on 2.9, I believe this issue can be marked as complete. I will close it.


Writing a Test cases for E-filing, Scrutiny and Admission , Pending Tasks

Please refer to the test cases for  E-filing.
E-filing test cases.xlsx


feat(katana): forked provider fetch remote non-state data

This PR adds the following functions to fetch non-state data from the fork network.
pub fn do_get_events(
        &self,
        filter: EventFilter,
        continuation_token: Option<String>,
        chunks_size: ChunkSize,
    ) -> Result<EventsPage, ForkedBackendError>

pub fn do_get_block_with_tx_hashes(
        &self,
        block_id: BlockIdOrTag,
    ) -> Result<starknet::core::types::MaybePendingBlockWithTxHashes, ForkedBackendError>

pub fn do_get_block_with_txs(
        &self,
        block_id: BlockIdOrTag,
    ) -> Result<starknet::core::types::MaybePendingBlockWithTxs, ForkedBackendError>

pub fn do_get_transaction_by_block_id_and_index(
        &self,
        block_id: BlockIdOrTag,
        index: TxNumber,
    ) -> Result<Transaction, ForkedBackendError>

pub fn do_get_transaction_by_hash(
        &self,
        transaction_hash: TxHash
    ) -> Result<Transaction, ForkedBackendError>

pub fn do_get_transaction_receipt(
        &self,
        transaction_hash: TxHash
    ) -> Result<starknet::core::types::MaybePendingTransactionReceipt, ForkedBackendError>

Closes #1466
Closes DOJ-128

sorry for the late review @fishseabowl i forgot about this 🙏 .
i realized that this is a bit more complex than what i had anticipated. due to how katana's storage provider works, we cant exactly map the RPC calls to one of the forked backend requests. it needs to call the same provider traits methods just like the other provider types.
for example, when the getBlockWithTxs is called on the RPC server,
https://github.com/dojoengine/dojo/blob/e9fab2caf970621ed8311330f3752dc9635f60d7/crates/katana/rpc/rpc/src/starknet.rs#L290
it needs to go through all these provider methods (ie BlockHashProvider::block_hash_by_id, BlockProvider::block, and BlockStatusProvider::block_status) to build the right return type that the RPC expect.
https://github.com/dojoengine/dojo/blob/e9fab2caf970621ed8311330f3752dc9635f60d7/crates/katana/rpc/rpc-types-builder/src/block.rs#L22-L33
this works well for non-forked provider types but for forked provider, ideally we want to just call the appropriate RPC methods to the forked network and just return the result without any modification as the type is already in the RPC format.
i still think this is possible, we just have to do some redundant storage operations in order to conform to the provider abstraction that we have. though, it may requires storing the fetched remote data, which involves refactoring the ForkedProvider storage layout.
does that make sense ?

@kariy Thank you for your comments. Let me try this.

@kariy how do you feel about this? Do you think we definitively need more rework to support that? Or @fishseabowl suggestion can be a good candidate?

@kariy how do you feel about this? Do you think we definitively need more rework to support that? Or @fishseabowl suggestion can be a good candidate?

@glihm @kariy I have updated the relevant code, can you help check what is missing? Thanks

Merged in #2592.
@fishseabowl thank you again for the time and effort here, please ping me on the dojo discord or telegram @glihm when you have a chance. 👍


datasource id changes

Whenever the grafana server changes the postgres datasource ID increments.
I.e. it is set to 10 - reset server, goes to 11.
Ugh.
Could do something like using the default datasource, probably?

this is my fault, I think.
There's a line in the provisioning script which deletes and recreates the datasource - removing that will fix the problem :)

been working ok after removing.


Update dependencies

Update dependencies without bumping versions in libs.versions.toml.
Freeze kotlinx.html and kotlinx.serialization versions until Kotlin itself is updated.

Most version changes affect test dependencies.

The Graal23 patch file needed updating. Did a git rebase-patch and resolved the conflicts. It's not all there yet, because running git apply --reject patches/graalVm23.patch && gw :pkl-config-java:shadowJar fails for me on a Unsupported class file major version 64, so there's some Java 21 creeping into the build.

so there's some Java 21 creeping into the build

That’s because some of the Graal JARs are multi-release JARs, i.e., they contain classes for multiple JDK versions (17-21) and the JVM picks the right ones at runtime. However, the shadowJar plugin needs to be able to process all of them. The solution is to update the shadowJar plugin. I did this in my JDK 21 PR, which also got stuck on the patch file.

I updated the shadowJar plugin, which should take care of the "unsupported class file version" problem. The patch still applies cleanly. I can't test the patch because it doesn't work on Linux x64. Can the team trigger another PR build?

Ready to squash and merge from my side.

Further dependency updates and JDK21/22 CI builds are blocked on this.

@bioball Some CI builds are failing. I believe jline should be locked to its previous version (3.23.0) in libs.versions.toml because newer versions have a problem with native image. I already filed an issue for jline.

Yeah, sounds good. Where is the issue that you filed? They're using an option that got introduced for GraalVM for JDK 21: https://www.graalvm.org/release-notes/JDK_21/

just sent a PR with a link to my jline PR


More on model optimization.

I wonder if anyone else is also working on model optimization to achieve faster speed and lower RTF of LPCNet. I have trained LPCNet with different parameters, and tried to add sparsity in the kernel matrix of RGU_A, but none of them gave me satisfying result.
Can anyone share some ideas and solutions?

https://arxiv.org/abs/2005.05551

@zhangjh915 what's your email address ? I wanna discuss this with you.


Error with the Gem?

Just got linked to this on twitter, ran gem install terraforming
and when running the command terraforming
I get this error:
/opt/rubies/2.0.0-p451/lib/ruby/gems/2.0.0/gems/terraforming-0.0.1/lib/terraforming/util.rb:1:in `<top (required)>': uninitialized constant Terraforming (NameError)
	from /opt/rubies/2.0.0-p451/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /opt/rubies/2.0.0-p451/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /opt/rubies/2.0.0-p451/lib/ruby/gems/2.0.0/gems/terraforming-0.0.1/lib/terraforming.rb:9:in `<top (required)>'
	from /opt/rubies/2.0.0-p451/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:73:in `require'
	from /opt/rubies/2.0.0-p451/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:73:in `require'
	from /opt/rubies/2.0.0-p451/lib/ruby/gems/2.0.0/gems/terraforming-0.0.1/bin/terraforming:3:in `<top (required)>'
	from /opt/boxen/rbenv/versions/2.0.0-p451/bin/terraforming:23:in `load'
	from /opt/boxen/rbenv/versions/2.0.0-p451/bin/terraforming:23:in `<main>'

Am I missing something?

Sorry, this is a bug of Terraforming.
I fixed this at #35 , and released it as v0.0.2.
Please try again with new version :bow:
$ gem install terraforming -v 0.0.2

@dtan4 this works thanks :)

:+1:
Thank you for reporting issue!


Add /v3/service_instance/:guid/permissions endpoint

A short explanation of the proposed change:

Add the /v3/service_instance/:guid/permissions endpoint that enables service_instance dashboard functionality with the V3 API

An explanation of the use cases your change solves

As a service developer there is a functionality called service dashboards (https://docs.cloudfoundry.org/services/dashboard-sso.html#checking-user-permissions). These dashboards call the CF API to check if a user is allowed to alter or read the service_instance. This is done via /v2/service_instance/:guid/permissions.
A corresponding V3 endpoint does not exist which means this functionality wont work anymore when V2 is disabled.
This PR adds this endpoint also in V3.
See also https://cloudfoundry.slack.com/archives/C07C04W4Q/p1657878898740599

[x] I have reviewed the contributing guide

[x] I have viewed, signed, and submitted the Contributor License Agreement

[x] I have made this pull request to the main branch

[ ] I have run all the unit tests using bundle exec rake

[ ] I have run CF Acceptance Tests

Superseded by #2946


fix: convert svg sprites to also use webpack instead of grunt

https://github.com/kisenka/svg-sprite-loader


Fix for p_width - 1=-1 out of size

Fixes a zero-size image creation by moving the profiler plot update call
to the Clear button handler so it is not called during initialization.
Fixes #21521

Thanks!


Is there a Unique attribute?

What I have in Sqlite flavored SQL
CREATE TABLE IF NOT EXISTS "users" (
    "id"	INTEGER NOT NULL UNIQUE,
    "name"	TEXT NOT NULL UNIQUE,
    "email"	TEXT,
    "login_count" INTEGER NOT NULL DEFAULT 1,
    PRIMARY KEY("id" AUTOINCREMENT)
);

I'd like to get away from writing raw SQL (i.e. migrations)... Most ORMs I think of (SQLAlchemy.py, GORM.go) support Models being code generated. (django.py, for instance, handles migrations for me)

What I'd like to see
#[derive(Debug, WeldsModel)]
#[welds(schema= "inventory", table = "products")]
#[welds(BelongsTo(seller, super::people::People, "seller_id"))]
pub struct Product {
    #[welds(rename = "product_id")]
    #[welds(primary_key)]
    pub id: i32,
    #[welds(unique)]
    /// Unique, Not Null, Text
    pub name: String,
    pub email: Option<String>,
    #[welds(default = 1i32)]
    /// Not Null, Default (This may be impossible, or maybe tie it to a closure?)
    pub login_count: i32,
}

Hi @Beanzilla,
so yes welds does have unique in its migrations. here is an example migration of the create table you have above.
use welds::errors::Result;
use welds::migrations::{create_table, types::Type, MigrationStep, TableState};

pub fn step(_state: &TableState) -> Result<MigrationStep> {
    let m = create_table("users")
        .id(|c| c("id", Type::IntBig))
        .column(|c| c("name", Type::String).create_unique_index())
        .column(|c| c("email", Type::String).is_null())
        .column(|c| c("login_count", Type::Int));
    Ok(MigrationStep::new("m20241202075544_create_table_users", m))
}

I have been taking the approach of trying to make welds modular.  The basic migration functionality is built into the welds library but the CLI tools for it are not. I have two other crates you might find interesting welds-cli and
gumbo
gumbo is a command-line tool focused on web development, but the model and migration tooling is generic. I use it all the time on another project that is not web related.
same migration created with gumbo (plus a couple small tweaks)
gumbo g migration user name:string email:string login_count:int

I am a little reluctant to have welds automatically change your database tables. I have a strong fear of data loss. What do you think of the idea of having it being able to "generate a migration from a Rust Model" or "generate a Rust model from a database table".
I would really like your input here.

so yes welds does have unique in its migrations. here is an example migration of the create table you have above.

I haven't looked at the examples and docs on writing migrations in Rust... but this is exactly what I was looking for. :+1:

use welds::errors::Result;
use welds::migrations::{create_table, types::Type, MigrationStep, TableState};

pub fn step(_state: &TableState) -> Result<MigrationStep> {
    let m = create_table("users")
        .id(|c| c("id", Type::IntBig))
        .column(|c| c("name", Type::String).create_unique_index())
        .column(|c| c("email", Type::String).is_null())
        .column(|c| c("login_count", Type::Int));
    Ok(MigrationStep::new("m20241202075544_create_table_users", m))
}

I had talked to a friend of mine who is also into Rust, they had mentioned to do up a unique index on the field. So I know this is the answer to how to do it. Thank you.

I have been taking the approach of trying to make welds modular. The basic migration functionality is built into the welds library but the CLI tools for it are not. I have two other crates you might find interesting welds-cli and gumbo
gumbo is a command-line tool focused on web development, but the model and migration tooling is generic. I use it all the time on another project that is not web related.
same migration created with gumbo (plus a couple small tweaks)
gumbo g migration user name:string email:string login_count:int

I'll have a look at gumbo, thank you for the extra idea.

I am a little reluctant to have welds automatically change your database tables. I have a strong fear of data loss. What do you think of the idea of having it being able to "generate a migration from a Rust Model" or "generate a Rust model from a database table".
I would really like your input here.

First off, I agree completely, modifying database tables is extremely dangerous for data. (In production environments, changing a database table should be rare)
When I hear both "generate a migration from a Rust Model" and/or "generate a Rust Model from a database table", I think of something like django.py.
Keeping this as brief and using Rust-like terminology... (The way, I think, django handles it's database)
You'd create you're Rust Model struct, ask welds-cli to generate a migration for the Model... subsequent changes would make a migration of the differences. (This I think would be the safest way to change the database tables, also maybe allow a human to step in and make changes if need be)
Obviously the above would require Welds (or welds-cli) to be able to interpret what's already there in the database, previous migrations, and then it could generate or update a existing Rust Model, or vice versa, update the database or generate a migration (or multiple) to bring the database up to the currently existing Rust Model / migration.
An aside, could we maybe add something to build an index. (I think if we could make that an attribute, that would also solve my "how to make something unique", plus make it a little easier to make indexes where you'd like)
A updated Rust Model that could show the adding index attribute...
#[derive(Debug, WeldsModel)]
#[welds(schema="", table="users")]
struct User {
  #[welds(primary_key)
  pub id: i32
  // Either by attribute
  #[welds(unique_index)]
  pub username: String
  pub first_name: String
  #[welds(index)]
  pub last_name: String
  pub email: Option<String>
}
// Or maybe, externally some how?
welds.create_unique_index(User::username);
welds.create_index(User::last_name);

Ah so there is a way to make normal indexes via migration Rust code... Looks like this issue is just requesting index and unique index to be created via attributes to fields in a Rust Model.
My code to make the latter User with the unique index and index
use welds::migrations::{TableState, create_table, types::Type, MigrationStep};
use welds::errors::Result as weldsResult; // I prefer aliasing this so it's clearer which Result,
// I also use anyhow::Result, and I think in some of my code Result (standard Result)
fn setup_User(_state: &TableState) -> weldsResult<MigrationStep> {
  let m = create_table("users")
    .id(|c| c("id", Type::IntBig))
    .column(|c| c("username", Type::String).create_unique_index())
    .column(|c| c("first_name", Type::String))
    .column(|c| c("last_name", Type::String).create_index())
    .column(|c| c(""email, Type::String).is_null())
  Ok(MigrationStep::new("create_users_table", m))
}


"Error when initializing model" - Model shards failed to start

I ran autotrain with the following command:
autotrain llm --train --project_name 'my-llm' --model meta-llama/Llama-2-13b-chat-hf --data_path db-atlas/fire_train --text_column text --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 18 --num_train_epochs 3 --trainer sft --push_to_hub --repo_id db-atlas/fire13b_0825 --merge_adapter
The model finished training way before the estimated runtime, just like explained in issue #212. As suggested in that solution I added --merge_adapter to my command. The model pushed to hub without problems and as far as I can tell it has the same structure and files as previously trained, fully working models.
My problem comes when initializing the model for inference, when the loading fails and returns error messages such as this:
2023-08-25T10:57:30.352453Z ERROR text_generation_launcher: Error when initializing model
2023-08-25T10:57:30.927965Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
RuntimeError: weight model.layers.0.self_attn.rotary_emb.inv_freq does not exist
2023-08-25T10:58:25.627712Z ERROR text_generation_launcher: Shard 0 failed to start

Following those messages I noticed that one of the files generated during training "pytorch_model.bin.index.json", contained in previously trained models mentions to for example:
"model.layers.0.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00003.bin",
This is missing in the models I'm currently trying to train, for all layers. I don't know if I'm missing some step while training, as I followed the same procedure some days back with no errors.

could you please update transformers version and then try again? if you are using TGI, switch to the latest version.

I tried again following your suggestions and got the same error. After that I started the training process from scratch with the following commands:
pip install autotrain-advanced
autotrain setup --update-torch

autotrain llm --train --project_name 'my-llm' --model meta-llama/Llama-2-13b-chat-hf --data_path db-atlas/fire_train --text_column text --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 18 --num_train_epochs 3 --trainer sft --push_to_hub --repo_id db-atlas/fire13b_0825 --merge_adapter

I am using TGI, but I'm already using the latest available version. The full error I'm getting is this:
2023-08-28T02:55:02.607405955-07:00 2023-08-28T09:55:02.607088Z  INFO text_generation_launcher: Args { model_id: " db-atlas/fire13b_0825", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "6474e648826f", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-08-28T02:55:02.608717046-07:00 2023-08-28T09:55:02.608606Z  INFO download: text_generation_launcher: Starting download process.
2023-08-28T02:55:05.568056463-07:00 2023-08-28T09:55:05.567766Z  INFO text_generation_launcher: Peft model detected.
2023-08-28T02:55:05.568091753-07:00
2023-08-28T02:55:05.568095543-07:00 2023-08-28T09:55:05.567835Z  INFO text_generation_launcher: Loading the model it might take a while without feedback
2023-08-28T02:55:05.568099593-07:00
2023-08-28T02:58:00.728586775-07:00 2023-08-28T09:58:00.728336Z  INFO text_generation_launcher: Loaded.
2023-08-28T02:58:00.728627235-07:00
2023-08-28T02:58:00.728629465-07:00 2023-08-28T09:58:00.728389Z  INFO text_generation_launcher: Merging the lora weights.
2023-08-28T02:58:00.728630825-07:00
2023-08-28T02:58:49.316117064-07:00 2023-08-28T09:58:49.315810Z  INFO text_generation_launcher: Saving the newly created merged model to  db-atlas/fire13b_0825
2023-08-28T02:58:49.316183964-07:00
2023-08-28T02:59:06.328275419-07:00 Error: DownloadError
2023-08-28T02:59:06.328369168-07:00 2023-08-28T09:59:06.327995Z ERROR download: text_generation_launcher: Download encountered an error:
Downloading shards:   0%|          | 0/3 [00:00<?, ?it/s]
Downloading shards:  33%|███▎      | 1/3 [00:42<01:25, 42.73s/it]
Downloading shards:  67%|██████▋   | 2/3 [01:25<00:42, 42.73s/it]
Downloading shards: 100%|██████████| 3/3 [01:53<00:00, 35.86s/it]
Downloading shards: 100%|██████████| 3/3 [01:53<00:00, 37.71s/it]
2023-08-28T02:59:06.328394648-07:00
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:04<00:08,  4.30s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:08<00:03,  3.95s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:10<00:00,  3.37s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:10<00:00,  3.56s/it]
2023-08-28T02:59:06.328399708-07:00 Traceback (most recent call last):
2023-08-28T02:59:06.328405038-07:00
2023-08-28T02:59:06.328408438-07:00   File "/opt/conda/bin/text-generation-server", line 8, in <module>
2023-08-28T02:59:06.328412638-07:00     sys.exit(app())
2023-08-28T02:59:06.328415818-07:00
2023-08-28T02:59:06.328419268-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 127, in download_weights
2023-08-28T02:59:06.328422968-07:00     utils.download_and_unload_peft(model_id, revision, trust_remote_code=trust_remote_code)
2023-08-28T02:59:06.328427148-07:00
2023-08-28T02:59:06.328429868-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/peft.py", line 41, in download_and_unload_peft
2023-08-28T02:59:06.328432808-07:00     model.save_pretrained(cache_dir, safe_serialization=True)
2023-08-28T02:59:06.328436118-07:00
2023-08-28T02:59:06.328438928-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1845, in save_pretrained
2023-08-28T02:59:06.328441958-07:00     safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
2023-08-28T02:59:06.328465897-07:00
2023-08-28T02:59:06.328469757-07:00   File "/opt/conda/lib/python3.9/site-packages/safetensors/torch.py", line 282, in save_file
2023-08-28T02:59:06.328473027-07:00     serialize_file(_flatten(tensors), filename, metadata=metadata)
2023-08-28T02:59:06.328476497-07:00
2023-08-28T02:59:06.328479417-07:00 safetensors_rust.SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })
2023-08-28T02:59:06.328483497-07:00
2023-08-28T02:59:06.328486307-07:00
2023-08-28T02:59:16.348667054-07:00 2023-08-28T09:59:16.348402Z  INFO text_generation_launcher: Args { model_id: "db-atlas/fire13b_0825", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "6474e648826f", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-08-28T02:59:16.348752263-07:00 2023-08-28T09:59:16.348639Z  INFO download: text_generation_launcher: Starting download process.
2023-08-28T02:59:18.910636653-07:00 2023-08-28T09:59:18.910409Z  INFO text_generation_launcher: Files are already present on the host. Skipping download.
2023-08-28T02:59:18.910685073-07:00
2023-08-28T02:59:19.307062668-07:00 2023-08-28T09:59:19.306850Z  INFO download: text_generation_launcher: Successfully downloaded weights.
2023-08-28T02:59:19.307473135-07:00 2023-08-28T09:59:19.307306Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2023-08-28T02:59:22.129166389-07:00 2023-08-28T09:59:22.128932Z ERROR text_generation_launcher: Error when initializing model
2023-08-28T02:59:22.129199019-07:00 Traceback (most recent call last):
2023-08-28T02:59:22.129202609-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_llama.py", line 40, in __init__
2023-08-28T02:59:22.129205369-07:00     tokenizer = LlamaTokenizer.from_pretrained(
2023-08-28T02:59:22.129207799-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1825, in from_pretrained
2023-08-28T02:59:22.129210579-07:00     raise EnvironmentError(
2023-08-28T02:59:22.129214229-07:00 OSError: Can't load tokenizer for ' db-atlas/fire13b_0825'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure ' db-atlas/fire13b_0825' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer.
2023-08-28T02:59:22.129218099-07:00
2023-08-28T02:59:22.129220259-07:00 During handling of the above exception, another exception occurred:
2023-08-28T02:59:22.129222679-07:00
2023-08-28T02:59:22.129224749-07:00 Traceback (most recent call last):
2023-08-28T02:59:22.129226869-07:00   File "/opt/conda/bin/text-generation-server", line 8, in <module>
2023-08-28T02:59:22.129229029-07:00     sys.exit(app())
2023-08-28T02:59:22.129231959-07:00   File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 311, in __call__
2023-08-28T02:59:22.129234259-07:00     return get_command(self)(*args, **kwargs)
2023-08-28T02:59:22.129236689-07:00   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
2023-08-28T02:59:22.129238829-07:00     return self.main(*args, **kwargs)
2023-08-28T02:59:22.129240929-07:00   File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 778, in main
2023-08-28T02:59:22.129260209-07:00     return _main(
2023-08-28T02:59:22.129262739-07:00   File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 216, in _main
2023-08-28T02:59:22.129264939-07:00     rv = self.invoke(ctx)
2023-08-28T02:59:22.129267019-07:00   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
2023-08-28T02:59:22.129269069-07:00     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-08-28T02:59:22.129271279-07:00   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
2023-08-28T02:59:22.129273359-07:00     return ctx.invoke(self.callback, **ctx.params)
2023-08-28T02:59:22.129275419-07:00   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 783, in invoke
2023-08-28T02:59:22.129277429-07:00     return __callback(*args, **kwargs)
2023-08-28T02:59:22.129279469-07:00   File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 683, in wrapper
2023-08-28T02:59:22.129281679-07:00     return callback(**use_params)  # type: ignore
2023-08-28T02:59:22.129284819-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 81, in serve
2023-08-28T02:59:22.129286979-07:00     server.serve(
2023-08-28T02:59:22.129289148-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 195, in serve
2023-08-28T02:59:22.129291668-07:00     asyncio.run(
2023-08-28T02:59:22.129293928-07:00   File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
2023-08-28T02:59:22.129296238-07:00     return loop.run_until_complete(main)
2023-08-28T02:59:22.129298378-07:00   File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
2023-08-28T02:59:22.129300508-07:00     self.run_forever()
2023-08-28T02:59:22.129302578-07:00   File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
2023-08-28T02:59:22.129304678-07:00     self._run_once()
2023-08-28T02:59:22.129307198-07:00   File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
2023-08-28T02:59:22.129309278-07:00     handle._run()
2023-08-28T02:59:22.129313538-07:00   File "/opt/conda/lib/python3.9/asyncio/events.py", line 80, in _run
2023-08-28T02:59:22.129315628-07:00     self._context.run(self._callback, *self._args)
2023-08-28T02:59:22.129317728-07:00 > File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 147, in serve_inner
2023-08-28T02:59:22.129320128-07:00     model = get_model(
2023-08-28T02:59:22.129322258-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/__init__.py", line 187, in get_model
2023-08-28T02:59:22.129324388-07:00     return FlashLlama(
2023-08-28T02:59:22.129326478-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_llama.py", line 48, in __init__
2023-08-28T02:59:22.129328528-07:00     tokenizer = AutoTokenizer.from_pretrained(
2023-08-28T02:59:22.129330598-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 720, in from_pretrained
2023-08-28T02:59:22.129332608-07:00     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
2023-08-28T02:59:22.129334638-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1825, in from_pretrained
2023-08-28T02:59:22.129336688-07:00     raise EnvironmentError(
2023-08-28T02:59:22.129338918-07:00 OSError: Can't load tokenizer for 'db-atlas/fire13b_0825'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'db-atlas/fire13b_0825' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
2023-08-28T02:59:22.129341138-07:00
2023-08-28T02:59:22.612630580-07:00 2023-08-28T09:59:22.612369Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
2023-08-28T02:59:22.612667420-07:00
2023-08-28T02:59:22.612671300-07:00 Traceback (most recent call last):
2023-08-28T02:59:22.612674470-07:00
2023-08-28T02:59:22.612676700-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_llama.py", line 40, in __init__
2023-08-28T02:59:22.612692280-07:00     tokenizer = LlamaTokenizer.from_pretrained(
2023-08-28T02:59:22.612694730-07:00
2023-08-28T02:59:22.612697170-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1825, in from_pretrained
2023-08-28T02:59:22.612699500-07:00     raise EnvironmentError(
2023-08-28T02:59:22.612702220-07:00
2023-08-28T02:59:22.612704410-07:00 OSError: Can't load tokenizer for 'db-atlas/fire13b_0825'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'db-atlas/fire13b_0825' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer.
2023-08-28T02:59:22.612707139-07:00
2023-08-28T02:59:22.612709079-07:00
2023-08-28T02:59:22.612711159-07:00 During handling of the above exception, another exception occurred:
2023-08-28T02:59:22.612713569-07:00
2023-08-28T02:59:22.612716339-07:00
2023-08-28T02:59:22.612718689-07:00 Traceback (most recent call last):
2023-08-28T02:59:22.612720869-07:00
2023-08-28T02:59:22.612722829-07:00   File "/opt/conda/bin/text-generation-server", line 8, in <module>
2023-08-28T02:59:22.612724899-07:00     sys.exit(app())
2023-08-28T02:59:22.612727259-07:00
2023-08-28T02:59:22.612729189-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 81, in serve
2023-08-28T02:59:22.612731649-07:00     server.serve(
2023-08-28T02:59:22.612734119-07:00
2023-08-28T02:59:22.612736239-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 195, in serve
2023-08-28T02:59:22.612738999-07:00     asyncio.run(
2023-08-28T02:59:22.612741329-07:00
2023-08-28T02:59:22.612743379-07:00   File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
2023-08-28T02:59:22.612745499-07:00     return loop.run_until_complete(main)
2023-08-28T02:59:22.612747599-07:00
2023-08-28T02:59:22.612749529-07:00   File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2023-08-28T02:59:22.612751729-07:00     return future.result()
2023-08-28T02:59:22.612753979-07:00
2023-08-28T02:59:22.612755939-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 147, in serve_inner
2023-08-28T02:59:22.612761899-07:00     model = get_model(
2023-08-28T02:59:22.612764109-07:00
2023-08-28T02:59:22.612766159-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/__init__.py", line 187, in get_model
2023-08-28T02:59:22.612768429-07:00     return FlashLlama(
2023-08-28T02:59:22.612770439-07:00
2023-08-28T02:59:22.612772559-07:00   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_llama.py", line 48, in __init__
2023-08-28T02:59:22.612774689-07:00     tokenizer = AutoTokenizer.from_pretrained(
2023-08-28T02:59:22.612776809-07:00
2023-08-28T02:59:22.612780019-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 720, in from_pretrained
2023-08-28T02:59:22.612782159-07:00     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
2023-08-28T02:59:22.612784269-07:00
2023-08-28T02:59:22.612786219-07:00   File "/opt/conda/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1825, in from_pretrained
2023-08-28T02:59:22.612788329-07:00     raise EnvironmentError(
2023-08-28T02:59:22.612790449-07:00
2023-08-28T02:59:22.612792539-07:00 OSError: Can't load tokenizer for 'db-atlas/fire13b_0825'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'db-atlas/fire13b_0825' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
2023-08-28T02:59:22.612795009-07:00  rank=0
2023-08-28T02:59:22.710589059-07:00 2023-08-28T09:59:22.710328Z ERROR text_generation_launcher: Shard 0 failed to start
2023-08-28T02:59:22.710616479-07:00 2023-08-28T09:59:22.710368Z  INFO text_generation_launcher: Shutting down shards
2023-08-28T02:59:22.710584179-07:00 Error: ShardCannotStart

whats the tgi command?

I'm actually running the commands in a runpod pod, which is probably where the problem comes from. With previous models, the command worked in version 0.9.4 of TGI.
pod = runpod.create_pod(
    name="Project",
    image_name="ghcr.io/huggingface/text-generation-inference",
    gpu_type_id=gpu_type,
    cloud_type="SECURE",
    docker_args="--model-id db-atlas/fire13b_0825",
    gpu_count=gpu_count,
    volume_in_gb=volume,
    container_disk_in_gb=disk,
    ports=ports,
    volume_mount_path="/data",
)

But now just creating the pod will throw the error about not being able to load the shards.


errors on trying to use the package with v1.1.0

I'm running Julia v1.1.0 on Debian9, and added the repo, and upon trying to 'use' the module have produced errors;
ulia> using EMD
[ Info: Precompiling EMD [3c2051a6-7368-5338-aae3-7c17c25c36d9]
ERROR: LoadError: syntax: space required before "?" operator
Stacktrace:
 [1] include at ./boot.jl:326 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] include(::Module, ::String) at ./sysimg.jl:29
 [4] top-level scope at none:2
 [5] eval at ./boot.jl:328 [inlined]
 [6] eval(::Expr) at ./client.jl:404
 [7] top-level scope at ./none:3
in expression starting at /home/xxx/.julia/packages/EMD/XUPSg/src/EMD.jl:46
ERROR: Failed to precompile EMD [3c2051a6-7368-5338-aae3-7c17c25c36d9] to /home/xxx/.julia/compiled/v1.1/EMD/FUPew.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853

a similarly natured issue arises with Splines

Thanks for the ping -- this package hasn't been touched / updated in several years and it looks like syntax has gotten stricter since then.
@mantzaris, if you want to try to fix this locally you could download the files and modify the lines in these files (and in splines as appropriate) to add a space before each "?".  I would bet that there are other issues that arise as well after this, however.
@bnels, maybe we should try the quick fix, and slap a deprecated sticker on this?  I would make changes but then I have to get code approved before submission...

Apologies for the delay.  Was travelling.
@victorminden I updated the package description and README to indicate that the package is no longer being maintained.
@gmaher If you are interested in updating the repository, you're welcome to
@mantzaris unfortunately I don't have the bandwidth to resurrect the repository right now.  However, you're welcome to make a pull request if you've fixed it up, or fork the repository for your own use.

@bnels and @victorminden thanks for responding to this. Unfortunately I do not have the bandwidth to make this a priority either.

Apologies for the delay. Was travelling.
@victorminden I updated the package description and README to indicate that the package is no longer being maintained.
@gmaher If you are interested in updating the repository, you're welcome to
@mantzaris unfortunately I don't have the bandwidth to resurrect the repository right now. However, you're welcome to make a pull request if you've fixed it up, or fork the repository for your own use.

I will look into it, over the weekend

Thanks, Brad!
On Thu, May 30, 2019, 6:27 PM gmaher<EMAIL_ADDRESS>wrote:

@bnels https://github.com/bnels and @victorminden
https://github.com/victorminden thanks for responding to this.
Unfortunately I do not have the bandwidth to make this a priority either.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bnels/EMD.jl/issues/1?email_source=notifications&email_token=AAN32E2YK4ZXDJYOMX7AXO3PYBIGVA5CNFSM4HOQECCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWTV54A#issuecomment-497508080,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAN32EYF3QH46KMIBUCQNNTPYBIGVANCNFSM4HOQECCA
.


Do not use p-value as a parameter of TopAnat analysis

In GitLab by @jwollbrett on Sep 3, 2018, 09:34
p-value is currently used as a parameter of a TopAnat analysis. It means that it is used to generate the unique key of an analysis.
If a user want to run the same analysis with a different p-value, TopAnat analysis has to be run again (which is time consuming).
In the code the p-value is used to filter the results after running topGO. It could be a better idea to remove the p-value from both the generation of the unique key of an analysis and from filter step of the code. The p-value could be used only to filter data shown in the website.
Then, rerunning a TopAnat analysis with a different p-value will be extremely fast.

In GitLab by @jwollbrett on Sep 3, 2018, 09:35
changed the description


C++/WinRT should document type system to projection mapping table

Tools that build on C++/WinRT may need a stable way to get the list of types that are mapped uniquely by the cppwinrt compiler. This list almost never changes so it may be sufficient to document somewhere on GitHub.

Moved to https://github.com/microsoft/cppwinrt/issues/353


allow deletions of snapshots

Changed title from "allow deletions of snapshots" to more accurately reflect the implementation


"Remember me" feature

Motivation and Context
Related issue: #1302
Description

Added a checkbox on the login page that allows users to select "Remember Me" before logging in
The state whether or not "Remember Me" is checked, is saved in a temporary cookie during the login process, and stored as a jwt claim after successfully logged in
In the middleware, check whether the jwt should be refreshed after validation, if so, extend the validity period in both jwt and the cookie's age
A MaxTokenLifetimeWithRememberMeInDays is set, to prevent the token from being valid forever
A MinUpdateIntervalInHours is set, so that (if "Remember Me" checked upon logging in) the token won't get refreshed too frequently

Steps for Testing

To be able to see the effect quickly, first change MinUpdateIntervalInHours in "\tools\middlewares.go" to a smaller value, e.g. 0.01, which corresponds to 36 seconds
Log in with "Remember Me" selected
Refresh the website within the period of MinUpdateIntervalInHours, see the cookie remain unchanged
Refresh the website after MinUpdateIntervalInHours, see both the value and the Max-Age of the cookie "jwt" is changed, and in the payload of jwt, the value of "exp" corresponds to the Max-Age of the cookie

Screenshots

Change of UI:

A 180-day session lifetime is way to long. I think one to two weeks is perfectly fine for a VoD platform.
We are already storing the cookie, so naming it "Remember Me" is a bit misleading. It is typically used when you want to use a session cookie, and a persistent cookie (with an expiration date).
A few examples:
Spotify has a "Remember Me" toggle sets the expiration date of the token to one month. When not toggled, only a session cookie is stored.
TIDAL does not have a "Remember Me" toggle and stores a sid cookie with a lifetime of 7 days.

A 180-day session lifetime is way to long. I think one to two weeks is perfectly fine for a VoD platform.

Why do you think a 180-day session is too long?
Just now, I inspected my YouTube and Netflix cookie.
Netflix has 6 months cookie expiration dates, while YouTube seems to have 2 years.
Furthermore, I don't think I have ever been logged out of either of those before without changing anything on my end. They probably even do session renewals to extend that period.

Thank you for the comments! As it comes to security, I do think we should be more careful.
But some thoughts on how a "remember me" feature, realised by refreshing the token for logged-in users, would actually improve security: With this feature, we can then shorten the default duration of validity of the token, or, provide an option to shorten it (lecturers and admins might want this behaviour).
And we are using jwt instead of a session cookie, which makes a great difference, since invalidating a jwt is more tricky.


Usage with near-operation-file-preset?

Hello! I was wondering if there's any way to use the plugin with near-operation-file-preset plugin, so that the factories are shaped/co-located with respect to individual .graphql documents.
Thanks!

Hey @leggomuhgreggo 👋 I never used near-operation-file-preset but it looks pretty cool and it shouldn't be too hard to support for it. Just to be sure, you'd like to create factories for operations and have them colocated with the source files?
So the following:
mutation CreateUser($name: String!) {
  createUser(name: $name) {
    id
    name
  }
}

Should lead to:
function createCreateUserMock(props: Partial<CreateUserMutation>): CreateUserMutation {
  return {
    ...props,
    __typename: 'Mutation',
    createUser: {
      __typename: 'User',
      id: '',
      name: ''
    }
  };
}

Is that correct or do you have a different use case?

Golly, thanks for being so generous!
Yes, that's the use case. That way the tests for a given query-connected component, can use the factories with strictly typed component interfaces. I was getting TS errors with the full-type factories.
Let me know if there's anything I can do to help! Thanks for the cool library!

I just merged #18, it adds support for split output which is required to fix this issue. The next step now is to implement support for operations. Hopefully it doesn't mean reimplementing the typescript-operations plugin, we will see.

That makes sense -- interesting to see the diff
(and I salute your attention to documentation!)

I spent some time looking into adding support for operations and it's more complex than I imagined. I will probably pause the development of this feature so I thought I would write down my notes here. In case someone wants to take over or for my future self.

Notes
The challenge here is that there is a lot of variety in how you can write an operation. For example, you can use fragments:
mutation CreateBlogPost($title: String!, $content: String!) {
  createBlogPost(title: $title, content: $content) {
    ...BlogPostFragment
  }
}
fragment BlogPostFragment on BlogPost {
  title
  content
  author {
    ...AuthorFragment
  }
}
fragment AuthorFragment on Author {
  name
  followers {
    ...FollowerFragment
  }
}
fragment FollowerFragment on Follower {
  name
}

In order to generate a createCreateBlogPostMutationMock factory we first need to resolve all the fragments and lookup their types in the schema. graphql-codegen does offer some utilities to help with that but they are mostly targeted at generating types so they are hardly compatible with what this plugin wants to generate.
If we were to go further, here is what is required:

The package must expose two plugins: one for the schema and another one for operations. Just like there is a typescript and a typescript-operations plugin. That's because the former visits the schema and the latter visits documents. If they are merged together, it would generate the factories for the schema as many times as there are documents.
Ideally the operations visitor would be based off BaseDocumentsVisitor but it generates types so we have to go with the lower level BaseVisitor. We probably have to duplicate some code from BaseDocumentsVisitor which integrates helpful utilities.
We should be able to leverage SelectionSetObject to deal with selection sets (e.g to resolve fragments). It seems to be targeted at generating types as well though, so we would need to give it a try. If it is not compatible, that's a deal breaker because this class does a lot and having to duplicate it would make the plugin super fragile.

With that being said, preferably we would contribute to graphql-codegen and see if we can make their utilities less types-oriented.

In the mean time, users can still create their own factories based on the ones generated from the schema. If we take back the example operation from above (see notes), here's what it could look like:
import { createBlogPostMock } from "./generated/factories";

function createCreateBlogPostMutationMock() {
  return {
    createBlogPost: createBlogPostMock({})
  };
}

It's not perfect but seems to be a decent workaround.

Since I'm also investigating this topic let me share my findings.
BaseDocumentsVisitor is the best visitor to start with. Although it's indeed targeted at types, the implementation covers all the needs that we have to generate factories.
I found a way to inject into the generation process to replace the currently serializes the operation into a type, wrote it down as an answer to this question.
We can create a subclass of SelectionSetToObject and make transformSelectionSet use the operation data to generate the factory, not a type. The original SelectionSetToObject implementation includes all the helpers to resolve the selection complexity regarding fragments.
class SelectionSetToFactory<Config extends ParsedDocumentsConfig = ParsedDocumentsConfig> extends SelectionSetToObject<Config> {
  public transformSelectionSet() {

    return 'function (){ }'
  }

  createNext(parentSchemaType: GraphQLNamedType, selectionSet: SelectionSetNode) {
    return new SelectionSetToFactory(this._processor, this._scalars, this._schema, this._convertName.bind(this), this._getFragmentSuffix.bind(this), this._loadedFragments, this._config, parentSchemaType, selectionSet);
  }
}

Thanks for sharing @ertrzyiks 🙏 I went ahead and dug deeper with your insights. I was able to generate something, it's not quite right yet but I am getting there. I had to tweak a few things though.
I chose to extend BaseVisitor instead of BaseDocumentsVisitor because with the latter, we need to override both FragmentDefinition and OperationDefinition methods. The only useful thing left is the handleAnonymousOperation method but it's private so we can't use it in an extended class (it should probably be a protected method instead). Also, this visitor expects a ton of configuration that probably does not make sense for our use case.
Then as you suggested, I had to create a subclass of SelectionSetToObject. I also had to create a subclass of BaseSelectionSetProcessor which is SelectionSetToObject's first parameter. I am now trying to make sense of all the configuration that is expected by SelectionSetToObject but it seems related to types for the most part. So we are probably fine providing default values.
I will open a PR soon 👌

I just published<EMAIL_ADDRESS>with support for operations and the near-operation-file-preset plugin. Feel free to give it a try and open an issue if you find any bug 🙏


Incorrect status code (403 instead 404) for unknown router

Do you want to request a feature or report a bug?

Bug

What did you do?
I try to reach an unknown router
curl -i --insecure https://unknown-router.local

What did you expect to see?
I expect to see a 404 error, as no service as served by Traefik
What did you see instead?
I see a 403 error
HTTP/2 403
accept-ranges: bytes
content-type: text/html
date: Sun, 19 Jul 2020 14:07:05 GMT
etag: "5f0df1ba-396c"
last-modified: Tue, 14 Jul 2020 17:56:10 GMT
server: nginx/1.19.1
content-length: 14700

Other example:
If i execute
curl -i --insecure https://unknown-router.local/john_doe

I have a 404 error, which is the correct answer as router doesn't exist
HTTP/2 404
accept-ranges: bytes
content-type: text/html
date: Sun, 19 Jul 2020 14:06:38 GMT
etag: "5f0df1ba-2cf5"
last-modified: Tue, 14 Jul 2020 17:56:10 GMT
server: nginx/1.19.1
content-length: 11509

Output of traefik version: (What version of Traefik are you using?)
2.2.6

Thanks

Hello,
Traefik never returns a 403 by it-self: he returns only 404 when a route doesn't exist.
Could you provide your configuration (full not partial) or a minimal reproducible example?

Here the configuration files.
Tests
curl -i --insecure https://unknown-router.local/john_doe --> return 404
curl -i --insecure https://unknown-router.local/ --> return 403

docker-compose.yaml
version: "3.8"

services:
  traefik:
    image: traefik:2.2.6
    environment:
      - "TZ=Europe/Paris"
    depends_on:
      - docker-proxy
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
    volumes:
      - ./volumes/traefik:/etc/traefik:ro
      - ./volumes/access-logs:/var/access-logs
      - ./volumes/certs:/certs:ro
      - ./volumes/acme:/acme
      - ./volumes/logs:/var/log
    networks:
      - traefik
      - docker-proxy
    healthcheck:
      test: ["CMD", "traefik", "healthcheck"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s
    deploy:
      resources:
        limits:
          memory: 200m
        reservations:
          memory: 50m
      mode: replicated
      replicas: 1

  docker-proxy:
    environment:
      CONTAINERS: 1
      SWARM: 1
      TASKS: 1
      NODES: 1
      DISTRIBUTION: 1
      SERVICES: 1
      EXEC: 1
      NETWORKS: 1
    image: tecnativa/docker-socket-proxy
    networks:
      - docker-proxy
    deploy:
      mode: replicated
      replicas: 1
      update_config:
        order: start-first
      resources:
        limits:
          memory: 200m
        reservations:
          memory: 50m
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  docker-proxy:
    driver: overlay
  traefik:
    external: true
    name: traefik_public

traefik.yaml
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
  traefik:
    address: ":8080"

ping:
  entryPoint: "traefik"

providers:
  docker:
    exposedbydefault: false
    endpoint: "tcp://docker-proxy:2375"
    network: "traefik_public"
    swarmMode: true
    swarmModeRefreshSeconds: 15
  file:
    directory: "/etc/traefik/"
    watch: true

http:
  routers:
    api:
      rule: Host(`traefik.localhost`)
      service: api@internal
      entryPoints:
        - traefik

  middlewares:
    redirect-to-nonwww:
      redirectRegex:
        regex: "^https://www.(.*)"
        replacement: "https://$1"
        permanent: true

api:
  dashboard: true

➜  issue-7073 curl -i --insecure https://unknown-router.local/
HTTP/2 404
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Mon, 20 Jul 2020 17:38:40 GMT

404 page not found
➜  issue-7073 curl -i --insecure https://unknown-router.local/john_doe
HTTP/2 404
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Mon, 20 Jul 2020 17:38:52 GMT

404 page not found
➜  issue-7073

I was unable to reproduce this issue given your configuration. Perhaps someone in the community could help further track down this issue if you post it on our forums? https://community.containo.us -- please be sure to post any debug logs from Traefik as well.
_kc

➜  issue-7073 curl -i --insecure https://unknown-router.local/

HTTP/2 404

content-type: text/plain; charset=utf-8

x-content-type-options: nosniff

content-length: 19

date: Mon, 20 Jul 2020 17:38:40 GMT

404 page not found

➜  issue-7073 curl -i --insecure https://unknown-router.local/john_doe

HTTP/2 404

content-type: text/plain; charset=utf-8

x-content-type-options: nosniff

content-length: 19

date: Mon, 20 Jul 2020 17:38:52 GMT

404 page not found

➜  issue-7073

I was unable to reproduce this issue given your configuration. Perhaps someone in the community could help further track down this issue if you post it on our forums? https://community.containo.us -- please be sure to post any debug logs from Traefik as well.
_kc

Is your unknown-router.local dns entry is routed to the correct ip ?

Is your unknown-router.local dns entry is routed to the correct ip ?

Yes, it was routed to <IP_ADDRESS>, which is what Traefik was bound to.

Is your unknown-router.local dns entry is routed to the correct ip ?

Yes, it was routed to <IP_ADDRESS>, which is what Traefik was bound to.

Ok. I will give you more details to help you reproduce this issue.
Thank you very much


Using only Cargo files for Rust/Cargo rule

Would it be possible to leverage dynamic dependencies to generate Buck dependencies so Reindeer is no longer needed?

I'm not going to say its impossible, but I haven't been able to figure it out, and I did give it a shot. Because dynamic dependencies are restricted to being within a target, you have to basically have each Rust target become a subtarget of a generic rule - which isn't impossible. You really want to reuse the existing Rust implementation, which means making it an anon_targets to reuse it, again not impossible. But when I started trying I failed, partly because anon_targets wasn't complete enough. Given there are improvements coming in that area, I was going to take another stab at it after.
But I didn't see any fundamental reason not to do it this way, and I think it would be quite elegant. Feel free to give it a go!

@ndmitchell were you sort of looking at this for writing rust rules in general?
For the usecase of reindeer where it's used to manage third-party dependencies, I think it should be fairly easy to get like a dynamic_reindeer() or whatever rule that required you to point it at a Cargo.toml file and to explicitly list out all the names of crates that you want to expose and then have subtargets for all of those crates (and then maybe implemented via dynamic_output and running reindeer at build time, but i haven't looked at reindeer's implementation too much... could also feasibly be implemented by using cargo to build, but idk).

@steveklabnik you mentioned a Discord in https://steveklabnik.com/writing/using-cratesio-with-buck but the invite there seems to have expired

Whoops! Fixed in https://github.com/steveklabnik/steveklabnik.com/commit/ff43c3d0d9f26ab0b3167da68b1a3ceaf8236a9f

anything happening there?

It's been a bit more quiet lately, but I imagine that invite expiring is at least part of it, ha! There's about 90 people, but it's fairly low traffic.

It looks like it tries to do fixups for scenarios that it understands already, would it be possible to have some fallback thing where the build script is executed in a sandboxed environment to get the last mile done?

You can in fact have it run build scripts. Fixups are for skipping that for scenarios where the build script is only used to replicate some feature that's simpler without running the script. For an example, check out the cxx BUCK file's entry for proc-macro2: https://github.com/dtolnay/cxx/blob/master/third-party/BUCK#L213-L241

I tried implementing a simple PoC where i just download archives where each url and  hash is specified in a json file. This is possible with dynamic dependencies but only with actions.download_file. I could not use an anonymous http_archive target (which i suppose is the only way to reuse rule implementations in other rules). Reusing just the inner rule implementation is also not possible because one cannot modify the ctx with values received the dynamic_output lambda.
Is there a special reason why anon_targets cannot be created in a dynamic output lambda and why an anon_target cannot call rules that have dynamic outputs? Is one of both ways already planned to be implemented? Is it even possible?


Rename existing rolefiles so the hooks make sense

Names like app, app_server, web,  and database are not very explanatory.
I propose that we agree upon a new list of steps that the provisioning and deployment process undergo and then change the role names to reflect these steps.

omnibox.yml tackles the following sections, omitting the custom ones:

database, which installs postgres. I propose we rename it to postgres_setup.
general, which sets up the deployer_user, does the apt-get update upgrade, handles swapping, and installs several base packages (Vim, Git, etc.). I think this is too large a set of operations to name properly. Can we do away with the general name and just call its separate parts on their own? I propose calling those, respectively, apt_update, configure_swap_file, and install_essentials. Split it off into new roles.
web, like general, also depends on deployer_user. It has the handler to restart nginx, plus it performs several nginx tasks and does some front-end installs/builds. I think we can break this into two parts. Take web and split it into two new roles, nginx_setup and front_end_setup.
legacy handles rbenv, installing a new one and removing the old one if necessary. Therefore, I think it can be renamed to rbenv_setup.
gems, big surprise, handles bundler and rbenv rehashing. I'd rename it to bundle_environment_setup.
app_server does a ~ton~, largely revolving around Unicorn, dropping/creating/migrating/seeding the DB, waiting for server config files (e.g. secrets.yml), and even installing gems. I think it can be broken into three roles: unicorn_setup (for Unicorn, natch); load_database, and rails_configs_setup. Is it possible to move the bundle install command over to the gems role?
delayed_job, obviously, configures Delayed Job. I think this is named just fine, but I think I want to stick with my theme of appending "_setup" to stuff and propose delayed_job_setup.

In sum, that means we'd move to having the following roles, (roughly) in order of their operation:
postgres_setup, apt_update, configure_swap_file, install_essentials, nginx_setup, front_end_setup, rbenv_setup, bundle_environment_setup, unicorn_setup, load_database, rails_configs_setup_, delayed_job_setup.
I'm also fine with (A) removing all the _setup garbage or (B) Moving it to the front, e.g. set_up_rails_configs or load_database.

Thanks for the explanation on each role.
I think database (postgres) and web (nginx) names are pretty descriptive and inline with "roles" on other frameworks: capistrano used those names. I like how you broke app_server
gems could disappear. zzet.rbenv includes a handler for rbenv rehash and a default-gems file that installs bundler, what other gems do we need?
That said, my changes on https://github.com/smashingboxes/tape/issues/31 are not going to be finished today, so don't wait for me. I can help implementing this once we settle on the names.

 The setup_ prefix was unnecessary.

Addressed in #37.