Stan refactor

Submisison Checklist

[x] Run unit tests: ./runTests.py src/test/unit
[x] Run cpplint: make cpplint
[x] Declare copyright holder and open-source license: see below

Summary
This pull request accomplishes two major things:

In the src/stan/services folder (the stan::services namespace), we now have functions that should be called by the interfaces.
Removes a lot of the old code.

It does this while maintaining all current behavior.
Intended Effect
The intended effect is to make the downstream interfaces easier to implement and maintain.
The code is much cleaner. Each of the functions in src/stan/services excluding the src/stan/services/util folder are meant to be called by RStan, PyStan, and CmdStan. This will also make it much easier to deal with the upcoming template build issue we will have with the introduction of forward-mode and RHMC.
How to Verify
(I'm sorry -- the diff is large.)

All tests should pass. (The upstream tests will fail due to CmdStan develop branch not having changes necessary, but I'll submit a pull request on the CmdStan branch.)
Take a look at all the new functions in src/stan/services. I've covered all the use cases of CmdStan, PyStan, and RStan. See side effects below.
There were a couple of additional utilities outside of src/stan/services that were added. Take a look.

If there are some oddly implemented things in there, most weird implementations are due to keeping the existing behavior intact. This way, we can roll these changes out with a minor release and keep moving forward.
Side Effects
CmdStan, RStan, and PyStan will break under the current branch, but will maintain the same behavior.
I have a branch of CmdStan that works. Allen has a branch of PyStan that's almost working. I have a branch of RStan that's almost working.
Documentation
The C++ code may need some more doc.
If anyone wants to help, there's that and adding more tests.
Reviewer Suggestions
This one's big, so it'd be helpful to have a few eyes on it.
@ariddell, @bgoodri: since this will effect both of you directly.
@sakrejda: since you've been looking at this for a while.
@bob-carpenter, @betanalpha, @rtrangucci: if you've got the time.
Although I'm confident that this behaves identically, this one shouldn't be taken lightly.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

Very exciting.  I'm happy to help out.  Should we
wait until we have working RStan and PyStan versions
before reviewing?  I don't want to commit something to
Stan's develop branch that might break the interfaces beyond
repair.

Bob

On Aug 8, 2016, at 10:05 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
Submisison Checklist
•  Run unit tests: ./runTests.py src/test/unit
•  Run cpplint: make cpplint
•  Declare copyright holder and open-source license: see below
Summary
This pull request accomplishes two major things:

In the src/stan/services folder (the stan::services namespace), we now have functions that should be called by the interfaces.
Removes a lot of the old code.

It does this while maintaining all current behavior.
Intended Effect
The intended effect is to make the downstream interfaces easier to implement and maintain.
The code is much cleaner. Each of the functions in src/stan/services excluding the src/stan/services/util folder are meant to be called by RStan, PyStan, and CmdStan. This will also make it much easier to deal with the upcoming template build issue we will have with the introduction of forward-mode and RHMC.
How to Verify
(I'm sorry -- the diff is large.)
• All tests should pass. (The upstream tests will fail due to CmdStan develop branch not having changes necessary, but I'll submit a pull request on the CmdStan branch.)
• Take a look at all the new functions in src/stan/services. I've covered all the use cases of CmdStan, PyStan, and RStan. See side effects below.
• There were a couple of additional utilities outside of src/stan/services that were added. Take a look.
If there are some oddly implemented things in there, most weird implementations are due to keeping the existing behavior intact. This way, we can roll these changes out with a minor release and keep moving forward.
Side Effects
CmdStan, RStan, and PyStan will break under the current branch, but will maintain the same behavior.
I have a branch of CmdStan that works. Allen has a branch of PyStan that's almost working. I have a branch of RStan that's almost working.
Documentation
The C++ code may need some more doc.
If anyone wants to help, there's that and adding more tests.
Reviewer Suggestions
This one's big, so it'd be helpful to have a few eyes on it.
@ariddell, @bgoodri: since this will effect both of you directly.
@sakrejda: since you've been looking at this for a while.
@bob-carpenter, @betanalpha, @rtrangucci: if you've got the time.
Although I'm confident that this behaves identically, this one shouldn't be taken lightly.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:
• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
You can view, comment on, or merge this pull request online at:
https://github.com/stan-dev/stan/pull/2004
Commit Summary
• Moving src/stan/services to src/stan/old_services
• Adding an empty_var_context
• Implemented a random_var_context
• Adding diagnose service method
• Adding optimize::bfgs service method
• Adding optimize::lbfgs service method
• Adding optimize::newton service method
• Simplifying mcmc_writer
• Adding sample::fixed_param
• Adding a stan::services::util::rng function to create rngs
• Updated service methods to use stan::services::util::rng()
• Updating diagnose to use initialize
• Updating initialize code
• Using stan::services::util::initialize
• Updating mcmc_writer
• Adding services::sample::hmc_nuts_dense_e
• Adding run_adaptive_sampler
• Adding services::sample::hmc_nuts_dense_e_adapt
• Adding sample service methods
• Adding rest of sample serivces
• Adding variational service methods
• cpplint
• Updating tests
• doxygen
• Moving mcmc_writer
• Moving mcmc_writer
• Moving error_codes
• removing progress call from services
• Updating makefile
• Adding an init_writer to all the service calls
• Adding init_writer
• Removing dependence on old_services
• Moving arguments to CmdStan
• Updating performance test to use the service method
• Renaming stan::interface_callbacks to stan::callbacks
• Moving stan::callbacks::interrupt
• Changing initialization on unconstrained scale to not transform roundtrip.
• Moving stan/old_services/variational/print_progress.hpp to stan/variational/print_progress.hpp
• Updating variational services call
• Removing old code
• Updating initialization code to match existing implementation
• cpplint
File Changes
• M make/tests (19)
• A src/stan/callbacks/interrupt.hpp (15)
• A src/stan/callbacks/noop_interrupt.hpp (18)
• R src/stan/callbacks/var_context_factory/dump_factory.hpp (8)
• R src/stan/callbacks/var_context_factory/var_context_factory.hpp (6)
• R src/stan/callbacks/writer/base_writer.hpp (6)
• A src/stan/callbacks/writer/chained_writer.hpp (88)
• R src/stan/callbacks/writer/noop_writer.hpp (8)
• R src/stan/callbacks/writer/stream_writer.hpp (8)
• D src/stan/interface_callbacks/interrupt/base_interrupt.hpp (18)
• D src/stan/interface_callbacks/interrupt/noop.hpp (20)
• A src/stan/io/empty_var_context.hpp (113)
• A src/stan/io/random_var_context.hpp (221)
• M src/stan/mcmc/base_mcmc.hpp (8)
• M src/stan/mcmc/fixed_param_sampler.hpp (6)
• M src/stan/mcmc/hmc/base_hmc.hpp (12)
• M src/stan/mcmc/hmc/hamiltonians/base_hamiltonian.hpp (40)
• M src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp (12)
• M src/stan/mcmc/hmc/hamiltonians/dense_e_point.hpp (4)
• M src/stan/mcmc/hmc/hamiltonians/diag_e_metric.hpp (12)
• M src/stan/mcmc/hmc/hamiltonians/diag_e_point.hpp (4)
• M src/stan/mcmc/hmc/hamiltonians/ps_point.hpp (4)
• M src/stan/mcmc/hmc/hamiltonians/softabs_metric.hpp (28)
• M src/stan/mcmc/hmc/hamiltonians/softabs_point.hpp (4)
• M src/stan/mcmc/hmc/hamiltonians/unit_e_metric.hpp (12)
• M src/stan/mcmc/hmc/hamiltonians/unit_e_point.hpp (2)
• M src/stan/mcmc/hmc/integrators/base_integrator.hpp (6)
• M src/stan/mcmc/hmc/integrators/base_leapfrog.hpp (22)
• M src/stan/mcmc/hmc/integrators/expl_leapfrog.hpp (12)
• M src/stan/mcmc/hmc/integrators/impl_leapfrog.hpp (20)
• M src/stan/mcmc/hmc/nuts/adapt_dense_e_nuts.hpp (6)
• M src/stan/mcmc/hmc/nuts/adapt_diag_e_nuts.hpp (6)
• M src/stan/mcmc/hmc/nuts/adapt_softabs_nuts.hpp (6)
• M src/stan/mcmc/hmc/nuts/adapt_unit_e_nuts.hpp (6)
• M src/stan/mcmc/hmc/nuts/base_nuts.hpp (10)
• M src/stan/mcmc/hmc/nuts_classic/adapt_dense_e_nuts_classic.hpp (6)
• M src/stan/mcmc/hmc/nuts_classic/adapt_diag_e_nuts_classic.hpp (6)
• M src/stan/mcmc/hmc/nuts_classic/adapt_unit_e_nuts_classic.hpp (6)
• M src/stan/mcmc/hmc/nuts_classic/base_nuts_classic.hpp (10)
• M src/stan/mcmc/hmc/static/adapt_dense_e_static_hmc.hpp (6)
• M src/stan/mcmc/hmc/static/adapt_diag_e_static_hmc.hpp (6)
• M src/stan/mcmc/hmc/static/adapt_softabs_static_hmc.hpp (6)
• M src/stan/mcmc/hmc/static/adapt_unit_e_static_hmc.hpp (6)
• M src/stan/mcmc/hmc/static/base_static_hmc.hpp (6)
• M src/stan/mcmc/hmc/static_uniform/adapt_dense_e_static_uniform.hpp (4)
• M src/stan/mcmc/hmc/static_uniform/adapt_diag_e_static_uniform.hpp (4)
• M src/stan/mcmc/hmc/static_uniform/adapt_softabs_static_uniform.hpp (6)
• M src/stan/mcmc/hmc/static_uniform/adapt_unit_e_static_uniform.hpp (6)
• M src/stan/mcmc/hmc/static_uniform/base_static_uniform.hpp (6)
• M src/stan/mcmc/hmc/xhmc/adapt_dense_e_xhmc.hpp (6)
• M src/stan/mcmc/hmc/xhmc/adapt_diag_e_xhmc.hpp (6)
• M src/stan/mcmc/hmc/xhmc/adapt_softabs_xhmc.hpp (6)
• M src/stan/mcmc/hmc/xhmc/adapt_unit_e_xhmc.hpp (6)
• M src/stan/mcmc/hmc/xhmc/base_xhmc.hpp (10)
• M src/stan/mcmc/stepsize_covar_adapter.hpp (4)
• M src/stan/mcmc/stepsize_var_adapter.hpp (4)
• M src/stan/mcmc/windowed_adaptation.hpp (4)
• M src/stan/model/util.hpp (6)
• D src/stan/services/arguments/arg_adapt.hpp (36)
• D src/stan/services/arguments/arg_adapt_delta.hpp (30)
• D src/stan/services/arguments/arg_adapt_engaged.hpp (27)
• D src/stan/services/arguments/arg_adapt_gamma.hpp (30)
• D src/stan/services/arguments/arg_adapt_init_buffer.hpp (25)
• D src/stan/services/arguments/arg_adapt_kappa.hpp (30)
• D src/stan/services/arguments/arg_adapt_t0.hpp (30)
• D src/stan/services/arguments/arg_adapt_term_buffer.hpp (25)
• D src/stan/services/arguments/arg_adapt_window.hpp (24)
• D src/stan/services/arguments/arg_bfgs.hpp (48)
• D src/stan/services/arguments/arg_data.hpp (24)
• D src/stan/services/arguments/arg_data_file.hpp (26)
• D src/stan/services/arguments/arg_dense_e.hpp (21)
• D src/stan/services/arguments/arg_diag_e.hpp (21)
• D src/stan/services/arguments/arg_diagnose.hpp (24)
• D src/stan/services/arguments/arg_diagnostic_file.hpp (26)
• D src/stan/services/arguments/arg_engine.hpp (31)
• D src/stan/services/arguments/arg_fail.hpp (21)
• D src/stan/services/arguments/arg_fixed_param.hpp (21)
• D src/stan/services/arguments/arg_history_size.hpp (30)
• D src/stan/services/arguments/arg_hmc.hpp (30)
• D src/stan/services/arguments/arg_id.hpp (30)
• D src/stan/services/arguments/arg_init.hpp (29)
• D src/stan/services/arguments/arg_init_alpha.hpp (29)
• D src/stan/services/arguments/arg_int_time.hpp (30)
• D src/stan/services/arguments/arg_iter.hpp (30)
• D src/stan/services/arguments/arg_lbfgs.hpp (24)
• D src/stan/services/arguments/arg_max_depth.hpp (30)
• D src/stan/services/arguments/arg_method.hpp (33)
• D src/stan/services/arguments/arg_metric.hpp (31)
• D src/stan/services/arguments/arg_newton.hpp (21)
• D src/stan/services/arguments/arg_num_samples.hpp (30)
• D src/stan/services/arguments/arg_num_warmup.hpp (30)
• D src/stan/services/arguments/arg_nuts.hpp (24)
• D src/stan/services/arguments/arg_optimize.hpp (28)
• D src/stan/services/arguments/arg_optimize_algo.hpp (31)
• D src/stan/services/arguments/arg_output.hpp (28)
• D src/stan/services/arguments/arg_output_file.hpp (26)
• D src/stan/services/arguments/arg_random.hpp (24)
• D src/stan/services/arguments/arg_refresh.hpp (30)
• D src/stan/services/arguments/arg_rwm.hpp (21)
• D src/stan/services/arguments/arg_sample.hpp (34)
• D src/stan/services/arguments/arg_sample_algo.hpp (29)
• D src/stan/services/arguments/arg_save_iterations.hpp (27)
• D src/stan/services/arguments/arg_save_warmup.hpp (28)
• D src/stan/services/arguments/arg_seed.hpp (28)
• D src/stan/services/arguments/arg_softabs.hpp (23)
• D src/stan/services/arguments/arg_softabs_alpha.hpp (29)
• D src/stan/services/arguments/arg_static.hpp (24)
• D src/stan/services/arguments/arg_stepsize.hpp (29)
• D src/stan/services/arguments/arg_stepsize_jitter.hpp (29)
• D src/stan/services/arguments/arg_test.hpp (27)
• D src/stan/services/arguments/arg_test_grad_eps.hpp (29)
• D src/stan/services/arguments/arg_test_grad_err.hpp (29)
• D src/stan/services/arguments/arg_test_gradient.hpp (26)
• D src/stan/services/arguments/arg_thin.hpp (30)
• D src/stan/services/arguments/arg_tolerance.hpp (32)
• D src/stan/services/arguments/arg_unit_e.hpp (21)
• D src/stan/services/arguments/arg_variational.hpp (50)
• D src/stan/services/arguments/arg_variational_adapt.hpp (25)
• D src/stan/services/arguments/arg_variational_adapt_engaged.hpp (26)
• D src/stan/services/arguments/arg_variational_adapt_iter.hpp (30)
• D src/stan/services/arguments/arg_variational_algo.hpp (30)
• D src/stan/services/arguments/arg_variational_eta.hpp (28)
• D src/stan/services/arguments/arg_variational_eval_elbo.hpp (33)
• D src/stan/services/arguments/arg_variational_fullrank.hpp (21)
• D src/stan/services/arguments/arg_variational_iter.hpp (33)
• D src/stan/services/arguments/arg_variational_meanfield.hpp (21)
• D src/stan/services/arguments/arg_variational_num_samples.hpp (33)
• D src/stan/services/arguments/arg_variational_output_samples.hpp (33)
• D src/stan/services/arguments/arg_x_delta.hpp (29)
• D src/stan/services/arguments/arg_xhmc.hpp (25)
• D src/stan/services/arguments/argument.hpp (92)
• D src/stan/services/arguments/argument_parser.hpp (229)
• D src/stan/services/arguments/argument_probe.hpp (31)
• D src/stan/services/arguments/categorical_argument.hpp (157)
• D src/stan/services/arguments/list_argument.hpp (184)
• D src/stan/services/arguments/singleton_argument.hpp (165)
• D src/stan/services/arguments/unvalued_argument.hpp (61)
• D src/stan/services/arguments/valued_argument.hpp (49)
• A src/stan/services/diagnose/diagnose.hpp (78)
• M src/stan/services/error_codes.hpp (4)
• A src/stan/services/experimental/advi/fullrank.hpp (85)
• A src/stan/services/experimental/advi/meanfield.hpp (84)
• D src/stan/services/init/initialize_state.hpp (447)
• D src/stan/services/io/do_print.hpp (51)
• D src/stan/services/io/write_error_msg.hpp (37)
• D src/stan/services/io/write_iteration.hpp (34)
• D src/stan/services/io/write_model.hpp (19)
• D src/stan/services/io/write_stan.hpp (21)
• D src/stan/services/mcmc/sample.hpp (52)
• D src/stan/services/mcmc/warmup.hpp (49)
• A src/stan/services/optimize/bfgs.hpp (201)
• D src/stan/services/optimize/do_bfgs_optimize.hpp (105)
• A src/stan/services/optimize/lbfgs.hpp (204)
• A src/stan/services/optimize/newton.hpp (136)
• A src/stan/services/sample/fixed_param.hpp (74)
• D src/stan/services/sample/generate_transitions.hpp (58)
• A src/stan/services/sample/hmc_nuts_dense_e.hpp (99)
• A src/stan/services/sample/hmc_nuts_dense_e_adapt.hpp (126)
• A src/stan/services/sample/hmc_nuts_diag_e.hpp (72)
• A src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp (92)
• A src/stan/services/sample/hmc_nuts_unit_e.hpp (72)
• A src/stan/services/sample/hmc_nuts_unit_e_adapt.hpp (86)
• A src/stan/services/sample/hmc_static_dense_e.hpp (100)
• A src/stan/services/sample/hmc_static_dense_e_adapt.hpp (125)
• A src/stan/services/sample/hmc_static_diag_e.hpp (72)
• A src/stan/services/sample/hmc_static_diag_e_adapt.hpp (90)
• A src/stan/services/sample/hmc_static_unit_e.hpp (72)
• A src/stan/services/sample/hmc_static_unit_e_adapt.hpp (85)
• D src/stan/services/sample/init_adapt.hpp (69)
• D src/stan/services/sample/init_nuts.hpp (46)
• D src/stan/services/sample/init_softabs.hpp (33)
• D src/stan/services/sample/init_static_hmc.hpp (45)
• D src/stan/services/sample/init_windowed_adapt.hpp (43)
• D src/stan/services/sample/init_xhmc.hpp (47)
• D src/stan/services/sample/progress.hpp (41)
• A src/stan/services/util/experimental_message.hpp (27)
• A src/stan/services/util/generate_transitions.hpp (68)
• A src/stan/services/util/initialize.hpp (147)
• R src/stan/services/util/mcmc_writer.hpp (48)
• A src/stan/services/util/rng.hpp (25)
• A src/stan/services/util/run_adaptive_sampler.hpp (83)
• A src/stan/services/util/run_sampler.hpp (67)
• D src/stan/services/variational/print_progress.hpp (72)
• M src/stan/variational/advi.hpp (50)
• M src/stan/variational/base_family.hpp (4)
• M src/stan/variational/families/normal_fullrank.hpp (2)
• M src/stan/variational/families/normal_meanfield.hpp (4)
• A src/stan/variational/print_progress.hpp (70)
• D src/test/integration/services/arguments/argument_parser_test.cpp (67)
• D src/test/integration/services/arguments/argument_test.cpp (63)
• D src/test/integration/services/arguments/categorical_argument_test.cpp (195)
• D src/test/integration/services/arguments/singleton_argument_test.cpp (161)
• D src/test/integration/services/arguments/unvalued_argument_test.cpp (116)
• D src/test/integration/services/arguments/valued_argument_test.cpp (102)
• M src/test/performance/logistic_test.cpp (20)
• M src/test/performance/utility.hpp (342)
• R src/test/unit/callbacks/var_context_factory/var_context_factory.data.R (0)
• R src/test/unit/callbacks/var_context_factory/var_context_factory_test.cpp (12)
• R src/test/unit/callbacks/writer/noop_writer_test.cpp (4)
• R src/test/unit/callbacks/writer/stream_writer_test.cpp (6)
• A src/test/unit/io/empty_var_context_test.cpp (54)
• A src/test/unit/io/random_var_context_test.cpp (83)
• M src/test/unit/lang/reject/reject_helper.hpp (8)
• M src/test/unit/mcmc/covar_adaptation_test.cpp (4)
• M src/test/unit/mcmc/hmc/base_hmc_test.cpp (8)
• M src/test/unit/mcmc/hmc/hamiltonians/base_hamiltonian_test.cpp (6)
• M src/test/unit/mcmc/hmc/hamiltonians/dense_e_metric_test.cpp (10)
• M src/test/unit/mcmc/hmc/hamiltonians/diag_e_metric_test.cpp (8)
• M src/test/unit/mcmc/hmc/hamiltonians/ps_point_test.cpp (6)
• M src/test/unit/mcmc/hmc/hamiltonians/softabs_metric_test.cpp (12)
• M src/test/unit/mcmc/hmc/hamiltonians/unit_e_metric_test.cpp (8)
• M src/test/unit/mcmc/hmc/integrators/expl_leapfrog2_test.cpp (10)
• M src/test/unit/mcmc/hmc/integrators/expl_leapfrog_test.cpp (6)
• M src/test/unit/mcmc/hmc/integrators/impl_leapfrog2_test.cpp (18)
• M src/test/unit/mcmc/hmc/integrators/impl_leapfrog_test.cpp (6)
• M src/test/unit/mcmc/hmc/mock_hmc.hpp (16)
• M src/test/unit/mcmc/hmc/nuts/base_nuts_test.cpp (34)
• M src/test/unit/mcmc/hmc/nuts/derived_nuts_test.cpp (2)
• M src/test/unit/mcmc/hmc/nuts/softabs_nuts_test.cpp (10)
• M src/test/unit/mcmc/hmc/nuts/unit_e_nuts_test.cpp (10)
• M src/test/unit/mcmc/hmc/nuts_classic/base_nuts_classic_test.cpp (30)
• M src/test/unit/mcmc/hmc/nuts_classic/derived_nuts_classic_test.cpp (2)
• M src/test/unit/mcmc/hmc/static/base_static_hmc_test.cpp (4)
• M src/test/unit/mcmc/hmc/static_uniform/base_static_uniform_test.cpp (4)
• M src/test/unit/mcmc/hmc/static_uniform/derived_static_uniform_test.cpp (36)
• M src/test/unit/mcmc/hmc/xhmc/base_xhmc_test.cpp (34)
• M src/test/unit/mcmc/hmc/xhmc/softabs_xhmc_test.cpp (10)
• M src/test/unit/mcmc/hmc/xhmc/unit_e_xhmc_test.cpp (10)
• M src/test/unit/mcmc/var_adaptation_test.cpp (4)
• M src/test/unit/model/util_test.cpp (6)
• D src/test/unit/services/arguments/arg_optimize_algo_test.cpp (15)
• A src/test/unit/services/diagnose/diagnose_test.cpp (39)
• D src/test/unit/services/init/command_init_test.cpp (90)
• D src/test/unit/services/init/initialize_state_test.cpp (1032)
• D src/test/unit/services/io/do_print_test.cpp (76)
• D src/test/unit/services/io/write_error_msg_test.cpp (23)
• D src/test/unit/services/io/write_iteration_test.cpp (55)
• D src/test/unit/services/io/write_model_test.cpp (16)
• D src/test/unit/services/io/write_stan_test.cpp (18)
• D src/test/unit/services/mcmc/sample_test.cpp (134)
• D src/test/unit/services/mcmc/warmup_test.cpp (134)
• A src/test/unit/services/optimize/bfgs_test.cpp (108)
• D src/test/unit/services/optimize/do_bfgs_optimize_test.cpp (96)
• A src/test/unit/services/optimize/lbfgs_test.cpp (109)
• A src/test/unit/services/optimize/newton_test.cpp (140)
• D src/test/unit/services/sample/generate_transitions_test.cpp (137)
• D src/test/unit/services/sample/init_adapt_test.cpp (3)
• D src/test/unit/services/sample/init_nuts_test.cpp (3)
• D src/test/unit/services/sample/init_static_hmc_test.cpp (3)
• D src/test/unit/services/sample/init_windowed_adapt_test.cpp (3)
• D src/test/unit/services/sample/mcmc_writer_test.cpp (484)
• D src/test/unit/services/sample/progress_test.cpp (45)
• A src/test/unit/services/util/rng_test.cpp (21)
• M src/test/unit/variational/advi_messages_test.cpp (8)
• M src/test/unit/variational/advi_multivar_no_constraint_test.cpp (6)
• M src/test/unit/variational/advi_multivar_with_constraint_test.cpp (6)
• M src/test/unit/variational/advi_univar_no_constraint_test.cpp (6)
• M src/test/unit/variational/advi_univar_with_constraint_test.cpp (6)
• M src/test/unit/variational/eta_adapt_big_test.cpp (4)
• M src/test/unit/variational/eta_adapt_fail_test.cpp (4)
• M src/test/unit/variational/eta_adapt_mock_models_test.cpp (11)
• M src/test/unit/variational/eta_adapt_small_test.cpp (4)
• M src/test/unit/variational/gradient_warn_test.cpp (8)
• M src/test/unit/variational/hier_logistic_cp_test.cpp (4)
• M src/test/unit/variational/hier_logistic_test.cpp (6)
• R src/test/unit/variational/print_progress_test.cpp (18)
• M src/test/unit/variational/stochastic_gradient_ascent_test.cpp (9)
Patch Links:
• https://github.com/stan-dev/stan/pull/2004.patch
• https://github.com/stan-dev/stan/pull/2004.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks for offering!
No, I don't think we need to wait, but I don't feel comfortable merging until we have RStan and PyStan versions ready to go.
I've implemented branches almost all the way through... enough to get samples out and into the Rcpp. I hadn't figured out how to configure the writers correctly, but that should be do-able.

One thing this doesn't is a good way to get default arguments.
Two reasonable options (and I'm sure there are more) include:

Having the default arguments in C++ as part of the function signature. This isn't enforceable, but it can be treated like doc.
Exposing functions for default arguments.

i didn't change all the tests; I'll work on that today.

Just FYI, the Jenkins tests pass; the failed test is from the upstream CmdStan branch trying to build with develop.

There are a couple more (minor) things that I need to do on this branch:

change stan::callbacks::base_writer to stan::callbacks::writer
remove src/stan/callbacks/var_context_factory. If any of the things need to move to RStan or CmdStan, then we should move them. (Or if they need to be in both, then we should leave it, but I don't think it's necessary anymore.)

@syclik I agree that getting the services well-defined in one directory makes this much easier to turn into interfaces, really nice to see this get worked out!  I looked through (all) the diffs and I didn't really find anything problematic.  I did see an opportunity to add doc in a bunch of places and I will have some time later this week to work on tests.  Can you suggest start and end points for the services tests?  Would it make sense to generate code for a specific simple model and test from the services entry points all the way to log-density and gradient calculations or should the end point be earlier?

I think we should have

functions for the defaults
functions to validate

I don't see how to do anything useful in C++ argument structure
because the defaults have to be some final subsequence, not an
arbitrary subset.

On Aug 8, 2016, at 10:32 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
One thing this doesn't is a good way to get default arguments.
Two reasonable options (and I'm sure there are more) include:

Having the default arguments in C++ as part of the function signature. This isn't enforceable, but it can be treated like doc.
Exposing functions for default arguments.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Any suggestions on how to deal with functions to defaults and validation?
One way I could see it happening is to move all the functions in src/stan/services/ to static member functions of a struct with the defaults for the arguments also being member functions within that struct. That seems less than ideal too, but at least we'd have the functions for default be tied to the function it belongs with.
For validation, do you think we should have validation outside of the function itself? Or should we just pass inputs into the function and have that validate inputs? (My fault... I should have taken care of that with this implementation.)

As a specific proposal, how about:
struct stepsize {
  static std::string description();
  static void validate(double x);
  static double default();
};

Maybe a short and long description if we need both.
The class is just a wrapper to keep things together, so this
could also be broken into functions stepsize_description(), etc.
General rule on validation is to always validate arguments to
functions called externally and never validate arguments from
functions called internally (caller's responsibility).
The only way to not have the caller have to specify everything
is with a builder pattern that involves an object per function
(this is a problem with having so many functions), and gets called
as
arg_builder().set_stepsize(0.01).set_int_time(1.0)....build();
Typically that build() call will do all the validation and return
a structure with defaults filled in.  I think this is overkill
with the functions expanded out as you have them, but would've
been critical with functions with complex interactions among
arguments.

I'm perfectly happy using struct. And for validating outside the calls.
One use case, which is possibly hypothetical, is the overloading of variable names and their meanings. We would have difficulty distinguishing the different uses and defaults if we had global defaults.
I just looked and that's currently a hypothetical use case. I think instead of global defaults, we can have the defaults live inside each subfolder / namespace. We can enforce that argument names are overloaded within one of those subfolders / namespaces. E.g. within sampling, there will only be one meaning for alpha.

I can see that we might have stepsize in both optimization
and sampling (or even in EHMC and RHMC) with different defaults.
I absolutely do not want to use includes to sort this out.
We can use namespaces or clutter up the names for defaults
with suffixes.
I was thinking struct per argument, but maybe it's better with
a struct per "route".  But there's so damn many of them that
it's offputting.  We could organize with inheritance, thereby
rebuilding the stuff Michael had in CmdStan into a C++
class hierarchy.  At some point, we'll just be back to having
the builder pattern.

Bob

On Aug 10, 2016, at 4:43 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
I'm perfectly happy using struct. And for validating outside the calls.
One use case, which is possibly hypothetical, is the overloading of variable names and their meanings. We would have difficulty distinguishing the different uses and defaults if we had global defaults.
I just looked and that's currently a hypothetical use case. I think instead of global defaults, we can have the defaults live inside each subfolder / namespace. We can enforce that argument names are overloaded within one of those subfolders / namespaces. E.g. within sampling, there will only be one meaning for alpha.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Yup. No need for include magic. I'm suggesting namespaces. I'll put together some code on the branch; struct per argument within a namespace.

On Aug 10, 2016, at 11:29 AM, Bob Carpenter<EMAIL_ADDRESS>wrote:
I can see that we might have stepsize in both optimization
and sampling (or even in EHMC and RHMC) with different defaults.
I absolutely do not want to use includes to sort this out.
We can use namespaces or clutter up the names for defaults
with suffixes.
I was thinking struct per argument, but maybe it's better with
a struct per "route". But there's so damn many of them that
it's offputting. We could organize with inheritance, thereby
rebuilding the stuff Michael had in CmdStan into a C++
class hierarchy. At some point, we'll just be back to having
the builder pattern.

Bob

On Aug 10, 2016, at 4:43 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
I'm perfectly happy using struct. And for validating outside the calls.
One use case, which is possibly hypothetical, is the overloading of variable names and their meanings. We would have difficulty distinguishing the different uses and defaults if we had global defaults.
I just looked and that's currently a hypothetical use case. I think instead of global defaults, we can have the defaults live inside each subfolder / namespace. We can enforce that argument names are overloaded within one of those subfolders / namespaces. E.g. within sampling, there will only be one meaning for alpha.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

+1 to struct per route.  When I was trying to organize the arguments in the wiki it became clear just how different the meanings of the names are dependent on route (especially when people start calling things "alpha" rather than descriptive names like "step_size").  Maybe we should require descriptive names at this level.

I still don't understand.  Are you suggesting a namespace per route
then overloading all the structures?  Because we'll only guarantee
that a name has a constant meaning on a per-route level.
Could you sketch one out before coding dozens of them so I can
understand what you're proposing?

Bob

P.S.  Can anyone think of a better term than "route"?

Yes, adapt_delta absolutely has to be replaced with something
more meaningful.  I think that means supporting the old name
and a new name until Stan 3 arrives.

Bob

On Aug 10, 2016, at 6:20 PM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:
+1 to struct per route. When I was trying to organize the arguments in the wiki it became clear just how different the meanings of the names are dependent on route (especially when people start calling things "alpha" rather than descriptive names like "step_size"). Maybe we should require descriptive names at this level.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Or just waiting for Stan 3 to make these changes.  That's
probably the better plan.

@syclik I tested out pushing to the branch with just one of the files I added but since it was originally cloned from my fork of this branch a second commit appeared that I didn't see.  Sorry if I messed something up... :/  It just looks like a merge of the changes you already made but I'm not sure why it's showing up at all....

That's fine. No problem. Doesn't look messed up. We tend not to worry at the commit level. (Even though we've been advised that we should.) just way too much effort.

On Aug 12, 2016, at 12:14 PM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:
@syclik I tested out pushing to the branch with just one of the files I added but since it was originally cloned from my fork of this branch a second commit appeared that I didn't see. Sorry if I messed something up... :/ It just looks like a merge of the changes you already made but I'm not sure why it's showing up at all....
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

ok, think I've got it figured out now :+1:

unit test failed from 1 new warning, I'll fix that with the next tests.

@syclik As I'm writing tests.... I think "init" is a weird argument name for the context on the C++ side.  Could just call it "context" or spell out "initial_values" or at least to match things like "num_warmup" call it "init_values".... except that it's not initial values, it's actually the data, so maybe just "data"...  pretty much anything other than "init", what do you think?

I'm fine with renaming it, but the type is a var_context and it's for initialization. We could call it init_context, but I'd prefer init since the context part is obvious from the type.
It's definitely not "the data." That's already part of the instantiated model.
Any other suggestions?

On Aug 12, 2016, at 6:27 PM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:
@syclik As I'm writing tests.... I think "init" is a weird argument name for the context on the C++ side. Could just call it "context" or spell out "initial_values" or at least to match things like "num_warmup" call it "init_values".... except that it's not initial values, it's actually the data, so maybe just "data"... pretty much anything other than "init", what do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Never mind then, init is a fine name but the model in the tests is just instantiated as stan_model model;... so I assumed "init" was for data.  Is there  a test somewhere that instantiates the model with data so I can figure out what the types are here?

The sense that some of the parameter names might be improved is one that
I share. I'll trust @syclik's judgment on whether or not such a feeling
is warranted.
On 08/12/2016 06:51 PM, Krzysztof Sakrejda wrote:

Never mind then, init is a fine name but the model in the tests is just
instantiated as |stan_model model;|... so I assumed "init" was for data.
Is there a test somewhere that instantiates the model with data so I can
figure out what the types are here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/pull/2004#issuecomment-239578609, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AABN7slHGebIFgdJvw5AYn94wZ2awRJaks5qfPkIgaJpZM4Jfpda.

I accept it with open arms!
But I think the init var_context was misinterpreted as being data when it is indeed initialization. That said, I support it.

On Aug 12, 2016, at 7:57 PM, Allen Riddell<EMAIL_ADDRESS>wrote:
The sense that some of the parameter names might be improved is one that
I share. I'll trust @syclik's judgment on whether or not such a feeling
is warranted.
On 08/12/2016 06:51 PM, Krzysztof Sakrejda wrote:

Never mind then, init is a fine name but the model in the tests is just
instantiated as |stan_model model;|... so I assumed "init" was for data.
Is there a test somewhere that instantiates the model with data so I can
figure out what the types are here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/pull/2004#issuecomment-239578609, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AABN7slHGebIFgdJvw5AYn94wZ2awRJaks5qfPkIgaJpZM4Jfpda.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

What I really need to do is make a flowchart for how a Stan model is put together! I think I get all the pieces now.

The main motivation to work at the commit level is to support
git bisect.  Given that we're much more careful on testing now,
it hasn't been such a big deal not to have every commit working.
But we like to have commits at least build, even if they don't pass
all the unit tests.  I've always been afraid to rebase because
I tend to push my commits to the origin.

Allen --- do you have suggestions for new names?  If so, now's
the time to bring them up!

And what the project really needs is doc.  The best we have
is this:
https://github.com/stan-dev/stan/wiki/Model-Concept
I think we want to move to where there's a base class and we
pass models by ordinary type rather than template parameter.
That makes everything easier to understand because then there's
an object-level spec in C++ as opposed to meta or comment level doc like
on a C++ "concept".

I've just pushed changes that take away the template parameter for the interrupt callback. I've also updated the finite diff code in diagnose mode so it takes an interrupt. (This will help with RStan; right now, there's no way to stop RStan when running finite diffs.)
Left on my plate is a prototype for defaults.

@sakrejda, if you want to see how this is all done in CmdStan, see feature/issue-287-stan-services. In particular, src/cmdstan/command.hpp. Lines 53 -- 118 set up most of the things needed to run any command.

@bob-carpenter I'll give the names some thought  I'll also double check for places where doc changes/additions would be helpful.. I'm more comfortable with the doc and var names for the writers than var_context but that could just be me.

@syclik Thanks---I had gotten confused with the fact that you need a var_context with data first for the model, and then another var_context to initialize sampling but now it's really clear.
@bob-carpenter now that I understand I think the "model concept" wiki is pretty good. The problem is that you totally loose perspective on what was hard once you get it.

@bob-carpenter: how about this for defaults?
https://github.com/stan-dev/stan/pull/2004/commits/f614952e2e0e615d8048fe80eda0d873edccdb3f
We can have one defaults.hpp per folder / namespace. I couldn't use the function name default() cause it's a keyword, so I made it more verbose. I'm happy to change it.

Exactly what I was imagining.
But don't you want to use our validation functions
rather than throwing exceptions directly?

Bob

On Aug 16, 2016, at 4:30 AM, Daniel Lee<EMAIL_ADDRESS>wrote:

@bob-carpenter: how about this for defaults?
f614952
We can have one defaults.hpp per folder / namespace. I couldn't use the function name default() cause it's a keyword, so I made it more verbose. I'm happy to change it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I do, but they aren't templated for exception type. I'm adding is_* checks on a branch. Once those are in, I'll update these.
I'll put together the rest of these. One per namespace.

On Aug 16, 2016, at 11:17 AM, Bob Carpenter<EMAIL_ADDRESS>wrote:
Exactly what I was imagining.
But don't you want to use our validation functions
rather than throwing exceptions directly?

Bob

On Aug 16, 2016, at 4:30 AM, Daniel Lee<EMAIL_ADDRESS>wrote:

@bob-carpenter: how about this for defaults?
f614952
We can have one defaults.hpp per folder / namespace. I couldn't use the function name default() cause it's a keyword, so I made it more verbose. I'm happy to change it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Oh, you want "std::invalid_argument"?  I suppose that makes
sense.
I don't think you need different namespaces for all these,
do you?  I'd go lighter on this and just put everything in
services.  Eigen puts almost everything in Eigen:: and Boost
is very light on namespaces.  Would there be conflicts otherwise?

Bob

On Aug 16, 2016, at 6:21 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
I do, but they aren't templated for exception type. I'm adding is_* checks on a branch. Once those are in, I'll update these.
I'll put together the rest of these. One per namespace.

On Aug 16, 2016, at 11:17 AM, Bob Carpenter<EMAIL_ADDRESS>wrote:
Exactly what I was imagining.
But don't you want to use our validation functions
rather than throwing exceptions directly?

Bob

On Aug 16, 2016, at 4:30 AM, Daniel Lee<EMAIL_ADDRESS>wrote:

@bob-carpenter: how about this for defaults?
f614952
We can have one defaults.hpp per folder / namespace. I couldn't use the function name default() cause it's a keyword, so I made it more verbose. I'm happy to change it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@syclik How about this for classes for testing writer/interrupt use (737520b27b9102ffd7a20317833853d8dc03b257). Classes are in src/test/unit/instrumented_callbacks.hpp and there's a usage example in src/test/unit/services/util/generate_transitions_test.cpp

there was a still one more missing include, fixed it and util.hpp now passes locally without warnings.  Hopefully that was it.

I added all the defaults with tests. They're in:

src/stan/services/diagnose/defaults.hpp
src/stan/services/experimental/advi/defaults.hpp
src/stan/services/optimize/defaults.hpp
src/stan/services/sample/defaults.hpp

@sakrejda, I'll need to look at your tests. I think once the tests are there, Stan will be code complete and we'll only need to worry about the interfaces.

These are the files that still don't have tests. We'll need to knock them off one by one. @sakrejda: do you have any tests written for any of this? If so, could you push soon?

[ ] src/stan/callbacks/chained_writer.hpp
[ ] src/stan/callbacks/interrupt.hpp
[ ] src/stan/callbacks/noop_interrupt.hpp
[ ] src/stan/services/experimental/advi/fullrank.hpp
[ ] src/stan/services/experimental/advi/meanfield.hpp
[ ] src/stan/services/sample/hmc_nuts_dense_e.hpp
[ ] src/stan/services/sample/hmc_nuts_dense_e_adapt.hpp
[ ] src/stan/services/sample/hmc_nuts_diag_e.hpp
[ ] src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp
[ ] src/stan/services/sample/hmc_nuts_unit_e.hpp
[ ] src/stan/services/sample/hmc_nuts_unit_e_adapt.hpp
[ ] src/stan/services/sample/hmc_static_dense_e.hpp
[ ] src/stan/services/sample/hmc_static_dense_e_adapt.hpp
[ ] src/stan/services/sample/hmc_static_diag_e.hpp
[ ] src/stan/services/sample/hmc_static_diag_e_adapt.hpp
[ ] src/stan/services/sample/hmc_static_unit_e.hpp
[ ] src/stan/services/sample/hmc_static_unit_e_adapt.hpp
[ ] src/stan/services/util/experimental_message.hpp
[ ] src/stan/services/util/generate_transitions.hpp
[ ] src/stan/services/util/initialize.hpp
[ ] src/stan/services/util/mcmc_writer.hpp
[ ] src/stan/services/util/rng.hpp
[ ] src/stan/services/util/run_adaptive_sampler.hpp
[ ] src/stan/services/util/run_sampler.hpp

@syclik I wrote classes I wanted to use for testing but they seem like the sort of complicated thing you guys sometimes dislike so I asked for your opinion (above) before I wrote a bunch of test code.  I also got busy with other stuff so it was kind of convenient that you didn't have time! Could you take a look at the classes in src/test/unit/instrumented_callbacks.hpp and the usage example in src/test/unit/services/util/generate_transitions_test.cpp?  I know they're not pretty but I think using these will make the test code shorter and easier to write.

Ok. Will do today.

On Sep 27, 2016, at 7:59 AM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:
@syclik I wrote classes I wanted to use for testing but they seem like the sort of complicated thing you guys sometimes dislike so I asked for your opinion (above) before I wrote a bunch of test code.  I also got busy with other stuff so it was kind of convenient that you didn't have time! Could you take a look at the classes in src/test/unit/instrumented_callbacks.hpp and the usage example in src/test/unit/services/util/generate_transitions_test.cpp? I know they're not pretty but I think using these will make the test code shorter and easier to write.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks, I'd rather get your opinion than push something you'll see as a nuissance in the future.

@sakrejda, looks fine to me. I would move it down to the test/unit/services folder since that's the only place it'll be used for now. We can leave it in the stan::test namespace. We don't need to be super-careful about the test namespace.
It definitely beats reimplementing that time and time again.
I don't see tests as a nuisance. I don't see most things as a nuisance if it has function now. I'll always annoyed by my past self, though. Never enough tests.

Ok, thanks! I have a couple of tests I need to update to use the class, I'll fix them up and push.

Thanks.

On Sep 28, 2016, at 9:17 AM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:
Ok, thanks! I have a couple of tests I need to update to use the class, I'll fix them up and push.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@syclik That's all the time I had today, I'll pick up again another day and check things off the list when I do.  hmc_nuts_dense_e_adapt.hpp has at least some tests implemented (counting that things get called, that the right columns show up in output, etc..) but I don't know when it's supposed to throw or other more detailed behavior.  I don't have snapshot output to do the exact regression tests against either so there's only a placeholder for that.

@sakrejda, I added the remaining tests! If you want to add any more doc, please do. Or we can at least mark what needs doc. For now, I think this is pretty code complete and needs a thorough review.
CmdStan is working with this branch.
I'm working on RStan and PyStan.

@syclik Congrats on getting it done! I'll take another look at code/doc this weekend.

I've marked this for v2.13.1++. This needs to get in sooner rather than later.
@bob-carpenter, mind reviewing?
@bgoodri, @jgabry: can you verify that the branch I put up for RStan works?
@ariddell: we'll talk in person next week, but let's make sure it works with PyStan too.
I'm confirming this works with CmdStan. I'll run tests with the right upstream branch and the tests will all be good.

I'll review.  I just need to find an old-fashioned turntable with a record changer to queue up enough albums.  And it looks like they made their diffs more robust by letting me pull them up file by file.  Yay!

Jenkins, retest this please.

Jenkins, retest this please.

Bump. @bob-carpenter, @thelseraphim, and anyone else: can you start reviewing this code?
We have branches working for RStan, PyStan, and CmdStan. Maybe not 100% yet, but definitely showing that it works.

NONONONONONONONONONONONONONONONO
to using RStan’s convention for num_samples as all
iterations and num_warmup as some subset.  It’s
confusing, awkward, and in general the worst.  I
wrote the sampling code to explicitly avoid this
(and CmdStan doesn’t use it).
On Dec 13, 2016, at 1:04 PM, Krzysztof Sakrejda<EMAIL_ADDRESS>wrote:

@sakrejda commented on this pull request.
In src/stan/services/sample/defaults.hpp:

    * @returns description

    */

   static std::string description() {

     return "Number of warmup iterations.";

   }

   /**

    * Validates num_warmup; num_warmup must be greater than or

    * equal to 0.

    *

    * @param[in] num_warmup argument to validate

    * @throw std::invalid argument if num_warmup < 0

    */

   static void validate(int num_warmup) {

     if (!(num_warmup >= 0))

       throw std::invalid_argument("num_warmup must be greater "

It also must be less than 'num_samples'.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@betanalpha that too but whatever you choose it needs to be clear in the doc because at the moment I do something stupid whenever I switch from CmdStan to rstan.

NOOOOO!  There is one sample consisting of a bunch of
draws.  The warmup iterations aren't draws from the posterior
and hence aren't kept as part of the sample.
Let's keep our language crisp at least in the doc and descriptions.
There should not be a "num_samples" parameter anywhere!

Yes, what is going on should be clear in the doc.
We don't want to change names in the interfaces for 2.x.  But
it'd be great to make everything consistent across interfaces for 3.0.
In the future, we don't want to conflate a draw with a sample or
indicate that what we get in the posterior iterations are independent
draws.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

We’re going to have to sit down and talk through this,
but from the math perspective “sample” does not mean
independent.  “Correlated samples” are exactly what
a Markov chain generates.  There are rigorous definitions
for all of these terms.
On Dec 13, 2016, at 2:16 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

Yes, what is going on should be clear in the doc.
We don't want to change names in the interfaces for 2.x. But
it'd be great to make everything consistent across interfaces for 3.0.
In the future, we don't want to conflate a draw with a sample or
indicate that what we get in the posterior iterations are independent
draws.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi, now I'm confused.  rstan uses "iter," not "num_samples," for number of iterations.

On Dec 13, 2016, at 1:23 PM, Michael Betancourt<EMAIL_ADDRESS>wrote:
NONONONONONONONONONONONONONONONO
to using RStan’s convention for num_samples as all
iterations and num_warmup as some subset. It’s
confusing, awkward, and in general the worst. I
wrote the sampling code to explicitly avoid this
(and CmdStan doesn’t use it).

@andrewgelman rstan uses the names iter and warmup, but I think the convention Michael was referring to doesn't have to do with the names but rather the fact that iter is used for the total number rather than total = iter + warmup.

Hi all, let's not derail this pull request due to naming disagreements. There is now a discussion on discourse, so please refrain from having that discussion on this pull request.
This pull request is a refactor. I've painstakingly maintained behavior from the develop branch. Even though I wanted to fix a lot of things along the way, I did not. This is not the right place for changes to old behavior. (I realize it's hard to tell what's what; if it's come along due to old behavior, I'll let you know and you can create a new issue for it.)
Please, let's keep this moving. The 4 things it needs:

code approval on Stan. This is what I'm asking people to do now.
confirmation that CmdStan works. I'm confirming.
confirmation that RStan works. I'm waiting for @bgoodri to confirm.
confirmation that PyStan works. I'm waiting for @ariddell to confirm.

I really need those looking at the C++ to approve the Stan code independently of the interfaces.

@syclik So, a) this is amazing; b) I did look over all the code and most of the changes to tests.  I don't see any specific problems or cut-and-paste problems, having the interface doc consolidated to these interface functions makes it easier to understand, I read through the logic of the interface functions last time and we wrote tests for most of it so I didn't go through it again in detail.  Last time the argument defaults/doc static functions weren't there but I think they're good too--it's clear and makes all that stuff programatically accessible.... I checked that doc matched functions and that the defaults were reasonable for most of them.  Are there specific other things I should look at? I'm sure (hope!) somebody else looks over this as well but I think it's good to go.

OK. I'm on notice for checking that everything works. I'm going to push
this off until after the PyStan release if that's alright. (There are a
variety of new tests to add.)
On 12/13/2016 09:21 PM, Daniel Lee wrote:

Hi all, let's not derail this pull request due to naming disagreements.
There is now a discussion on discourse, so please refrain from having
that discussion on this pull request.
This pull request is a refactor. I've painstakingly maintained behavior
from the |develop| branch. Even though I wanted to fix a lot of things
along the way, I did not. This is not the right place for changes to old
behavior. (I realize it's hard to tell what's what; if it's come along
due to old behavior, I'll let you know and you can create a new issue
for it.)
Please, let's keep this moving. The 4 things it needs:

code approval on Stan. This is what I'm asking people to do now.
confirmation that CmdStan works. I'm confirming.
confirmation that RStan works. I'm waiting for @bgoodri
https://github.com/bgoodri to confirm.
confirmation that PyStan works. I'm waiting for @ariddell
https://github.com/ariddell to confirm.

I really need those looking at the C++ to approve the Stan code
independently of the interfaces.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/pull/2004#issuecomment-266923962, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AABN7n7HFvVvBgLte3_EvOycS8Add3RDks5rH1KugaJpZM4Jfpda.

Thanks, @bob-carpenter! I took a quick read through the comments. I'll try to address as many as I can today.
A couple of bigger points:

naming. We should definitely fix that now. I'm happy going with no_op instead of noop. But I think I'd rather have a base implementation that acts as a no_op. Thoughts?
I'll try to generalize your comments to other places in the code.
Places where I can't change, I'll leave comments in place.

On Dec 27, 2016, at 9:51 AM, Daniel Lee<EMAIL_ADDRESS>wrote:
You're absolutely right. Any suggestions for names? tee_writer or teed_writer? forked_writer?

I like tee_writer from the unix:
See:  https://en.wikipedia.org/wiki/Tee_(command)

The doc is inherited in doxygen.

Fantastic.  Then if the doc doesn't change in the subclass, I'd rather
just inherit.  I don't think you need to write "teed version of ..." in
front of all those.  Maybe with all the time and editorial staff in the
world, but not worth it for us.

The reason for the pointer was to keep in line with the old pull request. I really think IO should be handled better, so I'm going to scale back the writers to what we use now:
• writer()
• writer(std::string)
• writer(std::vectorstd::string)
• writer(std::vector)
It'll cut down on our overhead for now. I think it's the right way to go and we won't have pointers floating around.

Nice!

+1 and thanks. Given that doxygen inherits doc from superclasses, should we just doc the superclass and not the subclass if it's trivial?

Yes, see above.

Absolutely. I'll add virtual destructors.

Thanks.  That'll cover us if anyone creates a subclass
that frees heap memory in the destructor.

yeah... I tried to make add the const keyword at one point; it doesn't work. We're not there yet. But this is making it easier.

I know.  Can't wait.  I've felt bad about not getting
const right before and I spent days in the past on
more than one occasion trying to fix it.  I think I could
do it now.  Skills training from compiling C++ night
and day for 5 years!

I completely agree that we need sampler-state specific writers rather than overloading a generic writer. We'll get there, but after this pull request.

Understood.

In src/stan/mcmc/hmc/base_hmc.hpp:

@@ -34,7 +34,7 @@ namespace stan {
epsilon_jitter_(0.0) {}

Yes, but it shouldn't hurt too much here. I think git will handle the merge fine.

OK.  And I think we should still support it, just present a
deprecation warning until Stan 3.

Bob

On Dec 27, 2016, at 9:19 AM, Daniel Lee<EMAIL_ADDRESS>wrote:
Thanks, @bob-carpenter! I took a quick read through the comments. I'll try to address as many as I can today.
A couple of bigger points:
• naming. We should definitely fix that now. I'm happy going with no_op instead of noop. But I think I'd rather have a base implementation that acts as a no_op. Thoughts?

I commented at length on that in one of the comments.  Short version:
absolutely if you can delegate calls to one method to another.  Makes
it easier and less error prone to implement subclasses.

• I'll try to generalize your comments to other places in the code.

Thanks.

• Places where I can't change, I'll leave comments in place.

Sounds good.

Bob

On Dec 28, 2016, at 9:30 AM, Daniel Lee<EMAIL_ADDRESS>wrote:
@syclik commented on this pull request.
In src/stan/io/empty_var_context.hpp:

+#include

+namespace stan {

namespace io {

/**

* This is an implementation of a var_context that doesn't contain

* any variables.

*/

class empty_var_context : public var_context {
public:

 /**

  * Destructor

  *

  */

 ~empty_var_context() {}

I'll make sure the base class, var_context, has a virtual destructor. As far as I can tell, the subclass's destructor doesn't have to be virtual. I don't think there will be a measurable hit if there's a performance difference: we only create this once in CmdStan for the lifetime of the program.

The subclass's destructor will be virtual if the base class's
is declared as virtual.
Performance hit or no, it's necessary for a properly formed
base class if you want to extend it generically.  Otherwise,
the destructor and class should be carefully documented to
ensure nobody extends it with a class that manages its own
memory.

I just pushed a lot of changes. I need to walk through the src/stan/services folder for doc next. This is going to affect RStan, PyStan, and CmdStan, so the upstream tests won't pass, but I'll take care of that next.

On Dec 28, 2016, at 11:21 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
@syclik commented on this pull request.
In src/stan/services/optimize/bfgs.hpp:

     model.write_array(rng, cont_vector, disc_vector, values,

                       true, true, &msg);

     if (msg.str().length() > 0)

       message_writer(msg.str());

     values.insert(values.begin(), lp);

     parameter_writer(values);

   }

   int ret = 0;

   while (ret == 0) {

     interrupt();

     if (refresh > 0

         && (bfgs.iter_num() == 0

             || (bfgs.iter_num() + 1 % refresh == 0)))

       message_writer("    Iter "

No. Of course not. But I'm trying hard not to fix everything that's wrong with our whole code base or we'll never get this in. This has been in since it was written -- it bothers me too, but I'm really trying to fix the things that need to be fixed. We can walk through the code base later.

Isn't the behavior going to be wrong here given that % binds
more tightly than +.  If so, could you please create an issue
for this?  Thanks.  I can't actually understand the logic or I'd
do it myself.

This one hurts. It's in ADVI and it's propagated through to CmdStan. I don't think all our developers are on board with the terminology.

Michael's OK with it, and I think we can just enforce it.
But I'm OK taking a pass through this later.  We can't actually
remove the existing naming until Stan 3, but we could replace
it and deprecate the old behavior.
Sorry if the comments are overwhelming.  I'm just noting
everything that pops out at me reviewing the code.

I don't think it's hopeless, but I think fixing this should come with fixing all of ADVI. I'm going to create a new issue for all the things you've written up.

Thanks!

      * @param[in] tol_rel_obj argument to validate

We haven't been too consistent with this. I can either add them everywhere or remove them. Or just let it be haphazard until we know what we want.

OK.  It's very inconsistent in the code base.  I often
forget because it wasn't in Javadoc when I used it (or I
didn't know about it).
As to what we want, we can have the [in] everywhere if
you think it's helpful to appear more consistent.  It's
exactly like adding const to a primitive---it doesn't do
anything to the external API because C++ pass by value.
That is, it has to be [in] by construction if it's a primitive
passed by value, the same way it's const by construction.

In src/stan/services/experimental/advi/fullrank.hpp:

     stan::variational::advi<Model,

                             stan::variational::normal_fullrank,

                             boost::ecuyer1988>

       cmd_advi(model,

                cont_params,

                rng,

                grad_samples,

                elbo_samples,

                eval_elbo,

                output_samples);

     cmd_advi.run(eta, adapt_engaged, adapt_iterations,

                  tol_rel_obj, max_iterations,

                  message_writer, parameter_writer, diagnostic_writer);

     return 0;

It can throw. ADVI is a mess, so I can't really track through all the different types of things it can or can't do, so I'm going to stick it on the ADVI issue I'm creating.

Thanks.  I'm not too worried about ADVI until it's a bit more
robust and we can recommend it to users in some situations.

Jenkins, retest this please.

sorry -- I misread the original comment. It does happen in a lot of places. Let me see how many spots and get this going if it's not too disruptive.

Or just create a new issue so we don't forget.  I really don't want to drag this one out---I was just commenting on all the stuff that I saw.

From CmdStan, RStan, PyStan. I'm pretty sure that's what we decided. Personally, I would have wanted to have (redundant) validation inside the service functions, but I think at some point I was convinced that it was better practice to validate arguments outside.

We wanted to do this outside so that the interfaces could produce error messages in their own language.

Yup. That's right. I think the current implementation isn't perfect, but
it'll work for now.
On Fri, Dec 30, 2016 at 12:12 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

From CmdStan, RStan, PyStan. I'm pretty sure that's what we decided.
Personally, I would have wanted to have (redundant) validation inside the
service functions, but I think at some point I was convinced that it was
better practice to validate arguments outside.

We wanted to do this outside so that the interfaces could produce error
messages in their own language.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/pull/2004#issuecomment-269796647, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F_lMbEafmm7KkGEDycBXfnwB0r-vks5rNTuSgaJpZM4Jfpda
.

We wanted to do this outside so that the interfaces could produce error messages in their own language.

I think one source of error messages (via throwing exceptions) would be fine. I thought there were other reasons pushing validation outside the function.

Bob, action is back on you to review the pull request.
I made most of the changes you suggested. If I missed anything major, please remind me. The ADVI issues, I've created a new issue: #2195.

@syclik  Thanks!  That's awesome.  I'll be able to take another pass tomorrow (1 January 2017!).

This pull request was too large---it overwhelmed my mac and ipad trying to review.  I lost about ten comments, mostly about all the repeated code snippets that should be oulled into functions.
Never mind.  I just approved it because as much as I hate to see all this code duplication, our code doesnt need to be perfectly factored going forward and I want to be pragmatic and get this in ASAP

On Jan 3, 2017, at 3:48 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
@syclik commented on this pull request.
...

     Eigen::VectorXd cont_params;

     cont_params.resize(cont_vector.size());

I'm changing both instances to construct with the right size. We can't use Eigen::Map -- it doesn't compile.

It compiles and passes tests on my machine with both assignment:
          Eigen::VectorXd cont_params
            = Eigen::Map<Eigen::VectorXd>(&cont_vector[0], cont_vector.size(), 1);

and construction:
          Eigen::VectorXd cont_params(Eigen::Map<Eigen::VectorXd>(&cont_vector[0], cont_vector.size(), 1));

Not sure it's faster or clearer, though.  Here's the lowdown on Map:
https://eigen.tuxfamily.org/dox/group__TutorialMapClass.html

Bob

Just communicated with @ariddell and he gave it a thumbs up.
This is now ready to merge as soon as the final tests pass. We have:

a review of the Stan branch by @bob-carpenter
sign-off by a CmdStan maintainer, me
sign-off by an RStan maintainer, @bgoodri
sign-off by the PyStan maintainer, @ariddell

As soon as the tests pass, I'm merging.

I think it is fine.
On Sun, Jan 8, 2017 at 9:45 PM, Daniel Lee<EMAIL_ADDRESS>wrote:

Just communicated with @ariddell https://github.com/ariddell and he
gave it a thumbs up.
This is now ready to merge as soon as the final tests pass. We have:

a review of the Stan branch by @bob-carpenter
https://github.com/bob-carpenter
sign-off by a CmdStan maintainer, me
sign-off by an RStan maintainer, @bgoodri
https://github.com/bgoodri
sign-off by the PyStan maintainer, @ariddell
https://github.com/ariddell

As soon as the tests pass, I'm merging.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/pull/2004#issuecomment-271202487, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADOrqgsIfyyL9M8hE-erL1OXzo9_Na8jks5rQZ8-gaJpZM4Jfpda
.

W00T!

On Jan 9, 2017, at 12:38 AM, Daniel Lee<EMAIL_ADDRESS>wrote:
Merged #2004.


On submit "Failed to send to server. Check your internet connection" but connection is fine

On submit "Failed to send to server. Check your internet connection" but connection is fine.
http://www.codewars.com/kata/54e6533c92449cc251001667/train/ruby
All tests on "Run tests" passed before hitting submit.
Logged in and can access next kata.
Tried reset and tried again, same failure to send to server message.

Does reloading the page fix the problem? See #154.

I found some script still being blocked, and now it works. Sorry. (Thanks for replying.)


https://github.com/elastic/examples/tree/master/ELK_nginx-json

For https://github.com/elastic/examples/tree/master/ELK_nginx-json, the highlighted text below in README should be nginx_json_logstash.conf instead of apache_logstash.conf
Note: Included nginx_json_logstash.conf configuration file assumes that you are running Elasticsearch on the same host as Logstash and have not changed the defaults. Modify the host and cluster settings in the output { elasticsearch { ... } } section of apache_logstash.conf, if needed.

@kunalvjti good catch. Thanks. Fixed ReadMe.


Add support for Rasberry Pi V.1

Hi,
I run the simple-client installation like mentioned here and arriving at this point (4) had some problems:
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
Because of Raspberry Pi V.1 (arm6) I had to install node manually.
No problem, did it like this:
wget https://nodejs.org/dist/latest/node-v10.5.0-linux-armv6l.tar.gz
tar -xvf node-v10.5.0-linux-armv6l.tar.gz
cd node-v10.5.0-linux-armv6l
sudo cp -R * /usr/local/
OK, installation went on fine until point 15:
node p2p-vps-client.js
The script ( I think it is write-files.js ) starts and runs the first steps successfully, but then stops at the same point with the same message mention before (4) and I tried modifing the write-files.js script changing the node install part like mentioned above - but this time no luck!
So I can't get a docker instance running until now, without solving that issue. Don't think it should be a big coding problem?
Thanks!
Regards

Good feedback. The P2P VPS client software depends on Docker, and I wasn't aware that Docker could run on the RPi v1. So even if you get past this point, I'm not sure if that part will work.
I also notice that you installed node v10 from source. The code has only been tested with node v8 at this point. v10 should work, but it's untested and may be the source of the problem.
What was the error message you were getting?

OK, maybe Docker won't run on RPi v1 - I didn't know that. Thanks for the hint.
But I'll give ist a try ;-)
I run this script (before editing "write-files.js")
node p2p-vps-client.js
and at stage 12 of 33 it brings this message (like already mention above at step 4 and I could solve by installing node manually):
## Installing the NodeSource Node.js 8.x LTS Carbon repo...

## You appear to be running on ARMv6 hardware. Unfortunately this is not currently supported by the NodeSource Linux distributions. Please use the 'linux-armv6l' binary tarballs available directly from nodejs.org for Node.js 4 and later.

followed by this message:
The command '/bin/sh -c bash nodesource_setup.sh' returned a non-zero code: 1
Error while trying to build Docker image! { Error: Command failed: ./lib/buildImage
    at Promise.all.then.arr (/home/pi/p2pvps-client/client/rpi/simple/node_modules/execa/index.js:236:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  code: 1,
  killed: false,
  stdout: null,
  stderr: null,
  failed: true,
  signal: null,
  cmd: './lib/buildImage',
  timedOut: false }
Error while trying to build Docker image! [ { Error: Command failed: ./lib/buildImage
      at Promise.all.then.arr (/home/pi/p2pvps-client/client/rpi/simple/node_modules/execa/index.js:236:11)
      at process._tickCallback (internal/process/next_tick.js:68:7)
    code: 1,
    killed: false,
    stdout: null,
    stderr: null,
    failed: true,
    signal: null,
    cmd: './lib/buildImage',
    timedOut: false } ]
error: Error while trying to build Docker image! code=1, killed=false, stdout=null, stderr=null, failed=true, signal=null, cmd=./lib/buildImage, timedOut=false
{
  "code": 1,
  "killed": false,
  "stdout": null,
  "stderr": null,
  "failed": true,
  "signal": null,
  "cmd": "./lib/buildImage",
  "timedOut": false
} []
error: {
  "code": 1,
  "killed": false,
  "stdout": null,
  "stderr": null,
  "failed": true,
  "signal": null,
  "cmd": "./lib/buildImage",
  "timedOut": false
}

So thought I can modify the script and repeat the steps I did on step 4 of the client install.
I tried to edit write-files.js with these lines:
#RUN curl -sL https://deb.nodesource.com/setup_8.x -o nodesource_setup.sh
#RUN bash nodesource_setup.sh
RUN wget https://nodejs.org/dist/latest/node-v10.5.0-linux-armv6l.tar.gz
RUN tar -xvf node-v10.5.0-linux-armv6l.tar.gz
RUN cd node-v10.5.0-linux-armv6l
RUN sudo cp -R * /usr/local/

But it ended up in a mess :-)))
Regards

That's exactly what I would have recommended you do. It looks like Docker is building and running. It's just having trouble getting node installed into the container.

The sudo command on the last line might be the problem.
What's the error you're getting when the Docker container fails the build?

Deleted sudo and the script stops here:
Step 11/33 : RUN wget https://nodejs.org/dist/latest/node-v10.5.0-linux-armv6l.tar.gz
 ---> Running in 38ce5a82e1d4
/bin/sh: 1: wget: not found
The command '/bin/sh -c wget https://nodejs.org/dist/latest/node-v10.5.0-linux-armv6l.tar.gz' returned a non-zero code: 127
Error while trying to build Docker image! { Error: Command failed: ./lib/buildImage
    at Promise.all.then.arr (/home/pi/p2pvps-client/client/rpi/simple/node_modules/execa/index.js:236:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  code: 127,
  killed: false,
  stdout: null,
  stderr: null,
  failed: true,
  signal: null,
  cmd: './lib/buildImage',
  timedOut: false }
Error while trying to build Docker image! [ { Error: Command failed: ./lib/buildImage
      at Promise.all.then.arr (/home/pi/p2pvps-client/client/rpi/simple/node_modules/execa/index.js:236:11)
      at process._tickCallback (internal/process/next_tick.js:68:7)
    code: 127,
    killed: false,
    stdout: null,
    stderr: null,
    failed: true,
    signal: null,
    cmd: './lib/buildImage',
    timedOut: false } ]
error: Error while trying to build Docker image! code=127, killed=false, stdout=null, stderr=null, failed=true, signal=null, cmd=./lib/buildImage, timedOut=false
{
  "code": 127,
  "killed": false,
  "stdout": null,
  "stderr": null,
  "failed": true,
  "signal": null,
  "cmd": "./lib/buildImage",
  "timedOut": false
} []
error: {
  "code": 127,
  "killed": false,
  "stdout": null,
  "stderr": null,
  "failed": true,
  "signal": null,
  "cmd": "./lib/buildImage",
  "timedOut": false
}
root@raspberrypi:/home/pi/p2pvps-client/client/rpi/simple#

OK, so thinking about the script more clearly ;-) there should be wget and tar installed on the docker image before trying to install node - right?
So maybe this should be modified here before?
https://github.com/P2PVPS/server-deployment2/blob/master/sshd-container/Dockerfile

Yes. You beat me to it.
Except this is the file you should edit (assuming you're running the simple client):
https://github.com/P2PVPS/p2pvps-client/blob/master/client/rpi/simple/lib/write-files.js
These lines:
...
RUN echo "export VISIBLE=now" >> /etc/profile
RUN curl -sL https://deb.nodesource.com/setup_8.x -o nodesource_setup.sh
RUN bash nodesource_setup.sh
RUN apt-get install -y nodejs
RUN apt-get install -y build-essential
RUN apt-get install -y inetutils-ping
...

Should look more like these lines:
...
RUN echo "export VISIBLE=now" >> /etc/profile
#RUN curl -sL https://deb.nodesource.com/setup_8.x -o nodesource_setup.sh
#RUN bash nodesource_setup.sh
#RUN apt-get install -y nodejs
RUN apt-get install -y wget tar
RUN wget https://nodejs.org/dist/latest/node-v10.5.0-linux-armv6l.tar.gz
RUN tar -xvf node-v10.5.0-linux-armv6l.tar.gz
RUN cd node-v10.5.0-linux-armv6l
RUN cp -R * /usr/local/
RUN apt-get install -y build-essential
RUN apt-get install -y inetutils-ping
...

Yeah, did it and had to run it serveral times, because it stoped at different stages.
E.g. when running tar it told me, that the directory was renamed (?!). But after starting the script for the third time, it unpacked everything but stops now at the step, when doing
RUN cp -R * /usr/local/
with this error:
Step 16/35 : RUN cp -R * /usr/local/
 ---> Running in 61e7ab9b0fd4
cp: error reading ‘proc/1/task/1/mem’: Input/output error
cp: failed to extend ‘/usr/local/proc/1/task/1/mem’: Input/output error
cp: error reading ‘proc/1/task/1/clear_refs’: Invalid argument
cp: failed to extend ‘/usr/local/proc/1/task/1/clear_refs’: Invalid argument
cp: error reading ‘proc/1/mem’: Input/output error
cp: failed to extend ‘/usr/local/proc/1/mem’: Input/output error
cp: error reading ‘proc/1/clear_refs’: Invalid argument
cp: failed to extend ‘/usr/local/proc/1/clear_refs’: Invalid argument
cp: error reading ‘proc/1/timerslack_ns’: Operation not permitted
cp: failed to extend ‘/usr/local/proc/1/timerslack_ns’: Operation not permitted
cp: cannot stat ‘proc/5/task/5/fd/3’: No such file or directory
cp: cannot stat ‘proc/5/task/5/fdinfo/3’: No such file or directory
cp: error reading ‘proc/5/task/5/mem’: Input/output error
cp: failed to extend ‘/usr/local/proc/5/task/5/mem’: Input/output error
cp: error reading ‘proc/5/task/5/clear_refs’: Invalid argument
cp: failed to extend ‘/usr/local/proc/5/task/5/clear_refs’: Invalid argument
cp: cannot stat ‘proc/5/fd/3’: No such file or directory
cp: cannot stat ‘proc/5/fdinfo/3’: No such file or directory
cp: error reading ‘proc/5/mem’: Input/output error
cp: failed to extend ‘/usr/local/proc/5/mem’: Input/output error
cp: error reading ‘proc/5/clear_refs’: Invalid argument
cp: failed to extend ‘/usr/local/proc/5/clear_refs’: Invalid argument
cp: cannot open ‘proc/sys/net/ipv4/route/flush’ for reading: Permission denied
cp: error reading ‘proc/sys/net/ipv6/conf/all/stable_secret’: Input/output error
cp: failed to extend ‘/usr/local/proc/sys/net/ipv6/conf/all/stable_secret’: Input/output error
cp: error reading ‘proc/sys/net/ipv6/conf/default/stable_secret’: Input/output error
cp: failed to extend ‘/usr/local/proc/sys/net/ipv6/conf/default/stable_secret’: Input/output error
cp: error reading ‘proc/sys/net/ipv6/conf/eth0/stable_secret’: Input/output error
cp: failed to extend ‘/usr/local/proc/sys/net/ipv6/conf/eth0/stable_secret’: Input/output error
cp: error reading ‘proc/sys/net/ipv6/conf/lo/stable_secret’: Input/output error
cp: failed to extend ‘/usr/local/proc/sys/net/ipv6/conf/lo/stable_secret’: Input/output error
cp: cannot open ‘proc/sys/net/ipv6/route/flush’ for reading: Permission denied
cp: cannot open ‘proc/sys/vm/compact_memory’ for reading: Permission denied
cp: cannot open ‘proc/kmsg’ for reading: Operation not permitted
cp: error reading ‘proc/sysrq-trigger’: Input/output error
cp: failed to extend ‘/usr/local/proc/sysrq-trigger’: Input/output error
^C

I'm not shure if this is a issue with my Rpi installation...

Hmm. The permission errors are odd. Try adding the sudo  back into that command and see if helps.

Yes, already tried adding sudo... no change, same errors.


Pasting a JSON object

Hi, this is a great project! Would love to know if it's possible to copy and paste JSON objects to and from the editor?
Thanks!

Yup, just pasting it with () surrounded. eg. ({"k1":123,"k2":"123","k3":false}). You could try it online: https://jv.yysuni.com/


tosiewytnie.pl

::: /etc/pihole/list.41.hosts.ubuntu101.co.za.domains (1 results)
<IP_ADDRESS> tosiewytnie.pl

Thanks @xxcriticxx added to false positives on https://github.com/mitchellkrogza/CENTRAL-REPO.Dead.Inactive.Whitelisted.Domains.For.Hosts.Projects and will be stripped out on next build. Keep em coming 👍

@xxcriticxx @funilrys if either of you can think of any domains which should be permanently whitelisted always, please review the list here and either send a PR with more domains to be whitelisted or simply list them in Issues on that repo and I will add them.

@mitchellkrogza I didn't look into the code but do you use regex match? Cause for example:
It'll save us some bytes to have

google.com ==> can match dl.google.com and drive.google.com
apple.com ==> can match itunes.apple.com
cnn.com ==> can match money.cnn.com
ubuntu.com ==> can match archive.ubuntu.com

I don't know what you think about it but I think it can be great assuming that only legitimate (big company or well-known website) domains are on that whitelist.

I agree totally, haven't figured out a regex in my cleaning functions to handle that. Much better to have just root domains and then *.domain.com get's stripped out of hosts lists. Do you want to review those list cleaning functions for me ??? You're smarter with Regex than me 😁

https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212

Thanks @xxcriticxx those have now all been whitelisted in https://github.com/mitchellkrogza/CENTRAL-REPO.Dead.Inactive.Whitelisted.Domains.For.Hosts.Projects/commit/047231eb58dc03149753fb7f8146d1de939047a2

@xxcriticxx we still have about another 5 days of dead domain testing to go before fresh links get pushed out again.

@mitchellkrogza no problem!

https://github.com/mitchellkrogza/Ultimate.Hosts.Blacklist/commit/199b042ab442a092702c3ce7bda7055e1be4f502 Fix: #55

@xxcriticxx fresh raw links pushed out today. Continuation of dead domains testing continuing in a separate environment.


Community Versions Documentation and Implementation

I am aware there have been a number of Community version related issues eg #85 and #219 but I was wondering how you would feel about a PR adding some more documentation and maybe configuration to this stack.
As a user I would love to have it documented specifically that I can utilize community-only things by setting certain parameters. Also exclude certain packages from the install stack.
And I would love to be able to switch of the metrics-reporter from the inventory.
If you guys would be willing to get something like that merged I would draft a PR :)

@Fobhep Thanks for reaching out, and I think this is a really great point.
Let me take this back to the team to discuss, to see if we can understand the pros/cons best way to approach this.
Personally, I am always in favour of more documentation, and even better when I don't have to write it. :)

happy to get your feedback then :)
let me maybe add one point - from my point as a consultant this would be very useful for "virgin kafka" clients who want to do a PoC using purely FOSS versions. Switching from there to a licensed deployment would then be even smoother.
And as I mentioned earlier this would also be a great thing for pure Apache Kafka users :)

@JumaX Are Confluent docs a monorepo?
I recently stumbled on this Azure video which might be useful for y'all's docs team
https://youtu.be/EpYYe6aQjJM

@Fobhep We would be happy to review the PR you submit regarding more open source documentation.  My suggestion would be for there to be an MD file that we can link to the readme.  I think it makes sense to keep the OpenSource docs in the repo itself.
@OneCricketeer Our official docs are handled by another team, and they are currently going through a redesign, so the short answer is not really, but it could change. :)

Dear all,
So what's about this PR ? Is there a chance to get it merged ?
Best,
Jerome

@jrevillard We are waiting on @Fobhep to finish his work before we can merge this.  If you are able to contribute, it would be greatly appreciated.

Closing this off based on conversation on https://github.com/confluentinc/cp-ansible/pull/299
Thanks!


strange mouse problem

Information
VIM version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled May  9 2018 23:29:27)
Patch incluse: 1-1806.
Operating System: linux debian stretch
:ALEInfo
Current Filetype: vim
Available Linters: ['vint']
Enabled Linters: ['vint']
Suggested Fixers:
'remove_trailing_lines' - Remove all blank lines at the end of a file.
'trim_whitespace' - Remove all trailing whitespace characters at the end of every line.
Linter Variables:
let g:ale_vim_vint_executable = 'vint'
let g:ale_vim_vint_show_style_issues = 1
Global Variables:
let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = 0
let g:ale_command_wrapper = ''
let g:ale_completion_delay = v:null
let g:ale_completion_enabled = 0
let g:ale_completion_max_suggestions = v:null
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%code: %%s'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'always'
let g:ale_lint_on_insert_leave = 0
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let g:ale_linters_explicit = 0
let g:ale_list_window_size = 10
let g:ale_list_vertical = 0
let g:ale_loclist_msg_format = '%code: %%s'
let g:ale_max_buffer_history_size = 20
let g:ale_max_signs = -1
let g:ale_maximum_file_size = v:null
let g:ale_open_list = 0
let g:ale_pattern_options = {}
let g:ale_pattern_options_enabled = 0
let g:ale_set_balloons = 1
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 0
let g:ale_sign_error = '>>'
let g:ale_sign_info = '--'
let g:ale_sign_offset = 1000000
let g:ale_sign_style_error = '>>'
let g:ale_sign_style_warning = '--'
let g:ale_sign_warning = '--'
let g:ale_statusline_format = ['%d error(s)', '%d warning(s)', 'OK']
let g:ale_type_map = {}
let g:ale_use_global_executables = v:null
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1
Command History:
(executable check - failure) vint
(executable check - failure) vint
What went wrong
Using the mouse before a command leads to erratic behaviours: can't yank lines, pressing d deletes a line instead of waiting for a motion, and normal d +leftclick to delete up to cursor doesn't work anymore.
Reproducing the bug
Steps for repeating the bug:
Tested in viml and python, also with minimal vimrc, I can reproduce it easily by doing:

left click somewhere
yy
left click somewhere else
d
either immediately or after a short delay, the line is deleted.
repeat 3-4, 5 should happen again

I'm sure it's this plugin because it reverts to normal behaviour if I disable it. In neovim the bug doesn't seem to happen, though.
I started having this problem after last time I updated the plugins with vim-plug, but I had not updated them for a while, before that.

Could you try disabling all other plugins and repeating your steps again?

It happens with this minimal vimrc:
let mapleader = "\\"
let maplocalleader = "\\"
set nocompatible

call plug#begin('~/.vim/plugged')
Plug 'w0rp/ale'
call plug#end()

Try setting ttymouse to xterm and check if your mouse is operational. I think this is vim bug because when I use mouse with sgr option my vim is segfaulting, I also compile vim quite often.

Thanks, set ttymouse=xterm seems to work :). At least d + leftclick works normally, so I guess does the rest.

Thank you @Szero for the suggestion. It sounds like this might be a Vim bug which ALE can trigger by running the right code from balloon functions. I might file an issue on the Vim repo for this if I can repeat the bug myself with a minimal example.

Mouse support is also broken for me when using putty-latest+linux+vim-latest+g:ale_set_balloons=1. (TERM=putty-256color, although I've tried many other settings, including screen-256color)
My .vimrc has:
set mouse=a
set ttymouse=sgr
packloadall
And ale is loaded as a plugin via vim pack feature.
Interestingly if I start a tmux session with mouse support enabled ('set -g mouse on' in .tmux.conf) the mouse works ok inside tmux. Within the tmux session,TERM=screen-256color.
I also cannot get terminal mouse balloons working even without ale. I tried the latest build of vim (e76c4b237d41d0e62ca5d4fc76d1dd163fe222c0)

I think this is a bug in Vim, not in ALE. All ALE does is show balloons using balloonexpr. I think someone should report a bug in Vim so it detects a better default for ttymouse.

@smhc Could you try my workaround for this problem? I was going through issues on official vim repo and some user pointed out that he managed to get balloons working in terminals by wrapping the balloon_show function inside the timer_start function and thus running it asynchronously. It works for me inside and outside tmux session. I'm using Debian Testing.

@Szero - my problem is that when the balloon support is enabled in ALE the mouse doesn't work for anything in vim, i.e resizing windows etc. It may be a bug with vim + enabled balloons + putty, rather than a bug with ALE specifically.
As I am using putty, balloon support in general is broken as putty does not support mouse tracking (yet).  So I am unable to test whether balloons are 'fixed'.
Due to balloon support being unsupported in my terminal I am better off with it disabled in ALE anyway.


lisa-install: Fix typo in install-nbextensions

An extra line-return escape character was preventing lisa-install to
complete.
Signed-off-by: Alessio Balsini<EMAIL_ADDRESS>

Ah, we really need Travis back on, sorry for the breakage :(


Mode display

Close #7
The length of the box is now relative to the length of the MODE.
I also changed the default color of MODE to white for more readability

No problem


Have account controller strip leading and trailing spaces from all inputs

Coverage remained the same at 92.666% when pulling 74e6f9431b2c7ea0b8a8405432e5b805cc107a5c on strip-email-address into 22be6941f1ca3ab35bc751d216ce0c24e224a052 on master.

As before, the failure is a pre-existing rubocop metric problem that is outside of the scope of this PR to fix.


checkpatch: ignore TRAILING_SEMICOLON and COMPLEX_MACRO

We are heavy users of macros and those rules keep coming up as false
positives that need to be overriden, so lets ignore them.

@julien-massot FYI.

@julien-massot FYI.


Remove BaseData underlying symbol from Future symbol

Just a thought I had and not sure if it matters, but there may be places in
the code where we check the IsDerivative property on Symbol, as this will
now return false for futures.
On Apr 17, 2017 12:52 PM, "Jared"<EMAIL_ADDRESS>wrote:

Merged #850 https://github.com/QuantConnect/Lean/pull/850.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/QuantConnect/Lean/pull/850#event-1045214348, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgb2LyplVloEFS3ySxAj_nXL2VKqT4aks5rw5jTgaJpZM4M_NYT
.

@mchandschuh There is no IsDerivative property on Symbol, maybe you meant IsCanonical ?
This one (and a few similar checks somewhere else) has been updated to reflect the change.
https://github.com/QuantConnect/Lean/blob/master/Common/Symbol.cs#L165

As always, sounds like you've got it covered :)
For some reason I thought there was an IsDerivative property some where,
must have been thinking of the canonical check. Sounds like you did the
necessary analysis here. Thanks for easing my concern :)
On Apr 17, 2017 2:58 PM, "Stefano Raggi"<EMAIL_ADDRESS>wrote:

@mchandschuh https://github.com/mchandschuh There is no IsDerivative
property on Symbol, maybe you meant IsCanonical ?
This one (and a few similar checks somewhere else) has been updated to
reflect the change.
https://github.com/QuantConnect/Lean/blob/master/Common/Symbol.cs#L165
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/QuantConnect/Lean/pull/850#issuecomment-294559960,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgb2A1dAPJuw88ox7G4gsm_hM_vAn-gks5rw7ZZgaJpZM4M_NYT
.


Consistent arg-name

Testing and validation are different tasks for a neural network. Since the latter is being done during training, arguments that interact with the user should be appropriately named.

By the way, just realized that you keep the validation data entirely separate, which emulates the definition of testing as well:
https://github.com/Deep-MI/FastSurfer/blob/c5e96778a237c7e21b1c525b8cf8cf1fc95cfdf2/FastSurferCNN/models/solver.py#L360
I was under the impression that validation data are also used in training.
Nevertheless, the proposed change holds for clarity and consistency.

I agree, consistent naming is important. I already changed this a while back so I am going to close the pull request. Thanks!


Fix AppReproducersTest#timezonesBakedIn

With GraalVM prior to 24.2.0 the default locale (and included locale) for a native image was determined by the -Duser.language and -Duser.country settings as image build time. With GraalVM >= 24.2.0 the build settings of language/country have no effect in terms of included locales in the resulting native image. Therefore, -H:IncludeLocales, needs to be used at build time instead. Also at image runtime the default language is determined by the system locale. For the purpose of the test we'd want the French language which we need to set at image runtime when the test runs.
Tested with GraalVM community 23+37 (which uses the old scheme) and with a latest JDK 24-based mandrel build which uses the new scheme. Both pass.
24.1:
[...]
Command: native-image -J-Duser.country=CA -J-Duser.language=fr -jar target/timezones.jar target/timezones
[...]
Command: ./target/timezones
jeu. sept. 26 12:48:51 CEST 2024
heure normale de l’Europe centrale

24.2
[...]
Command: native-image -H:IncludeLocales=fr-CA -jar target/timezones.jar target/timezones
[...]
Command: ./target/timezones -Duser.language=fr
jeu. sept. 26 12:53:57 CEST 2024
heure normale d’Europe centrale

Closes: #285

LGTM.

Thanks for the review.

I am just wondering whether we should instead fix this on the Quarkus side by adding user.language-user.country to the included locales like we do for quarkus.default-locale and quarkus.native.user-language-quarkus.native.country in quarkusio/quarkus#43448

I'm not sure we should. Having the build system locale included by default isn't something which I think all users are aware of. Maybe raise as an issue and see what the desired outcome of it is.

Maybe raise as an issue and see what the desired outcome of it is.

Done in https://github.com/quarkusio/quarkus/discussions/43533

@Karm OK with this? CI looks good too :)


Introduce AllocateTableSpace in Storage interface

Summary
We introduce a new method in the storage interface as follows:
Storage {
  String allocateTableSpace(...)
}

which returns the tablelocation where table objects will be created.
Implementations of this api should also execute one time operations such as creating buckets, assigning permissions etc.
Please note the comment:
this method should avoid creating TableFormat specific directories (ex: /data, /metadata for Iceberg or _delta_log for DeltaLake),it describes what shouldn't be done as part of this implementation.
Layering with internal implementations
Internal implementations such as LiHdfsStorage will now have flexibiliity to do one-time operations such as table provisioning.
Changes

[X] New Features
[X] Tests

For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done

[X] Existing tests run ✅
[X] Gradle build succeeds ✅
[X] E2E local tests work as well ✅

cc: @sumedhsakdeo , @jainlavina , @ctrezzo