Adopt AbstractMCMC.jl interface

Things to discuss

[x] AFAIK, the way to customize the logging in AbstractMCMC.jl is to pass progress=false to the underlying AbstractMCMC.mcmcsample and then use the callback keyword argument to log the progress. So the question is: should we do this so as to preserve the current logging functionality?
[x] To replicate the current summarization functionality (e.g. inform the user of average acceptance rates and EBFMI) as a post-sample step, we can overload StatsBase.sample and then perform this step after the call to AbstractMCMC.mcmcsample. Should we do this?

So the question is: should we do this so as to preserve the current logging functionality?

I think we should do this as to keep the functionality unchanged.
We can decide later if that's hard to maintain or not being as useful.

To replicate the current summarization functionality ...

Does it make sense to add this functionality in AbtractMCMC first? Seems to be a useful thing to have in general.

* [ ]  To replicate the current summarization functionality (e.g. inform the user of average acceptance rates and EBFMI) as a post-sample step, we can overload `StatsBase.sample` and then perform this step after the call to `AbstractMCMC.mcmcsample`. Should we do this?

Yes, that's the intended use case. :+1:

* AFAIK, the way to customize the logging in AbstractMCMC.jl is to pass `progress=false` to the underlying `AbstractMCMC.mcmcsample` and then use the `callback` keyword argument to log the progress. So the question is: should we do this so as to preserve the current logging functionality?

Yeah, I think the AHMC logging stuff should be moved to a callback.

Does it make sense to add this functionality in AbtractMCMC first? Seems to be a useful thing to have in general.

Not needed; with https://github.com/TuringLang/AbstractMCMC.jl/pull/56 merged we can implement this as a callback very easily, and just make this callback the default.

Btw, waiting for https://github.com/TuringLang/Turing.jl/pull/1579 since tests depend on Turing.

Is there anything missing from this PR? If there is nothing major, let's try to merge this PR since it will benefit several downstream tasks.

I've been working off the branch for this PR to support tempering on AHMC via the new MCMCTempering package and noticed that upon finishing sampling, even if I have using MCMCChains, I still just get back a vector of transitions from sampling, I think it would be nice to add a function similar to the one in Turing/Inference that implements AbstractMCMC.bundle_samples to actually make use of Chains, what do others think @torfjelde @yebai ?

I see yes, that is a fair assessment, I am using just AHMC, haven't tried with Turing but I agree it perhaps makes less sense to have something specific here for working with AHMC. On another note, I agree regarding Tor's comments on the state of the model currently, it would be better perhaps to rely on some standard definition in AMCMC.

What is the minimal expectation for the return from sample though? As currently it appears to be this vector of transitions rather than the more standard vector of thetas you see on master?

Glad you brought this up! I'm not certain tbh :confused: @xukai92 @yebai What do you think? Should it return PhasePoint or just thetas? I'm with whatever, but made it return PhasePoint initially.

What do you think? Should it return PhasePoint or just thetas?

I think we should not change the return at least for this PR.
Plus now stuff like ComponentArrays is supported, you can even have simple "variable bindings".

Not happy with how we have a DifferentiableModel in this repo. This should preferably go somewhere else, e.g. AMCMC. Alternatively we can make use of LogDensityProblems.jl, which might be a good approach (me and @devmotion discussed this on slack a couple of weeks ago).

Both are OK but I don't think we have to make it perfect within this PR as long as it's used behind the scene (i.e. users don't need to create it explicitly).
Just hope this PR doesn't keep growing than interface binding.

I've noticed something that might be a bug, so need to debug that first.

What's the potential bug?

What do you think? Should it return PhasePoint or just thetas?

I think we should not change the return at least for this PR.

Sounds good; I'll get that sorted out.

Plus now stuff like ComponentArrays is supported, you can even have simple "variable bindings".

Sure, but then we'd still need to depend on ComponentArrays.jl to be able to implement a bundle_samples(..., MCMCChains.Chains) method + assumes that the user is using it. IMO AMC shouldn't care, and this should be implemented somewhere else (alternatively we can provide an example somewhere).

Not happy with how we have a DifferentiableModel in this repo. This should preferably go somewhere else, e.g. AMCMC. Alternatively we can make use of LogDensityProblems.jl, which might be a good approach (me and @devmotion discussed this on slack a couple of weeks ago).

Both are OK but I don't think we have to make it perfect within this PR as long as it's used behind the scene (i.e. users don't need to create it explicitly).

True :+1: But currently the users do need to instantiate the DifferentiableModel (well, they don't have to, but the intention is that they can use this), so it is exposed to the user.
I'll take a look at what it would take to use LogDensityProblems.jl's approach, and then make a decision from there. I agree that we don't need to solve this completely right now.

I've noticed something that might be a bug, so need to debug that first.

What's the potential bug?

I don't think it is after all:) I observed some "strange" behavior in a problem I'm working on where the tree-depth during sampling was always one less than the maximum tree-depth, while in adaptation you'd see the "expected" behavior where it changed between iterations. But I increased the adaptation phase and then observe the same behavior at the end of adaptation, so it seems like it's just an artifact of the particular problem I'm working on. You've seen this before? It just seems a bit strange to me that it's so "deterministic" (also this is a very high-dimensional problem).

Sure, but then we'd still need to depend on ComponentArrays.jl

Aren't the implementation to bundle stuff is just for AbstractArray?

currently the users do need to instantiate the DifferentiableModel

Can we keep the same interface as before and convert Hamiltonian to DifferentiableModel under the hood?

You've seen this before?

It indeed sounds a bit weird but I don't have any idea why this could happen.
Did you try to increase the maximum and see if the depth still trys to "catch up" to max minus one?

where the tree-depth during sampling was always one less than the maximum tree-depth

@xukai92 I remember the counter for tree-depth was a bit confusing. Do we still record the actual tree-depth minus one internally? If so, it might be that NUTS is reaching the tree-depth limit during the sampling phase for Tor's problem.

Aren't the implementation to bundle stuff is just for AbstractArray?

Yes. My point is that we can't make use of keys in bundle_samples when we can't guarantee they exist. But I'm guessing you just meant that if we're using ComponentArray, and AHMC bundles these into a vector, then this can then be converted into a Chains later? If so I'm with you; I'm just saying that the last part shouldn't go into AHMC:)

Can we keep the same interface as before and convert Hamiltonian to DifferentiableModel under the hood?

Yeah, so this is what we're doing right now. Both using Hamiltonian and DifferentiableModel is directly is possible.

It indeed sounds a bit weird but I don't have any idea why this could happen.
Did you try to increase the maximum and see if the depth still trys to "catch up" to max minus one?

Didn't, but will; thanks :+1:

@xukai92 I remember the counter for tree-depth was a bit confusing. Do we still record the actual tree-depth minus one internally? If so, it might be that NUTS is reaching the tree-depth limit during the sampling phase for Tor's problem.

We found it's a bit unclear in PPLs regarding what exactly tree_depth means and how it should relate to n_steps. e.g. n_steps and tree_depth seem to relate to each other the same way in Stan and here but differently than in PyMC3. See https://github.com/arviz-devs/arviz/pull/1063#discussion_r499258271. Is tree_depth=1 a single-node tree, a 2-node tree, or a 4-node tree? Does a tree_depth of n mean that n tree doublings were accepted or proposed? perhaps here we are accidentally using two different definitions, one to count tree_depth and another to relate it to the maximum depth allowed?

@sethaxen thanks for the information - we're happy to adopt a more standardised terminology if that helps. Also happy to collaborate more closely with ArviZ devs on improving MCMC workflows.

@sethaxen thanks for the information - we're happy to adopt a more standardised terminology if that helps.

I'm not certain there is a standardised terminology here. The most important thing I think is to clearly define what the terms mean in this package and then include tests that the implied relationships are as expected.

Also happy to collaborate more closely with ArviZ devs on improving MCMC workflows.

ArviZ's focus is mostly on analysis of posteriors, focusing on diagnostics and visualizations, while we devs individually contribute to different PPLs. What sorts of collaborations RE MCMC workflows did you have in mind?

ArviZ's focus is mostly on analysis of posteriors, focusing on diagnostics and visualizations, while we devs individually contribute to different PPLs. What sorts of collaborations RE MCMC workflows did you have in mind?

For example, we are happy to transfer some parts of MCMCChains (e.g. diagnostics) to ArviZ. But ideally, we want to keep maintain them so that after transferring these implementations remain compatible with Turing. Additionally, it is helpful to have a suite of challenging tasks where current MCMC algorithms are known to perform poorly; such a suite is useful for testing the diagnostic tools developed by ArviZ, and for testing novel MCMC sampling methods. Just some initial thoughts.
cc @devmotion

ArviZ's focus is mostly on analysis of posteriors, focusing on diagnostics and visualizations, while we devs individually contribute to different PPLs. What sorts of collaborations RE MCMC workflows did you have in mind?

For example, we are happy to transfer some parts of MCMCChains (e.g. diagnostics) to ArviZ. But ideally, we are happy to keep maintain them so that after transferring these implementations remain compatible with Turing.

Yes, I proposed this in https://github.com/arviz-devs/ArviZ.jl/issues/129#issue-893730587 and https://github.com/TuringLang/MCMCChains.jl/issues/266#issuecomment-843010728, and I think this makes sense. And absolutely Turing devs should keep co-maintaining them with the ArviZ devs. A good way to ensure that works smoothly would be for one or more Turing core devs to join the ArviZ devs. There's no commitment or requirement there; it might just help facilitate better communication and integration between Turing and ArviZ, which I think would be a good thing anyways. Since @torfjelde has previously contributed to ArviZ.jl and @devmotion has a keen interest in the diagnostics and has already started work on https://github.com/devmotion/InferenceDiagnostics.jl, I'd nominate them, but perhaps that's something you can discuss amongst yourselves and who might be interested in joining.

Additionally, it is helpful to have a suite of challenging tasks where current MCMC algorithms are known to perform poorly; such a suite is useful for testing the diagnostic tools developed by ArviZ, and for testing novel MCMC sampling methods. Just some initial thoughts.

I think this is definitely useful, and Julia PPLs should have something like this. A few such efforts exist: e.g. Stan has posteriordb, TFP has inference-gym, and then there's PPL Bench for Jags, Pyro, NumPyro, PyMC3, and Stan. Necessarily, these efforts tend to be a bit PPL-specific, so I doubt there could be a formal collaboration here between ArviZ and Turing. But I would be interested in contributing to such a suite, and we could of course use ArviZ's diagnostics to critique the models.
Sorry @torfjelde for how off-topic this has gotten! Perhaps we should continue this discussion elsewhere.

Yes. My point is that we can't make use of keys in bundle_samples when we can't guarantee they exist. But I'm guessing you just meant that if we're using ComponentArray, and AHMC bundles these into a vector, then this can then be converted into a Chains later? If so I'm with you; I'm just saying that the last part shouldn't go into AHMC:)

For this PR we don't worry about interfacing with Chains.
We don't even want to bundle them into a vector as for now;
the current interface doesn't do this and we should keep it the same.

Just wanted to note, when I dev this PR and try to use latest version of Turing, the AHMC samplers don't seem to work:
julia > using Turing
[ Info: Precompiling Turing [fce5fe82-541a-59a6-adf8-730c64b5f9a0]

julia > using StatsPlots

julia > # Define a simple Normal model with unknown mean and variance.
        @model function gdemo(x, y)
          s² ~ InverseGamma(2, 3)
          m ~ Normal(0, sqrt(s²))
          x ~ Normal(m, sqrt(s²))
          y ~ Normal(m, sqrt(s²))
        end
gdemo (generic function with 1 method)

julia > #  Run sampler, collect results
        chn = sample(gdemo(1.5, 2), HMC(0.1, 5), 1000)
Sampling 100%|███████████████████████████████████████████████████████████████████████████████████████████████| Time: 0:00:22
ERROR: MethodError: no method matching step(::Random._GLOBAL_RNG, ::AdvancedHMC.Hamiltonian{AdvancedHMC.UnitEuclideanMetric{Float64, Tuple{Int64}}, Turing.Inference.var"#logπ#52"{DynamicPPL.TypedVarInfo{NamedTuple{(:s², :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s², Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s², Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{Normal{Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}}, Turing.Inference.var"#∂logπ∂θ#51"{DynamicPPL.TypedVarInfo{NamedTuple{(:s², :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s², Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s², Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{Normal{Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}}}, ::AdvancedHMC.HMCKernel{AdvancedHMC.FullMomentumRefreshment, AdvancedHMC.Trajectory{AdvancedHMC.EndPointTS, AdvancedHMC.Leapfrog{Float64}, AdvancedHMC.FixedNSteps}}, ::AdvancedHMC.PhasePoint{Vector{Float64}, AdvancedHMC.DualValue{Float64, Vector{Float64}}})
Closest candidates are:
  step(::AdvancedHMC.AbstractLeapfrog{T}, ::AdvancedHMC.Hamiltonian, ::P) where {T<:(Union{AbstractVector{var"#s55"}, var"#s55"} where var"#s55"<:AbstractFloat), P<:AdvancedHMC.PhasePoint} at /Users/harrisonwilde/Library/Mobile Documents/com~apple~CloudDocs/PhD/Pumas/AdvancedHMC.jl/src/integrator.jl:58
  step(::AdvancedHMC.AbstractLeapfrog{T}, ::AdvancedHMC.Hamiltonian, ::P, ::Int64; fwd, full_trajectory) where {T<:(Union{AbstractVector{var"#s56"}, var"#s56"} where var"#s56"<:AbstractFloat), P<:AdvancedHMC.PhasePoint} at /Users/harrisonwilde/Library/Mobile Documents/com~apple~CloudDocs/PhD/Pumas/AdvancedHMC.jl/src/integrator.jl:58
Stacktrace:
  [1] initialstep(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, spl::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, vi::DynamicPPL.TypedVarInfo{NamedTuple{(:s², :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s², Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s², Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{Normal{Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}; init_params::Nothing, nadapts::Int64, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/hmc.jl:198
  [2] initialstep(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, spl::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, vi::DynamicPPL.TypedVarInfo{NamedTuple{(:s², :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s², Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s², Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{Normal{Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64})
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/hmc.jl:153
  [3] step(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, spl::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}; resume_from::Nothing, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DynamicPPL ~/.julia/packages/DynamicPPL/emAbx/src/sampler.jl:87
  [4] step(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, spl::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}})
    @ DynamicPPL ~/.julia/packages/DynamicPPL/emAbx/src/sampler.jl:62
  [5] macro expansion
    @ ~/.julia/packages/AbstractMCMC/ByHEr/src/sample.jl:123 [inlined]
  [6] macro expansion
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
  [7] (::AbstractMCMC.var"#21#22"{Bool, String, Nothing, Int64, Int64, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Random._GLOBAL_RNG, DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, Int64, Int64})()
    @ AbstractMCMC ~/.julia/packages/AbstractMCMC/ByHEr/src/logging.jl:11
  [8] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:491
  [9] with_logger(f::Function, logger::LoggingExtras.TeeLogger{Tuple{LoggingExtras.EarlyFilteredLogger{TerminalLoggers.TerminalLogger, AbstractMCMC.var"#1#3"{Module}}, LoggingExtras.EarlyFilteredLogger{Logging.ConsoleLogger, AbstractMCMC.var"#2#4"{Module}}}})
    @ Base.CoreLogging ./logging.jl:603
 [10] with_progresslogger(f::Function, _module::Module, logger::Logging.ConsoleLogger)
    @ AbstractMCMC ~/.julia/packages/AbstractMCMC/ByHEr/src/logging.jl:34
 [11] macro expansion
    @ ~/.julia/packages/AbstractMCMC/ByHEr/src/logging.jl:10 [inlined]
 [12] mcmcsample(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, sampler::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, N::Int64; progress::Bool, progressname::String, callback::Nothing, discard_initial::Int64, thinning::Int64, chain_type::Type, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AbstractMCMC ~/.julia/packages/AbstractMCMC/ByHEr/src/sample.jl:114
 [13] sample(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, sampler::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, N::Int64; chain_type::Type, resume_from::Nothing, progress::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:156
 [14] sample(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, sampler::DynamicPPL.Sampler{HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}}, N::Int64)
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:155
 [15] sample(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, alg::HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}, N::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:142
 [16] sample(rng::Random._GLOBAL_RNG, model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, alg::HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}, N::Int64)
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:142
 [17] sample(model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, alg::HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}, N::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:132
 [18] sample(model::DynamicPPL.Model{var"#5#6", (:x, :y), (), (), Tuple{Float64, Int64}, Tuple{}}, alg::HMC{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.UnitEuclideanMetric}, N::Int64)
    @ Turing.Inference ~/.julia/packages/Turing/TbEZL/src/inference/Inference.jl:132
 [19] top-level scope
    @ REPL[4]:2

Not entirely sure what is going on, I have tried this with a clean setup so I am pretty sure it should be reproducible if someone else tries but I may just be being stupid, thought it was worth flagging up.

I believe we will need a patch on the Turing side as well to make this branch works.

I believe we will need a patch on the Turing side as well to make this branch work.

Yeah, there are breaking changes in the current master, unrelated to this PR.

Hey! I have also been trying out this branch but Turing free. And I faced one main issue. I tried to use ForwardDiff and I defined my DifferentiableDensityModel myself by passing function and gradient, except I did it wrong by not expecting that ∂ℓπ∂θ should return both the value and the gradient.
Also I noticed that the Hamiltonian created in the test is never used...
It would be nice to have constructors for DifferentiableDensityModel like the one for Hamiltonian, which given an AD package will do the right thing.
By the way the code failed because of the DualValue constructor expecting Scalar + Gradient which failed since ∂H∂θ passed x[1], x[2]. Maybe one should check that the res from ∂H∂θ is indeed a Tuple
Very cool stuff otherwise :D I am looking forward to see this merged

Everything is ready to go now, once #270 is merged.
IMO, we should make it a breaking release since there might exist some code that expects to receive the old trajectory type but which now receives a HMCKernel, e.g. https://github.com/TuringLang/Turing.jl/pull/1660, even though AFAIK none of the exported functions are doing anything different. I've therefore bumped the minor version here.

The CI workflow also fails here. Any idea?

The CI workflow also fails here. Any idea?

It might just be that the settings I have for the sampler is bad because it passes most of the time.

It might just be that the settings I have for the sampler is bad because it passes most of the time.

They were fine in the other PR so I suppose you mean it's due to randomness?
Seems like there are also conflicts.
Let's resolve them and see if the CI passes with a re-run.

Btw, one annoying thing is that find_good_stepsize isn't supported by DifferentiableDensityModel yet. There's a bit too much functionality that uses the hamiltonian in there atm, so I haven't bothered trying to refactor that part yet. But this is something we can always do later anyways.

Btw, one annoying thing is that find_good_stepsize isn't supported by DifferentiableDensityModel yet. There's a bit too much functionality that uses the hamiltonian in there atm, so I haven't bothered trying to refactor that part yet. But this is something we can always do later anyways.

Doing it in another PR sounds more reasonable. Can you create an issue for this along with other missing thing from DifferentiableDensityModel or even other alternatives (I believe we discussed a few).

How could IntegrationTests also pass even Turing is not compatible?

Doing it in another PR sounds more reasonable. Can you create an issue for this along with other missing thing from DifferentiableDensityModel or even other alternatives (I believe we discussed a few).

Will do :+1:

How could IntegrationTests also pass even Turing is not compatible?

Because this is a new minor version for AHMC, which won't be compat with Turing.jl since it has compat-bounds on AHMC + it indicates it's a breaking release and so Turing.jl should be expected to have to make some changes to be compatible with the new version. If you look at the integration tests you see Info: Not compatible with this release. No problem..
So it's just a question of whether we make the tests pass or fail in this case; I went with letting it pass since this is also what we do in other Turing.jl-packages, e.g. DPPL.

Okay this is just strange. It fails on macOS but seems to do fine otherwise.

So it's just a question of whether we make the tests pass or fail in this case; I went with letting it pass since this is also what we do in other Turing.jl-packages, e.g. DPPL.

I see. Makes sense to me now.

Okay this is just strange. It fails on macOS but seems to do fine otherwise.

Could be randomness again but I feel using the same parameters as the non-AMCMC version is enough. Otherwise it might indicate some issues.
If I give same initial position and same random seed, do I expect to get exact same samples from the interface implementation? If so maybe we can add a test to check if samples are exactly the same.

I realized what was causing the issue: I was including the adapted samples in the mean-computation:)
Now that those are dropped, it works. But now there's a different issue? This one seems completely unrelated to the changes in this PR though. You got any clue @xukai92 ?

Looks like the tests are fine now. Is turning off the progress (the last commit) the fix?

Looks like the tests are fine now. Is turning off the progress (the last commit) the fix?

No, that was just to make the logs nicer. It seems like one of the adaptation tests can fail if we're unlucky with the random seed, but as I said, this has nothing to do with this PR so IMO we merge and I'll make an issue reminding us that the adaptation test is fickle.


Sync Java 17 changes with main

Purpose
Java 17 migration
Examples
Checklist

[ ] Linked to an issue
[ ] Updated the changelog
[ ] Added tests
[ ] Updated the spec
[ ] Checked native-image compatibility

Closing this since the changes have been merged with another PR.


Deprecate Weight::from_{ref_time, proof_size}

Changes:

Deprecate Weight::from_ref_time and Weight::from_proof_size
Update weight templates

TODO:

[ ] Update weight files
[ ] Double-check templates
[ ] Companions

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2452966

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2452961

bot bench $ pallet dev pallet_balances

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453099

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453506

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453507

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453512

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453505

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2453969

bot merge


Cannot run build.sh to setup xenserver build env

Hi,
I am trying to build a xenserver build env, but after I added "DOCKER_OPTS="--dns <IP_ADDRESS> --dns <IP_ADDRESS> -s devicemapper" in /etc/default/docker file and restarted docker service, when I run build script,  I got the following error message
=======================================================================
sudo ./build.sh
Sending build context to Docker daemon 24.06 kB
Sending build context to Docker daemon
Step 0 : FROM xenserver/xenserver-base
Pulling repository xenserver/xenserver-base
341996e79147: Download complete
Status: Downloaded newer image for xenserver/xenserver-base:latest
---> 341996e79147
Step 1 : MAINTAINER John Else<EMAIL_ADDRESS>---> Running in b61312e4dad7
---> 84ceb7156562
Removing intermediate container b61312e4dad7
Step 2 : COPY files/yum.conf /etc/yum.conf
---> 6c638d82fbab
Removing intermediate container 85001f1af008
Step 3 : RUN mkdir -p /etc/yum.repos.d.xs
---> Running in c6dd7db30020
---> d3693d5e1076
Removing intermediate container c6dd7db30020
Step 4 : COPY files/Citrix.repo.in /tmp/Citrix.repo.in
---> 3bd36c5ae68b
Removing intermediate container 4158ae7e08e0
Step 5 : COPY files/RPM-GPG-KEY-Citrix-6.6 /etc/pki/rpm-gpg/RPM-GPG-KEY-Citrix-6.6
---> 2a4943d8dbf4
Removing intermediate container 4954b2344a4d
Step 6 : RUN yum install -y             git             make             mercurial             mock             rpm-build             rpm-python             sudo             yum-utils
---> Running in 49750d8c5407
Loaded plugins: fastestmirror
Loaded plugins: fastestmirror
http://www.uk.xensource.com/linux/distros/CentOS/7.0.1406/os/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://www.uk.xensource.com/linux/distros/CentOS/7.0.1406/os/x86_64/repodata/repomd.xml: (28, 'Resolving timed out after 30386 milliseconds')
Trying other mirror.
http://www.uk.xensource.com/linux/distros/CentOS/7.0.1406/os/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://www.uk.xensource.com/linux/distros/CentOS/7.0.1406/os/x86_64/repodata/repomd.xml: (28, 'Resolving timed out after 30387 milliseconds')
Trying other mirror.
...
One of the configured repositories failed (base),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
 1. Contact the upstream for the repository and get them to fix the problem.

 2. Reconfigure the baseurl/etc. for the repository, to point to a working
    upstream. This is most often useful if you are using a newer
    distribution release than is supported by the repository (and the
    packages for the previous distribution release still work).

 3. Disable the repository, so yum won't use it by default. Yum will then
    just ignore the repository until you permanently enable it again or use
    --enablerepo for temporary usage:

        yum-config-manager --disable base

 4. Configure the failing repository to be skipped, if it is unavailable.
    Note that yum will try to contact the repo. when it runs most commands,
    so will have to try and fail each time (and thus. yum will be be much
    slower). If it is a very temporary problem though, this is often a nice
    compromise:

        yum-config-manager --save --setopt=base.skip_if_unavailable=true

...
=======================================================================
Could you help to check what is problem?

Hi, and thanks for the bug report.
At the moment this environment uses a private yum repo to download build dependencies.
If you want to try to get it working, I recommend using buildroot to build the dependencies and then host the built RPMs in your own yum repo.
Hope this helps!

xenserver-build-env now works against a public yum repo - please take a look at the updated instructions in https://github.com/xenserver/xenserver-build-env/blob/master/README.md


sagittarius-scheme: 0.9.8 -> 0.9.9

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Please change commit and PR title to sagittarius-scheme: 0.9.8 -> 0.9.9

Please change commit and PR title to sagittarius-scheme: 0.9.8 -> 0.9.9

Done, sorry!


How to use  LazyLoadedTableDataAdapter with Master/Detail?

How to use  LazyLoadedTableDataAdapter with Master/Detail?
Is there any example how to do this?
I have to use large data sets(approx 5k rows) and  CollectionTableDataAdapter just freezes on them

Hello,
now it is only possible use LazyLoadedTableDataAdapter only for first level of Master/Detail GridView.
So fo example you can have LazyLoadedTableDataAdapter for Persons but for their orders you must have CollectionTableDataAdapter .
I am working on this kind of limitaion.

Now you can find in demo project example of using LazyLoadedTableDataAdapter with Master/Detail


Speaker fill does not work with desktop app

OS: Windows 10
Issue Descriptions: Speaker fill on 5.1 system does not work, and only front L/R speakers are playing, while all others are silent.
Steps to Reproduce: Press Play. Disconnect front left/right speakers from amp. Ta da!
Workaround: Create a shortcut to "Google Play Music Desktop Player.exe" with "--try-supported-channel-layouts" flag. Ta da!
Unfortunately workaround does not work with the default shortcut to "Update.exe"
Would be nice to get this to work out of the box, or have this as a setting in the app itself.

@duplicate-issue-bot[bot] Heh, those are a bit off the mark 😆
@sergesyrota I would accept a PR to add an option to the "Audio" settings tab to append this launch flag at startup.
--try-supported-channel-layouts
Would be a simple Settings.get call inside this file https://github.com/MarshallOfSound/Google-Play-Music-Desktop-Player-UNOFFICIAL-/blob/master/src/main/configureApp.js 👍

@MarshallOfSound, I'll try this out, and if it works, will submit a PR.

After an hour trying (first time with nodejs), can't get the build to work. I'll get back to this some time in the future when I have more time, but if anyone else can try it, please do :)

@sergesyrota If you need help getting the build going feel free to ask in Gitter.
A good starting point is https://github.com/MarshallOfSound/Google-Play-Music-Desktop-Player-UNOFFICIAL-/wiki/Development-FAQ

Fixed in #2715


Revive in-Docker builds

Nix should provide reproducible builds in the same way the Docker builds did.


NIFI-818: Initial implementation of Apache Accumulo on NiFi

Based on https://github.com/phrocker/nifi-accumulo with connectors to Apache Accumulo 2.x.
I have a branch for 1.9 if interested, but my opinion was that there is no need to maintain 1.x Accumulo connectors. Note that this requires a profile to be selected for assembly.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Note that if testing this, https://issues.apache.org/jira/browse/NIFI-6941 is an issue I encountered so having PutAccumuloRecord create the table is not a viable option in the interim. I'll dive into 6941 as soon as I can

After taking a look at this on a PR I think I may add some unit tests with mocked accumulo connectors.

Please avoid 'nifi.version' and 'project.version'. The maven release plugin takes care of this conversion for us and doing it without explicit values has create problems in the past. It is fine to have 'accumulo.version' though as presumably that is in a properties entry somewhere and that isn't part of this reactor build itself.

Thanks! I moved from a separate repo for the PR so I imagine I have more of these types of oversights. Appreciate that you were able to take a look.

@phrocker happy to help review, but you want to pull this out after a review and make a separate repo or offer to the Accumulo team since we're running on fumes in terms of space in our convenience binary for bundling new features. Last release, the graph database work that Matt, @mans2singh  and I did had to be left as an optional compile option because of that. Just a heads up.

as long as the nar is profile activated for inclusion into the convenience binary it is fair game to have here.

@joewitt I posted that before jumping into the commit. @phrocker feel free to disregard that if you want. We can bundle it as a profile if you want.

@MikeThomsen Thanks for the help. i'm getting back into the swing of things after Holiday travel -- sorry for my delay. Take a look at https://github.com/apache/fluo-uno -- it can get the stack up and running with a single command.

Thanks, I'll try to make time for that. Busy too with a lot of things so no worries.


fix(README.md): use CircleCI badge instead of TravisCI badge

We no longer build this project on TravisCI.

LGTM


feat: video link add

This is an awesome package for route and very useful. So, I created one video tutorial of this package. please add it in the doc. so, it helps the community. Thanks for this great package for the community.

Hi Shailesh! We prefer to only link to our own documentation, as we've found this is the easiest way to avoid any issues with external links going down or becoming out of date. I'd suggest submitting your tutorial to Laravel News as another way to get it in front of a lot of people. Thanks a lot for the PR and for making this video!


Add ecosystem doc

@felixmosh I've noticed that you wrote some plugins for knex, would you mind if we feature them on our knex tooling list? And if you are okay with that, which ones would you like mentioned?

I've 2 plugins that I'm using right now,
knex-paginate - this one allows you to handle pagination tasks
knex-on-duplicate-update - which probably will be deprecated once I will migrate my code base to use onConflict

@felixmosh Got it, thanks!


Error handling

Added at https://github.com/shunkosa/knowledge-category-list-view/commit/2c107e6e6e1943bdeb63037bbcb60cf414b10a49


add resource name property for views

if a view has a resourceName, the view will use the index
of the resource with that name for the view.
defaults to 0 if no resourceName is available.

Deployed.


CB-11442 Sdx upgrade: even if image change fails, Sdx flow continues

See detailed description in the commit message.

@horadla23  Pls rebase this to 2.39 as we will deliver it as a hotfix.

@pdarvasi sure, opened a new PR for that:https://github.com/hortonworks/cloudbreak/pull/10268/files


Save MTRs and ETRs

To ease post-model run analysis, marginal and effective tax rates for all model agents in all years should be saved in the ss_output and tpi_output dictionaries.

Resolved in PR #423.


[SUGGESTION] Add Questionably Immersive

The pack currently doesn't have any option for a more advanced coke oven. In fact, a more advanced coke oven is the only thing that mod adds.

No


Offline maps and custom maps data not loading in project.

When loading a project from the server, if there are offline maps or custom maps, they do not load.
To Reproduce
Are you able to reproduce the problem in a new project after restarting the app?
Yes
Steps to reproduce the behavior:

Download and save a map
logout then login
go to check Offline Maps and/or Custom Maps
See error

Expected behavior
To load any offline maps or custom maps no matter what project is loaded

fixed with commit 6a3947e564b0e46ddc2182a516576c1b0c114493 and df21908e4d4744746778bf6c7f18dafccc2258fa


TypeError: 'CommandCursor' object is unsubscriptabl

launched the command
python recipe.py "Tesla Model S" 100
and received this error.  MongoDB database is filled in fine - I see 98 tweets, everything is fine.
Traceback (most recent call last):
File "recipe.py", line 340, in
main(sys.argv[1], int(sys.argv[2]))
File "recipe.py", line 43, in main
print_results()
File "recipe.py", line 303, in print_results
avg_pos_score = mean_results['result'][2]['avgScore']
TypeError: 'CommandCursor' object is unsubscriptable

and there is a bug on your page
http://www.alchemyapi.com/developers/getting-started-guide/twitter-sentiment-analysis
instead of "analyze.py" it should be "recipe.py"

+1

This is an error in the code. Line 303 should be
avg_pos_score=mean_results['result][0]['avgScore']
change the 2 to 0 and the code should run properly.


Custom origin server and s3 origin options for SSR site plan

Hi. My app has an s3 bucket to hold media uploads and an additional lambda Function to process those uploads. I wanted to route to them using the CDN distribution created by my SSR site. I could create another Router and route from there to the SSR site CDN. That seems a bit wasteful. But would that be a better solution?
#996 proposes a solution to accept custom origins for the SSR site plan.
I published a demo in a public repo. The code to add the s3 origin and function are commented out here https://github.com/rmarscher/waku-sst-demo/blob/main/infra/website.ts#L28-L70
My SSR site component has a beforePlanValidate arg to hook into updating the plan. https://github.com/rmarscher/waku-sst-demo/blob/main/infra/components/waku.ts#L615-L617
Thank you!

Appreciate the PR! We have something planed.
In the meanwhile, i believe u can transform the cdn and modify the origins. Here's a thread of someone trying to transform the cdn - https://discord.com/channels/983865673656705025/1227658579046174872. Hope this helps until the feature gets added.

OK sounds good. It's possible I missed orderedCacheBehaviors as the right property to transform. I can go ahead and close this.


AXE Checker gets confused if there are multiple writes of the same data to the same address

I am seeing an AXE failure in tracegen. The summary given by axe-shrink.py is as follows
0: { M[1] == 51; M[1] := 114} @ 1613:
1: M[1] := 51 @ 1619:
1: { M[1] == 51; M[1] := 55} @ 1620:1678

I don't think this is actually a consistency violation. I think AXE is confused because it thinks both atomic swaps read the old value of 51. However, it's more likely the second swap read the 51 that was just written by the same processor.

Hi @zhemao, thanks for the report.
A few points in response:

If tracegen is generating multiple writes of the same data to the same address then that is a bug in tracegen.  (Axe requires that all writes have a unique address-value pair, and should give an error message if not.)

This trace doesn't actually contain multiple writes of the same data to the same address (there is one write of 51, one write of 114, and one write of 55).

I confirm that axe is behaving as intended by rejecting this trace.  Two atomic RMWs have read the same value which, combined with unique writes, cannot be atomic under any of the consistency models supported.

Q. What is the output of "axe check wmo trace.axe" where "trace.axe" is the original failing trace before shrinking?  If the output is "NO" then trace satisfies the unique writes restriction but fails wmo.  If the output is an error message then the trace doesn't satisfy axe's requirements on a trace, and there is probably a bug either in tracegen or the memory subsystem.

Oh, I see. You're saying that the write of 51 from core 1 is actually coming logically before the AMO from core 0. That makes much more sense.
I ran into this issue when attempting to run multiple trace generators against the same L1. I now realize that this might violate some assumptions the AXE checker has (since it assume generators 0 and 1 are separate cores). I'll close this issue for now and Henry and I will revisit what the semantics of multiple clients to the same L1 are.


upgrade marvin

issues w/ 1.4; try 1.5 when out, diagnose issues, etc

1.4 and 1.45 still have issues, limited to 1.3 for now


Time warps

Time warps are continous time transformations (a.k.a. non-linear functions of time). Because they are non-linear, they can not be represented as spans.
They can be represented as type Warp = Behavior Time (or Time -> Time). Intuitively, they map a "local" or "notated" time to a "global" or "sounding" time.
Time warps is the natural way to represent musical notions rubato, ritardando, agogics etc. They also generalize Span. For example assume that we have a function warp :: Warp -> a -> a similar to transform, then warp (transform s mempty) = transform s. In other words, for every span, there is a warp that perofrms the equivalent transformation, and that warp can be generated by transforming the empty warp.
Note that for Behaviors, warp is localRep from adjunctions.
I first found this idea in Nyquist, see http://www.cs.cmu.edu/~rbd/doc/nyquist/part5.html#37
Investigate:

How they compose: what do we lose by not having linearity?
Is there a form of invert?
Relation to derivatives/integrals.

Might reopen.


Add .build(nnArchive, config) to NeuralNetwork node

Added the function as well as python binding.

LGTM when tested, thanks!


[pwrmgr] V2(S) Signoff

Description
Ensure V2(S) signoff criteria are still maintained (this is not a focus area block).

pwrmgr V2(S) Signoff
This is not a focus area block.
Commits since Earlgrey-ES tapeout
$ git rev-parse --short HEAD
af85dad7af
$ git log Earlgrey-M2.5.2-RC0..af85dad7af --oneline -- hw/ip/pwrmgr hw/ip_templates/pwrmgr

dd117372ca [doc,pwrmgr] Document wakeup signal requirements

Document-only clarification

b16179dd74 [ipgen,rstmgr] Fix BUILD files

Change related to ipgen transition

ab4b36f6b9 [ipgen,rstmgr] Fix paths to rstmgr in md files

Change related to rstmgr ipgen transition

ce648ca68e [ipgen.pwrmgr] Change core files to vlnv naming and label as virtual

Fix related to pwrmgr ipgen transition

4a9a881d44 [topgen] Pass topname key/value pair to ipgen

Fix related to ipgen transition

da087bbddd [rtl,pwrmgr] Fix escalation timeout persistence

Fixes RTL issue #20516 needed for PROD

de31bdf1c2 [reggen] Remove the devmode input

devmode was tied to 1'b1 and the change is a net improvement with no functional impact

963a5006cc [doc] Minor tweak to md sanitisation code

Documentation-only change

748235cbb6 [dv,pwrmgr] Fix pwrmgr_escalation_timeout test

Fixes the test that verifies issue #20516

729f744642 [dv/pwrmgr] Add escalation_timeout test

Adds a test to verify #20516

c87db05473 [pwrmgr,ipgen] Remove the hw/ip/pwrmgr and their autogen files

Change related to pwrmgr ipgen transition

7215f8d9cb [pwrmgr,ipgen] Fix hjson files

Change related to pwrmgr ipgen transition

78c7edd733 [pwrmgr,ipgen] Fix pwrmgr template and autogen core files

Change related to pwrmgr ipgen transition

988b584da3 [pwrmgr,ipgen] Fix BUILD files

Change related to pwrmgr ipgen transition

3f88a55120 [pwrmgr,ipgen] Generate pwrmgr ip_autogen files with topgen

Change related to pwrmgr ipgen transition

8034e000d6 [pwrmgr,ipgen] Turn a few more files into templates

Change related to pwrmgr ipgen transition

a8529ce8a9 [pwrmgr,ipgen] Copy all hw/ip/pwrmgr files to ip_templates

Change related to pwrmgr ipgen transition

c288cc3968 [pwrmge,ip_gen] Add tpldesc hjson file

Change related to pwrmgr ipgen transition

77278de68f [pwrmgr] Replace tab with spaces in test plan

hjson formatting change

709d4e7ca0 [doc/pwrmgr] Improve theory of operation doc

Documentation-only improvement

9e006e2554 [sival,pwrmgr] Add features in pwrmgr.hjson.tpl

Adds features needed for the sival flow

1b16ca2122 [reggen] Add mubi support SWAccess that sets/clears a reg

Fixes a bug on MuBi CSR RTL

f4f5d98064 [dv/pwrmgr] Allow transitions to invalid fsm state

DV fix for SVA

59f8142826 [doc] Moved badges over to using hosted images

Documentation-only improvement

13c6152a0e [doc] pwrmgr registers and interfaces now use CMDGEN

Documentation-only improvement

f7edbbb6a3 [pwrmgr,dv] Correct name of disabled assertion

DV fix for SVA

51d31a1aae [dv/pwrmgr] Complete SVAs for rom integrity checks

DV fix for SVA issue #18236

7688e714e8 [reggen] Add initial support for version and cip_id hjson fields

RTL fix for issue #15333

fbd888eea8 Revert "[reggen] Add CIP_IDs and bump all major versions"

Reverts the change below

0ba10b3cd3 [reggen] Add CIP_IDs and bump all major versions

This change was reverted

9a61b02fe1 [pwrmgr] Fix xcelium compile error

Fix for xcelium, no functional change

143faa6be5 [pwrmgr] Synchronize lc signals

DV fix for SVA

e47df29f3e [misc] Use lc_tx_t testing functions at endpoints

RTL cleanup that doesn't affect functionality

Issues closed since the Earlgrey-ES tapeout

https://github.com/lowRISC/opentitan/issues/21451 [chip-test, pwrmgr] chip_sw_pwrmgr_escalation_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21450 [chip-test, pwrmgr] chip_sw_pwrmgr_b2b_sleep_reset_req Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21449 [chip-test, pwrmgr] chip_sw_pwrmgr_sleep_power_glitch_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21448 [chip-test, pwrmgr] chip_sw_pwrmgr_deep_sleep_power_glitch_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21447 [chip-test, pwrmgr] chip_sw_pwrmgr_random_sleep_power_glitch_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21446 [chip-test, pwrmgr] chip_sw_pwrmgr_main_power_glitch_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21445 [chip-test, pwrmgr] chip_sw_pwrmgr_aon_power_glitch_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21444 [chip-test, pwrmgr] chip_sw_sensor_ctrl_deep_sleep_wake_up Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21443 [chip-test, pwrmgr] chip_sw_pwrmgr_normal_sleep_all_wake_ups Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21442 [chip-test, pwrmgr] chip_sw_pwrmgr_external_full_reset Component:ChipLevelTest IP:pwrmgr Priority:P2 SiVal:Autogen

https://github.com/lowRISC/opentitan/issues/21026 [rstmgr] V2(S) Signoff Component:DV IP:rstmgr

https://github.com/lowRISC/opentitan/issues/21024 [clkmgr] V2(S) Signoff Component:DV IP:clkmgr

https://github.com/lowRISC/opentitan/issues/21014 [lc_ctrl] V2(S) Signoff Component:DV IP:lc_ctrl

https://github.com/lowRISC/opentitan/issues/20991 [rstmgr] D2(S) Signoff Component:RTL IP:rstmgr

https://github.com/lowRISC/opentitan/issues/20989 [clkmgr] D2(S) Signoff Component:RTL IP:clkmgr

https://github.com/lowRISC/opentitan/issues/20979 [lc_ctrl] D2(S) Signoff Component:RTL IP:lc_ctrl

https://github.com/lowRISC/opentitan/issues/20858 [doc,pwrmgr] Improve documentation of wakeup recording Component:Doc

https://github.com/lowRISC/opentitan/issues/20700 [ip,doc] Fix inaccurate references to reset behavior Component:Doc Milestone:V3 Priority:P2

https://github.com/lowRISC/opentitan/issues/20580 [sival,test] Flakiness in detecting PASS! in Rust test harnesses Component:ChipLevelTest Component:SiliconValidation Help Wanted Priority:P1 Type:Bug

https://github.com/lowRISC/opentitan/issues/20516 [rtl,pwrmgr] escalation timeout should persist Component:RTL Component:Security Earlgrey-PROD Candidate IP:pwrmgr Priority:P1

https://github.com/lowRISC/opentitan/issues/20415 [top_darjeeling] watch dog reset should be mapped to bit 0 Component:Darjeeling Component:DV Component:TestTriage

https://github.com/lowRISC/opentitan/issues/20392 [sysrst_ctrl] Unable to reset OT Priority:P1

https://github.com/lowRISC/opentitan/issues/20158 [dv,pwrmgr] Add test for escalation_timeout testpoint Component:DV Earlgrey-PROD Candidate

https://github.com/lowRISC/opentitan/issues/20135 [chip-test] chip_sw_pwrmgr_random_sleep_all_reset_reqs Component:ChipLevelTest Component:SiliconValidation

https://github.com/lowRISC/opentitan/issues/20134 [chip-test] chip_sw_pwrmgr_sleep_por_reset Component:ChipLevelTest Component:SiliconValidation

https://github.com/lowRISC/opentitan/issues/20133 [chip-test] chip_sw_pwrmgr_deep_sleep_por_reset Component:ChipLevelTest Component:SiliconValidation

https://github.com/lowRISC/opentitan/issues/20058 [chip-test] chip_sw_pwrmgr_random_sleep_all_wake_ups Component:ChipLevelTest

https://github.com/lowRISC/opentitan/issues/19858 [chip-test] chip_sw_sysrst_ctrl_combo_reset Component:ChipLevelTest Component:SiliconValidation IP:sysrst_ctrl SiVal:BazelLinkNeeded SiVal:draft/WIP

https://github.com/lowRISC/opentitan/issues/19819 [top_englishbreakfast,util] Fix paths to ip-specific ips in prepare_sw.py

https://github.com/lowRISC/opentitan/issues/19403 [top_level,tests] Convert sim_dv sysrst_ctrl test to use opentitantool Component:ChipLevelTest Earlgrey-PROD Triaged

https://github.com/lowRISC/opentitan/issues/19051 [pwrmgr/sram_ctrl] Use synchronizers on all LC signals IP:pwrmgr IP:sram_ctrl Priority:P1 Type:Bug

https://github.com/lowRISC/opentitan/issues/18696 [test-triage] rom_e2e_* Component:ChipLevelTest Component:TestTriage

https://github.com/lowRISC/opentitan/issues/18510 [i2c] Unexpected data in ACQ FIFO after deep sleep wakeup Earlgrey-PROD Candidate IP:i2c Priority:P2

https://github.com/lowRISC/opentitan/issues/18341 [aon_timer, rtl/doc] No wakeup signalled on watchdog bite if it occurs before bark Component:Doc IP:aon_timer

https://github.com/lowRISC/opentitan/issues/18236 [dv/pwrmgr] verify rom integrity check transitions Component:DV

https://github.com/lowRISC/opentitan/issues/17085 [top_darjeeling] fake pinmux or direct connection Component:Darjeeling

https://github.com/lowRISC/opentitan/issues/16960 [ci] Airgapped builds failing Component:CI Type:Bug

https://github.com/lowRISC/opentitan/issues/16735 [pwrmgr] Potential improvement on escalation path to avoid crossing slow FSM Earlgrey-PROD Candidate Hotlist:Security IP:pwrmgr

https://github.com/lowRISC/opentitan/issues/16689 [dv/cdc] Enable CDC instrumentation in all ips Component:DV Priority:P2 Triaging:MultipleBlocks

https://github.com/lowRISC/opentitan/issues/16562 Flow error that dc is unsupported for job runtime extraction

https://github.com/lowRISC/opentitan/issues/14867 [escalation / rma entry] Better cross handling in the future IP:alert_handler IP:lc_ctrl IP:pwrmgr Triaging:MultipleBlocks Type:FutureRelease

Currently open issues

https://github.com/lowRISC/opentitan/issues/21706 [sival,triag] Some SiVal ROM_EXT are broken

https://github.com/lowRISC/opentitan/issues/21222 [chip-test] Incomplete SiVal Stage 1 and 2 tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21083 [sival] test_locked_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21081 [sival] test_unlocked1_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21076 [sival] test_unlocked_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21075 [sival] raw_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21074 [sival] dev_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/21025 [pwrmgr] V2(S) Signoff Component:DV IP:pwrmgr

https://github.com/lowRISC/opentitan/issues/21012 [flash_ctrl] V2(S) Signoff Component:DV IP:flash_ctrl

This is unrelated, probably bad labelling

https://github.com/lowRISC/opentitan/issues/20990 [pwrmgr] D2(S) Signoff Component:RTL IP:pwrmgr

https://github.com/lowRISC/opentitan/issues/20983 [otp_ctrl] D2 Signoff Component:RTL IP:otp_ctrl

This is unrelated, bad labelling?

https://github.com/lowRISC/opentitan/issues/20798 [sival,sensor_ctrl] Fix unexpected alert 5 Component:RTL Component:SiliconValidation

Unrelated to pwrmgr

https://github.com/lowRISC/opentitan/issues/20518 [rtl,pwrmgr] escalation reset requests should not be delayed so much Component:RTL Component:Security Earlgrey-PROD Candidate IP:pwrmgr Priority:P2

To be done, has no counter-measure impact

https://github.com/lowRISC/opentitan/issues/20194 [sival] prod_all_tests

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/19861 [chip-test] chip_sw_sysrst_ctrl_ulp_wakeup Component:ChipLevelTest Component:SiliconValidation IP:sysrst_ctrl SiVal:BazelLinkNeeded SiVal:draft/WIP

Does not affect pwrmgr

https://github.com/lowRISC/opentitan/issues/19810 [doc,dif] Fix link in dif header files

Documentation-only

https://github.com/lowRISC/opentitan/issues/19797 [bazel, dvsim] Convert dvsim-executed tests to the new rules Earlgrey-PROD Triaged Priority:P1 SW:Build System Type:Task

All pwrmgrs tests are done

https://github.com/lowRISC/opentitan/issues/19124 [pwrmgr] Error FSM state is not observable in pwrmgr_fsm Earlgrey-PROD Candidate Priority:P2

There is an alternative way to implement the assertion for formal.

https://github.com/lowRISC/opentitan/issues/19059 [aon_timer] Consider requirements for clk_aon_i vs clk_i Earlgrey-PROD Candidate IP:aon_timer Priority:P2 Type:FutureRelease

It has no impact on pwrmgr

https://github.com/lowRISC/opentitan/issues/18513 [doc] AON Timer wdog bite reset missing sw initialization documentation Earlgrey-PROD Candidate

Documentation-only issue.

https://github.com/lowRISC/opentitan/issues/17086 [dv/mem_bkdr_util] Improve memory write performance Component:DV Earlgrey-PROD Candidate Type:Icebox

No impact on D/V2(S)

https://github.com/lowRISC/opentitan/issues/16636 [chip-test] Move C tests without external host from tests/sim_dv/ to tests/ Component:ChipLevelTest Earlgrey-PROD Triaged IP:alert_handler IP:csrng IP:entropy_src IP:flash_ctrl IP:lc_ctrl IP:otp_ctrl IP:pwrmgr IP:sram_ctrl Priority:P3 Subsystem:Entropy Triaging:MultipleBlocks Type:Enhancement

pwrmgr tests are done

https://github.com/lowRISC/opentitan/issues/16148 [dv/rstmgr] Apply different reset kinds where applicable Component:DV Earlgrey-PROD Candidate IP:rstmgr Priority:P2

It does not apply to pwrmgr

https://github.com/lowRISC/opentitan/issues/15986 [rtl,reset] Improve handling of main domain power glitches Component:DV Component:RTL Earlgrey-PROD Triaged IP:ast IP:pwrmgr IP:rstmgr Type:FutureRelease

This is not staged for PROD

https://github.com/lowRISC/opentitan/issues/14887 [chip/prim_esc_receiver] check if prim_esc_receiver is out-of-sync Component:DV Earlgrey-PROD Candidate IP:alert_handler IP:prim IP:pwrmgr Priority:P2 TOP:earlgrey

I think this should not block D/V2(S)

https://github.com/lowRISC/opentitan/issues/12848 [top/doc] Autogenerated tables for top-specific IP Component:Doc Earlgrey-PROD Triaged IP:alert_handler IP:clkmgr IP:pinmux IP:pwrmgr IP:rstmgr IP:rv_plic Triaging:MultipleBlocks Type:Cleanup Type:FutureRelease Type:Icebox

Documentation-only issue

https://github.com/lowRISC/opentitan/issues/11520 [dv/exclusions] Change handling of full-chip exclusions Component:DV Earlgrey-PROD Candidate IP:clkmgr IP:pwrmgr IP:rstmgr Priority:P3

Not a D/V2(S) issue

https://github.com/lowRISC/opentitan/issues/10323 [rom_ext] Possibly check ROM digest against a known value? Component:Software Earlgrey-PROD Triaged Priority:P3 SW:ROM_EXT

No impact on pwrmgr, since it only looks at the result of the checks as reported by rom_ctrl

https://github.com/lowRISC/opentitan/issues/9206 [top, doc] Top level documentation for keymgr seed flow Component:Doc Earlgrey-PROD Triaged IP:flash_ctrl IP:keymgr IP:lc_ctrl IP:otp_ctrl IP:pwrmgr Priority:P4 TOP:earlgrey Triaging:MultipleBlocks

Documentation-only issue

https://github.com/lowRISC/opentitan/issues/8440 [tracking] Use ipgen to produce templated IPs Component:Tooling Earlgrey-PROD Candidate IP:clkmgr IP:flash_ctrl IP:pwrmgr IP:rstmgr Priority:P2 Triaging:MultipleBlocks Type:Cleanup

The pwrmgr part of this is done

https://github.com/lowRISC/opentitan/issues/7978 [ast] Make ast a comportable module or at least a comportable front-end Component:RTL Earlgrey-PROD Triaged IP:ast Priority:P2 Type:FutureRelease

Not pwrmgr related

https://github.com/lowRISC/opentitan/issues/7612 [pwrmgr] Documentation update Component:Doc Earlgrey-PROD Triaged IP:pwrmgr Priority:P3 Type:Task

Documentation-only

https://github.com/lowRISC/opentitan/issues/3353 [flash_ctrl] TODO list for open source portion of flash controller Component:RTL Earlgrey-PROD Candidate IP:flash_ctrl Priority:P2 Type:Task

Not pwrmgr related

https://github.com/lowRISC/opentitan/issues/2083 [topgen] topgen flow Component:MultiTop Component:Tooling Earlgrey-PROD Triaged Priority:P3 TOP:earlgrey Type:FutureRelease

Not staged for PROD

@matutem's currently working on improving the failing nightly regressions, because pwrmgr at this point doesn't fulfill V2 requirements. We'll revisit this on Thu and may decide to postpone to M4.

The expectation is that https://github.com/lowRISC/opentitan/pull/22120 will take all tests above 90% passing.

Closing this since all pending items are done. The block version 1.0.0 is already at V2S.

Closing this since all pending items are done. The block version 1.0.0 is already at V2S.


bootstrap.ps1 fails on "Detecting host network interface ..."

What kind of request is this:
BUG: Provisioning a Windows node using the command provided in the UI produces errors.

INFO: Detecting host CPU reservation ...
INFO: Detecting host RAM reservation ...
INFO: Detecting host DISK reservation ...
INFO: Detecting host system locale ...
INFO: Detecting host Docker name pipe existing ...
INFO: Detecting host Docker release version ...
INFO: Detecting host network interface ...
Select-Object : Property "Subnets" cannot be found.
At C:\etc\rancher\bootstrap.ps1:109 char:34
+ ... rs = Get-HnsNetwork | Select-Object -ExpandProperty "Subnets" | Selec ...
+                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (@{ActivityId=95...on=38654705666}:PSObject) [Select-Object], PSArgumen
   tException
    + FullyQualifiedErrorId : ExpandPropertyNotFound,Microsoft.PowerShell.Commands.SelectObjectCommand

Other details that may be helpful:
The issue is clearly caused by line 156 in bootstrap.ps:
`$vNetAdapters = Get-HnsNetwork | Select-Object -ExpandProperty "Subnets" | Select-Object -ExpandProperty "GatewayAddress"

The following should be a reasonable fix:
`$vNetAdapters = Get-HnsNetwork | Where-Object -Property "Subnets" | Select-Object -ExpandProperty "Subnets" | Select-Object -ExpandProperty "GatewayAddress"
"GatewayAddress"

Environment information

Rancher version rancher/rancher:2.3.2
Installation option: single install

Cluster information

Cluster type: Custom
Machine type: VM, Microsoft Windows Server 2019 Standard, 10.0.17763 Build 17763
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:11:03Z", GoVersion:"go1.12.10", Compiler:"gc",
Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc",
Platform:"linux/amd64"}

Docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:37 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:36:50 2019
  OS/Arch:          windows/amd64
  Experimental:     false

Same issue here with rancher/rancher:2.3.1 and rancher/rancher:2.3.0

This script has verified via the following steps:

Import module from https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/start-kubelet.ps1
Create two fake HNS network via New-HNSNetwork

Get version info from the testing host, Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion':
SystemRoot                : C:\Windows
BuildBranch               : rs5_release
BuildGUID                 : ffffffff-ffff-ffff-ffff-ffffffffffff
BuildLab                  : 17763.rs5_release.180914-1434
BuildLabEx                : 17763.1.amd64fre.rs5_release.180914-1434
CompositionEditionID      : ServerDatacenter
CurrentBuild              : 17763
CurrentBuildNumber        : 17763
CurrentMajorVersionNumber : 10
CurrentMinorVersionNumber : 0
CurrentType               : Multiprocessor Free
CurrentVersion            : 6.3
EditionID                 : ServerDatacenter
EditionSubManufacturer    :
EditionSubstring          :
EditionSubVersion         :
InstallationType          : Server
InstallDate               :<PHONE_NUMBER>
ProductName               : Windows Server 2019 Datacenter
ReleaseId                 : 1809
SoftwareType              : System
UBR                       : 805
PathName                  : C:\Windows
ProductId                 : 00430-00000-00000-AA884
DigitalProductId          : {164, 0, 0, 0...}
DigitalProductId4         : {248, 4, 0, 0...}
InstallTime               :<PHONE_NUMBER>44329094
PSPath                    : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows
                            NT\CurrentVersion
PSParentPath              : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT
PSChildName               : CurrentVersion
PSDrive                   : HKLM
PSProvider                : Microsoft.PowerShell.Core\Registry

@goldsam , @bpaskalev Could you share your host version info here, please?
On the other hand, as a workaround, you could skip this check via -e WITHOUT_VERIFICATION=true:
PowerShell -NoLogo -NonInteractive -Command "& {docker run -e WITHOUT_VERIFICATION=true -v c:/:c:/host rancher/rancher-agent:<version> bootstrap --server <server> --token<token> --ca-checksum <ca-checksum> --worker | iex}"

@thxCode
Guest VM (target of install) :Microsoft Windows Server 2019 Standard, 10.0.17763 Build 17763
Host: Microsoft Windows Server 2016 Standard, 10.0.14393 Build 14393

@thxCode As a note, the guest VM had an external vSwitch bound to the only virtualized NIC. Perhaps this was a contributing factor.

@goldsam, I mean, yeah, my testing VM is a guest too, I bring it up on GCE. We have verified this on AWS/Azure as well.
I don't know what is the difference between those cloud-provided and yours, I only curios about this: it looks like the Get-HNSNetwork returns a wrong value / unexpected value or doesn't return anything on Subnets field.
My questions are:

Is there an HNS network without a subnet?
Or any differences between our Get-HNSNetwork methods?
Or Windows returns different struct of items when getting the HNS networks?
Or something else?

Your reasonable fix is as expected: give a filter to confirm the Subnets field, but I want to figure out why it doesn't work on your env.

I had this issue as well.
I remove everything and reinstall docker and this error is gone
https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server


TASK/DETAIL

Automatically generated comment ℹ
This comment is automatically generated and will be overwritten every time changes are committed to this branch.
The table contains an overview of files in the content directory that have been changed in this pull request. It's provided to make it easy to review your changes on the staging site. Please note that changes to the data directory will not show up in this table.

Content directory changes
You may find it useful to copy this table into the pull request summary. There you can edit it to share links to important articles or changes and to give a high-level overview of how the changes in your pull request support the overall goals of the pull request.

Source
Preview
Production
What Changed

get-started/quickstart/hello-world.md
fptghecghes@ 3.5 3.4 3.3 3.2 3.1 ghae
fptghecghes@ 3.5 3.4 3.3 3.2 3.1 ghae

Originally posted by @github-actions[bot] in https://github.com/github/docs/issues/18843#issuecomment-1168760164

Disc/files/requestedpagestimeout


experimental: new fira code spinner

[!WARNING]
Experimental may not get merged to made codebase

Depends on: https://github.com/ryanoasis/nerd-fonts/issues/1345
https://github.com/mikesmithgh/kitty-scrollback.nvim/assets/10135646/17317035-dc9a-4d25-8961-998c5e32e0ff

You dont need to depend on fonts anymore: https://github.com/kovidgoyal/kitty/commit/7d787e6c22728ebedcc715421c6006af61b1930a

Updated demo based on kovidgoyal/kitty@7d787e6
https://github.com/mikesmithgh/kitty-scrollback.nvim/assets/10135646/e13087f3-2d9a-4944-a042-63a3334c847d
Thanks @kovidgoyal! Very nice.

closing as won't do. if you are interested in this feature please open an issue


Correcting the ms.technology tagging

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/actionable-messages/actionable-email-dev-dashboard.md
:white_check_mark:Succeeded

docs/actionable-messages/actionable-messages-via-connectors.md
:white_check_mark:Succeeded

docs/actionable-messages/actionable-messages-via-email.md
:white_check_mark:Succeeded

docs/actionable-messages/actions-and-inputs.md
:white_check_mark:Succeeded

docs/actionable-messages/card-reference.md
:white_check_mark:Succeeded

docs/actionable-messages/connectors-dev-dashboard.md
:white_check_mark:Succeeded

docs/actionable-messages/get-started.md
:white_check_mark:Succeeded

docs/actionable-messages/index.md
:white_check_mark:Succeeded

docs/actionable-messages/security-requirements.md
:white_check_mark:Succeeded

docs/index.md
:white_check_mark:Succeeded

docs/rest/compare-graph-outlook.md
:white_check_mark:Succeeded

docs/rest/dotnet-tutorial.md
:white_check_mark:Succeeded

docs/rest/get-started.md
:white_check_mark:Succeeded

docs/rest/index.md
:white_check_mark:Succeeded

docs/rest/ios-tutorial.md
:white_check_mark:Succeeded

docs/rest/java-tutorial.md
:white_check_mark:Succeeded

docs/rest/javascript-tutorial.md
:white_check_mark:Succeeded

docs/rest/node-tutorial.md
:white_check_mark:Succeeded

docs/rest/php-tutorial.md
:white_check_mark:Succeeded

docs/rest/python-tutorial.md
:white_check_mark:Succeeded

docs/rest/recurrence-rules.md
:white_check_mark:Succeeded

docs/rest/reference.md
:white_check_mark:Succeeded

docs/rest/ruby-tutorial.md
:white_check_mark:Succeeded

docs/rest/terms-of-use.md
:white_check_mark:Succeeded

docs/rest/webhooks.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


constellation resource command

added constellation resource command to return selected resource in the constellation of the provided system

Looks great!  Thank you


_DefaultValueSerializer  issue

The _DefaultValueSerializer  class (in data_class.dart) did not cover all primitive variable type
for example if you call  int x = fromJson<int>('10);  it crashes
is it possible to add all primitive variable type like int, double, float, bool
class _DefaultValueSerializer extends ValueSerializer {
  const _DefaultValueSerializer();

   @override
  T fromJson<T>(dynamic json) {
    if (T == DateTime) {
      if (json == null) {
        return null;
      } else {
        return DateTime.fromMillisecondsSinceEpoch(json as int) as T;
      }
    }

    if (T == double && json is int) {
      return json.toDouble() as T;
    }

    // blobs are encoded as a regular json array, so we manually convert that to
    // a Uint8List
    if (T == Uint8List && json is! Uint8List) {
      final asList = (json as List).cast<int>();
      return Uint8List.fromList(asList) as T;
    }

    return json as T;
  }

  ...
}

I'm not sure if it should handle those cases at all. The List<int> special logic is because that's how Uint8Lists are encoded in json, we obviously loose their type. It's reasonable that 0 can be read as a double, but I'd say that '0' should not. In general I prefer to keep the implementation as simple as possible, and special cases for other primitive types shouldn't be necessary.
If you need support to read strings as ints or other primitive types, you can always override the default serializer with moorRuntimeOptions.defaultSerializer. I'll close this issue as I don't think that should be the default. If you disagree and have a common scenario that would require this, let me know.

Sadly there a lot of common scenario that would require this, howover as you know we have IntColumn() ..., and if we try create an object fromJson() and one of column has column type as IntColumn and in json we have that value as String this will cause a problem.

I see that, but why would the json have a string when an int is expected? Other json generators like json_serializable or built_value wouldn't accept that either. It sounds like an error on the server-side or whoever is generating the json if it represents integers as strings.

No, it's not a server, because in make case , i have a csv files that represents tables data, so i created a generic function that return T, by generate a json and call T.fromJson() to get my dataClass entity

Ok, but you'll agree that this is a very specific use-case, right? Since you know that you only get strings, it might make sense to map manually based on the column type:
extension MapFromCsvExtension<TableDsl extends Table, D extends DataClass>
    on TableInfo<TableDsl, D> {
  D mapFromCsvRow(List<String> values) {
    final typedValues = <String, dynamic>{};

    for (var i = 0; i < values.length; i++) {
      final rawValue = values[i];
      final column = $columns[i];

      dynamic typedValue;

      if (column is TextColumn) {
        typedValue = rawValue;
      } else if (column is BoolColumn) {
        typedValue = rawValue == 'true';
      } else if (column is IntColumn) {
        typedValue = int.parse(rawValue);
      } else if (column is DateTimeColumn) {
        typedValue =
            DateTime.fromMillisecondsSinceEpoch(int.parse(rawValue) * 1000);
      } else if (column is RealColumn) {
        typedValue = double.parse(rawValue);
      } else if (column is BlobColumn) {
        // don't know how you represent blobs in CSV
      }

      typedValue[column.$name] = typedValue;
    }

    return map(typedValues);
  }
}

This would allow you to use table.mapFromCsvRow from your database. I think the json converter isn't really a suitable tool when reading untyped values like CSV.

Very nice, seems a good solution , thank you for both your fast replys and your clever solution.

is it possible to make that extension returns a UpdateCompanion instead of DataClass ?
extension MapFromCsvExtension<TableDsl extends Table, D extends DataClass>
    on TableInfo<TableDsl, D> {
  Future<List<D>> loadFromCsv(File csvFile) async {
    if (csvFile == null || !await csvFile.exists()) return List();

    //Get all LINES from CSV Table
    List<String> csvRows = await csvFile.readAsLines(encoding: latin1);
    //Get all COLUMNS NAME from CSV Table
    List<String> csvColumns = CsvParser.decompose(csvRows[0], ';');

    if (csvColumns.isNullOrEmpty || $columns.isNullOrEmpty) return List();

    //each row  is a table data
    List<D> result = List();
    for (int row = 1; row < csvRows.length; row++) {
      //get row values
      List<String> values = CsvParser.decompose(csvRows[row], ';');

      //generate a data class for each row of data
      final typedValues = <String, dynamic>{};
      for (var i = 0; i < $columns.length; i++) {
        final localColumn = $columns[i];

        //escape the non existent columns
        if (!csvColumns.contains(localColumn.$name)) continue;

        //get cell value
        final cellValue = values[csvColumns.indexOf(localColumn.$name)];

        dynamic typedValue;

        if (localColumn is TextColumn) {
          typedValue = cellValue;
        } else if (localColumn is BoolColumn) {
          typedValue = cellValue == 'true';
        } else if (localColumn is IntColumn) {
          typedValue = int.parse(cellValue);
        } else if (localColumn is DateTimeColumn) {
          typedValue =
              DateTime.fromMillisecondsSinceEpoch(int.parse(cellValue) * 1000);
        } else if (localColumn is RealColumn) {
          typedValue = double.parse(cellValue);
        } else if (localColumn is BlobColumn) {
          // represent blobs in CSV
        }

        typedValues[localColumn.$name] = typedValue;
      }

      result.add(map(typedValues));
    }
    return result ?? List();
  }
}

In moor 3.1 (which will hopefully release later today, fingers crossed), you can call toCompanion(false) to convert a data class into a companion.


Dashboard Color Scheme keeps changing every refresh

Bug description
I've created a dashboard with the color scheme set as Superset Colors.
But every time I hit refresh, the colors of the dashboard change, however the color scheme does not change in the dashboard properties.
How to reproduce the bug

Click on 'edit dashboard' on any existing dashboard
Change color scheme to 'Superset Colors'
Save dashboard
reload the page

Screenshots/recordings
No response
Superset version
4.0.2
Python version
3.9
Node version
16
Browser
Chrome
Additional context
No response
Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

@dosu my labels are dynamic in nature. I'm unable to define the entire list of labels

CC @geido


[stable/owncloud] bump dependency mariadb 1.0.2

See https://github.com/kubernetes/charts/issues/1944#issuecomment-327856581


Corrected plugin documentations based on Issue #6752 (#6798)

backport #6752

Coverage remained the same at 78.215% when pulling b208b10bc221d65e3530e1a83f0e708c8563b142 on FinalAngel:release/3.7.x into 877c5c271198f32642333d5f9d59515ba8931836 on divio:release/3.7.x.


spirv-fuzz: Move ApplyTransformation to .cpp file

Sometimes, you need to change these functions during debugging (e.g., figure out why the transformation is inapplicable). When that happens, you need to recompile the whole fuzzer just because these functions are in the header file. This PR fixes the situation.

@afd ping.

@afd Can be merged.

@afd Can be merged.


Modify the artifact manager to receive magnet links and use them to receive artifacts.

This is to be done after #278
This task is to modify artifact manager to receive magnet link from a pubsub mechanism and use them with Synapse to download artifacts. This processing of magnet links should happen asynchronously of any incoming warp requests.

Closing due to bittorrent stories being out of scope / replaced with bitswap.


deprecate duplicate legacy dashboard

What does this PR do?
This is to deprecate the duplicate Legacy Kong dashboard.
Motivation
While checking the Kong dashboards to build an OpenMetrics version, I noticed that there seemed to be a duplicate OOTB dashboard for Legacy metrics. Got confirmation from Shri that this dashboard can be deprecated.
Jira card
Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Closing in favor of https://github.com/DataDog/integrations-core/pull/12811


Generated API should use async/await

To keep in line with the current node js versions, move the generated API code from Promises to Async / Await.

fixed by #97


CONFIG_RAM_SIZE may not be correct way to determine RAM size

See https://github.com/PX4/Firmware/pull/6451#pullrequestreview-18961944

@LorenzMeier In thinking about the use of this
Should we consider adding system integrity vault.
There are certain assets stored in ram that are critical and need to be tested for integrity at runtime.
Assuming we consider the OS+NSH vetted (if not we add CONFIG_BOARD_INITIALIZE and call it earlier from board_initialize),  in board_app_initialize we call system_integrity_init() this will fetch the os structs we want to check for corruption. I.E. the results mem.arena from mallinfo. etc  We save the values in the vault and crc it.
system_integrity_init() - populates the vault from compile time list of assets.
system_integrity_verify() can get assets again and check against the vault  (done in pieces if necessary)
system_integrity_verify_vault() - check the crc of the vault.
system_integrity_verify_item(enum handle, *void value) - checks the value return 0 for OK and -1 not OK
Then from places like  we can call system_integrity_verify_item(INTEGRITY_MEM_HEAP, &mem.arena).  If the result is not 0 call system_integrity_verify_vault() if it is OK then the heap is corrupted if it is not things are worse

Yep, that makes perfect sense. Can you add that to your list?

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 30 days. Feel free to reopen this issue if you deem it appropriate.
(This is an automated comment from GitMate.io.)


chore(main): release 1.0.10

:robot: I have created a release beep boop
1.0.10 (2024-10-04)
Features

add missing getAllCharacters call, reorganize chars & achievements (6ac276f)
attempt to auth automatically when needed (e611e9c)
characters: add getAllLogs, extract create and destroy data (2a41c19)
support specifying HTTP timeout (#6) (e3c34d2)
update gen'd types, add new task endpoints and unpaged reqs (#18) (8eb05e0)

Bug Fixes

account bank: fix http method in getDetails() (8f17997)
account endpoints: extract message from create() and changePassword() responses (9f9cba2)
auth no longer required in config (#28) (fe6ad3c)
extract maps.get() data property from response like all other infoCalls (1c15499)
getToken() (6ac1dba)
make node-inspect-extracted prod dependency (#14) (97461a8)
missed loading tasks info into client (#27) (225532d)
only assert needed path tokens (#10) (2a116b8)
remove redundant exported ArtifactsApi type (#19) (0f658b5)
remove redundant exported ArtifactsApi type (#21) (a1e1daf)
return type in meta.getLeaderboard() (3740a11)
still struggling with exporting class against type (#23) (f7b7a05)
typos affecting info.tasks.getRewardsPage and info.monsters.* (#25) (aa9df7e)

Documentation

fix publish badge (#8) (7b8a473)

Miscellaneous Chores

release 1.0.1 (#4) (baf47de)

Code Refactoring

move types (#16) (082cf0c)

Build System

enable noUncheckedIndexedAccess (#30) (244455f)

This PR was generated with Release Please. See documentation.

The creation of this release seems to be a bug as mentioned here: https://github.com/googleapis/release-please/issues/2384


impress service should have unix line endings

I had to convert the service script /etc/init.d/impress with dos2unix otherwise it would not run.

How do you install it? Using github or npm? How it can work for me on CentOS, Ubuntu and Debian and don't work for you? I think npm converts line endings in all sources and scripts, but I'll check it.

I install it, like you have it in the centos6x64.sh. I just did the last steps, as I had Node and npm installed already. For mongo I followed the instructions on the mongodb page, which are similar to yours in that bash script. These are the steps I did manually:
mkdir /impress
cd /impress
npm install mongodb
npm install nodemailer
npm install websocket
npm install geoip-lite
npm install impress

Nothing special.


Retire variants?

#2 started out being mostly about removing the distinction between quantities and units. I was reluctant then, saying:

Separating units and quantities was a very conscious decision, and one that I am quite happy with and feel has worked quite well in practice (YMMV). Converting quantities to values is performed, by necessity, very close to NIST guidelines (see 7.1 and 7.8) and I think this is a good thing.

Now, in light of e.g. #38, I'm reconsidering. Perhaps the added complexity of separating 'Quantity' and 'Unit' is not worth the extra complexity (Variant phantom type) and occasionally convoluted workarounds (e.g. Prelude. uses) that it can cause.
Relaxing would allow some ugly stuff:
x = kilo (meter + mile)
y = second * x

But even the current design isn't perfect with regards to NIST, allowing, e.g.:
u = kilo (kilo (meter / second))

so I'm not sure it is worth defending.
I would like to hear if you, Doug, have had any more thoughts on this topic. I know that you have actually gone even further with the distinction in the named-units branch, and imagine the impact would be treating everything as a unit (i.e. keep tracking the unit name) until multiplying with a Num (e.g. with (*~)).

Closing this as adequately and more comprehensively discussed under #46 and #48.


Conda installation does not work.

Getting the following error.
Solving environment: failed

ResolvePackageNotFound:
  - libffi==3.4.2=h7f98852_5
  - python==3.7.12=hf930737_100_cpython
  - libstdcxx-ng==12.2.0=h46fd767_19
  - libgcc-ng==12.2.0=h65d4601_19
  - libsqlite==3.40.0=h753d276_0
  - _openmp_mutex==4.5=2_gnu
  - ca-certificates==2022.12.7=ha878542_0
  - readline==8.2=h8228510_1
  - ncurses==6.3=h27087fc_1
  - xz==5.2.6=h166bdaf_0
  - libzlib==1.2.13=h166bdaf_4
  - openssl==3.1.0=h0b41bf4_0
  - tk==8.6.12=h27826a3_0
  - libnsl==2.0.0=h7f98852_0
  - _libgcc_mutex==0.1=conda_forge
  - libgomp==12.2.0=h65d4601_19
  - sqlite==3.40.0=h4ff8645_0
  - ld_impl_linux-64==2.40=h41732ed_0

Even after removing =h*** suffix from the packages, some of the packages are not found.

I am a user who was able to implement this successfully. You can't actually install this in Mac (which I believe you did). You need somewhat Linux machine like CentOS or other distributions to be able to make this work.

@hiyin Thanks for your suggestion. I finally managed to run the whole pipeline on Ubuntu 22.04 in x64 Win10 VMware.
Any suggestions to run the entire pipeline natively on Mac or Windows? Just for reference:

Native M1 version anaconda on Mac, same error
Ubuntu 22.04 arm64 in Parallels Desktop, same error
Ubuntu 22.04 x86_64 with Rosetta in Parallels Desktop, rosetta error, and failed to install anaconda
Native Windows version anaconda on x64 Win10, same error

docker on Mac also failed

Thanks for your interest in our system. Unfortunately ScienceBeam only supports Linux. Let us know if you find a way to install ScienceBeam on other systems.

We now added a new system that supports texts as input:
Usage:
python main_from_text.py  # If you are using non-linux operating systems or want to generate feedback from the parsed paper in text format


RDL-operations for EOS-wallets.

#36
Overview

[x] I have added Read-operation for EOS-wallets.
[x] I have added Delete-operation for EOS-wallets.
[x] I have added List-operation for EOS-wallets.

Comments

From acid-state point of view, we store EOS-wallets as the same HdRoots as FO-wallets, so implementation of Read, Delete and List operations is almost copy-n-paste. That's why I put it in one PR.

@KtorZ I've added checks for HdAccounts in EOS-wallets, with corresponding errors.

@KtorZ All checks have passed (hlint is happy again).


Rendering issue at GithubAnalytics(MyStreak)

I have Identified a bug in streak  is sometimes coming and sometimes not coming when i had visited your profile.
I have checked all the alternatives but still coming in profile.

Yeah I am already faced this issue. It's not an issue in code It's happening because of the link is not supposed to render all the time. For more updates you have to check out this repo in frequent.


Default small value for PUREDNS_PUBLIC_LIMIT

Would it be nice to set PUREDNS_PUBLIC_LIMIT as small value like 500 as default, add new flag to unleash the unlimited power instead.
From my experience, i blow up many router. Not sure if thats common issue.
https://github.com/six2dez/reconftw/blob/5448e7cfd063ab107e2eacfc4e23143f22d0d670/reconftw.cfg#L152

Most of the ppl using reconftw uses it on VPS, where there is no problem with this value. That's why by default there is no limit on this tool.


Search preview endpoint

Related to https://github.com/CartoDB/product/issues/243
Added /api/v3/search_preview endpoint to use while the user types in the search box. It finds visualizations and tags, returning only the required information.
Parameters:

q: search pattern (mandatory)
limit: max results. 4 by default (first all the found tags and then visualizations)
types: "derived,table,remote,tag" by default

Response example:
{
  "result": [
    {
      "type": "tag",
      "name": "spain",
      "url": "http://user.carto.com/dashboard/search/tag/spain"
    },
    {
      "type": "derived",
      "name": "Madrid traffic",
      "url": "http://user.carto.com/viz/107b672b-6f57-4a9e-b982-bbf1e041124a/map"
    }
  ],
  "total_count": 2
}

It includes visualizations/tags owned by or shared with the user, with any kind of privacy and locked/unlocked.

Acceptance is OK! 👌


Typo fix in concourse/README.md

line 66: tables lists->table lists
There are many such errors in stable directory.

/assign @william-tran

/ok-to-test
/lgtm


ZNE works with dynamic one shot

Context:
ZNE is not working with dynamic one shot, because it does not support to be applied on a classical function containing QNodes.
Description of the Change:

Add Function on dynamic_one_shot for a correct tracing.
Add the case of boolean constant of a JaxPrs in the lowering.
Remove the only QNode restriction.
Update the MLIR lowering to handle the case of ZNE on classical function containing QNodes calls. When folding the classical functions, the folding is applied individually on each qnode. Later when looping over scale factor we call the folded classical function containing calls to the folded qnodes.

Benefits:

ZNE is can be applied on any classical functions calling QNodes.
ZNE can be run with dynamic one shot.

[sc-68756]

@dime10 yes I will work on that


Asynchronous & long running functions

In the MVP, we are limiting our active support for immediate response functions.
However, we will need to work through how to pass appropriate console targets so that long-running asynchronous javascript methods can run and output can reasonably be directed to the associated console div. I still see that as ONLY maintaining the console & error output, NOT automatically doing any other IPC kinds of things.
If a user really wants to, they can spin up a socket or http endpoint, even within a code section, but we don't do anything other than maintaining the process. FWIW, this is a must-have, just not MVP. When interacting with blockchain stuff, the delays to verify or post transactions can be significant, so its pretty vital for our catalyzing use case.
Let's discuss how we might do that.
My initial thought is that each code block is always executed with a custom console.log function, which directs the output to the correct div. That function should continue to work even when other code blocks are executed.
Note that all code blocks will/should share the same compute context and hence, could block each other. That's just standard node/js stuff.

Btw, in this case, I can easily see separate sections that each spin up a long-running asynchronous function, with suitable console output directly below that code block, then students can run code that pings against those processes externally.
If you mess it up, you can reload. If you want to exclude some sections from execution,that should be easy (see #12)
Actually, that sounds like an awesome way to build out microservices.

Here is the core of the Jupyter javascript kernel - it's worth looking at for architecture suggestions
https://www.npmjs.com/package/jp-kernel
and it really comes down to this file:
https://github.com/n-riesco/jp-kernel/blob/master/lib/jp-kernel.js
I think this is an example of the 2nd landing point after the MVP.  In fact, we can give satyrn notebooks a mime type and then they can be fully integrated in the jupyter ecosystem as well as in our own viewer.
If we follow this pattern, we also get an API that supports all of our needs, and we get a javascript runtime out of the box.
In essence:
Stage 1: MVP => electron managed subprocess linked to an open doc.  subprocess coupling is via stdin/stdout/stderr, using the out-of-the-box event model from node's child_process package.
Stage 2:Replace the stdin/stdout/stderr model with a kernel adapter, which handles the features we need - we just need to connect it to our UI.  If we package up our satyrn notebooks in a mime-type and provide a React component (the browserWindow application) you've got something particularly powerful:
a) a re-usable UI that can participate in an existing ecosystem
b) a desktop educational app tailored to and directly supporting a specific, known, userbase (the RWoT, MyData, IIW, DIF, SSI, jada jada community of which we are members)
The reason to separate them into stages is the devops overhead of the build in switching to case 2 - we need solid control over the build-and-distribute pipeline, and while that is straightforward, it takes a while to spin up, Stage 2 is really about stabilizing that against a relatively mild, but deep, formalization of the UI / kernel relationship.
BTW - i recommend calling the sub-processes Kernels - here is the definition from Jupyterland: A notebook kernel is a “computational engine” that executes the code contained in a Notebook document  multi-kernel support is under active development - e.g. connecting to different engines w/in the space of a single document.  The challenge is mostly the UX of managing that relation - and shifting from a "document peers w/ engine" model to "document peers w/ engine-map"
it is also worth noting that ZMQ is pretty well positioned to bind into DID-based connection - and that makes this whole model feel a lot like Agents - which is pretty cool.  An opportunity for a bit of dogfooding for the broader community w/o blocking progress.
but first Stage 1! ;)
then some armor plating of the CI/CD pipeline (i suspect driven by the need to use gyp - although it would be nice to avoid that step)

/move to satyrnjs/satyrn