Integrating Turing and MarginalLogDensities

Moving a discussion with @yebai from Slack to here. @PavanChaggar asked if there was a way to do Laplace approximation in Turing, and I gave this little example of how it could be accomplished with MarginalLogDensities.jl:
using MarginalLogDensities
using ReverseDiff, FiniteDiff
using Turing

# simple density function
f(x, p) = -sum(abs2, diff(x))
# define a MarginalLogDensity that integrates out parameters 5 thru 100
mld = MarginalLogDensity(f, rand(100), 5:100, (), LaplaceApprox(adtype=AutoReverseDiff()))

@model function MLDTest(mld)
    θ ~ MvNormal(zeros(njoint(mld)), 1.0)
    Turing.Turing.@addlogprob! mld(θ)
end

mod = MLDTest(mld)
chn = sample(mod, NUTS(adtype = AutoFiniteDiff()), 100)

This issue is to discuss how this capability might be integrated better into Turing, probably via a package extension. (See also https://github.com/TuringLang/Turing.jl/issues/1382, which I opened before I made MLD). From a user perspective, an interface like this makes sense to me:
@model function Example(y)
    a ~ SomeDist()
    b ~ AnotherDist()
    mu = somefunction(a, b)
    x ~ MvNormal(mu, 1.0)
    y ~ MvNormal(x, 1.0)
end

fullmodel = Example(ydata)
marginalmodel = marginalize(fullmodel, (:x,))
sample(marginalmodel, NUTS(), 1000)
maximum_a_posteriori(marginalmodel, LBFGS())

I think there are two basic ways to implement this:

marginalize constructs a new, marginalized DynamicPPL.Model, or
It returns a MarginalLogDensities.MarginalLogDensity, with new methods for sample, maximum_a_posteriori, and maximum_likelihood defined for it.
I'm not very familiar with Turing's internals, so happy to be corrected if there are other approaches that make more sense....

The other current roadblock is making calls to MarginalLogDensity objects differentiable (https://github.com/ElOceanografo/MarginalLogDensities.jl/issues/34). This is doable, I just need to do it.

This looks neat 👀
Can you elaborate a bit on what mathematical model the following line should correspond to?
marginalmodel = marginalize(fullmodel, (:x,))

Would this marginalize out x?
(And MarginalLogDensities.jl looks really nice btw. If people are interested in model comparison, this will probably be a place we'll point people:))

IIUC you effectively want marginalize(...) to result in
$$p(y, a, b) = \int p(y, a, b, x) dx = \bigg( \int p(y \mid x) \times p(x \mid a, b) dx \bigg) \times p(a) \times p(b)$$
with the Laplace approx. replacing the integrand on the RHS?
But does this mean that you want to drop both the terms $p(y \mid x)$ and $p(x \mid a, b)$ in the original joint model, and replace is all with a simple addlogprob! call? I.e.
@model function MarginalizedExample()
    a ~ SomeDist()
    b ~ AnotherDist()
    Turing.@addlogprob! MarginalLogDensity(...)
end

?

Yes, your math is correct. If f(u) is a function that returns the log-posterior, and u = [a; b; x], then calling mld = MarginalLogDensity(f, u, i_x, y) makes a callable struct. When you call mld(v) (where v corresponds to  [a; b] == u[setdiff(eachindex(u), i_x)]), it will find the mode of $p(x | a, b, y)$ and the curvature at that mode, and use that Hessian to calculate the LA and integrate out $x$.
To do this using @addlogprob!, you'd need to write your own log-density function (as a function of all the parameters and data), wrap it in a MarginalLogDensity to integrate out $x$, and pass that into a Turing model, similar to my first example. That's basically possible now, but requires writing the log-posterior function from scratch.
To do it the other way, starting with a Turing model, my hypothetical marginalize function would do something like:

Convert the DynamicPPL.Model to something that takes a flat parameter vector and returns a log-probability (a LogDensityFunction ?  OptimLogDensity? Something else?)
Get the indices of the variables to marginalize within that flat parameter vector
Use 1 and 2 to construct a MarginalLogDensity, or maybe a new TuringMarginalLogDensity type that stores some extra metadata
Pass that to specialized methods for sample, maximum_a_posteriori, etc.
Do any extra bookkeeping needed at the end to report the report the parameters with their correct names, etc.

All this should be basically possible, no?

I guess I'm not clear why any of the internal details of the model need apply here...if a Turing model can be turned into a black-box function $p([a, b, x] | y)$  for the purposes of optimization or sampling, then the same function should work in a MarginalLogDensity.
Here's a concrete example of what I'm suggesting:
using Turing
using Distributions
using MarginalLogDensities
import Zygote

a = 0.5
b = 3
c = 2
d = 3
s = collect(-5:0.5:5)
μ = a.*s.^2 .+ b.*s
x = rand(MvNormal(μ, c))
y = rand(MvNormal(x, d))

@model function demo(y, s)
    a ~ Normal(0, 10)
    b ~ Normal(0, 10)
    logc ~ Normal()
    c = exp(logc)
    μ = a.*s.^2 .+ b.*s
    x ~ MvNormal(μ, c)
    y ~ MvNormal(x, 3.0)
end

m = demo(y, s)
ctx = Turing.Optimisation.OptimizationContext(Turing.DefaultContext())
old = Turing.Optimisation.OptimLogDensity(m, ctx)

# marginalize the `x` variables, indices 4:24
mld = MarginalLogDensity((u, p) -> -old(u), randn(24), 4:24, (), LaplaceApprox(adtype=AutoZygote()))
mld(randn(3))

I guess I'm not clear why any of the internal details of the model need apply here

Ah sorry, that probably wasn't clear from my end :grimacing:
I was thinking of more convoluted scenarios, e.g.
@model function demo()
    s ~ InverseGamma(2, 3)
    x ~ Normal(0, sqrt(s))
    y ~ Normal(0, sqrt(s))

    return (s, x)
end

model = demo() | (y = 1.0,)  # model over `s` and `x`
model()  # => (s = rand(InverseGamma(2, 3)), x = rand(Normal(0, sqrt(s)))

marginalized_model = marginalize(model, @varname(x))  # model is now only over `s`
marginalized_model()  # => (s = rand(InverseGamma(2, 3)), x = argmax(logjoint(model, x)))

where argmax(logjoint(model, x)) is representing the MAP of the posterior. Similarly, log-probability computations, e.g. logjoint would also be altered to include the normalisation constant from the Laplace approx.
Does that make sense? It was a bit unclear to me how to do this + whether someone would actually want to do this.

Turing model can be turned into a black-box function

But yes, this would be very easy to do:)
using Turing, MarginalLogDensities, LogDensityProblems, Zygote

"""
    marginalize(model::Model, varnames::Vector; method=LaplaceApprox())

Returns a `MarginalLogDensity` with `varnames` marginalized out from the `model`.
"""
function marginalize(model::DynamicPPL.Model, varnames::Vector; method=LaplaceApprox())
    # Determine the indices for the variables to marginalise out.
    varinfo = DynamicPPL.typed_varinfo(model)
    varindices = DynamicPPL.getranges(varinfo, varnames)
    # Construct the marginal log-density model.
    # TODO(torfjelde): Should link and use optimization context to avoid inclusion jacobian corrections to the log-density.
    f = Base.Fix1(LogDensityProblems.logdensity, DynamicPPL.LogDensityFunction(model))
    mdl = MarginalLogDensity((u, p) -> f(u), varinfo[:], varindices, (), method)
    return mdl
end

a = 0.5
b = 3
c = 2
d = 3
s = collect(-5:0.5:5)
μ = a.*s.^2 .+ b.*s
x = rand(MvNormal(μ, c))
y = rand(MvNormal(x, d))

@model function demo(y, s)
    a ~ Normal(0, 10)
    b ~ Normal(0, 10)
    logc ~ Normal()
    c = exp(logc)
    μ = a.*s.^2 .+ b.*s
    x ~ MvNormal(μ, c)
    y ~ MvNormal(x, 3.0)
end

model = demo(y, s);
mdl = marginalize(model, [@varname(x)]; method=LaplaceApprox(adtype=AutoZygote()));
mdl(randn(3))

At the moment, this requires a "bugfix" to DynamicPPL.jl (I'm using an internal function in a way that isn't quite meant to be used this way currently):
# FIXME(torfjelde): This is an internal function that isn't CURRENTLY meant to be used in this way,
# but we can add the following missing def to make it work.
# It takes in a vector of variable names and returns the corresponding indices.
function DynamicPPL.getranges(varinfo::DynamicPPL.TypedVarInfo, vns::Vector{<:DynamicPPL.VarName})
    # Here we need to keep track of the offset.
    offset = 0
    vns_all = DynamicPPL.keys(varinfo)
    return mapreduce(vcat, vns_all; init=Int[]) do vn
        # First we need to get the range so we can add it to the total offset.
        r = DynamicPPL.getrange(varinfo, vn)
        length_vn = length(r)
        # Then we check if `vn` is one of the variables we're extracting the ranges for.
        # TODO(torfjelde): Maybe use `findall` + `subsumes` instead?
        index = findfirst(isequal(vn), vns)
        return if index === nothing
            # If none exist, we shift the offset and return an empty array.
            offset += length_vn
            Int[]
        else
            # Otherwise, we return the (offseted) range and update the offset.
            r = r .+ offset
            offset += length_vn
            r
        end
    end
end

Btw, it's quite unfortunate that we have to use Zygote.jl here to get 2nd order information :confused: It's really not well-suited for Turing.jl models.

Oh, and if you wanted to use this for sampling, you could just add the following:
#########################
## To sample from this ##
#########################
# 1. Add LogDensityProblems.jl interface to it and we can suddenly use samplers.
LogDensityProblems.logdensity(mdl::MarginalLogDensity, u) = mdl(u)
LogDensityProblems.dimension(mdl::MarginalLogDensity) = length(mdl.iv)
function LogDensityProblems.capabilities(mdl::MarginalLogDensity)
    return LogDensityProblems.LogDensityOrder{0}()
end

# 2. Unfortunately, we have to use the sampler packages explicitly.
using AbstractMCMC, AdvancedMH
spl = AdvancedMH.RWMH(LogDensityProblems.dimension(mdl))
samples = sample(
    mdl, spl, 1000;
    chain_type=MCMCChains.Chains,
    # HACK: this a dirty way to extract the variable names in a model; it won't work in general.
    # But general methods exist, so we can fix that.
    param_names=setdiff(keys(DynamicPPL.untyped_varinfo(model)), [@varname(x)])
)

Results in
julia> samples = sample(
           mdl, spl, 1000;
           chain_type=MCMCChains.Chains,
           # HACK: this a dirty way to extract the variable names in a model.
           param_names=setdiff(keys(DynamicPPL.untyped_varinfo(model)), [@varname(x)])
       )
Sampling 100%|█████████████████████████████████████████████████████████████| Time: 0:00:03
Chains MCMC chain (1000×4×1 Array{Float64, 3}):

Iterations        = 1:1:1000
Number of chains  = 1
Samples per chain = 1000
parameters        = a, b, logc
internals         = lp

Summary Statistics
  parameters       mean       std      mcse   ess_bulk   ess_tail      rhat   ess_per_sec
      Symbol    Float64   Float64   Float64    Float64    Float64   Float64       Missing

           a     0.6280    0.3893    0.0576    40.8026    18.2906    1.4236       missing
           b   -15.2641    4.2187    2.0435     4.2878     4.6395    1.4834       missing
        logc     1.0824    1.1025    0.5967     4.0816     4.4534    1.4062       missing

Quantiles
  parameters       2.5%      25.0%      50.0%      75.0%     97.5%
      Symbol    Float64    Float64    Float64    Float64   Float64

           a    -0.6893     0.6643     0.6970     0.6970    1.3287
           b   -17.2110   -17.1433   -17.1433   -16.4625   -2.2291
        logc     0.4505     0.4505     0.4505     1.2029    3.7519

Cool, thank you for the clarification, I knew I was missing something! I would not have thought to try using any of the fancy conditioning syntax. Making a Turing model into a MarginalizedLogDensity is very analogous to making it into an OptimizationProblem, and I don't think users would expect any of the more advanced conditioning stuff to work.
And thanks for the worked example. I was trying to figure out how to get the variable names and indices automatically but couldn't do it myself. This feels like a good outline of a usable extension...does it make more sense as an MLD extension in Turing, or a Turing extension in MLD?

Btw, it's quite unfortunate that we have to use Zygote.jl here to get 2nd order information 😕 It's really not well-suited for Turing.jl models.

We shouldn't need to, MarginalLogDensities is totally backend-agnostic (in theory anyway).

I was trying to figure out how to get the variable names and indices automatically but couldn't do it myself. This feels like a good outline of a usable extension...does it make more sense as an MLD extension in Turing, or a Turing extension in MLD?

I think, given that it returns a MLD object, that it makes more sense as an MLD extension maybe?
And yes, I do think this is pretty close to being a functional extension:) If you want, I'm happy to make a PR or you can make a PR and I can support?
The only annoyance is that you end up touching some internals, e.g. OptimizationContext, which means that the extension could technically break even when Turing.jl makes non-breaking releases :confused:

We shouldn't need to, MarginalLogDensities is totally backend-agnostic (in theory anyway).

Yeah no I saw this, which is very nice:) It was more a comment on the AD packages; I tried both ForwardDiff.jl and ReverseDiff.jl, but both failed upon hessian computation.

I think, given that it returns a MLD object, that it makes more sense as an MLD extension maybe?

That's kind of what I was thinking as well. If there are ways to make the interface a bit more robust to changes in those Turing internals that would be great, but MLD is an experimental package with few users at the moment, so I'm okay with a moderate risk of breakage. If you want to make a PR go for it, otherwise I will try to get to it in the next week or so.

Yeah no I saw this, which is very nice:) It was more a comment on the AD packages; I tried both ForwardDiff.jl and ReverseDiff.jl, but both failed upon hessian computation.

Try again with ForwardDiff, there was a bug that just got fixed here: https://github.com/ElOceanografo/MarginalLogDensities.jl/pull/36. But in general yes, things are a bit more brittle in practice than they should be in theory...

If there are ways to make the interface a bit more robust to changes in those Turing internals that would be great,

Yeah, maybe it's better to do it as an extension to Turing.jl given that it access Turing.jl internals :thinking:  I think the likelihood of us breaking the functionality is greater than you doing so, so seems sensible to put it in Turing.jl (at least for now).
I'll open a PR with some minor tweaks to the above then :+1: Buuut will likely need some help to get it merged (a bit limited on time these days); maybe you could help add some tests @ElOceanografo ?:)

Yeah, definitely. Just ping me when you've got a PR and let me know where you'd like the tests to go.

Done! See #2421:)


Feature image wraps around on mobile

Just like the title states - feature images for blog posts can wrap if viewed on mobile:

Is there a way to clamp them instead, so they stop scrolling when they reach the edges?

Yeah that's a tough issue this one, it mostly depends on the image size.
It would need some more javascript for it. As of now it is done via CSS.

I see... I didn't even know you can do this without JS!

Yes whenever the image is not large enough on small scale it will do that (for example if you reduce the size of your browser it should happen).
Background-repeat will do as in the picture according to me, unless I misunderstood what you wanted to do with it? (or you meant to stop the background repeat?)

@Sylhare Re: "Yes whenever the image is not large enough on small scale it will do that (for example if you reduce the size of your browser it should happen)."
Looking through the documentation, I didn't see an ideal/recommended image size or proportions for the features. Do they exist? Thank you.

Ah True, I haven't documented that part, I will do that asap ;)

Is there any way to enlarge the avatar picture a little bit.
Don't know if this is the right place to ask.

Hi @ritajitk for the avatar picture, you can open a new issue.


Infused slimes

What will this feature do?
I think if the glitch slime maker can infuse a slime with a report of a bug, it might be able to infuse other things too,
imagine infusing a slime and some bread, bread slime? or it could work for already existing slimes like take a rock material then infuse it with a slime, you got a rock slime. this could work for making new slimes not able to be found in the wild and only made or / and it could work to make slimes that already exist, put in a strange diamond, and a slime bang crystal slime. more on the "a slime" thing, I think you could make more special nonwild slimes if you keep combing on a string of infused slimes, an example that would work for non-natural slimes too. ok so let's say you want a crystal slime, you can't just shove a strange diamond and a pink slime in there, you gotta make / find a rock slime and then put the rock slime with the strange diamond. you might think this isn't very useful but if you have just started out and have not gotten the indigo quarry it would be a way to get crystal slimes, this would take a bit of creative thinking for a lot of slimes like a rad slime, you can't put in just pure radioactive energy, so yeah it has a few problems and for abuse like putting in a rock slime and some graviton energy, it could only make a meteor slime if you put in a moon material to upgrade it to take in graviton energy, so you would have to get certain things before you can do that making it more of a quality of life since you don't gotta go all the way to the moon rather than a way of getting rare slimes instantly. as another idea, it could also be used to make normal slime sized largos and slime sized tarr since if you put in 2 slimes then it would make a largo that only has the DNA of both and not the mass. this post is too long to continue for foods so ill go and make a new post
How it should integrate into the mod? (Optional)
After getting Viktor more glitch slime plorts 5 days after he would show you his new infuser that can do the job of real glitch slime maker AND infuse slimes
Possible names for the feature? (Optional)
Fusions and lab-grown slimes
Additional context

Other infusions are already possible in the infuser, nothing like you are describing in most cases, there will be no actual fusion of slimes besides making glitch slimes and possibly digi-tarrs. Anything else is planned in a way or another.
Thanks for the suggestion though.


Sentry: Events query support

Adding support for https://docs.sentry.io/api/discover/query-discover-events-in-table-format/
More details :
https://raintank-corp.slack.com/archives/C02G20APV36/p1644914434851329?thread_ts=1644861430.433519&cid=C02G20APV36
https://raintank-corp.slack.com/archives/C02J0L4V4QL/p1653992132542589

duplicate https://github.com/grafana/sentry-datasource/issues/37


MINOR: Reduce log level to Trace for fetch offset downgrade

The log for fallback isn't very helpful for debugging purpose, reducing to a trace log.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

LGTM!

Cherry-picked to 2.5.


New and improved library and macro API

Library API
pub trait RlpEncodable: RlpLength {
    fn rlp_encode<T: BufMut>(&self, encoder: &mut Encoder<T>);

    fn rlp_encode_raw<T: BufMut>(&self, encoder: &mut Encoder<T>);

    fn rlp_len(&self) -> usize {
        let raw_len = self.rlp_len_raw();
        length_of_length(raw_len) + raw_len
    }

    fn rlp_len_raw(&self) -> usize; // Important: no default!!
}

// Might not be needed.
// pub trait DynRlpEncodable: RlpLength { /* ... */ }
// pub trait RlpLength { /* rlp_len, rlp_len_raw */ }
// impl<T> DynRlpEncodable for T where T: RlpEncodable { /* ... */ }

pub trait RlpDecodable<'de> {
     fn rlp_decode(decoder: &mut Decoder<'de>) -> Result<Self>;

     fn rlp_decode_raw(decoder: &mut Decoder<'de>) -> Result<Self> {
          Self::rlp_decode(decoder)
     }
}

pub struct Encoder<T: BufMut> {
    out: T,
}

// `struct Rlp` is replaced with `Decoder`.
pub struct Decoder<'de> {
    buf: &'de [u8],
}

impl<'de> Decoder<'de> {
    // Methods moved from `Header` ...
}

Other, not pictured:

[ ] Add trait implementations for tuples, with the same behavior as structs with derived impls

This with the improved macro attributes should remove the need of dyn Encodable, enabling use of a generic Encoder.

Improvements:

trait and method names prefixed with rlp to avoid conflicts on common types
separate methods for "raw" encoding/decoding, meaning without header, to allow #[rlp(flatten)], see below

Derive macros
New attributes:

[ ] (field) #[rlp(flatten)]: encode and decode using raw
[ ] (container) #[rlp(tagged)]: impls for enums

tagged: defaults to the variant's discriminant value, or the variant can have an #[rlp(tag = <expr>)] attribute.

Example: P2PMessage

not specifying this fails, to leave room for future enum representations

[ ] (container) #[rlp(transparent)]: replaces the separate *Wrapper macros

also account for #[rlp(skip)] in field number calculation

Hi can I take on this ?

Before assigning this, can you unpack a little more about how you're planning to approach this? Dani wrote a great initial spec above, worth splitting this into sub-tracking issues and sub-tasks that you may own and we can split with others.

A. Trait and Struct Implementations
RlpEncodable and RlpDecodable Traits:
Implement basic methods and ensure compatibility with tuples  (this could be in the form of derivable macros that output an trait impl  so it can be used for other structs  or just a native impl block)
Provide detailed documentation and examples for each method.
B. Macro Attributes and Derive Macros
Derive Macros for Auto-Implementation:
Develop macros that automatically implement the RlpEncodable and RlpDecodable traits for structs and enums.
Ensure that the macros handle various attributes correctly, such as rlp(flatten) and rlp(tagged).
Attribute Handling:
Implement and test new attributes like rlp(transparent) and rlp(skip).
Ensure proper handling of tagged enums and transparent wrappers.

@gakonst

@gakonst @DaniPopes ; is there any adjustments to be noted before taken on this ?


Seems like this only works if YAG has admin?

Just wanted to check if there is a reason why YAG will only function with these custom commands only if it has admin perms.
I get an error if I remove admin perms from YAG even if everything else is enabled.

hm I honestly don't know why that could be, I made these custom commands 2 years ago or so so maybe something was changed with yag ¯_(ツ)_/¯
I haven't been really using discord much except for chatting with friends, so I kind of "retired" from making CCs and such. I'm very sorry I can't answer your question, I'd need to revisit custom commands and YAGPDB and see all that was changed in the past 2 years. Maybe one of the VS knows why this problem is occuring.
I hope you have a nice day/evening and sorry again that I can't help with this :(


app-template: using chart in an air-gapped network

Details
The app-template chart currently uses a values.schema.json file that references the common library.
This reference ($ref) is a hard-coded link to the common library's GitHub repository.
As a result, the chart cannot be used in an air-gapped network without internet access.
I understand that the intention behind this implementation is to avoid duplication.
However, this renders the chart unusable in air-gapped environments.

That is a good point and to be honest I haven't thought about this use case. At the same time I'm not sure what a good solution would be that keeps the maintenance burden low as well. I'll see if I can think up something better


add optional template_path parameter to help template/output separation

I found it helpful to be able to maintain templates and "master"/output files in separate locations with an additional argument

@arc12, thanks for the PR. I apologize for neglecting it.
I usually keep nbrun in the same folder as template notebooks and save the output notebook in a different folder. We don't need the new argument template_path for this.
Your PR allows keeping nbrun in a single folder, while executing notebooks in different folders. This is useful in projects having notebooks in many subfolders. In this case nbrun doesn't need to be copied in each sub folder.
Also, the addition of the option to hide input cells and the addition of the path of the template notebook are a nice cherry on the cake.
Thanks :)
🍰


Add script for quick testing using FPGA SRAM only

Add a script that allows for quick turnaround testing of Icestation-32 software directly from SRAM, with no need for flashing, extra peripherals, or for special software running on the ESP32.

add a boot ROM variant with disabled IPL, boot_multi_noipl
make nextpnr output a JSON file ecp3s_pnr.json with placement data.
add script to inject a binary bootrom, and initial RAM contents into an
existing FPGA configuration.

I think it is quite elegant, everything considered.
Example:
ubuntu@build:~/icestation-32/software/hello_world$ make clean
rm -f prog.elf prog.bin main.o ../lib/vdp.o ../common/font.o     ../common/vectors.o ../common/start.o main.d ../lib/vdp.d ../common/font.d ../common/vectors.d ../common/start.d
ubuntu@build:~/icestation-32/software/hello_world$ time ( make && ../../scripts/run)
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -c main.c -o main.o
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -c ../lib/vdp.c -o ../lib/vdp.o
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -c ../common/font.c -o ../common/font.o
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -c ../common/vectors.S -o ../common/vectors.o
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -c ../common/start.S -o ../common/start.o
riscv-none-embed-gcc -std=c11 -Wall -Wextra -Os -MMD -MP -flto -march=rv32i -mabi=ilp32 -I../common/../common/ -I../common/../lib/ -I./ -ffreestanding -nostdlib -DASSERT_ENABLED -Wl,-Bstatic,-T,sections_p.lds,--strip-debug main.o ../lib/vdp.o ../common/font.o     ../common/vectors.o ../common/start.o -o prog.elf -lgcc
riscv-none-embed-objcopy -O binary prog.elf prog.bin
Running prog.bin (65540)...
Replacing BRAM data...
Running ecppack...
Uploading to SRAM through FTP...
Name (ulx3s.lan:ubuntu):
Completed, cleaning up.

real    0m5.639s
user    0m2.493s
sys     0m0.393s

Nice! This is great for fast iteration and not wearing the flash with quick repeated tests is another bonus.
For the optional IPL, I figure the skip-IPL variant can just be another target in boot.mk, then the boot ROM could be replaced entirely to avoid any error with patching. Looks like you've already done the work to enable that.

Nice! This is great for fast iteration and not wearing the flash with quick repeated tests is another bonus.

Yea exactly !

For the optional IPL, I figure the skip-IPL variant can just be another target in boot.mk, then the boot ROM could be replaced entirely to avoid any error with patching. Looks like you've already done the work to enable that.

Great idea !
I've added a new target boot_multi_noipl that passes a preprocessor flag BOOT_NO_IPL. This skips the IPL but otherwise initializes the flash in the same way (which is still useful to access sounds and such which tend to change less often).
The resulting binary is passed to the script along with the program. The script now replaces both the boot ROM and initial RAM contents.

I tried it out with the demos and it's much nicer even when just programming SRAM with fujprog. Only noticed a minor issue on my macos setup where wc adds leading spaces, which throws off the cut so SIZE ends up being empty. Looks like it doesn’t do that on Linux.
Running prog.bin ()...
../../scripts/run: line 24: [: -gt: unary operator expected

Just piping through xargs fixes it on my mac, if it doesn't cause issues on your end could it be updated with this? (or whichever preferred method of stripping leading whitespace).
SIZE=$(wc -c "${PROGFILE}" | xargs | cut -f 1 -d ' ')

Running prog.bin (65540)...
Replacing BRAM data...

Thanks for testing! I'm glad it works almost out of the box on MacOS as well.

../../scripts/run: line 24: [: -gt: unary operator expected

Ah yes the horrors of cross-platform shell scripting. When I realized it would be useful to check the size, then reading there are three methods for getting file size, all with support for different operating systems,  I regretted not writing it in Python 😅
In any case I've added in xargs , thanks for the suggestion.


AMY Arduino drum and fm examples on ESP32S3 cause restart

I'm experimenting with AMY and ran into the problems mentioned in the title. I'm building with PlatformIO using a project template that I've used many times before. The I2S device is a PCM5102A board which I got to work with some effort and the two sine wave and the voice_chord examples run as expected.
Since I'm using PlatformIO I ran the fm and drum examples under the debugger and found the source of the crashing for the fm one. This is because in algorithms.c at line 123 the order of the arguments is reversed from what is expected causing an array out of bounds error that is doing something bad. Changing that fixed the crashing but there is no sound generated.
I couldn't find the source of the crashing for the drum example so I built the desktop example for MacOS. In this case the drum example does run fine. The fm example however gives a sawtooth like sound without the fix and no sound when the fix is applied.
I wondering if the examples got out of sync with the library somewhere along the way? I did go through the commits over the past few months but couldn't see anything obvious.
BTW I really like the Tulip concept and have ordered a Tulip CC from Makerfabs. I also have some other things I would like to try with AMY itself or maybe a scaled down Tulip with no user interface except for configuration and control via MIDI.

Nice, thanks for checking out AMY!
I'll take a look at the Arduino examples asap. It's most likely they got out of sync -- we've been fully on Tulip the past couple of months and probably messed that up.
The FM example in examples.c was incorrect, that was a bug that popped up a couple of weeks ago and didn't check in examples.c. Try that again! (it's quiet but you should hear a bell-like tone)

Still no joy with getting sound from the fm example on either of my projects after replacing examples.c. There must be something else that needs changing.

Just to make sure, change the velocity to much higher -- it's a very faint tone and may not show up on your DAC. Like change line 321 of examples.c to e.velocity=50.0f;

I've confirmed that examples.c in both of my projects are the current one and changed the volume as suggested but no luck with either project. I downloaded the AMY repo a couple of days ago and was working with that but now I've done a git clone for my latest test. Tomorrow morning I will start fresh in case I've messed something up along the way :-)
Thanks for pursuing this!

I looked at this again to make sure I'm building the fm example with the modified examples.c and have added the velocity statement and there is no sound with either the esp32s3 or the MacOs application.

I've gone and made the amy-example binary play the voice chord (juno-6 patch 0) and then the constructed 2-op FM tone one after another (at a more reasonable volume.) On my Mac, when I run make in the AMY repo root, and then run ./amy-example, I hear a three note chord and then a bell-like FM tone at about the same volume. Can you try this and tell me what you hear on macOS? Then we can debug what's going on in Arduino.

No sound on Mac could be using the wrong audio device, I.e. needs -d 1 or something.

The MacOS version works perfectly now. I built my previous version using XCode and maybe I messed something up or Clang compiles something that is incompatible?
I tried again with Xcode and the updated amy-example.c and get no sound from the fm tone (but do get sound from the voice chord). I don't know why I used Xcode as I much prefer the Makefile/Gcc method. Sorry Brian if I sent you on wild goose chase with that!

Sorry I spoke too soon. In the Xcode project I had changed the call to note_on_mod() in algorithms.c to agree with it's definition and the fm tone worked.
Just to be clear, in algorithms.c at line 123 this is the call:
note_on_mod(synth[osc].algo_source[i], osc);
and the definition in line 99 is:
void note_on_mod(uint16_t osc, uint16_t algo_osc)
which I thought was causing the crash in the Arduino example but perhaps it is something else.

The note_on_mod stuff seems correct to me. algo_osc in this instance is the "base" oscillator, the "control" oscillator that you send notes to. In the examples.c FM example, that's osc = 2. algo_source (not algo_osc) is the list of source oscillators (aka operators) that make up an ALGO voice. In algorithms.c's algo_note_on we take in the algo_osc as osc, and run through its set algo_sources to start sine note ons per operator. We pass note_on_mod the operator oscillator number as well as the algo_osc (base osc).

Yes everything is fine with the MacOS version now.
As for the Arduino example when I used the debugger I saw that the first parameter which is an index into the array was 0xFFFF so I thought that the order of the parameters was wrong but I guess I just jumped to that conclusion because of the naming. Probably a red herring.

ok great!! When I'm back up at my studio this week I'll test some Arduino examples. Thanks for your help, I'll be back!

Anything I can do to help let me know. I am slowly figuring out how things work and loving it! Both AMY and Tulip are amazing projects (and of course Alles too although I haven't spent much time with it yet).

Also pasting in a similar report from discord user mathias:

Cool! I guess I may have found the issue (but sadly not fixed yet) with "amy.drums(clock+5000, 2); " not working with the ESP32 example.
I am afraid that currently no time-based queuing of amy-events is working at all. So either the long clock = amy.sysclock();  does not work as expected or when in the main loop "short * samples = amy.render_to_buffer();" is execuded any e.time events are ignored.
Whenever I add events they are (at least that is what I kind of observe) played "at once" and for some reason with amy.drums() that is too fast (or similar) and crashes the program (when I said everything was up and running, I maybe was a bit to optimistic, because at first I always tested with one note only and not sequences of notes, yet 😉
I suspeced that there may be some kind of #define constant not set correctly and tried out ESP_PLATFORM, but that seemingly is alread set

I'll be taking a look at this stuff this week.

OK, I think I fixed the drums restarting. See #142  but we were not resetting the oscs before changing the wave type. It's not clear why this crashes only on ESP32, but it's not something you're meant to do, so I just added an explicit reset in example_drums and in the ESP32 Arduino example.
I've also tested the (Now louder) example_fm example, and everything seems good on ESP32. Thanks to the discord forum member Mathias for help on the newer ESP_I2S setup we need to do as well.

pcm.c:23 checks for logfreq[COEF_CONST] <= 0 which looks like a hangover from the pre-log-freq days. So I suspect it's the freq=220 rather than the wave=SINE that's triggering the crash. It would be interesting to see if freq=440 fixed the original problem (since I think the transition from negative to positive logfreq is at 261 Hz).
I'm not clear about the intent of the logic in pcm_note_on(), but it may be that we want == instead of <=.


WIP: DBZ-1292 Add option to export change events in "CloudEvents" format

This PR is work-in-progress
Issue: https://issues.jboss.org/browse/DBZ-1292

A new connector configuration field cloudevents.format. When it's set to true, connector will produce source records in CloudEvents format.

A new CloudEvents formatter for SQL Server connector.

Hey @Wang-Yu-Chao, thanks a lot for opening up this PR! Could you perhaps provide some description of its current state, what already works, where you maybe are struggling etc. That'd help a lot to get a better grasp for this. Btw. I just was asked the other day whether/when Debezium would support CloudEvents; so this will definitely be a very welcomed addition. Thanks a lot!

Hey @Wang-Yu-Chao, howdy? Should we have a quick call perhaps on where we are on this one? Would love to see this completed; if you don't think you'll get to it any time soon, then that's no problem either, someone else can take over. Just let us know :) Thanks a lot!

Hey @gunnarmorling! Sorry for the late reply. I've been applying for a job recently and haven't got much time on the project. Please give me some more time and I shall complete this asap based on the progress I made so far.

@Wang-Yu-Chao, no worries, all the best for your job hunt!
On the PR itself: perhaps you can describe where we are atm. and what's missing? As said, also very happy to do a call, if you think it'd help.

I first want to make it a converter but turned out very complicated. It's bad to tie formatting with converting/serializing in my opinion. So I took use of the ChangeEventCreator interface to format  SourceRecord before it's changed into DataChangeEvent.

Hey @Wang-Yu-Chao, just saw you pushed an update for this, very nice! Should we have a call some time next week to synch on this one? It seems the design has changed a bit, and I feel I'm a bit behind in terms of what's happening :) If you like the idea, when would it suit you?

That's great! I'm not sure about your arrangement, but I'm basically free from today to Oct. 2 and after Oct. 8 (GMT+8). Maybe next Monday or Tuesday?

@Wang-Yu-Chao, would this time tomorrow work for you: Oct. 2nd, 9:00 UTC? If not, Monday or Tuesday next week, same time?

Hi @gunnarmorling, tomorrow is all right for me. See you tomorrow.

@Wang-Yu-Chao, just sent you a hangout invite via your Google email address.

@gunnarmorling, haven't received yet. Seems there's some problem. I've sent a email to your email address too.

Hey @Wang-Yu-Chao, howdy? Did you have a chance to make progress with the CloudEvents support?

Hi @gunnarmorling! I happen to work on this right now. The code basically works but still need to be refined.

Hey @Wang-Yu-Chao, just saw you pushed a commit on this one earlier today. Exciting :) How is it going?

Hi @gunnarmorling, fortunately it works basically fine for either Json or Avro format now. Unit tests for SQL server have done and I'm going to write tests for other connectors.

Hey @Wang-Yu-Chao, howdy? I see you pushed some more commits; anything else you've planned? It definitely would need a rebase. Let me know I should help. Thanks!

Hi @gunnarmorling! This PR is almost done now, and I'm trying to pass all the CI checks.

Ok, that's great news. Let me know if you need any help. I'll also begin to review this PR.

@Wang-Yu-Chao, so there's a test failure here: PostgresConnectorIT.shouldOutputRecordsInCloudEventsFormat. Any idea what it is about?

Hi @gunnarmorling. All checks have passed now :) Looking forward to your review.

Excellent, thanks a lot! Will check it out in the course of the week.

@Wang-Yu-Chao, first of all, thanks a lot! I've just pushed one commit with some small fixes (and probably will do a few more as I continue my review). So please don't force push to this branch, so to not overwrite any of the commits.
I was looking at the produced JSON message. Here it's how it looks atm.:
{
    "datacontenttype": "application/json",
    "dataschema": null,
    "time": "1576245885912",
    "extrainfo": {
        "op": "r",
        "ts_ms": "1576245885919",
        "source": {
            "version": "1.0.0-SNAPSHOT",
            "connector": "postgresql",
            "name": "test_server",
            "ts_ms":<PHONE_NUMBER>912,
            "snapshot": "true",
            "db": "postgres",
            "schema": "s1",
            "table": "a",
            "txId": 642,
            "lsn": 53329920,
            "xmin": null
        }
    },
    "data": "eyJzY2hlbWEiOnsidHlwZSI6InN0cnVjdCIsImZpZWxkcyI6W3sidHlwZSI6InN0cnVjdCIsImZpZWxkcyI6W3sidHlwZSI6ImludDMyIiwib3B0aW9uYWwiOmZhbHNlLCJmaWVsZCI6InBrIn0seyJ0eXBlIjoiaW50MzIiLCJvcHRpb25hbCI6dHJ1ZSwiZmllbGQiOiJhYSJ9XSwib3B0aW9uYWwiOnRydWUsIm5hbWUiOiJ0ZXN0X3NlcnZlci5zMS5hLlZhbHVlIiwiZmllbGQiOiJhZnRlciJ9XSwib3B0aW9uYWwiOmZhbHNlLCJuYW1lIjoiaW8uZGViZXppdW0uY29ubmVjdG9yLnBvc3RncmVzcWwuRGF0YSJ9LCJwYXlsb2FkIjp7ImFmdGVyIjp7InBrIjoxLCJhYSI6MX19fQ==",
    "id": "name:test_server;lsn:53329920;txId:642",
    "source": "/debezium/postgresql",
    "specversion": "1.0",
    "type": "io.debezium.postgresql.datachangeevent"
}

A few questions/remarks:

Is there a specific reason for having the extrainfo intermediary struct, as opposed to having its contents at the top-level? One could even think of pulling out all the things from the source struct, but it might be nice to keep this as is in order to keep their grouping. I.e. have op, ts_ms and source on the same level as datacontenttype etc. I'm a bit undecided on that one.
The data block is a binary representation atm., whereas it should be JSON in case of "datacontenttype": "application/json" and binary in case of "datacontenttype": "application/avro".
Setting time to the value of source.ts_ms looks like the right thing to do as per its semantics in the spec. But its format shouldn't be a ms timestamp but a formatted String (check the spec for details). If it can go to full ms resolution, we could even get rid of source.ts_ms; otherwise we should keep both
dataschema must be non-null/non-empty when its key is present. I already addressed that in my commit
For source I think it'd make sense to add the "identity" of the connector/database, i.e. the "logical name" we use in the connectors. So its value could be something like "/debezium/postgresql/test_server"

So all in all, I'd suggest the record should look like this (for JSON):
{
    "datacontenttype": "application/json",
    "time" : "2018-04-05T17:31:00.499Z",
    "op": "r",
    "ts_ms": "1576249411506",
    "source": {
        "version": "1.0.0-SNAPSHOT",
        "connector": "postgresql",
        "name": "test_server",
        "snapshot": "true",
        "db": "postgres",
        "schema": "s1",
        "table": "a",
        "txId": 738,
        "lsn": 80106656,
        "xmin": null
    },
    "data": {
        "before" : null,
        "after": {
            "pk": 1,
            "aa": 1
        }
    },
    "id": "name:test_server;lsn:80106656;txId:738",
    "source": "/debezium/postgresql/test_server",
    "specversion": "1.0",
    "type": "io.debezium.postgresql.datachangeevent"
}

What do you think? Overall I think we're almost there. Would you be interested in doing these changes to the structure? If not, that's ok, too, just let me know and I can take over.

{
    "datacontenttype": "application/json",
    "time" : "2018-04-05T17:31:00.499Z",
    "op": "r",
    "ts_ms": "1576249411506",
    "source": {
        "version": "1.0.0-SNAPSHOT",
        "connector": "postgresql",
        "name": "test_server",
        "snapshot": "true",
        "db": "postgres",
        "schema": "s1",
        "table": "a",
        "txId": 738,
        "lsn": 80106656,
        "xmin": null
    },
    "data": {
        "before" : null,
        "after": {
            "pk": 1,
            "aa": 1
        }
    },
    "id": "name:test_server;lsn:80106656;txId:738",
    "source": "/debezium/postgresql/test_server",
    "specversion": "1.0",
    "type": "io.debezium.postgresql.datachangeevent"
}

that would mean two source fields, Gunnar?

I'd recommend also looking at the subject.
In Knative Eventing Source for Kafka, we pass the partiton/offset on there

that would mean two source fields, Gunnar?

LOL, are you saying that's not supported in JSON, ahaha. It's oversight, the latter is the source defined by CE, the former the sourceInfo block of DBZ change events. We'll rename the latter.

I'd recommend also looking at the subject.

Thanks, will take a look. Much appreciated!

Beginning this comment to compile a list of things that can be addressed as a follow-up (will be tracked in JIRA eventually):

[ ] Support for binary mode with AK 2.4
[ ] Move connector-specific record makers to indvidual modules to avoid dependencies from core to them

@gunnarmorling Thanks for reviewing! I'm about to solve these issues one by one.
About the extrainfo attribute, I think subject might be a good place to go for op, ts_ms and source fields, as @matzew mentioned. According to the description, subject works with CE source attribute to qualify a kind of specific events (to help subscriber filtering events or something). It's appropriate to put three fields are appropriate.

I just noticed that there's no more Map type for CloudEvents attributes, and subject is String type.
So maybe we need to combine these fields into one string.
Or, we put op, ts_ms and all the fields in source on the first level as extension context attributes. Like this:
{
    "datacontenttype": "application/json",
    "time" : "2018-04-05T17:31:00.499Z",
    "op": "r",
    "ts_ms": "1576249411506",
    "version": "1.0.0-SNAPSHOT",
    "connector": "postgresql",
    "name": "test_server",
    "snapshot": "true",
    "db": "postgres",
    "schema": "s1",
    "table": "a",
    "txId": 738,
    "lsn": 80106656,
    "xmin": null,
    "id": "name:test_server;lsn:80106656;txId:738",
    "source": "/debezium/postgresql/test_server",
    "specversion": "1.0",
    "type": "io.debezium.postgresql.datachangeevent",
    "data": {
        "before" : null,
        "after": {
            "pk": 1,
            "aa": 1
        }
    }
}

Or, we put op, ts_ms and all the fields in source on the first level as extension context attributes.

Yes, I'm no fan of concatenating multiple fields into a single one. What exactly does "extension context attribute" mean? I think @matzew was referring to that before, too.

Let me also try and summon @bluemonk3y (who worked on the Kafka binding for CloudEvents). Neil, what do you think of that last proposal above, would it be a reasonable way to export data change events as CloudEvents? For now, that's only considering the structured mode. This shows the Kafka record value, for the key I think we'd just go with the existing format and e.g. using the JSON converter. WDYT?

Pushed a commit to wrap up on converting (extension) attributes. Will push some more commits in a bit.

Made some good progress with this one today; focused on the format and these work now:

JSON for the wrapper as well as for data
JSON for the wrapper and Avro for data

Will look into Avro + Avro tomorrow, maybe it already works. Also did some clean-up, will push some more soon. Seems we're getting there eventually :)

Ok, I'm going to close this one in favor of the new PR https://github.com/debezium/debezium/pull/1198, as this needs rebasing and I don't want to force-push. @Wang-Yu-Chao, thanks a lot for your work. If you like, let's continue the discussion over on the other PR. It's mostly done, I'll do some more testing next.


fix(pkg): upgrade xml-crypto to latest version

Addresses:
npm WARN deprecated<EMAIL_ADDRESS>Deprecated due to CVE-2021-21366 resolved in 0.5.0

:tada: This PR is included in version 4.5.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Runtime Semantics: ForIn/OfBodyEvaluation should check nextResult is Object

https://tc39.github.io/proposal-async-iteration/#sec-runtime-semantics-forin-div-ofbodyevaluation-lhs-stmt-iterator-lhskind-labelset
In step 6

a. Let nextResult be ? IteratorNext(iterator).
b. If iteratorKind is async, then set nextResult to ? Await(nextResult).
c. Let done be ? IteratorComplete(iterator).

the type of nextResult is checked in IteratorNext, but not after Await.
https://tc39.github.io/ecma262/#sec-iteratornext

If Type(result) is not Object, throw a TypeError exception.

So the assertion in IteratorComplete can fail.
https://tc39.github.io/ecma262/#sec-iteratorcomplete

Assert: Type(iterResult) is Object.

Thanks very much (for this and the other bugs). The actual fix here was slightly different, to avoid checking twice for Object-ness (once before and once after Await).


Async mode: Add support for Redis logical DBs

Now the Async mode supports Redis logical databases, and honors URLs like redis://localhost:6379/6
The support has been also added for sentinels. A typical config would look like this:
CONFIG_REDIS_PROXY=redis://redis-master/6
CONFIG_REDIS_SENTINEL_HOSTS=redis://localhost:26379,redis://localhost:26380,redis://localhost:26381

Hmm, is there a JIRA for this?

Instead of monkey patching upstream protocol class, it is a much better approach to use our own class and use the upstream protocol class with the composition pattern. Like I commented here: https://github.com/3scale/apisonator/pull/350#issuecomment-2020501532
For me this worked very easy OOB. ALthough it's not necessary to be just a copy of it. Just to use the upstream protocol class instead of monkey patching it. Unless I miss some details...

Hmm, is there a JIRA for this?

No, Sorry. This comes from https://github.com/3scale/apisonator/pull/350. I forgot to mention it in the PR description. Anyway, this is not directly related to TLS so maybe it  should have it's own Jira.

@akostadinov @mayorova I renamed the branch to add THREESCALE-11061 to the name and Github automatically closed this issue. I opened https://github.com/3scale/apisonator/pull/390


Bug: Poor legibility of diff in lualine_b with solarized theme

Self Checks

[x] I'm using the latest lualine.
[x] I didn't find the issue in exsisting issues or prs.

How to reproduce the problem
Expected behaviour
Sufficient contrast between the foreground of the diff component and background of the lualine_b section.
Actual behaviour
There's currently very poor contrast between the text of the diff component, and the background color of the lualine_b section. Note that this is only an issue with this specific section.
Minimal config to reproduce the issue
require 'lualine'.setup({
   options = {
      section_separators = '',
      component_separators = '',
      theme = 'solarized'
   },
   sections = {
      lualine_a = {},
      lualine_b = {'diff'},
      lualine_c = {},
      lualine_x = {},
      lualine_y = {},
      lualine_z = {}
   },
})

Aditional information
Here's a screenshot to help visualize this issue:

You can change the colors to your liking with diff_color option .
Diff colors by default are grabbed from  GitSignsAdd/GitGutterAdd/DiffAdded/DiffAdd highlight definition . So the color usually goes well with the colorscheme but not always there's possibility o something like this happening . And it's not possible for me to make sure this always appears right on every section of every theme.  So diff_color option is exposed to customize colors.


Enable node retrieval without type

From what I can tell, there are 3 types of entities.
Node/Vertex: Has Target property that === Node
Edge: Has Target property that !== Node
Param: Has no Target property
Given, this... We can find a Node/Vertex using only the Node property and ensuring that Target === Node
This is useful for determining which Nodes an Edge is connected from/to because you don't know what types of Nodes they are.

This is a nice addon. I believe that on older versions of the library I did something of the sort.
Before merging the pull request, could you add a test for this use case? If not I'll add it myself.
Thanks again for your interest.

There you go @guzmonne!


Fix broken headings in Markdown files

GitHub changed the way Markdown headings are parsed, so this change fixes it.
See bryant1410/readmesfix for more information.
Tackles bryant1410/readmesfix#1

Thanks Bryant.
Btw are you using it? Do you need any new feature in Howerpoint?

I don't currently use it :sweat_smile: but thanks!


Add regression test for #1607

This depends on:

#1607 - the actual bugfix this will test regression of

Correct error handling of in task status in the presence of memoization serialization errors - I've got some work in progress on that but no PR

@benclifford This draft seems to passing tests and the PR it depended on is already merged. Do you want to go ahead and convert from draft to PR ?


feat(deployment): FSS Deployment tool for Azure

FSS Deployment tool for Azure
Change Summary

Introduction of a FSS deployment tool for Azure subscriptions
The deployment tool support 3 modes of deployment -

geographies - 1 FSS Scanner Stack per Azure geographyGroup, Default
one-to-one - 1 FSS Scanner Stack per Storage Account (1:1 relationship)
single - Just 1 FSS Scanner Stack for all Storage Accounts (not recommended for multi-region storage accounts)

PR Checklist

[x] I've read and followed the Contributing Guide.
Documents/Readmes

[x] Updated accordingly

Plugins that have versioning

[x] Version bumped and follows Semantic Versioning

Other Notes
:warning: Recommend to merge PR #87 #88 before merging this PR.

Hi @GeorgeDavis-TM, Thank you for this PR. This can help customers to protect lots of storage accounts at a time. I have few questions about the solution:

How long does the function take to deploy stacks for, like 10, storage accounts? Consumption function app has 10 minutes as maximal timeout, and deploying one set of stacks would take that long time. Can the deployment run in parallel? Or can we use ARM to deploy that many stacks?

This is an MVP attempt at solving the deployment automation for FSS Stacks in Azure. I am looking for validation of this approach. Once validated from the product team, we can design for parallel runs.

One scanner stack for serving 50 storage stacks looks a bit aggressive to me. And some storage accounts that have higher uploading frequency would need dedicated scanner for each.

The value is configurable in the config.json file, under cloudone.max_storage_stack_per_scanner_stack. The default is set to 50 as a maximum guideline. We can add a warning at the start of execution on rightsizing for the workload.

How to monitor the failure and recover the process from failure when deploying multiple stacks?

At the moment, failure of a deployment, throws an exception and exits the script. Complex handling logic can be introduced in later stages, where we are deploying multiple stacks in parallel.

The way to configure credentials are not documented. Managed identity with a proper role would be another choice for you to refer to. By using it you won't need to configure credentials.

At the moment, I am using an Azure registered app and secret from the same, similar to tm-fss-management.

Hi @GeorgeDavis-TM ,
Sorry that I've been busy and cannot fully review this huge and rich-functional PR. In prod team Q3 OKR we have this item in the secondary priority, and this PR could be a start point for building this. Thank you for submitting this PR.
Due to resources and priority, the prod team might not be able to fully validate this solution in a short time. I tried to understand how the plugin works, but some points are still unclear to me. The deploy.sh script looks like just to deploy a function app and doesn't trigger it, so I don't get how to trigger the function. And there's no serverless template in the PR, why is there serverless deploy -s dev command in the README? It looks like it's the function app who deploys the stacks not the script, so the timeout hard limit needs to be considered. Maybe you can use premium tier which has unlimited timeout.
Do you think a short demo video clip can help us understand it quickly? If you have run some tests, like the number of storage accounts you've protected by this plugin, that would also be helpful. Thanks.

Hi @GeorgeDavis-TM , Sorry that I've been busy and cannot fully review this huge and rich-functional PR. In prod team Q3 OKR we have this item in the secondary priority, and this PR could be a start point for building this. Thank you for submitting this PR.
Due to resources and priority, the prod team might not be able to fully validate this solution in a short time. I tried to understand how the plugin works, but some points are still unclear to me. The deploy.sh script looks like just to deploy a function app and doesn't trigger it, so I don't get how to trigger the function. And there's no serverless template in the PR, why is there serverless deploy -s dev command in the README? It looks like it's the function app who deploys the stacks not the script, so the timeout hard limit needs to be considered. Maybe you can use premium tier which has unlimited timeout.
Do you think a short demo video clip can help us understand it quickly? If you have run some tests, like the number of storage accounts you've protected by this plugin, that would also be helpful. Thanks.

I will work on updating the README and get back to you for review. Thank you for your feedback.


Add new graphql query to get a single user

Related Issue or Background Info

https://github.com/CDCgov/prime-simplereport/issues/1295

Changes Proposed

Add graphql query to get a single user from the organization so we can later reduce okta usage by not returning role and permission data in getUsers()

Additional Information

This is only the third bullet in the ticket and is enough to allow the frontend work to be added and tested
Once this query and the frontend changes have been deployed to production, the remainder of the work for this ticket will be added to a new pull request (from branch brmaden/okta--split-getusers-step-2).  The two-step deployment will ensure we don't break the site for any users while making these changes.

Added https://github.com/CDCgov/prime-simplereport/issues/1416 to track future changes to make permissions of site admins consistent.


Possibility to add own ca-certificate to cert-managers trust store

Is your feature request related to a problem? Please describe.
We're using our own internal ACME server to issue certificates from an internal CA. The internal ACME server is reachable under https://acme.company.com/directory. Certmanager however is unable to validate the https issued certificate of https://acme.company.com/directory.
This results in the following error:
E0612 13:09:14.949395       1 setup.go:208] cert-manager/controller/clusterissuers "msg"="failed to verify ACME account" "error"="Get https://acme.company.com/acme/acme/directory: x509: certificate signed by unknown authority" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod-acme-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"=""
E0612 13:09:14.949602       1 sync.go:81] cert-manager/controller/clusterissuers "msg"="error setting up issuer" "error"="Get https://acme.company.com/acme/acme/directory: x509: certificate signed by unknown authority" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"=""
E0612 13:09:14.949731       1 controller.go:131] cert-manager/controller/clusterissuers "msg"="re-queuing item  due to error processing" "error"="Get https://acme.company.com/acme/acme/directory: x509: certificate signed by unknown authority" "key"="letsencrypt-prod"

The resource, named letsencrypt-prod can be ignored.
Describe the solution you'd like
I would like to know if its either possible to:

Add a custom ca to certmanager via an argument or parameter

or:
use a sidecar, with helm support, to inject the ca certificate to cert-manager controller.
Describe alternatives you've considered
An alternative would be to build our own cert-manager images (which is not a good one) and patch the certmanager images ca-certificates.crt. However, the base image is unknown.
Environment details (if applicable):

Kubernetes version (e.g. v1.10.2): 1.17.2
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): vSphere
cert-manager version (e.g. v0.4.0): v0.13.0
Install method (e.g. helm or static manifests):

...
  controller:
    replicas: 1
    image:
      repository: quay.io/jetstack/cert-manager-controller
      tag: v0.13.0
      pullPolicy: IfNotPresent

    resources:
      requests:
        cpu: 100m
        memory: 30Mi
      limits:
        cpu: 300m
        memory: 50Mi

    affinity: {}
    nodeSelector: {}
    tolerations: []

    # Optional additional arguments. Use at your own risk.
    extraArgs: []
    # Must be a list of `--`-denoted args, e.G.:
    # - --foo-args=foo-value

    # Optional adddional env vars. Use at your own risk.
    extraEnv: []
    # Must be a list of valid env var definitions, e.G.:
    # - name: SOME_VAR
    #   value: 'some value'
...

/kind feature

Seems to be a duplicate of https://github.com/jetstack/cert-manager/issues/2332 ?

/close
closing as duplicate


Error

After updating MH yesterday, I'm getting the following error:

same

same

same

same

Sorry for the annoyance.
Will the update take weeks? or is it near done? Just curious.

The update is done, I've just been too lazy to make a release. I'll do it sometime today.

Thank you so much for you efforts it's really appreciated! <3

The newest release for 15.23 can be found here: https://github.com/Fexty12573/SharpPluginLoader/releases/tag/<IP_ADDRESS>


Make it easier to remove the carrier board

Removing the carrier is very hard. Maybe add pokey hole to push it out from the back?

Hm, could you describe your vision/requirements for this pokey hole?  This is a 2-layer board and the Pico is right under there... so what's a pokey hole that's better than just levering the thing out, but is doable without too much disruption?

updated title and comment.
the issue atm is that you can't flash it with the board in place, so every time I tried new firmware I had to lever it out
I bent the pins after a few levering outs

There's no good way to do this... I think it matters little, now that we can program the flash in place through HK SPI.  Suggest we close.


Pointer events cancelled by touch-action on Chrome

Pointer events (e.g. pointermove) are being cancelled by a browser's touch action. This breaks Chrome on Android (or desktop Chrome with touch simulation enabled).
An easy fix is to put the following property to the svg element:
touch-action: none;

Before

After

Dude, my hero. Just pushed! Could you confirm that it's working?
https://perfect-freehand-example.vercel.app/
https://biwyi.csb.app/

Works perfectly, thank you! 👍
(you should probably document the requirement, it's easy to miss if someone hasn't messed with pointer events before)


java.lang.IllegalArgumentException: Illegal base64 character 2d

I am using kafka api, the topic is persistent and partitioned topic, in the java consumer, the consumer.poll method  throws the following exception
java.lang.IllegalArgumentException: Illegal base64 character 2d
at java.util.Base64$Decoder.decode0(Base64.java:714)
at java.util.Base64$Decoder.decode(Base64.java:526)
at java.util.Base64$Decoder.decode(Base64.java:549)
at org.apache.kafka.clients.consumer.KafkaConsumer.getKey(KafkaConsumer.java:364)
In the producer, the key is "tenant-4",
final ProducerRecord<String, byte[]> data = new ProducerRecord<>(
"persistent://public/default/abc", "tenant-4", outStream.toByteArray());
In the pulsar admin api  /pulsar-client consume,  the message is displayed.

Seems another user reported a similar issue of java.lang.IllegalArgumentException: Illegal base64 character 2d.
This looks more like a convert issue between protobuf string (optional string partition_key   = 6) to bytes.

@jiazhai can you start a PIP discussion? I am in favor of making the partition key to bytes. but we need to think about the backward compatibility issue.

Closed as stale. Please open a new issue if it's still relevant to the maintained versions.


Class "Thepany\Template\Template" not found

We tried to use your package on Laravel 8 but when we follow your instruction the error "Class "Thepany\Template\Template" not found" appeara.

Sorry, the instructions are outdated. try with Cristiangomeze\Template\Template


Add references to GeoJSON in the RAML specifications

The description of the /locations/geojson end point in the file dd.v20.raml the contains a reference to the GeoJSON FeatureCollection specification.
In all descriptions that refer to GeoJSON objects, such a reference must be added.

Has been done partly. Will be further elaborated in version 2.0


Paragraphs

https://github.com/CU-ADCL/SURI/blob/bb2a865416b265fa973b198d788b47e15cc703b9/RunCode.md?plain=1#L48
For some reason, the markdown processor is treating different lines as different paragraphs throughout the website. That seems to go against the official markdown specification, so I am not sure why it is doing that. But it makes all of the paragraphs have very rough right edges.

Should be fixed in 6fdc04b0f0320777ceefea78c4e5e131c16da18a


stf looks for tools.jar which does not exist on Java 9

openjdk.test.debugging/.classpath contains an entry for tools.jar, which is required if tests are running on java 8.
tools.jar is absent on Java 9 - the classes have been merged into the jdk modules.
stf stops with an exception if any of the classpath entries cannot be found.
Tests still need to run against Java 8, so the classpath entry needs to be present in that situation.
Some code should be added to only resolve a tools.jar entry if we are testing Java 8.

PR submitted to fix: https://github.com/AdoptOpenJDK/stf/pull/4

@sxa555, please can you rerun the test with Java 9 and close if OK

Looks good. Closing


Dead link on how to build

https://github.com/LesleyLai/eml/blob/master/docs/build.md

Thanks. I haven't worked on this project for a long time and just picked it up. This issue gets solved now.


Support creation of DLC channel

Fixes https://github.com/get10101/10101/issues/126.
This currently single-patch PR includes too many things that I have not managed to split into smaller patches. I can spend some time doing so if the review is too hard.
Namely:

Add new test to prove that we can open a DLC channel.
Start refactoring the "production" code in ln-dlc-node.
Rework all the tests in ln-dlc-node. The objective was to simplify and better understand why things work. This led to fixing a couple of bugs, and it made the tests faster and easier to read.
Remove system_tests feature in favour of #[ignore] attributes.

The DLC functionality still needs to be tested more e.g. can we make a payment after opening a DLC channel.

bors r+


UTC date/time tokens

You currently support tokens for datetime parts (eg, year, hour, etc). These are supported in filenames/foldernames/etc.
It would be nice to have the option of specifying UTC values instead of local-time values.

Are you suggesting just one token? If so what would the granularity be? Right now you provide granularity, via multiple tokens, down to the microsecond (not that I use it…).
I would think that for file names, YYYYMMDD_HHMMSS would be sufficient.
And thanks very much for the quick response!
Ken Schwartz
From: Ola Hallengren @.>
Sent: Monday, November 18, 2024 3:47 PM
To: olahallengren/sql-server-maintenance-solution @.>
Cc: Kenneth Schwartz @.>; Author @.>
Subject: Re: [olahallengren/sql-server-maintenance-solution] UTC date/time tokens (Issue #839)
[EXTERNAL]
How about a parameter called @TimestampTokensInUTC?
It would be used for the tokens in the file names and directory structure.
It would not be used for the logging (as that is not using tokens).
—
Reply to this email directly, view it on GitHubhttps://github.com/olahallengren/sql-server-maintenance-solution/issues/839#issuecomment-2484096189, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BLYAN72DBTDSJWI27MAORED2BJG3FAVCNFSM6AAAAABSAMBF2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBUGA4TMMJYHE.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at @.***, and then delete it. To opt-out of receiving emails Please click herehttps://info.kaseya.com/email-subscription-center.html. The term 'this e-mail' includes any and all attachments.

This would be a new parameter that would make all the timestamp tokens UTC tokens.

There is another similar request, #679

Yes, I realized you meant a bit about 1 second after I hit “send”. 😊
And, yes, that would certainly take care of it, at least in terms of the use cases I need (and can think of).
Thanks again.
Ken
From: Ola Hallengren @.>
Sent: Monday, November 18, 2024 4:05 PM
To: olahallengren/sql-server-maintenance-solution @.>
Cc: Kenneth Schwartz @.>; Author @.>
Subject: Re: [olahallengren/sql-server-maintenance-solution] UTC date/time tokens (Issue #839)
[EXTERNAL]
This would be a new parameter that would make all the timestamp tokens UTC tokens.
—
Reply to this email directly, view it on GitHubhttps://github.com/olahallengren/sql-server-maintenance-solution/issues/839#issuecomment-2484127320, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BLYAN73KRP6C2GBSIALRQDL2BJI7XAVCNFSM6AAAAABSAMBF2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBUGEZDOMZSGA.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at @.***, and then delete it. To opt-out of receiving emails Please click herehttps://info.kaseya.com/email-subscription-center.html. The term 'this e-mail' includes any and all attachments.

I have a test version.
https://github.com/olahallengren/sql-server-maintenance-solution/pull/840
You pass a parameter like this:
@TokenTimezone = 'UTC'

Hi,
I just wanted to thank you for your swift response to this. I have already downloaded the latest version of your code, and tested to see if it met our requirements.
It was perfect.
Ken
From: Ola Hallengren @.>
Sent: Thursday, November 21, 2024 2:34 PM
To: olahallengren/sql-server-maintenance-solution @.>
Cc: Kenneth Schwartz @.>; Author @.>
Subject: Re: [olahallengren/sql-server-maintenance-solution] UTC date/time tokens (Issue #839)
[EXTERNAL]
Closed #839https://github.com/olahallengren/sql-server-maintenance-solution/issues/839 as completed via #840https://github.com/olahallengren/sql-server-maintenance-solution/pull/840.
—
Reply to this email directly, view it on GitHubhttps://github.com/olahallengren/sql-server-maintenance-solution/issues/839#event-15387178418, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BLYAN7ZKMYKZHGL24FR3TGT2BYYUFAVCNFSM6AAAAABSAMBF2CVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJVGM4DOMJXHA2DCOA.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at @.***, and then delete it. To opt-out of receiving emails Please click herehttps://info.kaseya.com/email-subscription-center.html. The term 'this e-mail' includes any and all attachments.

I am happy that the feature worked as you wanted it to.
Here is the documentation:
https://ola.hallengren.com/sql-server-backup.html#TokenTimezone


Button to toggle DL/UL limit

First I gotta say that I really love picotorrent!! Congrats to the devs.
Can we have a button next to the dl and ul info on the botton bar to toggle dl and ul limit?
Idk how to add the feature request tag, sorry for that.

upload limit should be removed since you are so concerned about seeders


Alias decoded_token as decodedToken

Thank you and sorry for the delay. You can use decodedPropertyName now since version v4.4.0


Redbiom interface is slow to load

Following steps in #2569 , click on + symbol to show the desired artifact to retrieve the sample of interest:
At first click:

And nearly 30 min later the screen looks the same:

Check URGENT email to the qiita-admin. Anyway, the issue here is: https://github.com/biocore/qiita/issues/2570. We are currently gonna fix via nginx. Closing as this is a duplicate of #2570.


Marathon Update 3.2

changelog
-redid a large portion of medbay (cryonics, genetics is now a new area its sick)
-added the retro nanotrasen sign to conf room
-decor.,
-skub
probably gonna redo science soon since i think i can clean up the design and make it a lot better, just need to think on it more.

put this in the wrong spot oops lol


Vertical Axis Values are clipped in scrollable chart

I have a scrollable chart and want to show vertical axis values from index. However these are clipped as the size allowed for the value is too small for the value from index - unless I make the width of the chart really wide. The values in the chart image below should be 12am, 5am, 10am etc.

If I make the text smaller, it shows more of the values.

Or if I make the chart width really wide, then I can get more of the values showing.

But how can I get the values to show without making the chart really wide? If the verticalAxisStep is set, is it possible to allow the values to show past the allowed space without clipping them? (Hope that makes sense)
The code for this chart is:
Expanded(
          child: SingleChildScrollView(
            scrollDirection: Axis.horizontal,
            child: SizedBox(
              width: 160 * SizeConfig.safeBlockHorizontal,
              height: widget.chartHeight,
              child:
                  Chart(
                state: _chartState,
              ),
            ),
          ),
        ),

_chartState = ChartState.line(
      ChartData.fromList(
        temperatures.map((e) => BubbleValue<void>(e.toDouble())).toList(),
        axisMax: maxY,
        axisMin: minY,
      ),
      behaviour: const ChartBehaviour(
        isScrollable: true,
      ),
      backgroundDecorations: [
        GridDecoration(
          showVerticalValues: true,
          showHorizontalValues: false,
          verticalAxisStep: 5,
          gridColor: CustomColors.darkBlue.withOpacity(0),
          textStyle: const TextStyle(
            color: CustomColors.darkBlue,
            fontSize: 14,
          ),
          verticalAxisValueFromIndex: (index) {
            var now = DateTime.now();
            var midnight = DateTime(now.year, now.month, now.day);
            var time = DateFormat('h').format(midnight.add(
                  Duration(minutes: index.toInt() * 60),
                )) +
                DateFormat('a')
                    .format(midnight.add(
                      Duration(minutes: index.toInt() * 60),
                    ))
                    .toLowerCase();
            return time;
          },
        ),
      ],
      foregroundDecorations: [
        SparkLineDecoration(
          lineWidth: 2.0,
          gradient: lineColor(minY, maxY),
          smoothPoints: true,
        ),
      ],
    );

Hey @roly151, thanks for report 😄
There was issue that text painter would cut off text on single item width. So even though you set step size to 5, text painter didn't take that into account.
Now that is fixed and text should also be centred as well. (You can change alignment)
You can find fix in this PR: #64

Since there's no additional comments, and we have a fix I will close this.


Introduce a rawField JSON schema

A field JSON Schema allowing fine control of the absence or presence (with a value, that can be null) of a JSON property.
Use case: Be able to implement endpoints conforming to rfc7386 JSON Merge Patch  where it is mandatory to distinguish between a null and an absent property.

Thank you Antoine for your pull request.
I won't be able to review it before a couple of weeks. If anyone else wants to review it in the meantime, feel free to do so.

Hey @ant8e, are you still interested in working on this?

Yes. I have something drafted. Need to find the time to finish
Le 6 oct. 2023 à 20:43 +0200, Julien Richard-Foy @.***>, a écrit :

Hey @ant8e, are you still interested in working on this?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Great, thank you! As alternative names for “raw”, I thought of “precise”. Maybe PreciseJson?

I want to add also an example of implementing a JSON Merge patch, but I'm not sure where to put it
Any suggestion ?

I want to add also an example of implementing a JSON Merge patch, but I'm not sure where to put it Any suggestion ?

Great! This file is the best place, IMHO:
https://github.com/endpoints4s/endpoints4s/blob/master/json-schema/testkit/src/main/scala/endpoints4s/algebra/JsonSchemasOptionalFieldsTest.scala
The testing infrastructure is a bit messy, but for now the simplest path is to add your test to the end of that file. How does that sound?

I want to add also an example of implementing a JSON Merge patch

After reading your comment again, maybe my suggestion was not the best one. Could you please elaborate on the content of your test?

What I have in mind is to code some example that does implement a patch endpoint, conforming to the JSON Merge patch RFC.  preciseField  will allow doing that in the presence of optional fields.  That could also be just documentation.
Anyway, I can do this in a subsequent PR, if you want to go forward with merging this one.

It would be cool to have that example!
You can add a JsonMergeTestApi trait here defining the endpoint that uses preciseField, and then write a client test suite and a server test suite that both use that endpoint. Finally, update the interpreters (at least pekko-http and http4s, if possible also the fetch and sttp interpreters) to mix-in your test suites.

@ant8e I propose that we go ahead and merge your work now. If you have time and interest, you are welcome to add the JSON merge patch example in a separate PR!


ackage System.Net.Security 4.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0)

Hey all!
I'm trying to install Microsoft.Azure.DocumentDB.Core on my project, and the error messages are:

Package System.Net.Security 4.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0)
Package System.Security.SecureString 4.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0)

If I try to install the same package on the project using another machine, it works well. This is why I think it's not a problem on DocumentDB SDK.
My setup is:
Microsoft Visual Studio Enterprise 2017 - Version 15.4.0 (VisualStudio.15.Release/15.4.0+27004.2002)
Microsoft .NET Framework - Version 4.7.02046
Installed Version: Enterprise
ASP.NET and Web Tools 2017 - 15.0.30925.0
NuGet Package Manager - 4.4.0
.NET Core Runtime - 2.0
.NET Core SDK - 2.0
My project is configured to user .NET Core 2.0 as the target framework. And the project is a class library.
Some more details from Package Manager:
Install-Package : Package System.Net.Security 4.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0). Package System.Net.Security 4.0.0 supports:

monoandroid10 (MonoAndroid,Version=v1.0)
monotouch10 (MonoTouch,Version=v1.0)
netstandard1.3 (.NETStandard,Version=v1.3)
xamarinios10 (Xamarin.iOS,Version=v1.0)
xamarinmac20 (Xamarin.Mac,Version=v2.0)
xamarintvos10 (Xamarin.TVOS,Version=v1.0)
xamarinwatchos10 (Xamarin.WatchOS,Version=v1.0)
At line:1 char:1

Install-Package Microsoft.Azure.DocumentDB.Core -Project MM.ArchCore

CategoryInfo          : NotSpecified: (:) [Install-Package], Exception

FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

Install-Package : Package System.Security.SecureString 4.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0). Package System.Security.SecureString 4.0.0 supports:

monoandroid10 (MonoAndroid,Version=v1.0)
monotouch10 (MonoTouch,Version=v1.0)
netstandard1.3 (.NETStandard,Version=v1.3)
xamarinios10 (Xamarin.iOS,Version=v1.0)
xamarinmac20 (Xamarin.Mac,Version=v2.0)
xamarintvos10 (Xamarin.TVOS,Version=v1.0)
xamarinwatchos10 (Xamarin.WatchOS,Version=v1.0)
At line:1 char:1

Install-Package Microsoft.Azure.DocumentDB.Core -Project MM.ArchCore

CategoryInfo          : NotSpecified: (:) [Install-Package], Exception

FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

Install-Package : Package restore failed. Rolling back package changes for 'MM.ArchCore'.
At line:1 char:1

Install-Package Microsoft.Azure.DocumentDB.Core -Project MM.ArchCore

CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
=================================================

@dsplaisted @terrajobst @emgarten any ideas. This looks like it should work and it works for me in my simple repro:
dotnet new console
dotnet add package Microsoft.Azure.DocumentDB.Core --version 1.5.1
dotnet publish

on
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.DocumentDB.Core" Version="1.5.1" />
  </ItemGroup>

</Project>

succeeds.

@FerHenrique can you try dotnet nuget locals all –clear on the machine in a bad state?

I was not able to repro this either, this package should be compatible due to the netstandard1.3 support.
"compile": {
  "ref/netstandard1.3/System.Net.Security.dll": {}
}

@FerHenrique would you make a copy of the obj folder for the project including project.assets.json before trying to fix the problem.
After that

Clear the caches as @Petermarcu mentioned
Delete the obj folder for the project to force restore to create a new project.assets.json file

If the above fixes your issue it would be great if you could share the original project.assets.json file, if it doesn't fix the issue then I would still like to see it to figure out what is wrong.

@emgarten do you think this is a NuGet issue and we should move it to nuget/home?

I think this should be tracked with https://github.com/NuGet/Home/issues/5995, it is likely that nuget is unable to see the ref file and that leads to this compat error.

Ok, I'll close this one here and we can use https://github.com/NuGet/Home/issues/5995 to track this.

Thanks for the update.

Hi all!
Clearing the cache worked for me.
My project.assets.json is attached.
Tks!
project.assets.json.txt

Sometimes just clearing the Nuget cache is not enough. I also deleted obj folder and that fixed my problem.


Default template contains misleading metadata

The default template contains some misleading metadata:

Specifically, the "template", and author, as well as some properties that aren't necessary:

While it doesn't affect the output, it can be a bit misleading when sending the created document to third-parties.
Glad to submit a pull request, but wanted to check in first before I did, in case it was purposeful.

Please do, I don't think any of that metadata is necessary in there (We are not using it).
Probably Cristina just created an 'empty' document.
I can the that the 'tax guide.dotx' comes from the current project we are using it for :)

Done via https://github.com/karnov/htmltoword/issues/5.


Extracted hardcoded Strings from

Explode.java
explode.xml
SettingsUtil.java

into strings.xml
Strings with html tags were wrapped in  ...html...  tags and output as formatted text when you use Html.fromHtml(htmlString);

Oh dear, runs fine but I made lint angry adding a bunch of untranslated strings.  Not sure what the best approach is.  Don't want to unilaterally ignore lint warnings.

@laurencemeynell I will take a look.
In terms of the Html strings I think it would be better to style the text using Spans. Styling shouldn't really be part of the strings. A library like https://github.com/binaryfork/Spanny would make this simple. I am happy to make this change.

Thanks, that would be good.  It does seem more elegant doing it programmatically.

@laurencemeynell
For now, don't worry about the Lint error. It'll go away once the translations are added for German. I'm sure there's a better way to deal with it but I'm not too worried since we know the cause of the error.


definition of fixed time files fails

There's a bug in setup_utils that makes fixed files fail:
as I'm trying to access tstep_delta{frequency] even for fixed timestep, where before we were skipping these lines completely.

solved in branch issue197

Actually there wa sanother bug linked to this in extract_var when trying to retrieve the time dimensions
  tdims = [d for d in array.dims if 'time' in d][0]
failed for fixed variable without time dimension as list is empty
Now changed to:
    tdims = [d for d in array.dims if 'time' in d]
    if len(tdim) >= 1:
         tdim = tdim[0]
    else:
        tdim = None

in issue199 branch


Update TypeScript to 3.9.3

Shouldn't we keep local typescript version in sync with CRA, which is still on 3.8?


Update map store to allow complex Map implementations such as Hazelcast

Currently the map store allows you to specify the class of the Map implementation to use, this is a really useful feature as there are a lot of different Map implementations and this gives the user a lot of flexibility. However, for some more advanced Maps, such as Hazelcast's implementation, it might be necessary to provide a MapFactory class or something similar to add extra logic to control how we fetch/create maps. For some map implementations we may also need to provide a configuration file.
The MapFactory interface could be something like:
public interface MapFactory {
    void initialise(final MapStoreProperties properties);

    <K, V> Map<K, V> newMap(final String mapName);

    default <K, V> Map<K, V> newMap() {
        return newMap("default");
    }
}

The default simple implementation of this could just be:
public class SimpleMapFactory implements MapFactory {
    private Class<? extends Map> mapClass;

    @Override
    public void initialise(final MapStoreProperties properties) {
        try {
            mapClass = Class.forName(properties.getMapClass()).asSubclass(Map.class);
        } catch (ClassNotFoundException e) {
            throw new IllegalArgumentException("Map Class is invalid: " + properties.getMapClass(), e);
        }
    }

    @Override
    public <K, V> Map<K, V> newMap(final String mapName) {
        try {
            return mapClass.newInstance();
        } catch (InstantiationException | IllegalAccessException e) {
            throw new IllegalArgumentException("Unable to create new map instance of type: " + mapClass.getName());
        }
    }
}

We may also need to make some other changes to the MapImpl to work with persistent maps implementations.
For the Hazelcast example, it is more efficient in Hazelcast to call a special 'set' method instead of 'put'. We should consider the possibility of allowing Map implementations to override some of this logic too to improve the performance.
We should then add a new integration test suite for the MapStore, that runs the integrations tests over a different map implementation. This is just a case of duplicating the MapStoreITs class and providing a different store.properties file.

I have done a bit of work on this on branch gh-892-map-store-impl but it looks like there are several issues that need investigating.
I branched off the gh-763-cache branch as that is due to be merged in soon and it contains some hazelcast bits in that I thought might be useful.

As part of this issue the following has been done:

added a MapFactory so users can provide a more complex map implementation like Hazelcast or MapDB.
elements are now split up into different maps based on their groups
non aggregated and aggregated elements are now in separate maps
added a store property to configure batching elements up and applying aggregation before adding them to the existing map store

Merged into develop.


Filter improvements

Acceptance Criteria:

changing any filtering criteria (add, remove, switch) should reset the pagination to 1; the results per page is not modified, see https://github.com/BlueBrain/nexus/issues/2614#issuecomment-918840584 for details
the filter menu should only display 5 entries (select all + 4 most common values - where common means count) and the rest (the list of possible values) should be accessible by scrolling, see https://github.com/BlueBrain/nexus/issues/2614#issuecomment-918833528 for details
the filtering section displays a user choice on how the filtering logic should be executed, in two variations, where the first option is automatically selected:

if the column is single-valued (array=false) the following filtering criteria will be proposed: is any of (OR), is none of, see https://github.com/BlueBrain/nexus/issues/2614#issuecomment-918838061
if the column is multi-valued (array=true) the following filtering criteria will be proposed: is any of (OR), is none of, is all of (AND), see https://github.com/BlueBrain/nexus/issues/2614#issuecomment-918837751

Notes:

properly label the operators https://github.com/BlueBrain/nexus/issues/2614#issuecomment-918841513
types can be filtered (not the case right now?)

The list of values in the filter is still not complete:

Types filter does not display properly:


Add setting to disable email management

This commit adds WAGTAIL_EMAIL_MANAGEMENT_ENABLED setting that defaults
to True, but when disabled, hides the 'Change email' button in account
management view, and disables the associated route. This is useful when
using external authentication method like LDAP or OpenID Connect where
email management is handled elsewhere.
Wagtail already includes WAGTAIL_PASSWORD_MANAGEMENT_ENABLED setting.
This is almost exact copy of that implementation.

Thanks @alatja! This is very timely - @kaedroho has been talking about the issues with email management and external authentication. I've asked him to review.

Thanks @alatja! Merged in dd0bb9a870525cb6a559d6027abee5af2e36ae88 (with a tiny addition to the tests, to check that the "Change email" option does exist by default).


Uncaught ImportError in init.py line 78

Aider version: 0.69.1
Python version: 3.11.10
Platform: Linux-6.6.63-x86_64-with-glibc2.39
Python implementation: CPython
Virtual environment: No
OS: Linux 6.6.63 (64bit)
OS: NixOS
Git version: git version 2.47.0
An uncaught exception occurred:
Traceback (most recent call last):
  File "aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "main.py", line 727, in main
    main_model = models.Model(
                 ^^^^^^^^^^^^^
  File "models.py", line 878, in __init__
    res = self.validate_environment()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "models.py", line 1125, in validate_environment
    res = litellm.validate_environment(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "llm.py", line 25, in __getattr__
    self._load_litellm()
  File "llm.py", line 35, in _load_litellm
    self._lazy_module = importlib.import_module("litellm")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "__init__.py", line 1001, in <module>
    from .cost_calculator import completion_cost
  File "cost_calculator.py", line 22, in <module>
    from litellm.llms.anthropic.cost_calculation import (
  File "cost_calculation.py", line 9, in <module>
    from litellm.utils import get_model_info
  File "utils.py", line 54, in <module>
    from tokenizers import Tokenizer
  File "__init__.py", line 78, in <module>
    from .tokenizers import (
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

I'm running on my PC NixOS 24.05 this is not duplicate I'm not using replit.

Thanks for trying aider and filing this issue.
This looks like a duplicate of #1313. Please see the comments there for more information, and feel free to continue the discussion within that issue.
I'm going to close this issue for now. But please let me know if you think this is actually a distinct issue and I will reopen this issue.
Note: A bot script made these updates to the issue.

I'm running on my PC NixOS 24.05 this is not duplicate I'm not using replit.