CHIP-12 -- Reduce Plot Filter

plot filter chip, open up at 8am 1/20

Thanks for writing such a thorough explanation of the problem and proposed solution @jmhands! A few clarifying questions that I have already asked you in other channels, but adding here for completeness:

Does reducing the plot filter by half also double plot "decompression" farming workload? Or is it not exactly proportional?
Upon reaching size 32 plot filter, does that imply increasing minimum k-size to k33 is the logical next proactive step? At that time would plot filter reset to 512 or stay at 32?
Is increasing the minimum k-size a soft fork?
In the event of a hard fork, would that be an opportunity to introduce other pending changes that require a hard fork? What such changes might those be?
Would increasing the number of signage points per subslot (to decrease the time between infusion points) be a viable option as well or is that too drastic a change to the consensus?

A few more thoughts to stir the pot in favor of Option 2 (k-size change):

If there were ever a time to declare timelines for a minimum k-size, it's probably now while folks are considering replotting anyway.
I don't think a proposal on plot filter decrease is complete without considering when/how minimum k-size requirement might fit into the ~10 year timeline.
Plot filter changes + decompression considerations adds a dimension that many farmers may have trouble grasping or considering whereas a minimum k-size timeline is easier to grasp (say, k33 phases in 2025, k34 phases in 2028 etc.)
Given the unpredictability of future hardware and plotting/farming advances, there's a chance another hard fork to change the timeline of plot filter changes may be necessary whereas as Bram has mentioned in the past a minimum a k-size change would be a soft fork (I'd like to confirm whether or not this is the case).

Between:
A. Halve the plot filter every couple years
B. Increase k value every couple years
As far as I understand, B will require many farmers to replot to k+1 every couple years, whereas A doesn't require such a periodic replot. If that is indeed the case, I think this CHIP is a great proposal, and I support it.

This is a good proposal to ensure there is a constant cadence in hardware requirements, however I think the two year implementation schedule is a bit aggressive. Why was two years chosen, was it arbitrary, or based on ASIC game theory?
I think it makes more sense to tie in the halving of the plot filter with the halving of coin emissions every three years to prevent them from occurring too close together. More specifically six months prior to the coin halving we would get a plot filter halving. This would allow farmers to upgrade their hardware prior to coin halving in what is typically a positive market environment. We could also do it six months after coin halving but I think prior to coin halving makes the most sense. regardless arbitrarily picking two years does not seem like a good idea and is a tad extreme.
So if we decide three years makes more sense and this CHIP passes, the first filter halving would be 10/1/2023, six months before coin emission halving on 4/1/2024. That is approximately eight months from today.

To me it sounds like just pushing the problem into the future. What will happen if in a few years the network size is 2 times bigger and the nodes are mostly at k32? There is no incentive to plot on k33, k34 or kXX. However, reducing the size of the filter does not reduce the memory requirements for creating a plot. At some point, ASIC miners will come and replace the GPUs.  Therefore, in my opinion, increasing the K size oder changing the plot format makes more sense, even if not everyone is willing to do it. By increasing the memory requirements when creating the plots, this ensures that no ASICs are developed in this direction because RAM memory is still expensive. Better to lose some network size now than face the same problem in 7 years. I myself have a 1.5PiB farm with about 70% k32 plots. Every farmer knew that the moment will come when the K32 size is not enough ;)

Here is how I see it -> lets say chia corp goes bankrupt. I do not see considerations for this scenario.
Startup tech companies have a very high likelihood (>90%) of going bankrupt. Ones who have less revenue than expenses are more likely to fail.
This CHIP will send us to a plot filter of 32 and stay there forever.
I suggest doing a plot filter change on a as needed basis and not schedule them as its (extremely) likely chia corp will not be around to reset the plot filter and set it to k33. Chia corp will not have to live w/ the problems they create. We will.

Reducing plot filter towards 32 seems sensible, and preferable as the first lever, but is difficult to consider in isolation.
At some point a lower filter necessitates an increase in k.
Cumulative 16x decrease in farmable space with a given setup will catch out many farmers, especially those about to plot with compression levels that appear to have plenty of headroom. They'll either have to upgrade hardware with increasing aggression or replot in some form - to higher k or lower compression.
If k=34 by the time filter=32 that reduces the increase in farming demands to 4x over ~8 years. Feels like a sensible balance on multiple fronts and provides a predictable enviornment for farmers.
Perhaps a cadence of filter-filter-k-filter-filter-k? With an 18 month or "half halving" interval.

I suggest doing a plot filter change on a as needed basis and not schedule them as its (extremely) likely chia corp will not be around to reset the plot filter and set it to k33. Chia corp will not have to live w/ the problems they create. We will.

The chain does not need Chia Network Inc to incorporate any changes. The client can be forked and the farmers can decide which version to run if CNI disappeared off the face of the earth tomorrow. Also the scheduled plot filter change is purposely more aggressive than needed because relaxing the plot filter schedule would be a soft fork while accelerating it would be a hard fork -- so you'd want to default to a more aggressive schedule to be conservative. It would be far more disruptive to make plot filter changes "as needed" since each would need to be a hard fork.

Thanks for writing such a thorough explanation of the problem and proposed solution @jmhands! A few clarifying questions that I have already asked you in other channels, but adding here for completeness:
1. Does reducing the plot filter by half also double plot "decompression" farming workload? Or is it not exactly proportional?

2. Upon reaching size 32 plot filter, does that imply increasing minimum k-size to k33 is the logical next proactive step? At that time would plot filter reset to 512 or stay at 32?

3. Is increasing the minimum k-size a soft fork?

4. In the event of a hard fork, would that be an opportunity to introduce other pending changes that require a hard fork? What such changes might those be?

5. Would increasing the number of signage points per subslot (to decrease the time between infusion points) be a viable option as well or is that too drastic a change to the consensus?

Yes, all other things being equal, this would double the farming workload. However, keep in mind that the workload is minuscule to begin with.
We have no plans for changing either the filter or the minimum k size after this proposal runs out. The reasons for this are that
a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and
b. It's difficult to predict the state of the storage industry that far into the future. Conventional wisdom says that SSD storage will be cheaper than HDD storage at that time, which will mean that most of the netspace will be on SSDs as well. If that's the case, there may not be a need for a filter at all, but we simply don't know if this prediction will come true yet.
Yes. A soft fork is a tightening of the rules, and a hard fork is a loosening of the rules.

Increasing k would mean that some plots (k-32) would no longer work, therefore the rules would have been tightened so it would be a soft fork.
Reducing the filter would mean that some plots that did not pass the filter before the change would pass the filter after the change. This would be a relaxation of the rules and therefore would be a hard fork. Following this logic, if we were ever to increase the filter, it would be a soft fork.

Potentially yes, but we don't have any such plans at this time. If such plans did emerge, they would be proposed in a new CHIP. This is because each CHIP needs to focus on a single proposal.
To my knowledge changing the number of signage points per subslot wasn't discussed, though other consensus changes were. My initial thought about this idea is that it would make plot grinding more expensive and more difficult, but it wouldn't achieve significantly more than simply decreasing the filter. This would also be a large consensus change, it could affect the stability of the network, and it could potentially open the network to new attack vectors. While alternate proposals are certainly welcome, I don't see this one being viable (unless I'm mistaken about something fundamental).

A few more thoughts to stir the pot in favor of Option 2 (k-size change):
* If there were ever a time to declare timelines for a minimum k-size, it's probably now while folks are considering replotting anyway.

* I don't think a proposal on plot filter decrease is complete without considering when/how minimum k-size requirement might fit into the ~10 year timeline.

* Plot filter changes + decompression considerations adds a dimension that many farmers may have trouble grasping or considering whereas a minimum k-size timeline is easier to grasp (say, k33 phases in 2025, k34 phases in 2028 etc.)

* Given the unpredictability of future hardware and plotting/farming advances, there's a chance another hard fork to change the timeline of plot filter changes may be necessary whereas as Bram has mentioned in the past a minimum a k-size change would be a soft fork (I'd like to confirm whether or not this is the case).

Increased hardware requirements for plotting. A k=34 plot would be very difficult to create in RAM without using a high-end server

Maybe it's my experience from early plotting days but I've always viewed the ability to plot all in RAM (and now to plot with GPU+RAM) as a nice-to-have and not something farmers should feel is necessary or feel entitled to be able to do.

These are all fair points and the community will need to carefully weigh the options, just as CNI already has done.

What is the plan after 2030?

No plans yet. We have a decent idea of what the storage and GPU industries will look like in a few years, but it's difficult to predict what will happen 7+ years from now. However, the two primary mitigations we have against plot grinding are -- and likely will remain -- reducing the filter and increasing the minimum k.

passes, the first filter halving would be 10/1/2023, six months before coin emission halving on 4/1/2024, which is approximately eig

JM is the expert here, but I can say that the timeline from this proposal was not chosen arbitrarily. It was chosen based on the likely advancements in the GPU, memory, and storage industries over the next few years. The exact block heights where the changes will be implemented is something that can be discussed on Discourse, which will be set up soon.

To me it sounds like just pushing the problem into the future.
Correct, this proposal is meant to make plot grinding uneconomical over the next 8 years, but not forever.

What will happen if in a few years the network size is 2 times bigger and the nodes are mostly at k32?
The network size has nothing to do with this proposal. Most plots will likely still be k-32 because they are the easiest to create, though larger plots do require less resources while farming.

At some point, ASIC miners will come and replace the GPUs.
We don't foresee this happening (assuming you mean "plotters" and not "miners"). If you have any data to back this up, please share it.

When considering K32 on K33:
Is it technically possible to convert 2xK32 plots to a K33?
Replotting 10,000 K33s fails the CHIA policy.

It might be possible but it will require more time and energy than simply creating a new k33. However, this proposal does not require replotting. That decision remains entirely yours, regardless of whether the proposal succeeds or fails to gain adoption.

This is the Plot Filter proposal from GPU Plotting is Real – and Very Fast right?
Am i reading correct if this plot filter proposal is implemented then there will be a hard fork in the blockchain?
So farmers have to update to a specific software version before a certain block is reached?
Would have no problem with it. Just want to clarify.
I think also with blockchain there is always a need to upgrade software to adapt to technology development.

This is the Plot Filter proposal from GPU Plotting is Real – and Very Fast right? Am i reading correct if this plot filter proposal is implemented then there will be a hard fork in the blockchain? So farmers have to update to a specific software version before a certain block is reached? Would have no problem with it. Just want to clarify.
I think also with blockchain there is always a need to upgrade software to adapt to technology development.

This is correct, yes.

a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and
Is the definition of a "low end machines" still a RPi4? Or will the minimum requirement increased due to this change. Obviously in this scenario, the farmer is not farming compressed plots.

This CHIP is now a Draft. If it is accepted, it will require a hard fork of Chia's blockchain. Therefore it is imperative that we have consensus from the community. Please continue to leave your reviews here.
You can also discuss this CHIP in the #chips channel on Keybase, or on Discourse:
https://developers.chia.net/t/chip-0012-plot-filter-reduction/810
Additionally, we will hold a public Zoom call to discuss this CHIP at 8 AM PST on Wednesday, February 22. See the #chips channel on our Keybase for details.
https://keybase.io/team/chia_network.public

a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and

Is the definition of a "low end machines" still a RPi4? Or will the minimum requirement increase due to this change. Obviously in this scenario, the farmer is not farming compressed plots.

The definition of "low end machine" will likely remain within the same genre for the foreseeable future. For example, the min spec machine could become a RPi5, but it won't be a workstation.

Here is a recording of the discussion of this CHIP:
https://youtu.be/hf01z8Uhj1Y
Please continue to leave your reviews here.

Thanks for the informative public zoom call today. Wanted to summarize some thoughts:

Option 1 (plot filter reduction) makes much more sense to me than Option 2 (increase minimum k-size) if only because Option 2 likely still requires a plot filter reduction, which almost makes Option 1 strictly better. But I would still like to see documented guidance about what situations would warrant a k-size increase? If I understand correctly from the call today, it is when multi-GPU plotting can economically do k32s entirely in combined VRAM.

There needs to be more definitive direction/education for farmers looking to do compressed plot farming at higher compression levels (C6+) because plot filter reduction will proportionally increase decompression workload and hence proportionally reduce maximum farm size (by up to 16x). Plotting and farming now requires longer term planning for farmers looking to min/max their compression.

Probably a moot point but re: validity of proofs from older clients. Plots that pass the 512 filter will also pass lower plot filters. It was brought up on the call that clients that don't upgrade have a high likelihood of farming on the wrong chain due to rejecting valid proofs that the most of the network sees as valid.
Seeing "Farm Sooner" on the 2023 roadmap is interesting because I think a "light wallet" farmer relying on full node peers might be able to continue farming 512 filter plots at their own disadvantage. This is only relevant if "Farm Sooner" is released well before a plot filter reduction.

Given the expected eventual SSD flippening, wouldn't it make sense to also put in a plot filter reduction to 1 in the distant future to reduce likelihood of another hard fork? e.g. plot filter to 1 reduction in 2040. This may be aggressive but again, this can be pushed out via soft fork if we realize HDD farming is still a non-negligible portion of the netspace in 2040.

Thanks for these thoughtful comments. We will take them all into consideration.

I have 100,000     k32   plots😓😓

I have 100,000 k32 plots😓😓

This CHIP does not affect the validity of k32 plots.

So, from what i understand from a recent blogpost:

Thankfully it requires a very large cluster of GPUs and a tremendous amount of power to perform something like this today, and it is not economical even with the extended leverage.

In this context, sure one is not likely to invest in hardware for plot grinding like this, but what about people who already own a lot of GPUs? Chinese mining farms?  Will it be economical to them and if so, will lowering the plot filter be likely to deter them?

GPU mining should generally be prevented. Then you have to drastically reduce the plot reporting time.
Reason: CHIA is still dependent on investors.
CHIA advertises with GREEN-IT. If CHIA becomes POW (even partially) then it will be detrimental to investors.

So, from what i understand from a recent blogpost:

Thankfully it requires a very large cluster of GPUs and a tremendous amount of power to perform something like this today, and it is not economical even with the extended leverage.

In this context, sure one is not likely to invest in hardware for plot grinding like this, but what about people who already own a lot of GPUs? Chinese mining farms?  Will it be economical to them and if so, will lowering the plot filter be likely to deter them?

In the next 4 years without this change, the only economically viable attack would be a short term rental attack. Such an attack would use a material percentage of all GPUs that exist. The plot filter change cuts the viability of any such attack in half (doubles the required amount of GPUs to attempt it.) As to farming over weeks or months, using GPUs instead of just using them to plot and fill space remains un economic for a while without this change and with this change, doubles the cost for good measure.

GPU mining should generally be prevented.

If an individual were able to construct a GPU farmer, they could keep it for themselves or rent/sell its usage to gain an economical advantage over everyone else. Our approach therefore is to create the best tech possible and keep it open sourced. That way, no individual has an advantage over everyone else.
That said, we don't believe there is a viable way to prevent GPU farming altogether. CNI certainly has no ability to censor anyone from using the Chia blockchain. If you do have an idea to prevent GPU farming from occurring (along with the technical details to implement your idea), you are welcome to submit a CHIP. CNI, along with the entire Chia community, will take your idea into consideration.

That said, we don't believe there is a viable way to prevent GPU farming altogether.

I believe the comment was in support of the plot filter reduction as a means to reduce the economics of plot grinding specifically with GPUs.

@lord-icon You may not understand why over 24 years ago Nvidia created the term.GPU and moved the current graphics chip from motherboards and made a discrete graphics card. This is because unlike CPUs, GPU are made for one thing, compute at extremely efficient and powerful levels.
I don't blame you though, I blame CNI for their campaign against GPUs and not putting out real numbers like I have showing the community how much more efficient GPU farming is. Yes this CHIP will be used to hold at bay GPU farming even though it is more efficient than buying more harddrives because they can't build their own GPU farmer.
My 5PB farm uses only 190 watts more to run C7 plots on a 3060 GPU. I gain 37% with an additional effective space of 1.85PB. This would be equvalent to adding 103x18TB enterprise drives. Since each drive uses 9W this would add another 927W to my farm instead of 190W for compressed plots using a GPU.

I don't blame you though, I blame CNI for their campaign against GPUs and not putting out real numbers like I have showing the community how much more efficient GPU farming is. Yes this CHIP will be used to hold at bay GPU farming even though it is more efficient than buying more hard drives and farming with a CPU. Either they can't or don't want to build their own GPU farmer possibly because of their ties to the hard drive industry.

To be clear, this CHIP is not to prevent GPU Farming. This is to prevent the possibility of GPU grinding. As far as I know GPU plotting + farming is on the roadmap and in fact encouraged for farmers (such as yourself) where the efficiency makes sense.
The plot filter reduction will, as a side effect, reduce efficiency of GPU farming however, but that is not its goal or purpose.

@lord-icon You may not understand why over 24 years ago Nvidia created the term GPU and moved the current graphics chip from motherboards and made a discrete graphics card. This is because unlike CPUs, GPU are made for one thing, compute at extremely efficient and powerful levels.
My 5PB farm uses only 190 watts more to run C7 plots on a 3060 GPU. I gain 37% with an additional effective space of 1.85PB. This would be equivalent to adding 103x18TB enterprise drives. Since each drive uses 9W this would add another 927W to my farm instead of 190W for compressed plots using a GPU.
I don't blame you though, I blame CNI for their campaign against GPUs and not putting out real numbers like I have showing the community how much more efficient GPU farming is. Yes this CHIP will be used to hold at bay GPU farming even though it is more efficient than buying more hard drives and farming with a CPU. Either they can't or don't want to build their own GPU farmer possibly because of their ties to the hard drive industry.

You are conflating the wrong things. This CHIP addresses the problem of people substituting only GPU grinding for storing plots on disk. We totally agree that plotting with a GPU is more efficient and you factually misstate our position on that as GPU plotting and farming are almost shipped to the beta program. This is about keeping GPU farms from simulating space in a proof of work style.

You are conflating the wrong things. This CHIP addresses the problem of people substituting only GPU grinding for storing plots on disk. We totally agree that plotting with a GPU is more efficient and you factually misstate our position on that as GPU plotting and farming are almost shipped to the beta program. This is about keeping GPU farms from simulating space in a proof of work style.

Then explain why you are proposing to halve the plot filter three times in a little more than 4 years when you just said the above statement

In the next 4 years without this change, the only economically viable attack would be a short term rental attack.

2024-256
2026-128
2028-64

Then explain why you are proposing to halve the plot filter three times in a little more than 4 years when you just said the above statement

It's all explained in the CHIP -- which part do you specifically have a question about?

is that data you are already running?  Or is that the extrapolation according to the Excel list?  Because hardly anyone achieves this.  I also doubt that you can manage 5PB with a 3060 (without ti?).  or: final 6.85PB.  But yes.  I don't really question its effectiveness either.  Much more the procedure, which is increasingly moving in the direction of POW.  And CHIA was never "born" for that.  What is the next level?  ASIC?  C30?

is that data you are already running? Or is that the extrapolation according to the Excel list? Because hardly anyone achieves this. I also doubt that you can manage 5PB with a 3060 (without ti?). or: final 6.85PB. But yes. I don't really question its effectiveness either. Much more the procedure, which is increasingly moving in the direction of POW. And CHIA was never "born" for that. What is the next level? ASIC? C30?

I implore you to read Chia.net's blog posts detailing plot compression and plot grinding. In particular understanding the exponential increasing workload of "decompression" to see why there is no "next level".

is that data you are already running? Or is that the extrapolation according to the Excel list? Because hardly anyone achieves this. I also doubt that you can manage 5PB with a 3060 (without ti?). or: final 6.85PB. But yes. I don't really question its effectiveness either. Much more the procedure, which is increasingly moving in the direction of POW. And CHIA was never "born" for that. What is the next level? ASIC? C30?

Yes this is real time farm data

Yes this is real time farm data

hmmm... I'm looking again for the 2 reddit threads that achieve completely different (much higher) values.  Also with the graphics card.  But I have to look it up again to be sure.  But I think it was also a 3060. And it was much more likely to be fully utilized.  (no 6.85PB).  May I ask which one you have exactly?  Manufacturer and GB.  6GB or 12GB?  ti or without.  Then I can check again whether your setup is better.  Thanks

Yes this is real time farm data, in theory my max farm size WITH a 512 plot filter is 8.4PB. I have plans to expand to 10PB this fall and move to a 3060Ti. However in 2024 if this passes my max farm size will be cut in half to 4PB. This CHIP is way to aggressive especially when even Gene is admitting it will not be cost effective for the next 4 years. I would much rather prefer 3 years just before every halving.

Again, Gene is talking about cost effectiveness of grinding not farming.
Thinking the plot filter reduction schedule is too aggressive is a valid point so thank you for your suggestion of 3 years instead of 2. Do keep in mind making the schedule less aggressive later (as we see how technology progresses and in particular the PCIe 5.0 adoption) can be done with a soft fork but it would be very disruptive to realize 3 years is not aggressive enough later and hard fork again to move it to 2 years.

hmmm... I'm looking again for the 2 reddit threads that achieve completely different (much higher) values. Also with the graphics card. But I have to look it up again to be sure. But I think it was also a 3060. And it was much more likely to be fully utilized. (no 6.85PB). May I ask which one you have exactly? Manufacturer and GB. 6GB or 12GB? ti or without. Then I can check again whether your setup is better. Thanks

We should probably keep the discussion in this thread focused on the CHIP and leave the farming optimization discussions to a separate conversation.

Yes this is real time farm data

hmmm... I'm looking again for the 2 reddit threads that achieve completely different (much higher) values. Also with the graphics card. But I have to look it up again to be sure. But I think it was also a 3060. And it was much more likely to be fully utilized. (no 6.85PB). May I ask which one you have exactly? Manufacturer and GB. 6GB or 12GB? ti or without. Then I can check again whether your setup is better. Thanks

While I prefer EVGA GPUs (I use them for plotting) I have an ASUS TUF GAMING GeForce RTX 3060 12GB. These are cards I used for mining Ethereum that I kept when I sold my farms. I kept the EVGA because they no longer make them and they always made the best GPUs. I farmed with their cards for 5 years 24x7 and never had any fans fail. My plan as the plot filters get cut in half is to use my mining motherboards where I can use up to 13XGPUs on a single rig to farm with. Ironically the GPUs don't need as much PCIe bandwidth when farming.

Again, Gene is talking about cost effectiveness of grinding not farming.
 You need to rethink the discussion and how it affects GPU farmers.

Thinking the plot filter reduction schedule is too aggressive is a valid point so thank you for your suggestion of 3 years instead of 2. Do keep in mind making the schedule less aggressive later (as we see how technology progresses and in particular the PCIe 5.0 adoption) can be done with a soft fork but it would be very disruptive to realize 3 years is not aggressive enough later and hard fork again to move it to 2 years.
If you were paying attention or are a bot you would have known I made this suggestion when it was CHIP-53. So going back to ignoring your rants.

Reducing the plot filter will actually have the side effect of making more farmers switch from CPU farming to GPU farming. I’m concerned that the known negatives of running really big farms are clouding some folks judgement.

Reducing the plot filter will actually have the side effect of making more farmers switch from CPU farming to GPU farming. I’m concerned that the known negatives of running really big farms are clouding some folks judgement.

I have been seeing quite the opposite. Small farmers would like to replot using the highest CPU compression possible but have say a 200TB farm using an old CPU and C5 compression puts them close to hitting max farm size but makes the most sense for them to do all the replotting. Fast forward only a year to the first plot filter halving and they are now below what their system is capable of farming. Most are deciding to not replot because it is simply not worth it when considering the aggressive plot filter halvings. But this is what you guys want apparently.
I have no issues keeping up and I am sure a lot of the other large farmers will be able to as well. Just understand who will still be around in a few years as we push down profitability of those who don't use compression. This aggressive plot filter halving will have huge implications on your farming community, mostly the small ones. I don't come here to campaign for myself, I will be just fine. Rather I prefer a strong and thriving community and we need to ensure a change like this is really needed.

Great points @Voodoo-Chia. There's also a high likelihood SSDs will make up much more of the netspace by the 3rd or 4th plot filter reduction, which also dampens the impact of look ups (though not decompression).
Now that we have established block height (date) proposed, I would like to see this proposal move forward in the CHIP process along with #57. I have not seen any objections to the CHIP that haven't already been addressed by the comments above.

@SlowestTimelord
I will 2nd that motion. My original objection I stated here was to extend plot filter having to two years. Now that this has been done, it has my support.

Thanks everyone! Given your your feedback, along with projections for the advancement of hardware, we have updated the proposal to use a 3-year cadence for reduction, starting in one year. We will be moving this CHIP along in the review process soon. To make it absolutely clear, here is the current proposed schedule:

Block height
Month/Year (approx)
Filter size

5 496 000
June 2024
256

10 542 000
June 2027
128

15 592 000
June 2030
64

20 643 000
June 2033
32

Additional feedback and reviews are still welcome!

This CHIP is in Review. Please make your final reviews soon.

Full support, like the updated filter reduction cadence. Glad to see it in review with specific blocks. Could we get estimated dates for those blocks?

From: Dan Perry @.>
Sent: Monday, May 22, 2023 9:12:27 AM
To: Chia-Network/chips @.>
Cc: Digital Spaceport @.>; Comment @.>
Subject: Re: [Chia-Network/chips] CHIP-12 -- Reduce Plot Filter (PR #53)
This CHIP is in Review. Please make your final reviews soon.
—
Reply to this email directly, view it on GitHubhttps://github.com/Chia-Network/chips/pull/53#issuecomment-1557295661, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAOIVH632JYA6MHMM2PR2D3XHNX4XANCNFSM6AAAAAAUBDQTOA.
You are receiving this because you commented.Message ID: @.***>

The month estimates (especially for later reductions) might be as accurate as we can get. With the ASIC timelords coming online the exact corresponding date could have multiple days/weeks of variation?

We will minimize the impact of the ASIC timelords by introducing them slowly. However, other events such as a rapid increase in netspace could move the block heights by a few hours or days. We won't be able to give a more granular estimate than month and year.

Let's call this what it is, a tendency away from PoST and towards PoW.

Let's call this what it is, a tendency away from PoST and towards PoW.

This is exactly the opposite. It's a lever to ensure PoST doesn't become PoW.

Let's call this what it is, a tendency away from PoST and towards PoW.

Can you explain in more detail why you hold this opinion?

PoST always had a POW component therein because there is no other way to make plots other than doing work. For example, doubling lookups requires doubling the work for this component as this CHIP clearly describes. Also, replotting means more work. Also, the new compression algorithms move work to the CPU compared to the old.
PoST advocates, as I still consider myself to be, should always recognize these facts. Proof of Space Time would have been better labelled Proof of Space Time and Work(PoSTW) from the beginning because work is NOT removed by Chia Consensus only being managed better, but the key to managing is recognizing it imo.
There is clearly a precedent in Bitcoin that POW works well, but over-reliance on work tends to centralize and Chia is tending in that direction with this CHIP and other updates imo. I'm not offering an alternative, because perhaps it is inevitable, except enhanced disclosure so more folks can understand why it is necessary. I think this is important for the sake of mass uptake and therefore security.

Thanks for the feedback!
Responses in line:

PoST always had a POW component therein because there is no other way to make plots other than doing work.

True, but irrelevant for this CHIP.

doubling lookups requires doubling the work for this component as this CHIP clearly describes.

Also true, but it only doubles the lookups a farmer must perform. No new hashes are generated. No new data is written. It's doubling the amount of work, but reading data from a disk is not the same as the Proof of Work consensus.

the new compression algorithms move work to the CPU compared to the old.

Irrelevant -- This CHIP is reacting to the new compression algorithms, but it is not the cause of those algorithms, which would exist regardless of the existence of this CHIP.

replotting means more work.

Irrelevant -- This CHIP does not necessitate any replotting.

There is clearly a precedent in Bitcoin that POW works well, but over-reliance on work tends to centralize and Chia is tending in that direction with this CHIP and other updates imo.

Do you have other reasons to believe that Chia is trending in the direction of PoW that are specific to this CHIP?

[...] enhanced disclosure so more folks can understand why at least a minimum amount of work is necessary, but even with incremental increases it is not necessary in the same abundance as Bitcoin. I think this is important for the sake of mass uptake and therefore security.

Agreed, full disclosure is very important here. That's why this CHIP has been open for five months and counting.

PoST always had a POW component therein because there is no other way to make plots other than doing work. Also, doubling lookups requires doubling the work for this component as this CHIP clearly describes. Also, the new compression algorithms move work to the CPU compared to the old. Also, replotting means more work.

Speaking holistically and not to just this CHIP - on net the cost of plotting in both time and CPU power has fallen/is being amortized over farming. When looking at a new entrant, the amount of time and CPU/GPU that they have to allocate to be up and providing material additional security is greatly decreased by all these changes.
As Chia blockchain scales, that next 25 EiB will be brought on much more cheaply than before the new methods to create smaller plots on disk existed.

If there's one thing I "know" it is that nobody knows the future. Where does it say in this CHIP why the filter adjustment can't be more like Bitcoin where the difficulty adjustment is based on feedback from the network, not the solely the calendar(e.g. one year out and then every 3 years)? Sorry if I missed it. Since it requires a hard fork, perhaps this is an opportunity to build something that presumes less about the future.

The designed from the start mechanism to defend against the ongoing increase in memory bus speed between parallel processors is the raising of the minimum K size. This was an opportunity to use a less blunt instrument, but if the plot filter decreases are not enough, then we revert to the original plan that's always been there.

ass the 512 filter will also pass lower plot filters. It was brought up on the call that clients that don't upgrade have a high likelihood of farming on the wrong chain due to rejecting valid proofs that the most of the network sees as valid.

I was wondering why this wasn't programmed in from the start since it seems so systematic, but I used up all of my adversarial stance question points today ;) Thank you for answering the implied question.

This CHIP is now in Last Call. If no further changes are required in the next two weeks (roughly), it will be moved to Final.

This CHIP is now Final. No further changes are allowed.


Texture improvements

See individual commits.

Breaking changes for renderers:

wlr_texture_init now takes the size of the texture
wlr_texture_impl.get_size has been removed

Thanks!


MDN content project: Modernizing the Learning Area JavaScript modules

This was originally filed by Chris in mdn/content. Although the project is now complete I wanted to keep a record of the original proposal for traceability.

This RFC proposes that we work on modernizing the Learning Area JavaScript modules
available on MDN.
Problem statement
The JavaScript learning area modules are pretty good as they stand, and still
helpful to aspiring web developers. The main problem is that many of the
component articles were written before the time that MDN's JS policy was
changed (from "don't use ES6 features, they are too modern" to "use modern
features").
As a result, a number of articles are a bit out of date, or teach the old way
of doing something first, with the new way added in as a new section at the end.
Examples:

In the Handling text — strings in JavaScript
article, old-school string concatenation is introduced centrally,
whereas template literals are introduced at the end as a newer syntax that
you'll also come across. This should really be done the other way round.
In the arrays introduction,
the basics are explained just fine, but there is no coverage of modern array
methods that are very commonplace in JS these days, such as map() or filter().
In the JS objects module,
old-school constructors and prototypes are introduced first, and ES class
syntax is tacked on the end as an afterthought. This should be the other way
round.
We did some work on making sure that const/let is used in appropriate
Places instead of var. We should check to see if there is any more work to
do here.
We should include a guide early on about targeting browsers, where we could
cover things like considering which browsers/non-browser runtimes you need to
support, making it clear that we only support modern browsers in these guides,
and setting up tools like Babel.js if you want to support ancient engines.

Priority assessment
This list checks this project against the OWD prioritization criteria.

Effort: Medium/High: There are over 40 JavaScript articles in the
learning area, and each one needs to be audited, a decision made about what we
should do with it, and then rewriting work done. Related examples also need
to be updated.
Dependencies: Not much to mention here.
Community enablement: Yes. Each page rewrite is a self-contained task, so
we could get community to help out here.
Momentum: Medium. A lot of features we'd be writing about are pretty
stable, but the technology is very popular.
Enabling learners: Yes!!
Enabling professionals: Yes, somewhat. Even professional web developers
can often benefit greatly from an authoritative source of information show
the modern way to write JS. We should aim to keep this pretty up-to-date.
Underrepresented topics / Ethical web: n/a
Operational necessities: n/a
Addressing the needs of the web industry: n/a

Proposed solutions
Audit existing articles, work out how to update them, get them rewritten.
Task list

Agree on an overall strategy for the updates. For example, is there a
particular set of ECMAScript that we want to stick to documenting in this
resource? How new is too new? Do we gate it on what is supported in at least
two rendering engines, or some such thing? How do we deal with
backwards-compatibility? Show examples of how to support older browsers, or
just teach everyone how to use Babel.js to begin with?
Audit each page. For each one, write a list of what needs to be done to get
that page updated in line with the above strategy.
Find writers to help do this work.
Divide the writing and reviewing work up between the available writers.

Fixed in https://github.com/mdn/content/issues/10337.


Can quic be used instead of http?

🚀 Feature Request
Relevant Package [REQUIRED]

 This feature request is for...
There is a large amount of http in the project. It is recommended to switch to quic to improve performance.
Description [REQUIRED]
 A clear and concise description of the problem or missing capability...
A lot of http is used in the project, such as core-command, core-data, and core-metadata. Each http request requires a three-way handshake. QUIC has been optimized to improve the efficiency of the request.
Describe the solution you'd like
 If you have a solution in mind, please describe it.
The client I want to use when writing app-service is QUIC, so that the request\response time can be reduced as much as possible.
Describe alternatives you've considered
 Have you considered any alternative solutions or workarounds?
I have considered gRPC, but it seems a bit heavy to use on the edge of the Internet of Things. I have considered Coap, but it is too simple and does not meet the security requirements.

@sixhj please fill out the issue template with more information, such as performance comparison and further pros / cons.
thanks

The client I want to use when writing app-service is QUIC, so that the request\response time can be reduced as much as possible.

@sixhj, do you have a particular latency or performance target in mind that EdgeX needs to meet?  Have you confirmed that switching to QUIC enables EdgeX to meet that target when today it does not?

The client I want to use when writing app-service is QUIC, so that the request\response time can be reduced as much as possible.

@sixhj, do you have a particular latency or performance target in mind that EdgeX needs to meet? Have you confirmed that switching to QUIC enables EdgeX to meet that target when today it does not?

The timely delivery of instructions to devices is crucial when controlling them in batches. With over 100 terminal devices, any significant delay would jeopardize the prompt execution and orderly coordination of actions.

@sixhj , EdgeX uses the MessageBus to push event data from Device Services to App Services. REST is no longer used for this.
Sending commands from App Service to Device services can now also be done via the MessageBus instead of REST.
See here to details of Commands via Messaging
And here for example of enabling the command client in App Service to use the MessageBus instead of REST.
https://github.com/edgexfoundry/app-rfid-llrp-inventory/blob/3.1/res/configuration.yaml#L9-L11
The command client is accessible from the ApplicationService API and from the AppFunctionContext API
The above eliminate the use of REST in the internal communication between Device Services and App Services.
Please see EdgeX MessageBus for more details about the internal MessageBus and options.

close this issue as Lenny already provided the solution


Fix zh_CN ID number generator to match correct format

Issue: #235

Nice writeup on the bug report!


Fix responsive markers to correspond to pagination

Adjust responsive markers logic to augment in multiples of 3

The function now takes the floor of the card augment/3 to figure out how many card to paginate through. This ensures that the feature is only being augmented by multiples of three to avoid issues in the pagination. In addition, the logic was updated to highlight the correct listing for each marker (there was an off-by-one error).


Thin volume documentation

Codecov Report

Merging #543 (2e325a6) into main (b67ed35) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #543   +/-   ##
=======================================
  Coverage   77.43%   77.43%
=======================================
  Files          31       31
  Lines        2198     2198
=======================================
  Hits         1702     1702
  Misses        496      496

Merging it in without approval as this is a minor doc change


Add custom sorting

Describe the feature
Repositories are now sorted by number of PRs, we can add a dropdown to add some more custom sorting
It should be implemented because
No response
Did you think about any alternative?
No response
Additional context
No response
Extra fields

[X] I have used the search function to check if an issue already exists
[ ] I'd like to work on this issue

Hi @Balastrong, for example sorted by PR status?
It's a good idea a dropdown menu for custom sorting, but it's necessary choose type of options.
Let me know what you think :)

Hey @DomeT99, yeah a dropdown like that should be fine for now :)
In case we want to make it more sophisticated in the future there's always time to do that :D

"Order by" is a small dropdown that has "Status" as its only option (algorithm to be defined).

Do you like it?

Looks good!
I'd probably move it to the right side, on the same line as the text filter


Add ability to self image_self_link in stemcell.MF

Windows images cannot be imported. For this reason
StemcellCloudProperties now has a new property
ImageURL which means that instead of creating a
new GCP image from a tarball (either inside the
stem cell or on GCS), the existing gcp image will
be used for new instances.
[#135362799]
Signed-off-by: Charlie Vieth<EMAIL_ADDRESS>Signed-off-by: Sunjay Bhatia<EMAIL_ADDRESS>

@dgodd could you make your CloudFoundry github org membership public to make the CLA bot happy?

@ljfranklin @evandbrown Any other queries about this PR?

@dgodd We'll get this merged now, sorry for the wait

@dgodd hit an issue when testing this. We think this PR needs updated to no-op on delete_stemcell if an ImageURL is specified. This integration tests shows the failure:
diff --git a/src/bosh-google-cpi/integration/config.go b/src/bosh-google-cpi/integration/config.go
index 3e66fd3..3714e8f 100644
--- a/src/bosh-google-cpi/integration/config.go
+++ b/src/bosh-google-cpi/integration/config.go
@@ -46,6 +46,7 @@ var (
 	instanceGroup        = envOrDefault("BACKEND_SERVICE", "cfintegration")
 	zone                 = envOrDefault("ZONE", "us-central1-a")
 	region               = envOrDefault("REGION", "us-central1")
+	imageURL        = envOrDefault("IMAGE_URL", "https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1404-trusty-v20161213")

 	// Channel that will be used to retrieve IPs to use
 	ips chan string
diff --git a/src/bosh-google-cpi/integration/stemcell_test.go b/src/bosh-google-cpi/integration/stemcell_test.go
index 37f98a1..5ae0da4 100644
--- a/src/bosh-google-cpi/integration/stemcell_test.go
+++ b/src/bosh-google-cpi/integration/stemcell_test.go
@@ -36,4 +36,32 @@ var _ = Describe("Stemcell", func() {
 		Expect(response.Result).To(BeNil())

 	})
+
+	It("executes the stemcell lifecycle with an image_url", func() {
+		var stemcellCID string
+
+		By("uploading a stemcell with image_url")
+		request := fmt.Sprintf(`{
+				"method": "create_stemcell",
+				"arguments": ["", {
+					"name": "bosh-google-kvm-ubuntu-trusty",
+					"version": "3215",
+					"infrastructure": "google",
+					"image_url": "%s"
+				}]
+			}`, imageURL)
+		stemcellCID = assertSucceedsWithResult(request).(string)
+
+		By("deleting the stemcell")
+		request = fmt.Sprintf(`{
+				"method": "delete_stemcell",
+				"arguments": ["%v"]
+			}`, stemcellCID)
+
+		response, err := execCPI(request)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(response.Error).To(BeNil())
+		Expect(response.Result).To(BeNil())
+
+	})
 })

• Failure [0.093 seconds]
Stemcell
/Users/pivotal/workspace/bosh-google-cpi-release/src/bosh-google-cpi/integration/stemcell_test.go:67
  executes the stemcell lifecycle [It]
  /Users/pivotal/workspace/bosh-google-cpi-release/src/bosh-google-cpi/integration/stemcell_test.go:66

  Expected
      <*dispatcher.ResponseError | 0xc4202bbb90>: {
          Type: "Bosh::Clouds::CloudError",
          Message: "Deleting stemcell 'https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1404-trusty-v20161213': Failed to find Google Image 'https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1404-trusty-v20161213': googleapi: Error 400: Invalid value 'https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1404-trusty-v20161213'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', invalidParameter",
          CanRetry: false,
      }
  to be nil

OK. I'm looking now. I think you're theory is good, and I am adding the above test and fixing it.

@ljfranklin @evandbrown The best answer I could find for not deleting imageURL stemcells was the stemcellCID string prefix.

@dgodd your commit is now on develop and will go out with the next release: https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/commit/b17d08ecccdc19af5adf0d024f85723c6123ce8e

Thanks for this feature, @dgodd, and thanks for taking the lead merging, @ljfranklin! It's out in today's release (25.6.2).


[Feature] Add support for Cloudstack

Thanks for taking the time to fill out this feature request! Before submitting this issue please check the open feature requests to ensure the feature has not already been reported. If it has been reported give it a 👍
Describe the feature request
Fill out the following questions and add any additional information.
Summary: Cloudstack has already a terrafrom provider that can spawn instances with offerrings and networking firewalling. Can you include Cloudstack support as a public/private cloud provider as well ?
Is this an enhancement to an existing resource or data source, if so which one?
If this is a new resource or data source describe what it should do.
The whole hypervisor provisioning process and related networking should be handled by a seperate Resources (E.g. citrix_cloudstack_hypervisor, citrix_cloudstack_resource_pool).
Link to any docs that cover this feature:
https://registry.terraform.io/providers/cloudstack/cloudstack/latest/docs
Link to any APIs that cover this feature:

Hi @fermosan ,
Just want to make sure we understand your request better. Do you want to use the Cloudstack provider together with our Citrix Provider side by side in one terraform workspace, i.e. having Cloudstack resource and Citrix resource configured in one .tf file?
Thank you,
Zhuolun

Hi @fermosan ,
Thank you for your suggestion. If I understand your request correctly, you want to use Cloudstack as hypervisor and use MCS to provision VDAs. Unfortunately Citrix MCS do not support Cloudstack provisioning at the moment.
Alternatively, you can use Cloudstack provider to create VDA vms. Then use Citrix provider to create a manual non-power managed catalog and add the VDA to the catalog using their machine account in AD. You can refer to the example below:
resource "citrix_machine_catalog" "example-manual-non-power-managed-mtsession" {
    name                        = "example-manual-non-power-managed-mtsession"
    description                 = "Example manual non power managed multi-session catalog"
    zone                        = "<zone Id>"
    allocation_type             = "Random"
    session_support             = "MultiSession"
    is_power_managed            = false
    is_remote_pc                = false
    provisioning_type           = "Manual"
    machine_accounts = [
        {
            machines = [
                {
                    machine_account = "DOMAIN\\MachineName1"
                },
                {
                    machine_account = "DOMAIN\\MachineName2"
                }
            ]
        }
    ]
}

Please let me know if you need further assistance on this.
Thank you,
Zhuolun

Thanks for your reply, we will try your suggestion and let you know.

Hi @fermosan ,
Just want to follow up with you on this. How did the manual provisioning machine catalog work for your use case? Do you have any suggestion / want any improvement for this workflow?
Thank you,
Zhuolun

Hello, thanks for checking on this. We haven't yet tried the deployment. Please keep this open so i can give you feedback when we  deploy.


Double-free if IMG_SaveJPG_RW_jpeglib() fails and IMG_SaveJPG_RW_tinyjpeg() is tried

To reproduce:

Compile SDL_image with dynamic runtime loading (dlopen) and libjpeg enabled
Somehow get SDL_image into a state where IMG_SaveJPG_RW_jpeglib() fails. I'm currently trying to backport SDL_image 2.8.2 to Steam Runtime version 1 'scout', an Ubuntu-12.04-based environment, and for whatever reason it's failing; I'll address that eventually when I've figured out what is happening, but for now, it provides a convenient way to exercise failure code-paths that are not normally reached.
Make IMG_SaveJPG_RW_jpeglib() fail cleanly (at the moment there is another bug in its use of setjmp/longjmp: #429)
Call IMG_SaveJPG_RW(surface, dst, 1, quality), for example via IMG_SaveJPG which is called by SDL_image's own test suite

Expected result: Saving with libjpeg fails. SDL_image tries to fall back to tinyjpeg. No crash.
Actual result: First SDL_image calls IMG_SaveJPG_RW_jpeglib(., ., freedst=1, .), and then it calls IMG_SaveJPG_RW_tinyjpeg(., ., freedst=1, .). The first call closes (frees) the destination RWops object (file). The second call is a use-after-free and then a double-free, which in my case is detected by glibc, causing an abort().
Suggested solution: Probably the entire implementation of freedst should be in IMG_SaveJPG_RW(), and the two backend-specific implementations IMG_SaveJPG_RW_jpeglib() and IMG_SaveJPG_RW_tinyjpeg() should never free the RWops object.

That sounds like the right solution.

I'll provide a PR when I've figured out what is even going on in my scout build...

It looks as though d2c7e0905 might have accidentally fixed this as well (it was intentionally fixing a missing close/free, but seems to have unintentionally fixed this double-free at the same time).

I think #430 resolves this, but I want to get to the bottom of what is happening in my scout build to have better confidence about that solution before undrafting it.


Fix markdown-history script

Works fine. Do we also want to add a script like send-markdown for sending messages?


Database Support for Shelley Era Blocks, Transactions and Ledger State Information [CD-6]

We need to adapt the existing database format and tools to support the needs of the new GraphQL dashboard/explorer and other internal or external tools.  In essence, this involves:

Agree on the data requirements with the explorer team

Adapt the existing PostgreSQL database schema for Byron to support the new Shelley era block, transaction and other data
including address format and other changes

Normalise the schema if necessary to optimise the data storage and database performance
extend the existing schema

Provide any necessary interfacing code
to abstract over raw SQL code (e.g. db_add_new_block(...))
(this may involve changing existing calls to deal with additional information or the new schema)

Include calls in the node to record basic on-chain data
block creation and embedded transactions

Include calls to record off-chain data obtained from the ledger state
rewards calculations
treasury and reserves levels
(may require code to be inserted at the epoch boundary transition - beware of data insertion cost
- might need a separate thread to achieve this efficiently)

Include basic abstractions over transactions
pool registration/deregistration
stake key registration/deregistration
delegations
...
(needs to be done during transaction insertion - make sure this is efficient - it will be on the critical performance path)

Extend to include more complex transactions
parameter protocol updates (may require replay of the voting mechanism)

Note that the database will need to be migrated to the new format.  We will not be able to use the existing database.

Closing this as it is basically done. Will track the remaining work in more fine grained tickets.


spanner: Make samples consistent with the style guide

The Golang samples for Cloud Spanner have some inconsistencies with the style guide. These need to be fixed.

@tbpg, I have a question with the command-line interface in https://github.com/GoogleCloudPlatform/golang-samples/blob/master/spanner/spanner_snippets/snippet.go.
Currently, in https://cloud.devsite.corp.google.com/spanner/docs/getting-started/go#create_a_database, we clearly tell users that they can run samples via commands. However, with this new style, it is impossible to do that because all sample functions are private. Also, there is no way to run a single function.
Is it possible to make all methods public? and keep snippet.go as a command-line interface wrapper. What do you think?

Ah, I didn't realize the command line interface was referenced in the docs.
It seems like we should put the snippets into a package main, either by renaming the package in the current directory or by moving them back to the spanner_snippets directory. Then, we don't need to export the functions.

But in this way, we can't run the command go run snippet.go createdatabase ~. We have to run go run *.go to include all necessary file and also it cannot contain *_test.go files. Otherwise, we got the error:
go run: cannot run *_test.go files (integration_test.go)

Also, there is no main.go so that we can't run go run . either:
runtime.main_main·f: function main is undeclared in the main package

I think exporting the sample functions as public would be the ideal solution. Do you have any other ideas?

@tbpg I will update sample functions to be public and call them from snippet.go. Does this sound reasonable? Do you have any concerns?

runtime.main_main·f: function main is undeclared in the main package

This indicates there is no func main. We'd have to write one (as we do to use go run). Exporting still doesn't seem ideal to me. It also adds indirection when someone wants to open the sample code and it's not in main.go or the current directory.

This indicates there is no func main.

I have defined func main in snippet.go. go run . does not work. It would be better to keep go run snippet.go ~ command.

It also adds indirection when someone wants to open the sample code and it's not in main.go or the current directory.

I am not sure if I understand your meaning here. When someone opens the sample code, it will open the corresponding code file.

@tbpg, just to give you some more context of why we're trying to keep snippet.go alive, it's because of the hand holding we do in the getting started guides for Cloud Spanner. For first time users, we ask them to run the commands from the command line tool to get setup. E.g. to create a database. They can do all this by checking out the repo without having to write code.
There's a question of whether this type of hand-holding is necessary for our users but this is just the way it's setup right now for all languages.
So one thing we could do is to maintain the command line tool just for the samples in the getting started guide. We don't reference the commands in the other documentation. Would you be open to that?

They should probably run gcloud commands instead.

They should probably run gcloud commands instead.

For some of it, like creating a database, sure, I agree. But some of the commands let users populate the database with some data to start with. E.g. dmlWrite.
I guess the idea of the getting started guide is to get familiar with how to use Cloud Spanner using the client libraries since that's what most people will end up using in the long run compared to gcloud which is mostly useful for managing an instance/database here and there.

Is dmlWrite covered by gcloud spanner databases execute-sql? (sorry, I don't know Spanner very well)
If anything is useful enough that it should run as a "go run" in a tutorial, it probably should be implemented in the gcloud surface, just on principle. I think it's fine to have it temporarily while the gcloud command is being implemented, but most of those calls could probably also be expressed as cURL commands (i.e., programming language-agnostic).

Aha, I looked at the tutorial now, that makes sense. We have a couple guided multi-step tutorials like that but I don't have current knowledge on what our style guide says about them.

I'd be curious to know if other products do something similar or if we're unique.
Anyway, our preference would be to keep these tutorials consistent between the languages. So only have this wacky command line tool just for the few samples that are on that page and not do that for all samples. Hopefully, that won't create too much toil. Especially because we won't be adding any new samples to those tutorial pages.

I don't know of any others off the top of my head, @tbpg might. Maybe something in the ML APIs has or had that.
If it is a one-off, it seems interesting to try it to see whether our users tend to like the format or not. @tbpg is a great person to talk to about this. I'll defer to him. Hopefully I'm not stirring the pot too much here!

I have two general concerns with sample CLIs:

Testing the commands documented on cloud.google.com.

This is really "testing the CLI" in general, but the most important part is ensuring what users see works.

Maintaining the CLI itself during sample updates (this is causing some toil now, sorry!)

I don't think I've seen any other APIs that document a CLI for snippets. Of course, there are quickstarts, but those are different.
Only having the CLI for the samples where we document the CLI makes sense to me. It's up to you. It may not be too much extra work to maintain the others.

Thanks for the discussion @tbpg and @cbro. I think we won't make a decision here right now as we need to do it consistently for all languages. As you know, most languages have this requirement to not use command line tools anymore for the samples. So we'll go back to the docs team and discuss our options. We'll then get back to you on if we need to maintain a CLI just for a smaller set of samples.

Sounds good! Thank you for looking into this!


Create Docker composition which includes GeoServer for testing

How would you change it? On my side I have copied all the AG input files in data/ag so it works. I added data/ag in the .gitignore so that we don't publish all this data here

How would you change it? On my side I have copied all the AG input files in data/ag so it works. I added data/ag in the .gitignore so that we don't publish all this data here

Maybe it is enough to add as a docu how it can be used, and/or add a user output in the script. This is what I tried so far:

run: scripts/test.sh -> gives an error, maybe we can tell the user here to download some data first in the directory mentioned
run: docker-compose -f tests/docker-compose.yaml up -> there are some errors "input contained no data", probably just a consequence of the missing directory
open in browser: http://localhost:8380/web/
login as admin/geoserver, click on "Layers" -> there is an error in the console "Error while getting feature type", probably just a consequence of the missing directory

I will retry after putting the data from aargau in an "ag" directory.

OK, I will also have look, because the data in data/ag should not be mandatory to run the test composition

I got a bit further as follows, but I am still missing something:
Preparations for tests:
start the lyrx2sld service (as in the README)
copy all AG data to a subdirectory data/ag
mkdir data/geoserver_data_dir/styles/ag (otherwise there is an error when running the script)
run ./scripts/test.sh
stop the lyrx2sld service
Now to run, I did this, but same effect as before:
docker-compose -f tests/docker-compose.yaml up
There are some errors in the console such as java.io.EOFException: input contained no data
http://localhost:8380/web/
login as admin/geoserver
Click on Layers -> warning in the console "Error while getting feature type, flushing cache and retrying: {0}", nothing happening in UI.

I think the docker composition is missing a database for the layer data. Should we add this as well?
The database connection should not be required to edit and preview the styling (you can try by going to the page "Styles" instead of "Layers") - there you can upload the SLD files (which you have been created by lyrx2sld when running the bash script)


Get-Help misrepresents non-advanced functions as advanced ones when comment-based help is present

Steps to reproduce
# Define a simple (non-advanced) function with comment-based help.
function foo {
# .SYNOPSIS
# Hiya
}

Get-Help foo

Expected behavior
...
SYNTAX
    foo
...

Actual behavior
...
SYNTAX
    foo [<CommonParameters>]
...

That is, the syntax diagram mistakenly suggests that foo is an advanced function.
Environment data
PowerShell Core v6.2.0-preview.3 on macOS 10.14.2
PowerShell Core v6.2.0-preview.3 on Ubuntu 18.04.1 LTS
PowerShell Core v6.2.0-preview.3 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.471)
Windows PowerShell v5.1.17134.407 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.471)

mklement0, my output of the function above in PowerShell Core v6.2.0 -preview.rc1 Windows 10 64-bit is:
´´
C:\Users\ic-sc> get-help foo
NAME
foo
SYNTAX
foo
ALIASES
None
REMARKS
None
``
I think thats fixed , update to rc.1

Interesting, @ChrSchultz, I still see the problem in RC1, both on Windows 10 (64-bit; Version 1809, OS Build: 17763.379) and on macOS 10.14.3


Fix license setting

Fix some problems with license file setting (too short timeout and incorrect process exit code).

Thank you!


Inline latex doesn't allow punctuation following the ending $

Hi
When I try to include an inline equation I can't seem to put punctuation after the ending $. The following examples don't work unless I put a space before the period or comma:
The algorithm has a time complexity of $O(log(n))$.

Let's consider $x$, $y$, and $z$.

Putting a space in any of those locations really doesn't look good, and I think it's incorrect to include the punctuation in the equation itself.
I took a quick look at the regex and I wonder if it's possible to change the final look-ahead from (=?\s|$) to (=?\W|$). The non-word class \W includes spaces, but also common punctuation, which would resolve my issue.
https://github.com/UziTech/marked-katex-extension/blob/afea812770314c7c251c79691a079117d7673dea/src/index.js#L4

Ya I think that could work. Can you create a PR and add a couple tests?

It looks like we would need to update the look ahead to (=?[^\w\}\$]|$)
This will prevent a match if one of the following are true:

A word character is found \w since we don't want to match $300 $400
A squiggly brace is found } since we want to be able to match nested katex $ \hbox{$x^2$} $
A dollar sign $ since we want the same number of delimiters so it won't match $ not katex $$

I think this might be too lenient and allow other types of markdown that is not katex or prevent matching other valid katex.
The other alternative could be to list specific punctuation that we do allow (=?[\s.,]|$). This is how other parts of markdown work specifically adding valid punctuation.
I am also not an expert in katex so I don't know if there is a valid case of nested katex that would have punctuation after $
For example: (im not sure if this is valid katex)
$ some nested katex $x$,  $
If we do decide to list valid punctuation, what punctuation should we allow?

Thanks for looking into this 👍 Good find about the autolink, I didn't consider that.
An explicit list of punctuation that's allowed, just like the autolink example, sounds like a better idea than the broad \W. I don't see why anybody would want to emphasize, bold, or strike-through an inline equation, and it probably wouldn't do what you want, so maybe keep ?, !, ., ,, and : and drop *, _ and ~? I.e. (=?[\s?!\.,:]|$)?
If this list is fine then I can create a PR with tests maybe today or tomorrow. The punctuation list can always be revised if there are any further issues.

Ya that list works for me 👍


when converting CFF => CFF2, tx leaves redundant FontMatrix in FontDict

Take an OTF with CFF 1.0 and a non-standard UPEM (i.e. not 1000), and convert it to CFF2 using cffsubr (which uses tx).
You will see a warning upon compiling the OTF from fontTools.cffLib::L2409, saying that:
Some CFF FDArray/FontDict keys were ignored upon compile: FontMatrix

The CFF2 table that is generated by tx seem to contain an extra FontMatrix operator in the FDArray[0].FontDict which fonttools doesn't expect it to be there and removes it while issuing the warning. This is weird since FontMatrix is a TopDict operator, so it should not be there.
I suspect this could be a bug in tx code or some support library elsewhere in afdko which is responsible for the CFF1 to 2 conversion.
Note the warning only happens when FontMatrix is different form the default [0.001 0 0 0.001 0 0], which for CFF is quite rare given most fonts are drawn at 1000 UPEM.
To reproduce, take a UFO, change its unitsPerEm in fontinfo.plist to something different from 1000, compile the font to OTF with CFF 1.0 e.g. with fontmake or ufo2ft. Then pass it through python -m cffsubr --cff-version 2 to convert it to CFF, and you should see the above mentioned warning appear.
To be honest, I actually am not sure what FontMatrix is used for when CFF/CFF2 are embedded in an OpenType font..

AFAIK, if the font does not use 1000 UPEM, FontMatrix must be used otherwise the glyphs will be incorrectly scaled.

I also think CFF has FontMatrix operator in FontDict (in case different CFF fonts had different FontMatris), but CFF table does not allow more than one font so having it in TopDict is enough.

See also https://github.com/adobe-type-tools/afdko/issues/137

As for CFF2, why is tx adding a FontMatrix to the FontDict (it wasn't there to begin with, it was only in the TopDict of the original CFF 1.0 input table), but then fonttools is dropping from it with a warning?

This looks like a bug indeed, the only operator allowed in CFF2 FontDict is Private, but may be tx was following and old version of the spec (see the note in Font DICT Operator Entries)

may be tx was following and old version of the spec (see the note in Font DICT Operator Entries).

thanks @khaledhosny, now it makes sense. For now, i'll just strip it (#14), but will revise if/when this will  be fixed upstream in afdko. /cc @josh-hadley

Thanks @anthrotype . We should definitely address this upstream. From discussions with @cjchapman that should be pretty straightforward.

@anthrotype & @khaledhosny Josh has fixed this issue in tx in https://github.com/adobe-type-tools/afdko/pull/1165


Cert gen fails

Hi there
for a domain the cert gen fails with the following log entry:
2016-05-20 12:27:20,871:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): <IP_ADDRESS>
2016-05-20 12:27:20,986:DEBUG:requests.packages.urllib3.connectionpool:"POST /enterprise/control/agent.php HTTP/1.1" 200 None
2016-05-20 12:27:20,989:DEBUG:letsencrypt_plesk.api_client:Plesk API-RPC response: <?xml version="1.0" encoding="UTF-8"?>
<packet version="<IP_ADDRESS>">
  <site>
    <get>
      <result>
        <status>error</status>
        <errcode>1023</errcode>
        <errtext>Unkown filter value : wie-gseehsch-us.ch</errtext>
      </result>
    </get>
  </site>
</packet>

2016-05-20 12:27:20,992:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/local/psa/var/modules/letsencrypt/venv.Cr2rM/lib/python2.7/site-packages/certbot/auth_handler.py", line 108, in _solve_challenges
    resp = self.auth.perform(self.achalls)
  File "/usr/local/psa/var/modules/letsencrypt/venv.Cr2rM/lib/python2.7/site-packages/letsencrypt_plesk/configurator.py", line 75, in perform
    responses.append(self.plesk_challenges[domain].perform(x))
  File "/usr/local/psa/var/modules/letsencrypt/venv.Cr2rM/lib/python2.7/site-packages/letsencrypt_plesk/challenge.py", line 28, in perform
    self._init_domain_props()
  File "/usr/local/psa/var/modules/letsencrypt/venv.Cr2rM/lib/python2.7/site-packages/letsencrypt_plesk/challenge.py", line 54, in _init_domain_props
    error_text = str(api_result['errtext'])
UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 32: ordinal not in range(128)

any idea on that?
Thanks for the fish

In version 2.0 we replaced Python-based certbot with PHP-based client: unicode issues should be fixed now.


URL lengthener fetch timeout is too short

When running test crawls on my machine, this is timing out, which kills the job.
I've bumped it to something much bigger, but the real solution is to figure out how to incorporate @vmagotra's work on supporting a max fetch duration in the actual fetcher code.

Actually the real problem is that the HttpClient being used for lengthening (a) isn't setting the maxSimultaneousRequests, so it's defaulting to 1, and (b) isn't setting maxConnectionsPerHost, so it's defaulting to 2. Since we're hitting (mostly) http://bit.ly, we need to have both of these set to fetcherspertask parameter for the job.

There's also the issue of getting the these two attributes of the other BaseHttpFetcherBuilder objects (i.e., for pages, site maps, and robot rules) set appropriately (at least for CrawlTool, if not by default in CrawlTopologyBuilder). Currently, I think only the page fetcher gets its maxSimultaneousRequests set, and none of the three get their maxConnectionsPerHost set.

Note that although FetchUrlsFunction for the page fetcher is constructed using _pageFetcherBuilder.getFetchDurationTimeoutInSeconds() * 2, the builder's instance field isn't being set up first.


Add dropdown checklist for difficulties in database

Fixes #98
Again, I can't test without credentials but it looks as correct as I can get it

I decided not to go with this because 1) bringing in react just for this one solution seemed like overkill; and 2) I was worried about the extra layout shift (especially on mobile) by populating the field with all the selected difficulties.


#126 added a grid-list view for sessions in archive

Pull Request Test Coverage Report for Build 298

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.545%

Totals

Change from base Build 296:
0.0%

Covered Lines:
2769

Relevant Lines:
2781

💛  - Coveralls


sas-cas-server in Init Status after Install

After running the deployment to install Viya 4,
All pods seems to be running fine

except for sas-cas-server-default which is in Init Status

This is seen in the describe output

Appreciate any suggestions.
Thanks,

Hi @thpang
We are using EFS in this environment and it was provisioned separately not using the IAC code. We map it by setting  V4_CFG_MANAGE_STORAGE: false in our ansible vars yaml file.

To add. one of the items we are wondering is where could this path be defined in the scripts:
our storage path is /viya-share/non-prod-dev-viya-ns however seems the deployment is reading it from //non-prod-non-prod-dev-viya-ns/viya-share/non-prod-dev-viya-ns

If you look in the terraform output you can you'll see these entries for your filestore:
rwx_filestore_endpoint and rwx_filestore_path these are the values that are used by the pods to mount into your storage.

@thpang
This is the value from our terraform output for the 2 entries.

Seems rwx_filestore_path configured is "/" directory. But seems still different from the path the deployment reads which is //non-prod-non-prod-dev-viya-ns
Is it possible we have missed something?
Thanks,

Hello i have the same exact problem, but i haven't changed anything in terraform vars neither bring up custom nfs storage server
what could the error be ?

@cceeci99  May I see the output of the command:
kubectl get pods -n
and
kubectl describe pod sas-cas-server-default-controller -n

yeah i know with that command i got the exact same description, i think the problem is that the pods can't find the NFS storage to mount volumes

The // comes from the fact that a variable is not set which would include /viya-share so not quite sure what's a miss here. We never create paths with the // prefix.

I found the problem is that if you configure terraform to make its own network security group you must specify vm_public_access_cidrs so it can open an SSH port to the jumpuser, and when running the deployment with ansible, it will be able to connect to the jumpuser and make the directories that are needed. If it can't connect the ansible somehow manages to skip that error and doesn't create the directories but create all the deployments and that's why the sas-cas-controller could not mount the volumes

That is correct if there is no access to the vm's from ansible it will simply skip those steps without erroring. It's guessing you've handled those directories outside of the DAC code base. Did you set the default_public_access_cidr value or did you leave this empty?

@cceeci99  @thpang
the default_public_access_cidr in our IAC .tfvars file is empty. Is it required for this to have a value?

Also we did not configure the IAC to create a jumpbox as this was already provided by customer separately.

I have this as default meaning empty, because I do not want my resources to be accessed by any other IPs, as I have a VPN In my resource group and everything in the vnet will have access to them

Marking as stale/inactive. If there are further questions please open a new GitHub issue.


package:openNDS

Implement a per-SSID embedded Captive Portal making use of OpenNDS
Authentication type:None
Fix for WIFI-596
Signed-off-by: Nagendrababu<EMAIL_ADDRESS>

merged into pending, Thanks !


The interplay between comments and string literals is flawed

This is relatively easy to fix. However, it is likely to affect work on the incremental parser, and so is best left for now.

Fixed!


feat: entryhuman: Log the caller's relative module path and function

At @terrastruct our codebase is currently structured in layers of similar package names and files.
So just printing the base file name for a log's location is vague.
This PR makes sloghuman print the module path to the file and even make it relative to the main module path
if possible. It also makes slog prints the originating function name.
See the test changes for how it looks for when the package is outside the main module.
Unfortunately go test currently doesn't set the binaries build info correctly yet (https://github.com/golang/go/issues/33976) so module paths in test logs will always be absolute. But this is how a relative path looks for cdr.dev/slog:
<cdr.dev/slog/sloggers/sloghuman_test/sloghuman_test.go:21>       TestMake

->
<./sloggers/sloghuman_test/sloghuman_test.go:21>       TestMake

cc @alixander

@nhooyr we use semantic to enforce standardized commit messages. Please update the title, and I'll merge!

@kylecarbs done!


The Tinymce component drop-down box incorrectly when the page is scrolling

The Tinymce component drop-down box incorrectly when the page is scrolling
Because in fixing this bug: scrollbar and modal component #151
overflow: hidden;
overflow-x: hidden !important;
delete is normal
https://github.com/tinymce/tinymce/issues/3592
It's mentioned at the bottom

src\design\index.less
overflow-x: hidden 与 Tinymce 冲突。
相关menubar 弹框定位不准

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days


"Define a working set" of Nuclide's tree-view is misplaced

[Short description of problem here]
The btn of "Define a working set" is too small。
Screenshots and GIFs

Atom version: 1.14.3
OS and version: macOS

resolve it using the css style
    .nuclide-file-tree-toolbar .btn-group .btn {     padding: 0;     width: 8rem;     border-radius: 0;     margin-bottom: 1rem; }

This is fixed in 2.0 available for Atom 1.17-beta. No more special styles for Nuclide, it works out of the box! 🎉


Errors when opening the window to add a shild node (Godot 4.1.3)

After installing and enabling the plugin, I get the following error traces when opening the window to add a new node to a scene:
  Attempt to open script 'res://addons/utility_ai/considerations/consideration.gd' resulted in error 'File not found'.
  Failed loading resource: res://addons/utility_ai/considerations/consideration.gd. Make sure resources have been imported by opening the project in the editor at least once.
  Attempt to open script 'res://addons/utility_ai/considerations/consideration_from_node.gd' resulted in error 'File not found'.
  Failed loading resource: res://addons/utility_ai/considerations/consideration_from_node.gd. Make sure resources have been imported by opening the project in the editor at least once.
  Attempt to open script 'res://addons/utility_ai/actions/action.gd' resulted in error 'File not found'.
  Failed loading resource: res://addons/utility_ai/actions/action.gd. Make sure resources have been imported by opening the project in the editor at least once.
  Attempt to open script 'res://addons/utility_ai/agent/agent.gd' resulted in error 'File not found'.
  Failed loading resource: res://addons/utility_ai/agent/agent.gd. Make sure resources have been imported by opening the project in the editor at least once.
  Attempt to open script 'res://addons/utility_ai/utility_ai_base.gd' resulted in error 'File not found'.
  Failed loading resource: res://addons/utility_ai/utility_ai_base.gd. Make sure resources have been imported by opening the project in the editor at least once.

Thanks for the report, and sorry that it's taken me a while to look at this.
I've just tried to reproduce this with a fresh install of Godot 4.1.3 and an installation of the plugin from the Asset Library, and I don't see any errors.  Could you please provide some more information about where you got the plugin from and how you installed it?
The paths in your trace are really surprising to me, because they refer to files that I thought I purged completely from the repository...  I can't find any reference to those paths anywhere.


Correct Productive weekday project to return correct svg fir the portfoliio

Resolved using https://github.com/jugshaurya/productive-weekday/commit/f46659fcfd4a6460fedaad781f10cd8395b2cd29


3.x Java 20 support

ASM 9.4.0 supports Java 20
ASM 9.5.0 supports Java 21
https://asm.ow2.io/versions.html
This likely requires upgrades to EclipseLink and Hibernate to support Java 20.
EclipseLink 2.7.12
Hibernate 6.1.7.Final
plugin.hibernate.enhance 6.1.7.Final

As of this writing, Hibernate is not ready yet; the mainline depends on Byte Buddy 1.12.18 which uses ASM 9.4: https://github.com/raphw/byte-buddy/blob/byte-buddy-1.12.18/pom.xml#L71

As of this writing, Eclipselink is ready for Java 20 but not ready yet for Java 21; the mainline depends on ASM 9.4: https://github.com/eclipse-ee4j/eclipselink/blob/8ac1935575d461d3a51672c92a16062645e11bb3/pom.xml#L188


Adding API Best Practices to Software Architecture

Thanks!
I'm not sure if this really qualifies as a book.

Agreed that it's not packaged as a book.  Nonetheless, it's a comprehensive post on the topic, more so than this other related book that is included.  So I thought it would be a valuable resource to share.


Local build failing to run (NO AVX2)

Following the guide for locally building as my test system does not have AVX2 - I am using dual E5-2637v2's.
root@lab:~/turbopilot/build/ggml/build/bin# ./codegen-serve -t 16 /root/turbopilot/models/codegen-6B-multi-ggml-4bit-quant.bin
Illegal instruction
Is it possible to compile without the need for AVX2?

Hi there, I'm working on shipping builds without AVX and AVX2 in the next release but if you are building it yourself locally you should be able to disable AVX 2 by passing -DGGML_AVX2=OFF at the cmake stage.
I.e. full command would be something like cmake -DGGML_AVX2=OFF ..

FWIW release 0.0.5 now contains separate AVX and AVX 2 builds :)

Re-opening as this still exists as an issue, i am running the AVX build, system has 128GB ram with plenty available.
root@lab:~/copilot# ./turbopilot -m wizardcoder -t 32 -f ./models/WizardCoder-15B-1.0.ggmlv3.q4_0.bin
[2023-09-07 22:10:08.248] [info] Initializing Starcoder/Wizardcoder type model for 'wizardcoder' model type
[2023-09-07 22:10:08.248] [info] Attempt to load model from wizardcoder
load_model: loading model from './models/WizardCoder-15B-1.0.ggmlv3.q4_0.bin'
load_model: n_vocab = 49153
load_model: n_ctx   = 8192
load_model: n_embd  = 6144
load_model: n_head  = 48
load_model: n_layer = 40
load_model: ftype   = 2002
load_model: qntvr   = 2
load_model: ggml ctx size = 25608.48 MB
Illegal instruction


Error in creating because of the long text (excerpt)

Error in creating because of the long text:
"SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'excerpt' at row 1 (SQL: insert into mavitm_estate_realty (user_id, title, excerpt, description, slug, category_id, price, status, published, updated_at, created_at) values (...

"excerpt" field can only contain 255 characters, short input text
exaline-ru<EMAIL_ADDRESS>9 Tem 2019 Sal, 14:46 tarihinde şunu
yazdı:

Error in creating because of the long text:
"SQLSTATE[22001]: String data, right truncated: 1406 Data too long for
column 'excerpt' at row 1 (SQL: insert into mavitm_estate_realty (user_id,
title, excerpt, description, slug, category_id, price, status, published,
updated_at, created_at) values (...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/MaviTm/oc_estate/issues/6?email_source=notifications&email_token=AAOW3HKS455HI2CFV7CUDWTP6R3B7A5CNFSM4H7EU6L2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6CRNHA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOW3HIY2NUS3D72ZBLAC73P6R3B7ANCNFSM4H7EU6LQ
.


cleanDestinationDir in the config file is ignored sometimes

hugo v0.81.0+extended linux/amd64 BuildDate=unknown
Specifically, it seems to be ignored if you have never used this option from CLI. After installing Hugo I just put cleanDestinationDir: true in config.yaml of the site I'm trying to make. However, the command hugo never deleted any directories or files that were becoming obsolete. For example, I first built the site with the default taxonomies, but later decided that I didn't need them, so I disabled them with
disableKinds:
- taxonomy
- term

However, after running hugo the directories "tags" and "categories" were still in "public". The same was the case with obsolete files/directories in "content". Only after I have commented the option cleanDestinationDir: true in the config.yaml and ran hugo --cleanDestinationDir from the command line did Hugo build the "public" directory without the obsolete files/directories.
After that, to test the issue, I uncommented the line in the config and ran just hugo. But now the option in config is working and the obsolete files are deleted from "public" as they should. It's like something has been initialized by running hugo --cleanDestinationDir once.

I am unable to reproduce this error. When I create a new site and run hugo with the default config, it creates the two folders. When I modify the config file with the following:
disableKinds = ["taxonomy", "term"]
cleanDestinationDir = true
It removes the two folders from the public folder as expected.

I can reproduce the problem.
TLDR --cleanDestinationDir fails to remove items from the public directory unless the static directory is present.
hugo new site mysite
cd mysite
hugo
find . -type d -name categories -o -name tags

The categories and tags directories are present (PASS).
hugo --disableKinds "taxonomy,term" --cleanDestinationDir
find . -type d -name categories -o -name tags

The categories and tags directories are gone (PASS).
hugo
find . -type d -name categories -o -name tags

The categories and tags directories are present (PASS).
rmdir static
hugo --disableKinds "taxonomy,term" --cleanDestinationDir
find . -type d -name categories -o -name tags

The categories and tags directories are present (FAIL).
mkdir static
hugo --disableKinds "taxonomy,term" --cleanDestinationDir
find . -type d -name categories -o -name tags

The categories and tags directories are gone (PASS).

So, this get skipped if there are no static directories.
I suggest that we in that case just do a "walk and Remove" in the DestionationFs -- but we need to respect the filter that we apply in the normal case:
https://github.com/gohugoio/hugo/blob/3d5dbdcb1a11b059fc2f93ed6fadb9009bf72673/commands/hugo.go#L653

With the help of @jmooring's string of commands and with respect to what @bep mentioned I made the pull request that atleast fixes observed failure


FileNotFoundError: [Errno 2] No such file or directory: '../config/redis_address'

Hello,
I ran into this exception when trying to execute learning_transforms.py. This exception was actually caught but throw an other into ray module. Find here the full stack trace:
/home/luc/anaconda3/envs/pytorch/bin/python /home/luc/PycharmProjects/learning-circuits/learning_transforms/learning_transforms.py
WARNING: Not monitoring node memory since `psutil` is not installed. Install this with `pip install psutil` (or ray[debug]) to enable debugging of memory-related crashes.
/home/luc/PycharmProjects/learning-circuits/butterfly/butterfly_multiply.py:17: UserWarning: C++/CUDA extension isn't installed. Will use butterfly multiply implemented in Pytorch, which is much slower.
  warnings.warn("C++/CUDA extension isn't installed. Will use butterfly multiply implemented in Pytorch, which is much slower.")
/home/luc/PycharmProjects/learning-circuits/butterfly/permutation_multiply.py:14: UserWarning: C++/CUDA extension isn't installed. Will use butterfly multiply implemented in Pytorch, which is much slower.
  warnings.warn("C++/CUDA extension isn't installed. Will use butterfly multiply implemented in Pytorch, which is much slower.")
INFO - Transform_factorization - Running command 'run'
INFO - Transform_factorization - Started run with ID "1"
2019-11-13 09:39:15,160	WARNING worker.py:1406 -- WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.
2019-11-13 09:39:15,160	INFO node.py:423 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-11-13_09-39-15_32391/logs.
2019-11-13 09:39:15,265	INFO services.py:363 -- Waiting for redis server at <IP_ADDRESS>:35395 to respond...
ERROR - Transform_factorization - Failed after 0:00:00!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/luc/PycharmProjects/learning-circuits/learning_transforms/learning_transforms.py", line 176, in run
    with open('../config/redis_address', 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../config/redis_address'

During handling of the above exception, another exception occurred:

Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/luc/PycharmProjects/learning-circuits/learning_transforms/learning_transforms.py", line 180, in run
    ray.init()
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/worker.py", line 1473, in init
    head=True, shutdown_at_exit=False, ray_params=ray_params)
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/node.py", line 103, in __init__
    self.start_ray_processes()
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/node.py", line 427, in start_ray_processes
    self.start_redis()
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/node.py", line 271, in start_redis
    redis_max_memory=self._ray_params.redis_max_memory)
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/services.py", line 576, in start_redis
    system_memory = ray.utils.get_system_memory()
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/utils.py", line 387, in get_system_memory
    memory_in_bytes = vmstat("total memory") * bytes_in_kilobyte
  File "/home/luc/anaconda3/envs/pytorch/lib/python3.6/site-packages/ray/utils.py", line 336, in vmstat
    raise ValueError("Can't find {} in 'vmstat' output.".format(stat))
ValueError: Can't find b'total memory' in 'vmstat' output.

Process finished with exit code 1

Have you got any idea on how to solve this issue?
Thank you
Luc

It might be an error on Ray's side. Can you check that you can run Ray's quick start example (https://github.com/ray-project/ray)?
Which OS are you using? We've only tested on Linux (Ubuntu and Debian).


Begin a non fork version of varnish

As our team is getting smaller, we dont have time to maintain a full varnish demo.
The goal of this ticket is to have a slimed down varnish project that we can more easily maintain.
We will have a theme and a small demo showing the theme and component usage.

[x] make a v2 branch
[x] remove tugboat and other unused stuff
[x] replace antd with mui (app layout and toolbars, etc)
[x] make sure all deps are as up to date as can be
[x] add linting
[x] ensure all skiff config stuff is up to date
[x] bring over mui varnish package as varnish-mui
[x] bring over varnish linting package
[ ] bring over mui varnish app
[ ] fix up all docs and repo comments about v1 vs v2
[ ] officially kill any antd support
[ ] fix dns to point to new site
[ ] deprecate mui repo
[ ] release to npm (as varnish2)

packages:

[x] eslint-config-varnish
[x] varnish-mui

menu and pages for new site:

[ ]  Welcome	| copy content from v1 and update
[x]  App	| -Header-
[ ]  Getting Started	| Getting Started
[ ]  Varnish1 to Varnish2	| Varnish1 to Varnish2
[ ]  Third-Party Libraries	| Third Party Libraries
[ ]  VarnishApp	| Varnish App
[x]  Patterns	| -Header-
[x]  App Layout	| App Layout
[ ]  Menu	| Menu
[ ]  How Do I...	| How Do I...
[x]  Components	| -Header-
[ ]  AI2 Header	| AI2 Header
[x]  AI2 Footer	| AI2 Footer
[ ]  AI2 Logos	| AI2 Logos
[x]  Colors	| Colors
[ ]  Copy To Clipboard	| Copy To Clipboard
[ ]  ErrorBoundary	| Error Boundary
[ ]  Markdown	| add a coming soon page
[ ]  Max Width Text	| Max Width Text
[ ]  Typography	| add a coming soon page
[ ]  Theme	| Theme
[x]  Extensions | -Header-
[ ]  MUI	| copy content from v1 (antd) and convert ot talk about MUI
[ ]  ESLint Config Varnish	| ESLint Config Varnish
[ ]  Nivo	| add a coming soon page
[ ]  Shellac	| Shellac
[ ]  React Router	| Varnish With React Router

proposed future lerna repo structure:

packages

eslint-config-varnish
varnish (stretch)

theme (JSON) (stretch)
shellac (CSS) (stretch)

varnish-mui

components (TSX)
theme (TS)

varnish-nivo (stretch)

theme (TS) (stretch)

apps

varnish-demo

eslint
varnish theme
varnish-mui
varnish-nivo (stretch)

broke this into muiltiple tickets


Build Iot Block app from CI/CD

@kotylamichal

@TomaszAIR can you look at this?

@kotylamichal LGTM, please merge it