[GH-#421] Persisting chainstates by given block hash

In order to get chainstates by given block height we have to persist not only the latest PMT root hash but all previous as well.
Closes #421

What is this feature needed for, I the future we would have to remove it for performance reasons anyways. State in Bitcoin is about 3GB in Ethereum over 50GB. We don't want to store this amount of data for every block.

What is this feature needed for, I the future we would have to remove it for performance reasons anyways. State in Bitcoin is about 3GB in Ethereum over 50GB. We don't want to store this amount of data for every block.

There is no other way to make the get_chainstate_by_height to work. If we do not need to get previous chainstates, we can just remove it.

I did ask what we need it for.

I did ask what we need it for.

@thepiwo there is bug in the get_chainstate_by_height. This issue is about that. The solution that I provided resolves the bug. If You think that we do not need it we can close this issue and make new one to remove get_chainstate_by_height from the chain worker.

Is this function used for something?
We could find a way to recompute older chainstates for reorganization purposes that is more efficient.

Is this function used for something?
We could find a way to recompute older chainstates for reorganization purposes that is more efficient.

@cytadela8 wrote initially this function, maybe he can give more info. I discovered that it is not working and @DanielaIvanova confirmed that. I'm okey if we do not need it at all.

okay, lets keep this fix for now and eventually revise later

Generally I think we should persist last "n" chainstates for some small n. Like about 10-50. That's approximately what Ethereum clients are doing currently AFAIK. Such chainstates are useful in case there is a fork. So we don't have to regenerate current chainstate form genesis. I would merge this for now and add some pruning maybe later to address thepiwo valid concern.
As for my get_chainstate_by_height function. It's not used by any code, but it's useful in debugging when one wants to explore old chainstates. (I would say it's part of our node interface.)


kore: init at 2.0.0

Things done
Include kore web framework

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Thanks for all the feedback, noticed some issues myself as soon as I submitted the PR.

Please review my changes in https://github.com/NixOS/nixpkgs/pull/29989/commits/717e6ac0c55afb075e718a0ec01aedc5a2158420
The patch is needed because without it kore run will fail when kore is not in PATH.

@orivej Is that actually necessary? It substitutes PREFIX in before that, and PREFIX is set to $out

In kore masterit is no longer necessary (src/cli.c#L1399), but in 2.0.0 it is (src/cli.c#L1231).

Ah, I see. Thanks for the help with this, both of you.

Thank you too!

@vyp I managed to get the username JohnMH anyway, so am I good now? :)

@JohnMH yes of course! And also to be fair to you, people commonly do change their github usernames so a lot of names eventually become out of sync anyway (so the @ mentioning wouldn't work), and there's not much nixpkgs can do about that. So I feel like the email address is a more reliable identifier as I would say people change that much less often. It was just really more of a convenience thing and a way to avoid any potential future conflicts. :raised_hands:


proposal: Go 2: tuples as sugar for structs

Updates:

only allow unpack when all fields are exported
unpack skips unexported fields of structs from different packages
include a vet rule
explicitly include a change to Go 1 compat agreement so that unpack is treated the same as unkeyed struct literals

This proposal adds basic tuples to Go with one piece of sugar and two builtins.
The sugar is for struct(T0, T1, …, Tn) to be shorthand for a struct type with n fields where the ith field has type Ti and is named fmt.Sprintf("F%d", i). For example, struct(int, string) is a more compact way to write struct { F0 int; F1 string }.
This gives us a tuple type notation for and answers all questions about how they behave (as the desugared struct they are equivalent to). By naming the fields F0 and so on, this both provides accessors for the individual elements and states that all tuple fields are exported.
The variadic pack builtin returns an anonymous tuple, with the appropriate types and values from its arguments. In particular, by ordinary function call rules, this allows conversion of a function with multiple returns into a tuple. So pack(1, "x") is equivalent to struct(int, string){1, "x"} and, given func f() (int, error), the statement t := pack(f()) produces the same value for t as the below:
n, err := f()
t := struct(int, error){n, err}

The unpack builtin takes any struct value and returns all of fields in the order of their definition, skipping _ fields and unexported fields from a different package. (This has to be somewhat more generally defined as tuples aren't a separate concept in the language under this proposal.) This is always the inverse of pack. Example:
// in goroutine 1
c <- pack(cmd_repeat, n)

// in goroutine 2
cmd, payload := unpack(<-c)

The struct() sugar let's us write pairs, triples, and so on for values of mixed types without having to worry about names. The pack and unpack builtins make it easier to produce and consume these values.
No changes are needed to the public API of reflect or go/types to handle tuples as they're just structs, though helpers to determine if a given struct is "tuple-y" may be useful. go/ast would need a flag in StructType noting when a struct used the tuple syntax but as long as the implicit field names are explicitly added by the parser. The only place this would be needed is for converting an AST back to a string.
The only potential danger here is unpack. If it's used on a non-tuple struct type from a different package it would be a breaking change for that package to add an additional exported field. Go 1 compat should be updated to say that this is a acceptable just as it says that adding a field that breaks an unkeyed struct literal is acceptable. Additionally, a go vet check should be added that limits unpack to structs with exclusively "F0", "F1", …, "Fn" field names. This can be relaxed at a later time.
This is a polished version of an earlier comment of mine: https://github.com/golang/go/issues/33080#issuecomment-612543798 In the years since I've written many one-off types that could have just been tuples and experimented with generics+code generation to fill in the gap. There have been multiple calls for tuples in random threads here and there and a few proposals:

#61920
#32941
#33080 (not tuples per se but related)

I'm not sure the struct() sugar is necessary, but the pack and unpack builtins are brilliant.
What if instead of struct(), we said type MyRecord record{ string, error } was sugar for type MyRecord struct{ F1 string, F2 error }? ch := make(chan record{ string, error }) reads better to me than ch := make(chan struct( string, error )).

I'd be fine with that but honestly I prefer struct because

emphasizes that it is still just a struct
no new keywords

If there is a new keyword it should probably be tuple as record generally means things more like structs

The only danger here is unpack. If it's used on an exported non-tuple struct type from a different package it would be backwards incompatible for that package to add an additional exported field. I think the only way around that is to say don't do that and not consider that an incompatible change. It is safe to unpack a value of type that is meant to be a tuple regardless of its source as the length of a tuple is necessarily part of its API.

I think this makes sense. It's similar to the existing situation with struct literals initialized without field names, i.e. v := Example{2, "something"}. That's technically legal, but it's not safe in terms of backwards compatibility and should be used with care.
One question I have though is whether it would be valid to directly use pack() with the results of a function call. For example, something like
func F1() int { return 1 }
func F2() (int, string) { return 2, "something" }

func main() {
  v := pack(F1()) // Allowed?
  v = pack(F1()) // Still allowed? Same type?

  v2 := pack(F2()) // Allowed?
}

Yes, unpack is basically the mirror image of unkeyed struct initialization.
@DeedleFake I copied your example and put the examples in inline:
func F1() int { return 1 }
func F2() (int, string) { return 2, "something" }

func main() {
  v := pack(F1()) // v is the same as if the call were pack(1) so: struct(int){1}
  v = pack(F1()) // this is still allowed and is the same type

  v2 := pack(F2()) // This is allowed by func call rules and is the same as pack(2, "something")
}

One question I have though is whether it would be valid to directly use pack() with the results of a function call.

I think it should work like the current system where you can do f(a()) if the arguments of f match the return types of a. So you could do:
func Foo() (a string, b int, c error)
func Bar(a string, b int, c error)

packed := pack(Foo())
// ...
Bar(unpack(packed()))

In Python, you need *arg, **kwargs to "unpack" arguments, but since Go is strongly typed, that shouldn't be necessary, and it should just always do the right thing.
One question is if this should work (I think it should):
func Foo() []string
func Bar(...string)

packed := pack(Foo())
// ...
Bar(unpack(packed()))

A vararg is just a final slice, so unpack should be able to just transparently unpack into it.
I am less sure if this should work:
func Foo() (a, b, c string)
func Bar(...string)

packed := pack(Foo())
// ...
Bar(unpack(packed()))

I think probably not, but it's a harder call.

@carlmjohnson You are correct about the return type matching (though you have an extra () in your example code).
Neither of those last two would work, though: changes to varargs would need to be a separate proposal.

This works now, so I guess the unpack version of it should too:
func Foo() (a, b, c string)
func Bar(...string)

Bar(Foo())

Varadics are different issue and I guess can be handled separately.

Interesting. I could have sworn that was an error! Yes that would work.
And I also misread your first example:
func Foo() []string
func Bar(...string)

Bar(unpack(pack(Foo())))

This will not work. It's the same as Bar(Foo()) which needs to be Bar(Foo()...) so you'd need Bar(unpack(pack(Foo()))...) which works. It's a little confusing because the tuple is type struct([]string) so unpack only has a single return which is a slice

In some languages with tuples, a tuple is a type that can be passed to a function and thus part of a function signature. In this design, that wouldn't be supported is that right?

How does struct(T0, T1) work across packages when there is no exported type Foo struct(T0, T1) that gets used? Is this out of scope on purpose? Pardon me if I overlooked the corresponding passage in the draft above.

How does struct(T0, T1) work across packages when there is no exported type Foo struct(T0, T1) that gets used? Is this out of scope on purpose? Pardon me if I overlooked the corresponding passage in the draft above.

IIUC, it would follow the usual rules about type T struct{ /**/ } vs. type T = struct{ /**/ }, where you can pass unnamed struct types between packages without needing a central definition.

@thediveo & @carlmjohnson
https://go.dev/ref/spec#Type_identity

Two struct types are identical if they have the same sequence of fields, and if corresponding fields have the same names, and identical types, and identical tags. Non-exported field names from different packages are always different.

By construction there are no tags and the field names are automated so as long as they have the same number of types in the same order they're the same
proof!: https://go.dev/play/p/Ek8vIDKndw-
this also works if one (but not both) of the packages define the type: https://go.dev/play/p/pUGiqhUxzQB

Okay. I thought it would work, but then I confused myself. This works because all the fields are public, but this does not because the fields are private.

@jaloren

In some languages with tuples, a tuple is a type that can be passed to a function and thus part of a function signature. In this design, that wouldn't be supported is that right?

You could include a tuple in a function signature like:
func F(n int, tup struct(Point, Color))

It's an ordinary type (specifically a struct!) so you can use it however or whenever you would any other type.

I think one common use will be dealing with configuration functions that take more than 4 arguments.

I do not think that will be common at all. As your own example shows it buys nothing over not doing it other than having to include an extra pack and unpack.
Using a defined struct is immensely superior for the use case specifically because you can name the fields and easily omit irrelevant ones by using a keyed literal.
The major use case for tuples is having some types that you need to bundle together but there's no real need for anything other than that. If you've ever written a type with no methods like
type locationWeightPair {
  L location
  W weight
}

just so you could use it as a map key or throw it down a channel, you could just use struct(location, weight) under this proposal.

I really like this proposal. The more I think about it the nicer it seems. I'm trying to figure out if it could help with two-value iterator problem over in #61405, and I think it probably could, but only partially. You could do something like
// t is a struct(string, error)
for t := range tupleSeq {
  v, err := unpack(t) // Needs an extra line, but nicer than needing to manually unpack a whole struct.
  // ...
}

The ability of unpack() to work with non-tuple struct types as well should make the above more straightforward for some cases, too, but that backwards compatibility complication might cause a few problems there.

It would only help with iterators if range auto-expanded tuples, which is what, for example, Python does. Since tuples aren't a separate distinct type in this proposal and there doesn't seem to be any interest in raising the number of loop variables past two so I don't see that happening under this or any other proposal, realistically.
You could write generic functions to convert a 2-iter into a pair and vice versa. That may be useful. The xiter funcs that are supersets of what is commonly called zip and zipLongest define types that are basically tuples.

The ability of unpack() to work with non-tuple struct types as well should make the above more straightforward for some cases, too, but that backwards compatibility complication might cause a few problems there.

That's more necessary than it is useful. I think it would make sense and be fine for some types like image.Point which are essentially just a pair with benefits. For others, to reiterate, it's just the same problem with using an unkeyed literal but in reverse. (If this were on paper I'd triple underline that bit.)

Go already has tuples in the special case where the type happens to be the same for all the items: arrays. For example, [2]int and struct(int, int) are both capable of containing the same amount of information as the other.
Given that, I think it would make sense to expand unpack to work for arrays as well. It's not fundamental to the proposal and it could be added later, so I'm not going to include it for now, but something to consider.

I think there is no need add new keyword or functions, just use () to pack tuple in the right hand side and unpack in the left hand side.
For example:
a := 100
b := "string"
c := []int{1,2,3}

// pack
t := (a, b, c)

// and unpack

(x, y, z) := t
// so x is 100, y is "string" and z is []int{1,2,3}

@leaxoy There are no new keywords in this proposal: just use of one keyword in a new context and two new predeclared identifiers. I don't think just using () can be made to work either syntactically or philosophically. Perhaps I am wrong but I'm not especially interested in that possibility myself as I like the explicit pack and unpack and think they fit with the language better.

Could we allow tuple as a field in a struct ?
type TupleInside struct{
   FieldOne string
   FieldTwo int
   FieldThree struct(int, string, float64)
}

If we allow, how do we marshal/unmarshal the tuple case ?

Do we allow tuple as a field in a struct ?

I think yes, because it’s just a struct with some sugar for the declaration.

If we do, how do we marshal/unmarshal the tuple case ?

When you unpack a struct containing a tuple, the target variable would be a struct of the appropriate type. Just like if you have a, b, c := f() and c is a struct.

@septemhill
The code in your comment is 100% equivalent to
type TupleInside struct {
  FieldOne String
  FieldTwo int
  FieldThree struct {
    F0 int
    F1 string
    F2 float64
  }
}

It's just syntax sugar.

@DeedleFake
Sure, I understand it's just syntax sugar.
For example, if we want to marshal the TupleInside to json
type TupleInside struct {
   FieldOne string                         `json:"field_one"`
   FieldTwo int                            `json:"field_two"
   FieldThree struct(int, string, float64) `json:"field_three"`
}

After de-sugar and marshalling, we would get json as following:
{
  "field_one": "field_one",
  "field_two": 123,
  "field_three": {
    "F0": 234,
    "F1": "f1",
    "F2": 345.345
  }
}

So, that means we cannot customize the tag name for each field in struct(int, string, float64)?
It would always be F0, F1 and F2.
Please correct me if I got something wrong, thanks.

That is correct.

Yes, it is true that struct tags are not a part of this proposal, and so I expect most folks will want to avoid using tuple-like structs in types intended for reflection-based marshalling.
I don't see that as a significant problem, though. Not all types are useful in JSON serialisation, and that's okay. If you are doing JSON serialisation then you will choose your types with that in mind.

There's nothing that would prevent adding tags after the types that I can see, but I don't see a lot of reason to add it at this point. I'm not sure why you would both want the version of structs where you don't have to choose the names and want to specify the names. You can just use a regular struct.

The only danger here is unpack. If it's used on an exported non-tuple struct type from a different package it would be backwards incompatible for that package to add an additional exported field.

This is my biggest concern with this proposal. Yes, it's similar to unkeyed struct literals, but more problematic because, first, it's more tempting to use, and second, writing a vet check for it is harder. The unkeyed-struct-literal vet check just needs to check that the package of the type differs from the package of the literal, but I think that would be too strict for many reasonable uses of unpack.
So two suggestions:

Define unpack to fail at compile time if the struct has any unexported fields. That won't interfere with the intended use case, and it lets type authors protect against this backwards-compatibility issue in the same way they would for unkeyed struct literals.
Add a vet check. The hard part is coming up with one that captures the idea that the type author intended it to be a tuple type. Maybe one simple idea is that the type being unpacked has no explicit definition: it is only defined indirectly by calls to pack.

The only danger here is unpack. If it's used on an exported non-tuple struct type from a different package it would be backwards incompatible for that package to add an additional exported field.

This is my biggest concern with this proposal.

I actually like that this will force downstream unpackers to adapt their code, it's a safety static check. unpack on tuples has a strict contract, or do I mistake the idea of unpack?

My understanding of the concern with unpack is that it would be valid to write something like this:
type Foo struct {
    Name string
}

foo := Foo{Name: "Emma"}
name := unpack(foo)

...and have name be "Emma", because it just destructured the fields in order.
Now if Foo were in a separate package and were to later be modified like this:
type Foo struct {
    Name string
    Age  int
}

...or like this:
type Foo struct {
    Name string
    age  int
}

...then both of them are different kinds of breaking change for the caller of unpack. In the former case, the result arity is now wrong -- two results instead of one -- and in the latter case presumably unpack would need to be forbidden altogether because the caller in another package is not allowed to depend on the presence of that unexported age field.
To me this feels more like a "just don't do that, then" sort of situation, but I do agree that if it's possible to have a compile-time check or lint for it then it'd be worth doing so. Making unpack only accept anonymous struct types and not named types whose underlying type is a struct type might be another reasonable heuristic, in addition to those already discussed above.

I agree that unpack should not work for types with unexported fields. It would just lead to confusion.
I don't think unpacking across packages  is necessarily a problem. It's the same as how this code will fail if foo.Foo is changed:
package foo

type Foo struct {
	A string
	B string
}

// elsewhere
import "foo"

type myFoo struct {
	A string
	B string
}

var _ = myFoo(foo.Foo{}) // breaks if foo.Foo ever changes without myFoo changing

Would pack work with "void" functions?
func foo() {}

v := pack(foo()) // v is struct{}? Or compile time error?

@jba yeah unpack is the hard part.
I don't have a problem with saying all fields must be exported. That seems like a reasonable rule and it could always be relaxed later if it turns out to be too aggressive. I don't imagine there would ever be a need for it to be relaxed.
The simplest vet check would be to only let it be used if all the field names follow the "F%d" naming convention. Another simple one would be to only allow it for types defined in the same module. Both of these fail the image.Point test, which, as I've stated earlier, is an example of a non-tuple that's completely valid to unpack.
I'm not sure how tempting it would be to misuse in practice, though. I can't really think of any struct you'd want to unpack that wasn't basically a tuple. It would get unwieldy quickly as the number of struct fields grows so even misuses would probably be on structs with, say, 6 or fewer fields and even then the majority would likely be 4 or fewer.
If that struct is under your control it's fine. If it's not, and you take on the burden of accepting the possibility of a breaking change, your code will fail to compile pretty early on. The only case where it could go undetected is if two exported fields of the same type were transposed.
If we go with the "no unexported fields are allowed" rule then anyone who feels really strongly can add a zero-sized unexported field and most cases where it would get iffy are already illegal.
It is definitely a theoretical danger but I'm not sure how dangerous it is practically.
@apparentlysmart I don't think named type is a good heuristic because, though the vast majority of the usecases are for leaving the type anonymous, you could do type stuff struct(a, b, c) even if it's just to avoid repeating it all over the place or to add a Less method so it can be stored in a generic container
@carlmjohnson you can't invoke a function with the result of a "void" function so it's already not allowed. Although now that you mention it I suppose you could use pack() as a shorter version struct{}{}, bonus!

@septemhill one more thought about json marshalling. There could be a separate proposal to allow any struct to serialize as a list of values. This is how most other languages handle tuples so it could help interop.
That would allow your example to be:
type TupleInside struct {
   FieldOne string                         `json:"field_one"`
   FieldTwo int                            `json:"field_two"
   FieldThree struct(int, string, float64) `json:"field_three,list"`
}

{
  "field_one": "field_one",
  "field_two": 123,
  "field_three": [234, "f1", 345.345]
}

The struct() sugar let's us write pairs, triples, and so on for values of mixed types without having to worry about names.

Is that the (sole) justification for these struct-tuples that warrants the language change?

@andig that's not a justification so much as its purpose/usecase.
Some justifications for tuples in general

having to write tiny structs just to bundle a few values together is annoying
having to read tiny struct definitions is at best annoying and at worst confusing. Naming a type is an important thing so you expect the definition to be enlightening but it turns out, no, it's just there so two values can be sent down a channel at once or store a pair of items in a collection. With a tuple syntax you can say "ah it's just a tuple" and move on to important details without the false detour
naming things is hard and these let you skip that part if it's not important
it's awkward to work with unnamed structs currently so you're forced to name the type and the fields (see previous points)
pack/unpack make it easy to package and unpackage multiple return values
incredibly common language feature so new users may expect it to exist in some form or just have to deal with it when interoperating with other languages

Some justifications for this proposal in particular

no new keywords
no new kinds of types
explicit but still concise
if you send a tuple to a package written before tuples and it uses reflection everything still works since it's just a struct
any question you have about how tuples work can be answered by desugaring it into a regular struct and then asking the question again

If anyone has others to add, this is the time and place!

I like the idea, but I'm bothered by the inability to unambiguously determine whether something was intended to be a tuple from Go reflection. There's an intent that visible in the Go source code that cannot be observed by reflection.
Consider the following:
type S struct {
    F0 int
    F1 string
}

type T struct(int, string)

From the perspective of Go reflection, both S and T look the same, but are just named differently. It's unclear whether S is definitely meant to be a tuple or not, while T obviously is based on the construction.
The problem is that you can write S today, and it will be JSON serialized as {"F0":0,"F1":""}.
I don't see how we can backwards compatibly JSON serialize T as [0,""] as that would be changing the behavior of S as well. What we need is some type of reflection-based marker to indicate that some struct type with sequential F%d fields was truly intended to be a tuple. It could be a magic method, a new Go kind, or whatever.
To be clear, I want to support serializing of tuples in JSON as I've dealt with REST-based APIs that do this sort of stuff and it's currently a pain to handle it well in Go. This proposal is almost there in terms of addressing this problem.
Regarding https://github.com/golang/go/issues/63221#issuecomment-1740075809 by @septemhill

So, that means we cannot customize the tag name for each field in struct(int, string, float64)?
It would always be F0, F1 and F2.

I don't think we should be able to specify struct tag names for a tuple.
The purpose of a tuple is a lightweight sequence of strongly typed values of disjoint types.
The moment someone wants to customize the names of anything, I'd argue that's what a regular struct is for.

I don't see how we can backwards compatibly JSON serialize T as [0,""] as that would be changing the behavior of S as well. What we need is some type of reflection-based marker to indicate that some struct type with sequential F%d fields was truly intended to be a tuple. It could be a magic method, a new Go kind, or whatever.

My interpretation of https://github.com/golang/go/issues/63221#issuecomment-1741544805 in terms of the approach of go-json-experiment was that a format:list option in the json tag would cause the encoder to marshal a struct field as a list of its exported fields in source order. Then it is independent of whether the struct is concretely written as a tuple, and the benefit is available to non-tuple structs as well.

The downside of a format:list tag is that it doesn't work well for higher orders of variance such as:
[]struct(int, string)
map[string]struct(int, string)

We could use format:list for a top-level struct(...), but anything beyond that either requires a complicated format:... expression.
In this situation, the intent is there in the code, so it seems like we should just bridge that gap into Go reflection. That would avoid any annotation in the struct field tags.

Perhaps the solution is that struct(a, b, c) is sugar for
struct {
    F1 a `tuple`
    F2 b `tuple`
    F3 c `tuple`
}

So it's still just a normal struct, but it comes with something for reflect to see.

If we had the concept of type tags (i.e., basically struct field tags, but for a type declaration), we could do:
struct {
    F1 a
    F2 b
    F3 c
} `tuple`

A type tag would provide other application-specific utility, but the existence of this feature would be beyond this proposal, but may be worth filing separately.

@zephyrtronium your interpretation is 100% correct :dart:
@dsnet

I like the idea, but I'm bothered by the inability to unambiguously determine whether something was intended to be a tuple from Go reflection. There's an intent that visible in the Go source code that cannot be observed by reflection.

reflection couldn't tell if it was written as struct() but it could tell if it could have been written as a struct() if:

it's a struct
each field is named "F" followed by a number that is the same as its position

That's the best you're going to get without creating a new kind of type entirely. The types are identical regardless of the syntax used. You shouldn't be able to tell them apart.
If there were some tags, even ones at the type level that you can't do currently [:+1: to that, btw] someone could just as easily add them to a faux tuple and now we're back in the same position. In fact, if you were transitioning to/from tuples and ordinary structs you'd have to add those in to keep stuff from working so you'd still be able to create a non-tuple that reflects the same. There's not much gained.
There is a good part to a struct() jsonifying as a regular struct: it encodes/decodes the same even in programs compiled with an older versions of Go that don't have the sugar yet.
I can definitely see a desire to automatically encode them as lists, especially when part of a composite. That will have to be a nongoal of this proposal.
Looking at the stated problem, and the larger problem discussed in the video, maybe there could be some mechanism where you tag a field with an identifier like json:"plan(X)" and have a map of identifiers to more complex options. That would let you say "look a lot's going on here, too much to write in a tag, refer to the plan named X for how to handle it" and that plan somehow contains the necessary information.
Regardless, since tuples are just structs anything would apply to structs in general even if this proposal is rejected so whatever the solution would be better as a separate proposal or proposals.

@carlmjohnson I'm not too concerned about 1 vs 0 indexing the field names. 1 is somewhat more traditional for tuples but that's not universal and pretty much everything in the language and most libraries are 0 indexed so it seems like it's a better fit overall. I do like that [2]int and struct(int, int) would be indexed the same. Ultimately it doesn't matter too much since there aren't a lot of cases where you'd actually be using the field names. It's important that they're there but it's not important what they are. They could be numbered "Fa", "Fb", …, "Fz", "Faa", "Fbb", … and it wouldn't really make much of a difference

If there were some tags, even ones at the type level that you can't do currently ... There's not much gained.

There's still something to gain. If type tags were released at the same time as this feature, then we know that there couldn't possibly be any false positives. I'm not worried about people adding the tuple tag after we give meaning to it, but I'm worried about false positives with existing code.

reflection couldn't tell if it was written as struct() but it could tell if it could have been written as a struct() if:

it's a struct
each field is named "F" followed by a number that is the same as its position

At the end of the day, my concern ultimately comes down to changing behavior of existing code.
Here's some hard data based on analyzing all Go source code on the module proxy as of 2023-07-01:

There are a total of ~30.7M struct types

~23.6M are named struct types
~7.1M are anonymous struct types

There are 153 struct types that match the F%d convention

144 are testing related
9 are non-testing related

2 actual unique types as many seem to be forks

1 contains no struct field tags
1 contains struct field tags with json

The testing only ones are uninteresting since it seems that the author just didn't want to come up with actual field names. The non-test ones are more interesting, but I didn't analyze whether they end up being introspected by Go reflection for use with JSON, XML, YAML, etc.
It seems that false positives are probably going to be very rare.
tuples.txt

I think we should adjust the definition of unpack from:

The unpack builtin takes any struct value and returns all of its exported fields in the order of their definition.

to:

The unpack builtin takes any struct value with only exported fields and returns all of the fields in the order of their definition.

Today, you can prevent order-dependent and size-dependent struct literals by including an unexported field.
The adjusted definition of unpack would follow that principle.

Regarding the "why" of this (as invited in https://github.com/golang/go/issues/63221#issuecomment-1741821681) my desire for it is focused on one specific annoyance with current Go:
Function arguments and function return values are currently different than everything else in Go. They are "struct-like" in that they are a sequence of independently-type fields, but the syntax for using them is significantly different than for structs and there are no language features for conveniently and generically bridging the two.
From a practical standpoint, that means it takes an unwieldy amount of adapter code to deal with implementation details such as:

An API which externally "looks like" function call syntax at both ends but internally sends the arguments and/or return values over a channel. (e.g. chan struct(int, string))
A debugging-oriented or testing-oriented wrapper or mock for some interface that wants to produce a slice representing the arguments and return values of all calls. (e.g. []struct(int, string))

In other words, arguments and return values are a non-orthogonal "wart" in a language that otherwise does a reasonably good job of being orthogonal, and aspires to be so. This "tuple struct" concept would not have been my first choice for healing that gap, but it seems like the most pragmatic way I've seen so far to heal it without changing too many fundamentals of the language.
Today it's typical to deal with these concerns by writing a named struct type with fields of the same names and types as the relevant part of the function signature and then hand-write what are essentially specialized versions of the proposed pack and unpack to translate between the function signature forms and the struct forms. Generics allow some reduction of that boilerplate, but still at least require one struct type and one pair of adapter functions for each arity.
This proposal "paves the cowpath" by introducing a more concise syntax for those structs that has a similar appearance and "essence" to an argument list, and by providing built-in pack and unpack functions that cannot be provided as generic library code within the constraints of Go's current generics features. This allows achieving the same effect as the most typical (as far as I know) solution to this problem today, with considerably less distracting boilerplate.
The other implications of this proposal -- such as being able to "unpack" a struct Point { X, Y float64 } into a pair of float64 variables, seem interesting too, but for me this bridge between two important-but-incompatible concepts in the language feels the most significant. My weighing of this particular problem far higher than any others is also why I previously argued that the JSON serialization of a "tuple struct" ought not to be a huge concern, but of course others seem to differ on that; I don't feel strongly about it either way.

For what it's worth, I independently imagined a very similar design in an earlier tuple discussion -- far more clunky in the details, but the same essence. The fact that two people starting from a similar problem statement independently devised essentially the same solution to that problem might be a useful signal, though of course with only two people it's not a strong signal, so I mention it only to complement the other argumentation here.

As another why: my interpretation of why maps.Entries (#54012) was rejected because there was fear that maps.Entry[K, V] would end up getting used everywhere as a generic pair type. This is really pointing at the fact that Go lacks easy tuple-like data structures.

I've updated the proposal per @jba and @dsnet's separate suggestions to limit unpack to structs that only contain exported fields

@dsnet thanks for the numbers and examples. I'm surprised there are so many F%d even in testing code, but that is an argument in favor of this proposal. If there are so many that happened onto the same arbitrary scheme there are likely many more using other schemes in addition to others using ad hoc names.

At the end of the day, my concern ultimately comes down to changing behavior of existing code.

What is the change that you are concerned about? There is nothing that changes in the proposal. There has been some discussion in this thread, mostly related to json, that would introduce changes and I've said we should not do that as it would introduce changes (without opting in).

As far as unexported fields go, currently code is permitted to read unexported fields in the same package and is not permitted to read unexported fields in different packages.  It seems clear to me that unpack should follow that same rule.  You can unpack a struct exactly when you are permitted to refer to each of the fields individually.  So (disregarding complex cases of struct embedding) you can unpack any struct defined in the same package, but when using unpack with a struct defined in a different package all the fields must be exported.

I am troubled by calling unpack on a type defined in a different package, as typically that will break if the type changes: if it adds or removes a field, or even if it rearranges the fields.  This restriction is similar to the current vet check on using a type defined in a different package in a composite literal: we require that they composite literal be keyed.  If we don't have a similar requirement for unpack, then any change to a struct type may be a breaking change.  I don't see how to resolve that except to say that vet should warn about using unpack with any type defined in a different package.  How troublesome would such a vet check be in practice?

I think there would need to be some kind of comment you could add to a struct that means “I promise not to add fields to this struct, so it’s okay to unpack it if you want”. The default would be that it’s not okay to unpack across packages, of course.

@ianlancetaylor using the usual exported/unexported rules would be fine by me.
So is a vet check no matter how strict (it can always be loosened with experience).
If there had been a way to make it strictly the inverse of pack I would have. It being necessarily more powerful than that does create hypothetical issues. Again, though, I don't believe there would be much misuse in practice. Objectionable uses don't seem especially useful and vice versa. That may be lack of imagination on my part, of course.

If there had been a way to make it strictly the inverse of pack I would have.

There was a suggestion up above to automatically add tags to the fields of a tuple. What if unpack() only unpacked, in order, all the fields of a struct that had a certain tag on them, and then just make the tuple syntax sugar add that tag? That way the usage is opt-in per struct field, new fields can be added without breaking it, and it which fields are unpacked should, I think, even be checkable by the compiler and not have any overhead from it.

@DeedleFake that sounds awkward to spec. A vet check is a simpler approach, if anything needs to be done here.

I'm curious how this proposal works alongside what go already calls tuples - function parameters and return values. Would this proposal cause confusion, such that people might be trying to define tuples as (int, string)  like in function definitions instead of specifying the struct type beforehand? Would there be an assumption that these tuples are the same?

That's a complicated question.
I don't think it would never happen, but I don't think it would happen that many times in a row, and I do think that anyone who runs into that would learn the lesson fairly quickly. A small bump in the road but nothing that will slow you down too much.
With the exception of some functional language families, most languages have more of a difference between "function tuples" and "value tuples" than you'd think. Using the same syntax kind of papers over those difference at the expense of a complicated grammar and leaving lots of little special cases all over the place.
Being explicit about construction, destructuring, and type construction avoids those special cases and makes tuple operations easier to spot when reading code. On the whole, I think being explicit would make tuples easier to use and fit into Go better.

That's interesting. Have you thought about how to handle generic code?
Notably for functions, if one wants to abstract over them, tuple will probably be required to describe function arguments and function returns.
So somehow, it might require some more work to make them types. (the return value needs to be assignable to some kind of variable too which has to be of a given "tuple" type)
But then some other questions may arise... What to do with functions which are variadics. What's the arity? i.e. What's the actual size of the tuple? Infinite might be problematic.
Some of the questions are far reaching so I don't necessarily expect an answer but it might be relevant when considering a new feature such as this.

@atdiar that is all far, far out of scope for this proposal.

Well it would still be a good litmus test I to think about it I believe.
Features tend to interact with each other and given that people seem to think about variadic generics too, might be good to take it into consideration as well.

more and more like this proposal, it can generate a lot of new associations such as lambda expression, error handle, etc...

Just today I find myself creating a Result struct just to pass two values through a channel and immediately thought of how much nicer it would be with this proposal.

I wouldn't know about the feasibility of that but it feels unlikely. Is there any precedent for builtins being used in that way?
Not having it certainly isn't ideal but doesn't seem too bad:
case v := <-c:
  x, y, z := unpack(v)

The fact that a select case can support a value, ok assignment to detect if the channel has been closed seems to complicate that.
But with that said, reading out the tuple and then passing it to unpack as a separate step still seems considerably better than the hand-written-every-time boilerplate that's required today. A shorthand that allows reading and destructuring at the same time would be nice, but even without it I'd still like to have what is proposed here.

I'm wondering how the following use of unpack would work:
type B struct{ F0 int }

type A struct{ B }

a := A{
	// The following should work because this works today:
	//  B: struct {
	//  	F0 int
	//  }{
	//  	F0: 42,
	//  }
	B: pack(42),
}

v := unpack(a)
fmt.Printf("%v", v) // Is this '42' or '{42}'?

Since, in this case, pack(42) is equivalent to struct{F0 int}{42} and unpack(a) is equivalent to a.B, it prints "{42}":  https://go.dev/play/p/PgPtYqH7Jhy

I've updated the proposal to include language/spec changes and to endorse a vet rule.
Language: Per @ianlancetaylor: Allow unpack to unpack any struct. unpack always skips _ fields. Unexported fields of a struct from a different package are also skipped.
That is, given:
package X
var V struct {
  A int
  b bool
  C string
  _ struct{}
}

the statement
a, c := unpack(X.V)

is equivalent to
a, c := X.V.A, X.V.C

Vet: There are many complicated rules we could derive to cover as many cases as possible but there are also simple rules that may be good enough:
option 1: do not allow unpack on any struct defined in a different package.
option 2: only allow unpack on a struct struct defined a different package iff all of its fields are exported.
option 3: limit unpack to structs that could be written as a struct() (that is, all fields are named F%d and the number corresponds to the field position)
option 1 is certainly appealing but then that would disallow use with a defined tuple type from a different package. This would break the symmetry of pack and unpack so it's a no go.
option 2 is my mild preference as it basically just enforces the old rule that you can't use it with unexported fields by covert means.
option 3 is probably the safest bet since it in effect only allows unpack to be used with tuples. This is the one that I have included in the proposal.
Regardless of which vet option is taken, I think it's important to add unpack to https://go.dev/doc/go1compat next to the carve out for struct literals to make it official that changes that break unpack are not considered breaking changes. I have made that more explicit.

unpack returning a different number of results depending on whether you use it from the package where the struct is defined seems like it would be surprising the first time someone encounters it, but I think it could be okay if combined with an actionable error and/or vet message when done, as you've proposed. I might've gone a little further and said that unexported fields are always skipped regardless of who's calling unpack, but I can't really defend that as any less surprising than what you proposed, just different.

If the number of results varies depending on the callsite then I expect a likely error is using the wrong number of symbols on the left side of the assignment:
a, b, c := unpack(X.V)

Is the error message that's returned for the wrong number of symbols able to vary depending on what's on the right side of the assignment, or is that just handled generally for all assignments? Ideally I'd like to see the above situation say something specific about how only two results are returned because the unexported fields are excluded, so that there's a hook for someone who encounters this problem to find the relevant section of the docs/spec.
(Perhaps the vet checks are sufficient, but I'm worried about someone who isn't using vet for some reason and so is relying entirely on the direct compiler errors.)

@apparentlymart it only unpacks fields it can see per the standard visibility rules. That seems like the least confusing way to me, but perhaps I am wrong. If you think of desugaring unpack into regular field accessors it helps because the right hand side can then only contain legal references.
Having special error messages for unpack would probably be a good idea in general as it could relate the number of returns and their types to the result of the expression in unpack as though it were desugared. That is of course up to the various compiler writers, though.

This might be a bad question, but what if there is no unpack? It at least none in this proposal.
Instead, what if a tuple, or a packed struct, is always destructed on assignment? This would look like this:
// in goroutine 1
c <- pack(cmd_repeat, n)

// in goroutine 2
cmd, payload := <-c

This would leave the ability to assign a tuple to a variable to be done only if you pack it before assignment.

@urandom u, v := <- c already has the meaning of checking whether c is closed. Similarly for maps. That would be an incompatible change.
I'm also not sure how to consistently represent the idea of forced destructuring except when the assignment is to pack. The assignment you're doing in goroutine 2 in your example is already the result of pack, it just went through a channel first.

@jimmyfrasche
In terms of definition, I think not trying to differentiate between the current package and other packages is simpler. For example, the godoc comment in builtin might say something like

unpack returns all of the exported fields of its argument, which must be a struct, in the order in which they appear in the struct type's definition.

Rewording that to try to allow unexported fields, too, but only if they are accessible from where it's called only makes it more complicated.

unpack returns fields of its argument, which must be a struct, in the order in which they appear in the struct's type definition. It returns all fields which are accessible from its call site, meaning that if the struct type was declared in another package, it only returns its exported fields, but if it was declared in the local package it returns all of it's fields.

I think it makes the most sense to just make it consistently skip unexported fields.

@DeedleFake unpack's specifics are certainly the stickiest bit. To confess, I really don't care what those specifics are as long as they contain the inverse of pack. Anything sensible is fine by me.
That said, I will make this argument for the current revision: it's always exactly the most you could write if you manually desugared it. It's always all the fields that you can access normally. If you can write s.f in normal code then unpack(s) includes s.f in the appropriate place. If you can't write s.f, then unpack(s) doesn't unpack it. It unpacks as much of s as you can, no more, no less.

I think it should just be a compiler error to unpack a struct with unexported fields. And a vet error to unpack a struct across packages if it doesn't have some kind of //go:unpack magic comment or struct tag.

Forbidding unpack on structs with unexported fields is fine by me. I don't endorse that plan and I'd prefer the rule I wrote, but I have no objection to it.
I do object to requiring directives, however. That's noisy and requires perfectly good code to be updated just to opt in to something that's technically allowed means there will be a lot of cases where you can't use it because no one got around to the ceremony or you end up having to vet a dependency update only to find out it just contains blessing for some structs. That's not a good use of anyone's time.

One problem with it being an error to fail if the struct has unexported fields:
If you have a dependency with this code at version N
package p
var S struct {
  X int
}

and this code at version N+1
package p
var S struct {
  X int
  y string
}

then unpack(p.S) is valid when written against version N but you update to N+1 and it's now illegal.
With the rule that only accessible fields are accessed unpack(p.S) is equivalent to writing p.S.X in either case. While it's true there would be an error if S had added a new exported field (or removed X), it is at least immune to changes that it cannot see.
Since S cannot be written with struct() syntax it would fail my proposed vet rule, regardless.

I don't love it being an error to unpack with unexported fields any more than I love the other two possibilities, but it is the one option of the three that constraints future evolution the least: future versions of Go could make it valid in either of the other two ways specified, if practical experience suggests it would be valuable to do so.
I think it's also worth keeping in mind that unpack is there primarily for reversing what pack does, and that its acceptance of arbitrary other types in this proposal is in the interests of a more straightforward spec, not because it's a recommended thing to do in general.
I would still like to see a vet rule that warns against using unpack with any type that doesn't match the conventions produced by pack, to help reinforce that doing so risks your code becoming broken if the struct changes to include an unexported field in future (and, viewed from the other direction, that adding an unexported field to a struct that was previously entirely exported should not be treated as any more bothersome as it is today).

@apparentlymart the current vet check in the top post is exactly that.

Indeed, sorry my intent was to say: even if it were made invalid to unpack a struct type with unexported fields I think the vet check remains important.
In other words, I don't think making unexported fields be an error would replace the need for the vet check, for the compatibility-related reasons we were discussing. (I think I'm just agreeing with you, as far as I can tell)

If it's feasible to spec that you can only unpack a struct that can be written with the struct() syntax that would allow the most leeway for future expansion and would require no vet checks and doesn't introduce any new ways to make a breaking change. I'm not sure if that's an option but if it is it may cause the least unease.

@urandom u, v := <- c already has the meaning of checking whether c is closed. Similarly for maps. That would be an incompatible change.

Even the current spec states that this form yields an additional untyped boolean result  . It doesn't state it is the second value. This should also not be an incompatible change, as u, v := <-c would cause destructing only if the value was packed in the first place. Thus, without even changing the spec, one could still write the special form as u, v, ok := <-c
At least that's my interpretation of it

@urandom
What if you don't want to unpack? And how is a tuple detected, then? The proposal wants pack() to just be a convenience for creating an instance of an otherwise normal anonymous struct type. With your suggestion, using pack() would creating a struct that was uniquely identifiable as having been created from pack(). Unless I'm missing something.

What if you don't want to unpack? And how is a tuple detected, then? The proposal wants pack() to just be a convenience for creating an instance of an otherwise normal anonymous struct type. With your suggestion, using pack() would creating a struct that was uniquely identifiable as having been created from pack(). Unless I'm missing something.

Detection should be possible if pack creates these anonymous struct types that also implement some sort of interface. If one doesn't want to unpack, I was thinking that it might be possible to pack the tuple again on assignment, something like:
tuple := pack(...)

This would however require some special handling in order to be able to produce the special assignment forms for channels and maps.
One reason I'm suggesting this is, to me it seems like the unpack is the harder part, with a lot more edge cases, according to the discussions here. So perhaps avoiding the unpack altogether for this proposal might be easier as a start.

@urandom

Detection should be possible if pack creates these anonymous struct types that also implement some sort of interface.

The tuples are defined to be identical to an anonymous struct. This proposal doesn't really work if they're not.
Also you're not avoiding unpack: you're making it implicit. That doesn't make any of the issues go away it just adds more, unfortunately.
I do understand where you want to go with that but you need to start from a different place to get there. I think you'd both need to introduce tuples as a separate kind of type unrelated to structs and define general destructuring. I like many languages with those things but the more explicit mechanisms in this proposal seem the most likely way to do this in Go.

@griesemer filed a sketch for tuples in #64457. That's not a proposal that is intended to be accepted, just documentation of a potential design. I evaluate it versus this proposal.
There are two bits to each of these proposals: syntax and semantics. They are largely separate and you could bolt the syntax of this proposal onto the semantics of that one and vice versa.
Syntax
The sketch uses the more common (x, y, z) syntax, for both types and values.
It notes that this "directly matches the commonly used mathematical notation for tuples". I don't think that's strictly true. Not all mathematicians use that notation for tuples, though it is pretty much standard in computer science. Regardless, matching mathematical notation isn't a goal in itself or we'd use juxtaposition for multiplication or at the very least write string concatenation with * rather than +. It does however match the syntax used by many languages with tuples, which is a big, big :+1: in its favor.
That does introduce some cons, though

a 1-tuple requires a trailing comma: (1,)
a function returning only an anonymous tuple requires double parens: func() ((int, error))
there is no equivalent to this proposal's pack(f()) where f returns multiple values
it requires a new type in go/ast to represent a tuple type/value

(1,) is inaesthetic but common. The double parens on returns is less common and could cause annoyance fixing compiler errors when forgotten, but that's something I could easily get used to.
The lack of a way to turn a multiple-return into a tuple is a major :-1: against the sketch, at least in my book. It could be rectified by introducing additional syntax specifically for the task but that just adds to the complexity and the number of things to learn.
Requiring a new type in go/ast means that all go tooling that parses source needs to be made aware of the new production making deployment of the feature more costly.
The sketch writes unpack(t) as t... which is shorter.
Semantics
Unlike this proposal, the sketch makes tuples an entirely new kind of type. This means that they can diverge from structs semantics. There is very little in the sketch that does, however, other than making tuple fields inaddressable. The major gain is that ... need not be defined for general structs.
A very large complication of introducing a new kind of type is the reflect package.
Say you have a tuple t and a third party dependency pkg with a function func F(any) and F both uses reflection and was written before reflect.Tuple become a valid reflect.Kind. What happens when you call pkg.F(t)? Well, anything. It could panic, or loop, or just do something slightly wrong that takes forever to trace back to its source.
One thing the sketch has that this proposal doesn't is rules for converting between tuples and structs. Roughly, you can convert a tuple to any struct that has the same types in the same order and vice versa.
That could be added to this proposal if unpack is always allowed and as a special case S{unpack(t)} is allowed. Then you could convert a tuple t to a struct type S with S{unpack(t)}. You could also convert a struct value s into a tuple with pack(unpack(s)). I'm not really sure how useful it is to convert between tuples and structs, though. Generally if I have a struct I'll just use that and the tuple is when I neither have nor need a struct.
Conclusion
I'd be happy with the sketch if it adds some simple way to lift the multiple returns a function invocation into a tuple. Other than that they are essentially the same mechanism. I still have a strong preference for this proposal as it is much simpler.

Personally, I see this proposal treating tuples under the hood as a Go struct as a feature. As the author of several reflect-based packages, I'd rather not think about how to support tuples for the most part. The differences between a tuple and a struct are not sufficiently different that it justifies a separate Go kind. That said, I would still like a single bit in the reflect information to at least distinguish between explicitly created Go struct types versus the implicit structs from tuple-like syntactic sugar if I do happen to care about that distinction in the reflect code.

It's pretty trivial to write a function that most likely tells you if it's a tuple under this proposal: Check if it's a struct and all of its fields match the naming convention and are untagged.  It does not tell you how it happens to have been written, but what important difference is lost there?

Worth noting that Rust also considers its tuples to be structs, but Rust also doesn't have the strict separation between types and data format that Go does. A struct type is defined as
struct Example {
  val: i32,
}

As far as I know, there isn't really an equivalent of Go's type Ex int, only an equivalent of type Ex = int.

there is no equivalent to this proposal's pack(f()) where f returns multiple values

It's not too hard to add one, though. The first thing that comes to mind for me is (f()...). It's a bit weird, but I think it would be pretty consistent.

Oh, another thing that I thought I'd mention: The tuple problem can also be somewhat solved using #56462. For example, it would be possible under that proposal to just do
type Tuple[T ...any] struct { v T }

func Pack[T ...any](v T) Tuple[T] { return Tuple[T]{v} }

func (t Tuple[T]) Unpack() T { return r.v }

func main() {
  c := make(chan Tuple[(int, string)], 1)
  c <- Pack(3, "Example")
  t := (<-c).Unpack()
  // Etc.
}

Use cases for tuples:
Communicating multiple values between goroutines
ch <- pack(1, 2)

Storing multiple values in a generic data structure
m := map[struct(string, string)]{}

Piping multiple values through generic functions that except a single value:
f[struct(int, string)](aTuple)

Lifting a function's multiple returns into a single value (usually in service of one of the above)
v := pack(multipleReturns())

Calling a function with multiple arguments using a single value (from one of the above)
f(unpack(<-ch))

Creating an intermediary group of/index on values within a function/data structure that is necessary for processing but does not escape to the outside world
(this is too hard to illustrate with a one liner)
Have I left any out?
Could you use structs instead? Yes, but then you need to come up with a name and put it somewhere. If you're lucky you can come up with good names for things but often the use case is small so it's simpler to just stick to numbering it (Pair.First, etc.) You can use generics to only define these once but then you still need a new one for every number used. For a large module you could probably stick these in an internal/ package. You could also write out the most commonly useful sizes and put it in a public package but then you need to put that somewhere and all the names in it are Pack2, Pack3, etc.
Should you always use tuples? No. They can make things harder to read and use if they're used injudiciously. It's rarely correct to use as part of an API. They're best used inside an encapsulation barrier when a name is more trouble than its worth and it doesn't matter if they need to be changed since you're in control over all the places that will change.  It's easier to say "ah a tuple of two values" then "what's this type? let's see, go to definition…. Ah, just a pair. Where was I?"

You can use generics to only define these once but then you still need a new one for every number used.

While a tuple may be okay, a triple is already a smell, and a 4-tuple is a stench. An internally defined generic (2-)tuple type should be enough for anybody.

All of my use-cases for tuples are variations on sending a set of function arguments and/or return values through anything that supports only a single value, as described in https://github.com/golang/go/issues/63221#issuecomment-1837604979. Given that, I would suggest that an N-tuple for any N is a "smell" only if a function with N arguments and/or N return values is also a "smell" for the same N.
I agree that a function with a large number of arguments and/or a large number of return values is a situation that would give me pause, particularly in an exported API. However, by my sensibilities N is greater than 4: a function with 3 or 4 arguments seems reasonable to me. I might start being concerned at 5 or more.
I have more tolerance for larger N for unexported functions because the caller and callee are defined closer together, which permits taking greater design liberties. I agree with @jimmyfrasche that I would use this only inside an abstraction boundary, and I would therefore tolerate a larger N before considering it problematic.

I would suggest that an N-tuple for any N is a "smell" only if a function with N arguments and/or N return values is also a "smell" for the same N.

Functions and their parameters almost always have names, whereas tuples and their fields don't. Multiple return values usually have names starting at 3 or when the second parameter is not an error. That's what makes the difference in "smelliness".

I've used 3 and 4 tuples within a function where nothing would have been clarified by using anything else. There is nothing wrong with that. I have certainly seen tuple-mad code and agree that's bad but that's not because of the idea of tuples it's due to their use when something better made more sense.
A pair is the most common n to the point where it would cover a majority of the times you need a tuple (75%+ easy) and I think it's fair to consider only supporting a pair on that basis alone, though I'd prefer a more general mechanism.

Just some comments regarding #64457: It does mention (end of comment) that a possible implementation of tuples would simply consider them as syntactic sugar for light-weight structs. In many ways this seems like the preferred approach because it would be fairly light-weight.
The concern I have with such an approach though is that the respective struct fields will have to be given "automatic" names that are always the same. This leads exactly to the kind of readability problems that others have pointed out with tuples - fields like F0 and F1 etc. are just not very informative. In such cases one should really use a struct with telling field names.
The approach I had investigated was mapping tuples to structs with all blank (_) fields. That would make those fields inaccessible but retain the benefit of the mapping. Because individual fields are inaccessible, there's a reduced danger of using tuples for building data structures where proper structs are the better tool.
Unfortunately this falls apart because at some point in the past we decided that _ fields are ignored when comparing structs. Thus such tuples couldn't be compared. (As an aside, in retrospect, adding the rule that _ fields are ignored when comparing structs seems like a mistake to me: it ties properties of blank field names together with properties of structs and thus connects these in non-orthogonal ways, violating one of the first rules for Go language features.)

If _ for the names worked, I'd be fine with that. Most languages offer some way to get to just the nth field of a tuple but it's not universal and it's rare to have a legitimate reason to access the fields individually. You generally either create one whole or break one into all it's pieces at once (pack/unpack) and if you need to do more you should probably be using something else.
Simulating unlabeled types via a labeling scheme may not be ideal and probably shouldn't be considered for a new language but it is a pragmatic solution for working it into an existing language and for the most part the names can just be ignored.
If I'm writing a package now that needs a single pair I'll just write the struct out and probably give the fields good names, but if I need multiple kinds of pairs I'll make the struct generic and name the fields something like first and second (if not fst and snd) as the only sensible name is its index into the container and not much is gained over F0, F1.
Of course, the Fn pattern is just what I've been using. Maybe FieldN would be more palatable? It could be anything as long as it's

legal
exported
deterministic

I agree that, for my purposes, accessing the individual fields by name is not important at all. The only parts of this idea that I really care about are:

Something like pack and unpack for concisely wrapping and unwrapping argument lists / return value lists send them through language features that only support individual types/values.
A concise way to describe the type of what a specific pack call would produce, so that it's possible to use it as a type parameter without having to define a named type.

Neither of the above require writing down individual field names anywhere, or even knowing that there are field names.

I want to re-emphasize the importance of making existing reflection-based code work well with tuples.
If fields were not individually accessible, it implies that there must be new reflect API to extract all of the elements of a tuple (functionally something that matches pack and unpack in reflection). One possible API would be something like:
package reflect

func Pack(...reflect.Value) reflect.Value
func Unpack(reflect.Value) []reflect.Value

but I suspect this API will perform poorly as it will incur allocations.
Furthermore, it goes against my prior comment that reflection should require close to zero changes to support this feature.
One of the benefits of type parameters is that it had no material impact to Go reflection, thus avoiding a large overhaul of reflection packages to support it. It is a noble goal if tuples, type unions, or any language addition avoid changes to reflection (if possible).

The concern I have with such an approach though is that the respective struct fields will have to be given "automatic" names that are always the same.

There is already an inherent "namespace" for the elements of tuple in that ordering is inherent to the structure of a tuple. Using a numeric index is the most natural representation for ordering. The choice of a "F" prefix in the name is fairly arbitrary, but at least the number portion is a natural outflow of the tuple structure itself.

@dsnet a reflect.Pack/Unpack may be useful when using reflect specifically with tuples even if they're not strictly necessary

Random thought: Make the field names something visible, but awful and non-idiomatic, like Tuple_Field_0, etc., and then add a go vet warning with a comment about pack() and unpack() for direct accesses of fields with that name pattern.

Arguably Fn already satisfies that.
I don't think it would be unreasonable to access the fields, for example, to sort a slice of tuples so I think effectively outlawing it overshoots somewhat.

Hmmm... What if structs allowed numeric field names, without syntactic support for it? You wouldn't be able to define a numeric field name in a normal struct, but when defining a tuple the resulting struct type definition would get numeric fields instead of named ones. Then, allow structs to be indexed into, i.e. v[1], but require the index to be a constant integer. Internally it would still be a struct. reflect would also see it as a struct, and reflect could be allowed to access struct fields with numeric names normally like any other struct field. pack() and unpack() could then be made to only work with these special numeric fields, removing the question of how to use unpack() with unexported fields from other packages.
In other words,
type Tuple struct(int, string)

would effectively be equivalent to
type Tuple struct {
  0 int
  1 string
}

except that the second would be syntactically illegal because you can't start an identifier with a number. pack() would work as proposed originally, but unpack() would, conceptually, start unpack all of the fields starting with the one named 0 and incrementing until it runs out of fields, ignoring all others. It shouldn't be possible for there to be others, but it would give it a clean rule that way. reflect deals with struct field names as strings anyways, so it could just be allowed to deal with them manually through reflect like they were any other exported field.

I could imagine tools and libraries doing the wrong thing by failing to special case numeric field names but not well enough to come up with a specific example.
In that vein I idly wonder if it would be possible to decree that the fields follow an unexported scheme like f%d BUT that anything all uses of the struct() syntax are treated as if they're declared in the same package so that their identity isn't bound to their origin. I don't think that's a great idea as the simpler solution of the F%d scheme being much simpler than alternatives.

I could imagine tools and libraries doing the wrong thing by failing to special case numeric field names but not well enough to come up with a specific example.

I suspect some reflection libraries may only look at exported fields by checking whether the first rune of a name is unicode.IsUpper since the Go spec calls out uppercase identifiers as exported.
There is now a reflect.StructField.IsExported method, but that was added more recently (Go 1.17).
(I'm assuming fields of a tuple should be exported, but that's debatable.)

One argument for numeric fields is that they could be unexported but made exported later. Though yes, I'd prefer they be exported for the rare case where you do want to just take a quick peek in the crate without having to take the whole thing apart.
The argument against doesn't just include reflect but go/ast and co. There is now an assumption that the fields are all legal identifiers so violating that could cause some fun problems.

A numeric field would still be stored as a string. Only go/parser would have to be adjusted if one were to allow actually writing such fields. If t is a tuple, the 1st field could be accessed as t.1. Again, only the parser has to be adjusted.

Only the parser has to be adjusted for the standard library to work. My concern, perhaps unfounded, is tools compiled against the new version whose logic hasn't been updated and doing something wrong because it expects the field name to be a valid Go identifier or as @dsnet points out not realizing that "2" is exported because it has an ad hoc check instead of using one of the various IsExported predicates. If that's the only change that needs to be made to such code it's probably not that bad.

Since the standard selector syntax code generation accessing fields would work if the name is left unaltered. There could be issues if it assumes that it can write out the field name in a new struct definition or create a variable using the field name as a prefix. Other than that the only bugs I can think of it causing are incorrectly categorizing it as exported.

Assuming numeric field names, presumably given
type tup struct(int, int)
var s struct {
  tup
 a, b int
}

both s.1 and s.a are legal

Would s.0777, s.0b10101, or s.5_543 be legal? Both are integer literals per the spec, but I'd be horrified to ever see that in source code. If no, it unfortunately, means we need to declare a subset of the integer literal grammar for this.

it's a natural grammar

@dsnet I'd just allow decimal integers. A leading zero is only permitted if the field name is 0. So s.0, s.1, s.42 but certainly not s.0o8 etc. These are names, not numbers, and we'd get to decide the syntax. The point would be to distinguish them from regular identifiers.

There are two proposals for field access in generics:

(implicit) any field shared by the type set is allowed
(explicit) require a new kind of F T constraint meaning types may only be in the type set if they have a field F of type T

The F%d scheme works with either. The t.1 syntax works with the implicit but not the explicit proposal unless you are allowed to declare numeric fields in an interface but not a struct.

Assuming numeric field names, presumably given
type tup struct(int, int)
var s struct {
  tup
 a, b int
}

both s.1 and s.a are legal

Would unpack(s) be legal and return the same thing as unpack(s.tup)?

Library changes for numeric struct fields:
add a predicate to go/token to check if a string matches ^[1-9][0-9]*$
change token.IsExported to check if the first rune is uppercase or the above predicate is satisfied (ast.IsExported just calls token.IsExported)
in go/ast, note that Ident may contain a numeric string when used in a Field in a FieldList used in a StructType parsed from a tuple or in a SelectorExpr.Sel; possibly add an IsTuple method to StructType
Make a note in go/types.Tuple that there are tuples now but these are unrelated and change the unexported isExported predicate to match the token version; possibly add an IsTuple method to Struct
go/parser,printer,format do not need any visible changes
The reflect IsExported predicates do not need to be changed and would work as-is. Possibly add an IsTuple predicate and Pack/Unpack helpers.
Pros:
No "accidental tuples" so encoding/json could always output a tuple struct as a list without having to worry about backwards compat.
Safe to define unpack to be tuple-only without any contortions.
Undeniably stylish.
Cons:
Possible to cause issues in tools that use go/ast and libraries that use reflect that do their own checks for exportedness or require that field names are identifiers.
More complicated.
Conclusion:
I am starting to lean toward this. It's costlier than the simple F%d scheme but it confers benefit in kind. It may require changes throughout the ecosystem but they'd be extremely niche and most likely very simple to change and could be changed in a way that continues to work with older versions of Go.
Good idea, @DeedleFake!

^[1-9][0-9]*$ implies that we are 1-indexed instead of 0-indexed, right?
Personally, it feels like tuple fields should be 0-indexed to be consistent with how slices and arrays work.

possibly add an IsTuple method to Struct

If tuple T is embedded in struct S, what does reflect.TypeFor[S]().IsTuple report?
I assume the answer is false? It does feel a bit odd that fields are forwarded to the parent, but not the tuple-ness of the child. Something about this just feels off.

A more interesting embedding case is two tuples of different length:
type T1 struct(int)
type T2 struct(int, int)
var s struct { T1; T2; f int }

There's an s.1 but no s.0 because that selector is ambiguous.
If unpack is defined to return only numeric fields unpack(s) would only return s.1.
[warning: this section is not something I'm recommending, just saying this as part of an argument] There's nothing technically blocking allowing users to add numeric fields to struct. A major reason to use them over an identifier scheme is that it avoids incorrectly opting in existing structs. If user defined numeric fields are added at the same time as or later than tuples, there is no such concern. That would allow
var n = struct {
  4 int
  2 int
}{-1, 1}

If unpack is defined  to return only numeric fields unpack(n) could be defined to return 1, -1 or -1, 1.
Given those two things it seems important to

not allow user defined numeric fields
define unpack to only work directly on a tuple

#64613 proposes s... for what is essentially unpack(s) in this proposal. It also extends composite literal rules to allow S{f()} which covers pack(f()) though it requires having a struct type to pack into. This generalizes some of the mechanism here but does not cover all of them. Notably it doesn't allow working with unspecified fields. It is simple to adapt this proposal:

just use ... instead of unpack
keep struct(T0, T1, ..., Tn) for tuple types
either keep pack without unpack or use struct(v0, v1, ..., vN) for constructing tuple values

I generally agree, but I think that I prefer unpack() over .... It reads much nicer in a few places and removes all question of operator precedence with, for example, something like <-c....

#66651 proposes variadic generics that would allow this proposal to be written as ordinary code: (along with generic type aliases which are coming soon, see: #46477)
package tuple

type Of[T... any] = struct { F T }

func Pack[T... any](v T) Of[T] {
  return Of[T]{v}
}

func Unpack[T... any](t Of[T]) T {
  return t.F
}


Frontend for the node graph

Done in #581:

[x] Create a Node Graph panel
[x] HTML/CSS for the graph
[x] HTML/CSS for the nodes
[x] Generating horizontal flow connection curves
[x] Styling horizontal flow connections
[x] Generating vertical composite connection curves
[x] JS for panning/zooming in the graph
[x] Visualizing grid dots neatly at all zoom levels
[x] Starting new flow connections by dragging from output ports

Done in #846:

[x] Swap placeholder HTML for data model-driven Vue templating
[x] Assisting new flow connections by snapping to hovered input ports
[x] Uniting the Vue data model for nodes and link connections with the backend node graph model (or tree-based folder/layer structure if the node graph backend isn't ready yet)
[x] Selecting/deselecting nodes by clicking
[x] Displaying parameters in the Properties panel

Done in #855:

[x] Synchronizing user-drawn links with the Vue data model
[x] Persisting new flow connections by dropping onto input ports
[x] Multi-selecting nodes by Shift-clicking them

Done in #863:

[x] Secondary inputs
[x] Dragging nodes
[x] Snapping the dragged node(s) to the grid

Done in #866:

[x] Exposing inputs to the graph

Done in #869:

[x] Node add menu
[x] Search box in add menu

Later (post-MVP) improvements:

[ ] Lambdas
[ ] Solve Z ordering so links are behind nodes, except when entering/exiting a node's port
[ ] Selecting/deselecting nodes by dragging a box around them (with Shift to append to an existing selection)
[ ] Smooth interpolation between grid squares when being dragged so grid snapping feels less jarring (look into using CSS transitions for this)

Closed by the various PRs listed in the issue description during Sprint 20.


nightly: update sgx stack version to 2.14

The nightly-ubuntu test is in link
The nightly-centos test is in link


Create SkypeWebSDKVersionUpdateNotifications.md

Document the bumped SDK and control versions

@aaronlo717,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

Skype/WebSDK/docs/SkypeWebSDKVersionUpdateNotifications.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[5.3] Query Builder::insert() doesn't handle different columns

Laravel Version: 5.3.26

Description:
So i have a batch insert where the two first rows have this columns a and b
the third row has a, b and c
c is nullable, so i'm not going to pass it to all rows.
I get this error SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 3
I saw that a similar issue has been brought up before and closed
https://github.com/laravel/framework/pull/15689
But please @taylorotwell consider fixing this. Other then the fact that this is a bug, Laravel is known for making things easier and less complex.
Fixing this manually before passing data to Builder::insert() isn't a pleasant thing to do and neither the expected behavior.
Graham please don't close it :D and give this issue a chance so taylor can see it.

Graham please don't close it :D and give this issue a chance so taylor can see it.

Taylor doesn't normally read the issues here. It looks like from the other PR, that this is more of a feature request, than a bug, given Taylor's comment was:

Just prefer to specify the keys explicitly.

Perhaps the best plan of action here is to open a proposal issue on the internals repo so you can get a discussion going. The issue tracker on this repo isn't geared towards that. Hope this makes sense. :)

https://github.com/laravel/internals/issues


Category mapping doesn't accept correct scientific name capitalisation

In the dialogue box for category name mapping: category name is a weed of species scientific name, scientific name does not accept the capitalisation of Genus.

Yes, currently our standard is to always have lowercase to ensure that there is consistency in multiple entries... Is that okay? Or should we use standard capitalisation? Perhaps we should transform the user's input to lowercase, or put a note in the form that lowrcase is expected.

The capitalisation of genus on the explore page is probably something that should be fixed in the long term so it is technically correct, but in terms of priority it's quite far down. As for the form, it took me a bit of playing around to work out why it wasn't liking my scientific name, so accepting any form of capitalisation and transforming it would be ideal I think and probably fairly important thing to change so people have a smooth upload experience.

So we should convert to lowercase for validation in frontend and backend, except for "UNSPECIFIED" which is expected uppercase.


Is it possible to use Nova for non-IVC circuits?

Great project! Thanks to the hard work of the project team!
If I understand correctly, the current Nova is for IVC circuits. That is, the output of the previous Step_Circuit will be used as the input of the next Step_Circuit.
I would like to ask if it is possible to use Nova for non-IVC circuits. Specifically, there are multiple Step_Circuits with the same structure, and the input of each Step_circuit can be customized.
Any insights are appreciated, thanks for the hard work!

Hi @ExcellentHH, thanks for your interest!
Yes, this is possible. Here's a simple approach (there are optimizations which I'm eliding to keep things simple):
Say, you have k instance-witness (x, w) pairs for the same circuit C (instance is sometimes also called public IO). We will design a new step circuit C' that does the following:

At step i, C' takes the ith (x, w) pair as non-deterministic input. It also takes a hash value h as input from the previous step i-1. At step 0, the hash value is set to 0.
Checks if C(x, w) = 1
Output hash(h, x).

So, the IVC proof attests to a statement of the form (k, z_0, z_k), where k is the number of steps of IVC, z_0 is the initial hash value (which should be checked to be 0) and z_k is a hash value that contains a recursive hash (i.e., hash chain) of all instances checked at different steps. So, the verifier can compute a hash chain of all public IOs and check that that hash value matches z_k.
This ensures that the prover checked k different instance-witness pairs such that the instances checked are exactly what the verifier desired.
Please let me know if you need any further information.

Closing this issue given the response. Please reopen if necessary.


Parameter automation of VST

Some things to remember/fix regarding the parameter automation.

[ ] When a SFZ file initialized a CC, this should automate the corresponding parameter in the plugin.
[x] It was reported by @kinwie that CC parameters don't do anything when they are manipulated with Hermann Seib's VSTHost generic UI.
Likely related to the Bitwig console log warning about the CC parameters.
In other VST3 example plugins, the CC parameters are hidden, unlike ours are.

Should be fixed now right?

Yes it should be all good. Some testing in a larger range of hosts would be appreciated.
On host side, there remains this problem that Juce's hosting classes don't notify the controller of changed CC when it happens by MIDI. So don't be surprised if Carla lacks this support. Reaper works well.

I'll close this then ! Thanks !


fish: environment.shellAliases are not available in non-login shell

Issue description
Fish usually has environment.shellAliases available, but the aliases are now missing, when the shell is not a login shell (presumably because of https://github.com/NixOS/nixpkgs/pull/24314).
Steps to reproduce
Have
  environment.shellAliases = {
      lsa = "${pkgs.coreutils}/bin/ls -lahFT0 --group-directories-first";
  };

Then launch fish as a not-login shell. Try to use lsa.
Technical details

System: 17.09pre107265.0afb6d789c (Hummingbird)

Agh, what a mess. Thanks for raising this issue.

I think this issue is a little subtler than one would think: aliases are sourced, but only when the parent shell has not already sourced the interactive configuration.
This is out of line with the behavior of the other shells on NixOS. I have a fix which I've tested with your example here. I will post a link to the relevant PR momentarily (after I put it up).

So apparently outgoing SSH is blocked from the WiFi on this cruise ship, so I can't push commits with the remotes I have set up, and my sister is nagging me to get off the computer so we can go out. If you need a quick fix on your own machine (I will be back later today, while I still have internet access, to submit the new PR), just change the set -gx <...> lines in the NixOS module for fish (/nixos/modules/programs/fish.nix) to  set -g <...> and your aliases will get sourced every time the shell starts, so changes will be reflected without a reboot or manually unsetting the relevant variable before starting a new shell (set -e __fish_nixos_interactive_config_sourced).
(The latter will do the trick if you want a quick fix without any changes to your Nixpkgs tree.)

there's a fix in this commit: https://github.com/NixOS/nixpkgs/pull/25835/commits/183082c1f872f3c6fe2aab10f176d623c8178f80

Thanks!
On Sun, May 21, 2017, 11:11 PM Jörg Thalheim<EMAIL_ADDRESS>wrote:

Closed #25800 https://github.com/NixOS/nixpkgs/issues/25800.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/25800#event-1091208654, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADCJbp8elSK3rvsFk5DUricwH61CTv2qks5r8SclgaJpZM4NbGdY
.


Feat/modal

please check my modal code. this one will be called by future edit-form-components. thanks!

Thanks for the review :)
Sorry for the confusion, i should have impelemented the first task of the edit form user story here. There will be a an edit-button next to each editable place. This solution here was just temporary to test the functionality.

I see, ye that edit icon is exactly what I'm talking about!
If that's something to come in later pull requests, I'm all good :)


MultiPoint.convex_hull doesn't contain all of the points

Given a setup like this (the numpy stuff just gives me a set of points inside a circle in a given coordinate system):
>>> xs, ys = numpy.mgrid[:20,:20]
>>> pts = np.transpose(np.where(((xs - 10)**2 + (ys - 10)**2 < 45).T))
>>> hull = shapely.geometry.MultiPoint(pts).convex_hull

One would assume that all of the points pts are contained within hull. However:
>>> all(hull.contains(shapely.geometry.Point(pt)) for pt in pts)
False
>>> sum(hull.contains(shapely.geometry.Point(pt)) for pt in pts) / len(pts)
0.7956204379562044

So about 20% of the points are missing. In my actual code case, almost all of the points are missing, and a whole bunch of points outside what I would define as the convex hull are included in the convex hull. Is there some explanation for what is going on?

contains() is defined such that a polygon does not "contain" a point that lies on its boundary. The missing 20% in your example are all the points that form the vertices of convex hull. covers will return True for points that are inside a polygon or on its boundary. Some more information here: https://en.wikipedia.org/wiki/DE-9IM
>>> all(hull.covers(shapely.geometry.Point(pt)) for pt in pts)
True

It's worth noting that the hull's exterior contains all of the missing points in this example
>>> points = [shapely.geometry.Point(pt) for pt in pts]
>>> not_contained = [p for p in points if not hull.contains(p)]
>>> all(hull.exterior.contains(p) for p in not_contained)
True

This isn't a bug, but rather a misunderstanding of the specific technical meaning of the "contains" predicate. From the manual:

object.contains(other)
Returns True if the object’s interior contains the boundary and interior of the other object and their boundaries do not touch at all.

This confusion isn't helped by the documentation for the convex hull method. The use of "contains" here isn't the DE-9IM definition, but the more casual meaning. I think that instead this should read "the smallest convex Polygon intersecting all the points".

object.convex_hull
Returns a representation of the smallest convex Polygon containing all the points in the object unless the number of points in the object is less than three. For two points, the convex hull collapses to a LineString; for 1, a Point.

The example below illustrates this. Points that are contained by the hull are coloured green, points that are not contained but do intersect (i.e. their boundaries touch) are coloured blue. Note that there are no red points (i.e. anything else).
import numpy as np
from shapely.geometry import MultiPoint, Point
import matplotlib.pyplot as plt

xs, ys = np.mgrid[:20,:20]
pts = np.transpose(np.where(((xs - 10)**2 + (ys - 10)**2 < 45).T))
hull = MultiPoint(pts).convex_hull

for pt in pts:
    pnt = Point(*pt)
    if hull.contains(pnt):
        style = 'og'
    elif hull.intersects(pnt):
        style = 'ob'
    else:
        style = 'or'
    plt.plot(pt[0], pt[1], style)

xs, ys = hull.exterior.xy
plt.plot(xs, ys, '-k')

plt.xlim(2, 18)
plt.ylim(2, 18)
plt.show()

@jwass You beat me to it!
I do wonder if we need to correct the documentation for convex hull. From Wikipedia:

Formally, the convex hull may be defined as the intersection of all convex sets containing X or as the set of all convex combinations of points in X.

Okay, so my example was bad. My actual code (which isn't working at all) is more like this:
>>> any([shape.geometry.box(x, y, x + 1, y + 1).intersection(hull).area for x in range(pts.shape[0]) for y in range(pts.shape[1])])
False

Basically, I'm trying to figure out how much of every unit square in a grid is covered by the convex hull. My actual code gives me this output:

The purple crosses are the pts, and the green discolouration is the output of the above list comp.

Hi @cyphar,
I'd expect the purple crosses in your example to be a circle centered at (10,10) like the points are in @snorfalorpagus's plot so I'd double check your plotting code.?
Also, we don't have much insight into your specific use here, but I think you're trying to use each point in pts as the corner of a 1x1 box. If so, that would look something like:
boxes = [shapely.geometry.box(x, y, x + 1, y + 1) for x, y in pts]
areas = [hull.intersection(box).area for box in boxes]

Probably worth pointing out that some points that get filtered out in your code have corresponding boxes that intersect the hull. This would explain why the total area of the intersections does not equal the area of the hull.
>>> sum(areas)
119.0
>>> hull.area
122.0

I'm not sure if that matters for you but either way I think we're likely straying beyond a Shapely issue. :smile:

@jwass Yeah, turns out if I remove the shapely code from my code the problem still exists. I'll close this, sorry about that!


Remove pint dependency from Datasette 1.0

Pint still hasn't made itself compatible with Python 3.13. More importantly, the functionality that it supports in Datasette (unit conversions) is not at all widely used, and adds up to some gnarly complexity in the codebase.
I've decided to remove unit functionality and Pint prior to Datasette 1.0
If there's demand for that functionality to come back I'd like to do that as a plugin, adding extra plugin hooks if needed to support it. Those hooks could be added in a post 1.0 version without breaking backwards compatibility, so I don't think we need to solve those prior to the 1.0 release.

Some relevant parts of the code and docs:
https://github.com/simonw/datasette/blob/d444b6aad568e3743199b44d4ae978f5a9ce36a4/docs/metadata.rst?plain=1#L267-L277
https://github.com/simonw/datasette/blob/d444b6aad568e3743199b44d4ae978f5a9ce36a4/datasette/views/table.py#L295-L302
https://github.com/simonw/datasette/blob/d444b6aad568e3743199b44d4ae978f5a9ce36a4/datasette/views/table.py#L1032
https://github.com/simonw/datasette/blob/d444b6aad568e3743199b44d4ae978f5a9ce36a4/datasette/filters.py#L416-L428

Look for ureg and units.

This unblocks:

#2320


ctrl+c is inconsistent

Issue Type: Bug
[ctrl]+c frequently does not copy. I do [ctrl]+c, [ctrl]+v a lot and it works perfect in VS2017 but not in VS Code. I throws me off when I copy something, arrow down and hit [ctrl]+v and it pastes something other than what I expected.
sometimes it works, sometimes it doesn't. I sometimes have to type [ctrl]+c a couple of times to make sure I got it after a couple of failures.
 I'm guessing that means one of the plugins is causing the problem. Is there any way to figure out which one is likely to be the culprit?

@ormico the only way is to selectively disable them. I'm guessing you could also look at their keybinding contributions to find one that conflicts with the standard ones.

I looked and I didn't see anything that was binding to [ctrl]+c though I could have missed something.
I have 34 extensions, so going through them one by one will be a pain.

I think I found it. I believe the issue is the following extension. I'm leaving this here in case anyone else has the same problem.
I had forgotten I had this extension, but after reviewing the description I do not think the behavior that I'm seeing is the intended behavior. I wasn't really using it, so i uninstalled.
Clipboard Ring
sirtobi.code-clip-ring
SrTobi