[HELP] Struggling to understand how to apply textures to faces

I don't understand how this is intended to be done. I think I almost get it? But I feel like I'm just grasping at the air so I don't know if I truly get it.
So
I need to iterate over all the quads, right? And then... I need to figure out what block they correspond to? Can I do that by indexing my chunk with "minimum"??? What does minimum mean? Is that the 3d index for the block that they correspond to? or is that like... one of the corners of the quad? I don't understand.
I need to get the corresponding block for each quad so i can use the block to figure out what texture to use, right? I feel like I could figure things out from there, but. I just. Don't get it.
What is a minimum. Why is it called that? Why is there not a proper example for this? It feels like a common thing users would want to do is map textures to their chunk meshes. There looks like there's an example for UV mapping, but it looks like it just cyclically assigns textures to faces?
Also, I can't even run the UV mapping example even if i do git lfs fetch.

actually, give me a moment, just realized i might want to play with giving the mesher something other than bool voxels and maybe looking at doing that will help me out some
of course, there's a good chance that i will remain clueless, but i'll go ahead and look at that first and then come back if i remain a stump without clue

no, that actually has nothing to do with my problem and confusion, because the only data the voxel trait stores is if its empty, transluscent, or opaque

i feel like the problem might be one of documentation? i dont even know
https://docs.rs/block-mesh/0.2.0/block_mesh/struct.UnorientedUnitQuad.html
does the minimum field refer to the coordinates of the voxel it corresponds to? if so, why doesn't it just say that? why is it called minimum? what is a minimum voxel?

Have you looked at this?
https://github.com/bonsairobo/block-mesh-rs/blob/main/examples-crate/uv_mapping/main.rs

Going to write something up here for anyone confused about the same thing I was.
My code is a modified version of the simple mesh generation.
https://github.com/jameshiew/block-mesh-rs/blob/fix-examples/examples-crate/render/main.rs
    let num_indices = buffer.num_quads() * 6;
    let num_vertices = buffer.num_quads() * 4;
    let mut indices = Vec::with_capacity(num_indices);
    let mut positions = Vec::with_capacity(num_vertices);
    let mut normals = Vec::with_capacity(num_vertices);
    let mut uvs = Vec::with_capacity(num_vertices);
    for (group, face) in buffer.groups.into_iter().zip(faces.into_iter()) {
        for quad in group.into_iter() {
            indices.extend_from_slice(&face.quad_mesh_indices(positions.len() as u32));
            positions.extend_from_slice(&face.quad_mesh_positions(&quad.into(), 1.0));
            normals.extend_from_slice(&face.quad_mesh_normals());

            let block_id = chunk[UVec3::from(quad.minimum) - UVec3::new(1, 1, 1)];
            let attributes = block_id.get_attributes();
            let normal = face.signed_normal();

            let tex_coord = if normal.x == 1 {attributes.tex_coords.east}
                         else if normal.x == -1 {attributes.tex_coords.west}
                         else if normal.y == 1 {attributes.tex_coords.top}
                         else if normal.y == -1 {attributes.tex_coords.bottom}
                         else if normal.z == 1 {attributes.tex_coords.north}
                         else {attributes.tex_coords.west};

            let quad_uvs = face.tex_coords(RIGHT_HANDED_Y_UP_CONFIG.u_flip_face, true, &UnorientedQuad::from(quad)).map(|uv| {
                let mut u = uv[0] * 8.0;
                let mut v = uv[1] * 8.0;
                if u != 0.0 {u -= 1.0};
                if v != 0.0 {v -= 1.0};
                u += tex_coord.x as f32 * 8.0;
                v += tex_coord.y as f32 * 8.0;
                [u/256.0, v/256.0]
            });

            uvs.extend_from_slice(&quad_uvs);
        }
    }

And later:
render_mesh.insert_attribute(
        Mesh::ATTRIBUTE_UV_0,
        VertexAttributeValues::Float32x2(uvs),
    );

To explain some magic numbers, 8x8 is the size of the textures in my atlas, while 256x256 is the size of the atlas itself.
https://docs.rs/block-mesh/latest/block_mesh/struct.UnorientedUnitQuad.html
Minimum is what you want for finding the coordinates of the voxel for the face so that you can figure out how to texture the face. The documentation is written confusingly, but this is what you're looking for.


IPLD Data Model

So, reading #4, there seems to be a few separate issues:

what gets encoded into the wire format
how that gets represented in a human-readable format (JSON, YAML, etc)
how IPLD maps to language-specific datatypes

A major deficiency in JSON is its lack of (user-defined) datatypes. Several workarounds to this issue have been proposed, by reserving a special key in each JSON object:

_type: https://www.npmjs.com/package/typed-json
__proto__: http://tobyho.com/2009/10/02/typed-deserialization-with/

It looks like the @context key proposed in #4 is trying to achieve the same thing.
Whilst this is a reasonable solution for encoding into JSON, I don't think it should be a fundamental part of IPLD, as other representations actually have proper support for representing type information:

CBOR tags: https://tools.ietf.org/html/rfc7049#section-2.4
YAML user-defined datatypes: https://en.m.wikipedia.org/wiki/YAML#Data_types

It would be nice if these features could be supported by IPLD.
So, on the wire, we could have CBOR-tagged data, encoding this into JSON would give something like:
{"@some_reserved_key":"identifier for Person type",
 "name":"David"}

or in YAML:
!Person { name: David }

and mapping into native (say, JS) datatypes:
class Person {
  constructor(name) {
    this.name = name;
  }
}
object = ipld.decode(data, {'person type': Person})

CC: @jbenet

The type information in CBOR is only meant for generic types, universally understandable. It allow tagging binary strings as bigint, or integer as timestamps. It doesn't allow arbitrary tagging of objects for application specific purpose like Person.
This is exactly why the Linked Data directives were created, to link object semantic (and not really typing) to the actual data. Semantic is defined by type URI, like XML namespaces. The URI is the unique identifier that tells how to interpret the data.
JSON-LD permits more than just the CBOR tags can (as for YAML, I don't know enough). The only problem with JSON-LD is that not anything can be encoded with it. Most JSON documents can be updated to fit the LD format, but this does not always come free.
This is why we are trying to construct a JSON dialect that can encode both arbitrary JSON (without LD information) and JSON-LD, while allowing us to add meta information of our own on the data.

Uh, the cbor spec says tags 256 to<PHONE_NUMBER>3709551615 are available for registration, so it's not that limited. In any case you only need one tag to specify (schema uri, data) pairs, which I doubt would be too hard to get registered.
All I'm saying is I don't think ipld should be hobbled by json's deficiencies (lack of typing information or metadata in general), given that json is only one of several possible human readable representation formats, and json isn't even involved if you're decoding wire format directly into native data structures.
If you want the full extensibility of jsonld, then you're free to use that on top of ipld. However, my personal opinion is that ipld at its core should only require the necessary features for it to be able to function properly, agnostic to any particular surface encoding or programming language data model (to the extent this is possible). Otherwise it's not future proof (first it was XML, now json, ...).

Another quote from the spec:

A secondary purpose is to allow optional tagging when
the decoder is a generic CBOR decoder that might be able to benefit
from hints about the content of items.  Understanding the semantic
tags is optional for a decoder; it can just jump over the initial
bytes of the tag and interpret the tagged data item itself.

Which I think fits with what we're doing.

Uh, the cbor spec says tags 256 to<PHONE_NUMBER>3709551615 are available for registration, so it's not that limited. In any case you only need one tag to specify (schema uri, data) pairs, which I doubt would be too hard to get registered.

Yes, that's not so limited indeed (I didn't knew there were so much available ids). And as you say, you can't just use one id per schema. These are just raw integers. You actually need to embed the data in an array with the first item being the schema identifier.
Do you know of a CBOR extension to allow that specifically ? If so, that would be better than using JSON-LD in the first place. And we could output JSON-LD using a conversion step.
I think our data model should allow the Linked Data model as a first class citizen. This model is quite universal, is used in many places, and there is an extensive vocabulary defined for it.
If you look in what I did already on IPLD, I think most of it could be implemented as tagging: https://github.com/ipfs/go-ipld/pull/7 (instead of going to the lengths of escaping the @ character in JSON keys to allow adding arbitrary directives)

Do you know of a CBOR extension to allow that specifically ? If so, that would be better than using JSON-LD in the first place. And we could output JSON-LD using a conversion step.

Not off the top of my head, but I can certainly investigate.

I think our data model should allow the Linked Data model as a first class citizen.

What about separating IPLD objects into metadata and data components (so you don't need to reserve special keys in the data), and you could use the metadata part for storing LD directives, etc?

If you look in what I did already on IPLD, I think most of it could be implemented as tagging: #7 (instead of going to the lengths of escaping the @ character in JSON keys to allow adding arbitrary directives)

Sounds good to me :)
By the way, I'm not trying to change any of the JSON(-LD) stuff you've been working on, I'm just trying to make sure the core IPLD data model is as general as possible. I guess we're both coming at it with different use cases in mind (I'm interested in transparently persisting native datastructures to ipfs).

What about separating IPLD objects into metadata and data components (so you don't need to reserve special keys in the data), and you could use the metadata part for storing LD directives, etc?

By the way, I'm not trying to change any of the JSON(-LD) stuff you've been working on, I'm just trying to make sure the core IPLD data model is as general as possible. I guess we're both coming at it with different use cases in mind (I'm interested in transparently persisting native datastructures to ipfs).

That's what I understood we wanted (and why JSON-LD was good but not quite what we wanted). Hence, this is why we came up until now to the JSON key escaping (with \ everywhere, yay). But if we could implement this more efficiently when using smarter serialization format, I'm all for it.

@mildred I can't see any existing CBOR extensions for dealing with this, so I think the best approach would be to register some tag(s) with IANA.
@jbenet Thoughts? I know you don't want any CBOR magic, but I think that making this separation clear in the wire format is a much nicer solution than reserved/escaped keys within the data itself. This would make it easier to map to encodings like YAML that do have extra support for object metadata. It's also fully compatible with JSON, we would just move the reserved/escaped keys into the JSON encoder/decoder rather than the wire format itself.

Having a data model that is trivially expressible in any format is key. I.e. i should be able to take any object and go into JSON, CBOR, XML, and so on, one-to-one (i.e. roundtrip).
The reason for the JSON data model is that JSON is super, super easy to work with and it's used all over. (i understand this was the same for XML once.) i want to make it extremely easy to use IPFS, particularly web devs. And embedding more complex representations -- i.e. typed -- onto json can be done, and is something we could solve with JSON-LD, JSON-Schema, and so on.
One thing to put you at ease is that multicodec allows us to upgrade the protocol to a new format some day, just as we're upgrading from the first protobuf fmt.
Another thing to keep in mind is that this is blocking IPNS improvements and a number of other things, so we decided to move fwd with JSON data model (which is a very safe choice) to make fwd progress. We can upgrade to add typing within that model, so it's likely fine even for wanting typed things. (If i'm not seeing what you mean though please give more examples?)

@jbenet Sure, something is better than nothing, and we can always fix it later. However, I think a small change to make the wire format a little less ugly would be easier to do now than later once entrenched.
So, my reading of one of your comments is that IPLD would require a @context key to be used for every IPLD object? And the purpose of this @context key is essentially to supply typing information (or schema if you like, but that's just arguing over words)? All the other metadata is optional, and can just be done on top of IPLD at the application level?
If I'm wrong and @context is optional, and not required for IPLD to function, then I agree that it shouldn't be built into the IPLD data model, and can just be done on top of the JSON data model.
Otherwise, if required for IPLD to function, it seems like it would make sense to make @context explicit at the wire level, rather than trying to shove it into an arbitrary key in a dictionary (and then handling the ensuing naming conflicts).
One of the examples @mildred posted could look like the following YAML (for example):
ipld_object:
  !unixfsdir
  attrs:
    mode: 0775
  entries:
    some-dir:
      !unixfsdir
      attrs: ...
      entries:
<EMAIL_ADDRESS>          !unixfsfile
          attrs: ...
          content: !mlink "/ipfs/Qm..."

Decoding into JS object (for example), you'd supply appropriate constructors for unixfsdir, unixfsfile and mlink, and it would give you back objects as instances of the proper class (rather than just a bunch of unstructured dictionaries). You could default standard types like mlink to handlers for transparently traversing the merkledag.

Otherwise, if required for IPLD to function, it seems like it would make sense to make @context explicit at the wire level, rather than trying to shove it into an arbitrary key in a dictionary (and then handling the ensuing naming conflicts).

I second this. Especially since the JSON-LD spec (where the idea of @context came from) specifically allows providing a context separately from the actual JSON file. They show an example using HTTP headers: http://www.w3.org/TR/json-ld/#interpreting-json-as-json-ld
@davidar The thing about adding a type information in YAML is very nice (and you could associate the type name to the type definition the context file). Unfortunately JSON doesn't support such type informaion, and adding a directive (@type for example) would still require to escape all the other JSON keys to avoid conflicts.
The other solution is do as we do now, consider that any object with a mlink property that is a string to be a merkle-link.
In any case, @jbenet, is it possible to take a decision on these points :

are @context required, or just a possibility given to the document author?

if so, what about having the context specified out of the JSON data structure, or do we want to keep it inside?

would we want to allow other directives (such as @type), in which case we must still escape JSON keys? or do we say that there are no other directives allowed (the context file can still contain some information)  and we remove all the weird character escape mechanism?

would we still want to add other IPLD specific @ directives to the JSON (and thus still require to escape @ characters by \@ in JSON keys), or is @context the only IPLD specific directive (and we can get rid of the weird JSON key escaping, I'd actually be happy with that even though #7 would be dramatically simplified)

Unfortunately JSON doesn't support such type informaion, and adding a directive (@type for example) would still require to escape all the other JSON keys to avoid conflicts.

Yes, but that would be a property of the JSON encoder rather than IPLD itself

@jbenet also, what actually exists at /ipfs/<hash-of-mlink>/mlink?

Edit: I'm not trying to make it harder for webdevs --- what I'm suggesting seems easier than handing back unstructured dictionaries that they have to manually decode?

though i agree with you, most of the web dev community does not. this is why the many js class things continue to be unused, json serializing is all still raw objects, and protobuf is also unused.
what i do agree with them on is that "the simple case should be as simple as possible" -- eg. i shouldn't have to create classes or constructors or anything to serialize simply {"hello": "world"}

Otherwise, if required for IPLD to function, it seems like it would make sense to make @context explicit at the wire level, rather than trying to shove it into an arbitrary key in a dictionary (and then handling the ensuing naming conflicts).

I second this. Especially since the JSON-LD spec (where the idea of @context came from) specifically allows providing a context separately from the actual JSON file. They show an example using HTTP headers: http://www.w3.org/TR/json-ld/#interpreting-json-as-json-ld

this is not trivial to do with nice programmatic interfaces. I don't see how this would be exposed to the user nicely. I think escaping is easier to reason about.
please keep in mind that the reason the web is using json (and not rdf, and not xml, and not protobuf, and not ASN.1, and not XDR, and ... ) is the  level of programmatic simplicity. this is paramount.

@davidar The thing about adding a type information in YAML is very nice (and you could associate the type name to the type definition the context file). Unfortunately JSON doesn't support such type informaion, and adding a directive (@type for example) would still require to escape all the other JSON keys to avoid conflicts.

i support types. it's why i was drawn to JSON-LD in the first place.
is there any cross-serialization-format typing definition? i.e. something that's the same in JSON, YML, and so on, and trivial re-coders would get right without any special work?

are @context required, or just a possibility given to the document author?

depends on how we want to handle mlinks.

i always want to allow user to input valid json without an @context.
if we require it, we would add it.
talking with @diasdavid we figured we could get mlink things working first, and go from there, since our mlink thing is doable with a context
i'm leaning towards not required, but not sure.

would we still want to add other IPLD specific @ directives to the JSON (and thus still require to escape @ characters by @ in JSON keys), or is @context the only IPLD specific directive (and we can get rid of the weird JSON key escaping, I'd actually be happy with that even though #7 would be dramatically simplified)

i think if we go the escape route, we should escape all @keys, and give the user two functions, one that accepts input and escapes it, and one that takes input as is (so users can do escaping themselves and manipulate @things etc.).

@jbenet also, what actually exists at /ipfs//mlink?

AFAIU, just a file like:
{
  "@context": {
    "mlink": "<uri-to-mlink>"
  }
}

i think. but <uri-to-mlink> would be content addressed so .... \o/. i asked about this in the #json-ld channel but was told there's no relative linking there, even to self...

Edit: also, I'm still a bit confused about the whole @context thing, and I suspect the the hypothetical webdevs will be too

Yeah... just handling mlink ourselves and ditching context altogether is certainly the simplest thing to do. sadface :/ :/

please keep in mind that the reason the web is using json (and not rdf, and not xml, and not protobuf, and not ASN.1, and not XDR, and ... ) is the level of programmatic simplicity. this is paramount.

Lots of webdevs are familiar with and use YAML though. I agree that APIs are usually JSON, but YAML is also quite common for data serialisation. I believe YAML to also have good library support, and am not aware of any concerns over programmatic simplicity.

i support types. it's why i was drawn to JSON-LD in the first place.
is there any cross-serialization-format typing definition? i.e. something that's the same in JSON, YML, and so on, and trivial re-coders would get right without any special work?

@jbenet Both YAML and CBOR both have a tagging system that can be used to specify types quite naturally. Some quotes from the YAML (1.2) spec:

section 3.1.1
Each YAML node requires, in addition to its kind and content, a tag specifying its data type. Type specifiers are either global URIs, or are local in scope to a single application. For example, an integer is represented in YAML with a scalar plus the global tag “tag:yaml.org,2002:int”. Similarly, an invoice object, particular to a given organization, could be represented as a mapping together with the local tag “!invoice”. This simple model can represent any data structure independent of programming language.

section <IP_ADDRESS>
YAML represents type information of native data structures with a simple identifier, called a tag. Global tags are URIs and hence globally unique across all applications. The “tag:” URI scheme is recommended for all global YAML tags. In contrast, local tags are specific to a single application. Local tags start with “!”, are not URIs and are not expected to be globally unique. YAML provides a “TAG” directive to make tag notation less verbose; it also offers easy migration from local to global tags. To ensure this, local tags are restricted to the URI character set and use URI character escaping.
YAML does not mandate any special relationship between different tags that begin with the same substring. Tags ending with URI fragments (containing “#”) are no exception; tags that share the same base URI but differ in their fragment part are considered to be different, independent tags. By convention, fragments are used to identify different “variants” of a tag, while “/” is used to define nested tag “namespace” hierarchies. However, this is merely a convention, and each tag may employ its own rules. For example, Perl tags may use “::” to express namespace hierarchies, Java tags may use “.”, etc.
YAML tags are used to associate meta information with each node. In particular, each tag must specify the expected node kind (scalar, sequence, or mapping). Scalar tags must also provide a mechanism for converting formatted content to a canonical form for supporting equality testing. Furthermore, a tag may provide additional information such as the set of allowed content values for validation, a mechanism for tag resolution, or any other data that is applicable to all of the tag’s nodes.

JSON doesn't have native support, but there are a few existing conventions using reserved keys that I mentioned in OP (_type and __proto__). Of course you then have to deal with naming conflicts, but this should be quite simple to do in the en-/de-coder, and people are unlikely to be using keys like __proto__ for some other purpose anyway.

In PR #7 We came to agreement that the @ character should be reserved. Normal JSON keys containing the @ character are escaped to replace it with<EMAIL_ADDRESS>For the moment, it is thought to use it for JSON-LD directives, but we can imagine adding some directives of our own.
@davidar what tagging would you like that isn't included in JSON-LD (the spec)? JSON-LD permits already to specify a type by adding a @context key. What's nice about it is that the context can be processed and the object type can be understood by the computer.
If you want something more, I suggest you imagine a scheme involving the special @ character (that will be escaped in IPLD) or the escape character \. Try to find something that wouldn't clash with JSON-LD directive names.

Also, JSON-LD has provisions for a @type key that specifies the type of a literal value (encoded in a string). This can be used for dates for example.

@mildred I don't have a problem with @directives, I just don't think they should be part of IPLD itself, but instead layered on top of IPLD.

@type key ... This can be used for dates for example.

And the CBOR spec recommends using tags 0 and 1 for this purpose. It seems kind of pointless using CBOR as the wire format if we aren't even going to follow the spec.
@jbenet A couple of other thoughts:

could we escape the literal @ as @@ instead of \@ ("\\@" in JSON)? That way, it would only introduce one special character instead of two. The way it is now, if I have a key named foo\@bar, I'd then have to escape it to foo\\@bar, which then becomes "foo\\\\@bar" in JSON, which is a little bit confusing compared to just "foo\\@@bar". [This is similar to the way SQL escapes quotes by writing two (single) quotes.]
renaming IPLD to something like IPS(tructured)D would probably make it less likely to be confused with RDF

And the CBOR spec recommends using tags 0 and 1 for this purpose. It seems kind of pointless using CBOR as the wire format if we aren't even going to follow the spec.

This could be a nice thing to use that would avoid escaping characters as we plan to do. Perhaps we can do as follows:

if we use CBOR backend, use the tag
if we serialize to JSON: escape the keys

@jbenet, what do you think if the keys to the Node are no longer strings bur a struct { string, tag bool }. This way we don't escape strings in the data structure, only when serializing (depending on the value of tag).

could we escape the literal @ as @@ instead of @ ("\@" in JSON)? That way, it would only introduce one special character instead of two. The way it is now, if I have a key named foo@bar, I'd then have to escape it to foo\@bar, which then becomes "foo\\@bar" in JSON, which is a little bit confusing compared to just "foo\@@bar". [This is similar to the way SQL escapes quotes by writing two (single) quotes.]

I would like that :-) But we ruled that out I think :-/ See discussion in PR #7

we can say we double the character to escape it (Ada style "I ""quote"" my words", I like it, it'simple).

This is common, but i've seen it confuse many people. So i've taught many programming classes in the past, so i've a sense of what trips people up, and having different escape characters across programs can be a very confusing thing. People can grasp double escapes much easier (though yes, it's a bit annoying too), than recalling which escape character is for which layered system.

This could be a nice thing to use that would avoid escaping characters as we plan to do. Perhaps we can do as follows:

if we use CBOR backend, use the tag
if we serialize to JSON: escape the keys

:+1:

I would like that :-) But we ruled that out I think :-/ See discussion in PR #7

Ah, fair enough. The discussion in #7 is quite long so I missed that point :)

@mildred I don't have a problem with @directives, I just don't think they should be part of IPLD itself, but instead layered on top of IPLD.

Yeah i would appreciate the separation, but we need something for making sense of the { mlink: <ipfs-path> } construction, and that's at the native level.

It seems kind of pointless using CBOR as the wire format if we aren't even going to follow the spec.

We're not using CBOR for all the CBOR features, just like we're not providing XML encoding for all the XML features. Everything we write MUST have a 1:1 mapping to JSON. That is a strict requirement.

And the CBOR spec recommends using tags 0 and 1 for this purpose.

This could be a nice thing to use that would avoid escaping characters as we plan to do. Perhaps we can do as follows:

if we use CBOR backend, use the tag
if we serialize to JSON: escape the keys

This might be fine, as long as we can guarantee a straightforward 1:1 mapping between the CBOR rep and the JSON rep.
Again, i'm not against using features of the native formats. I'm against breaking any 1:1 mapping across the formats. IPLD is not about maximizing the utility of each native format, it's about creating something extremely easy to work with across platforms and across physical computers. Supporting the pure JSON model is one reason we moved away from protobuf (though we kept a 1:1 mapping for those objects).

@jbenet, what do you think if the keys to the Node are no longer strings bur a struct { string, tag bool }. This way we don't escape strings in the data structure, only when serializing (depending on the value of tag).

No, keys like that are very, very annoying to work with. Instead, just make the datastructure have special functions defined that return things of interest. Or define a subpackage (in this repo) with lots of these nice utils.

renaming IPLD to something like IPS(tructured)D would probably make it less likely to be confused with RDF

Hmm possibly.
It is linked data, even if it isn't "Linked Data". One of my goals is making "linked data" much easier to work with, which might involve using IPLD as a stepping stone toward the RDF model (with the directives re-mapping arbitrary JSON structures to JSON-LD).
Also worth doing soon is a trivial layering of RDF and Turtle on top of IPLD. (i.e. using IPLD as a transport, probably a large array or something). People already asking for this.

Yeah i would appreciate the separation, but we need something for making sense of the { mlink: <ipfs-path> } construction, and that's at the native level.

This is difficult, mostly because there are so many ways to do that using JSON-LD. We have few solutions:

Implement a full JSON-LD processor in IPLD, and make sense of whatever is in the context to parse links
Require the links to be formatted in a JSON-LD compatible way, but do not allow full range of JSON-LD.

Now, there are two ways to implement this in JSON-LD:

consider the link to be a typed literal value. Those values are generally of the form { @type: <literal type>, @id: <literal value> }. We can arrange for the literal value to have the mlink key however.

advantage: there are not so many different ways to do it. We can be compatible with many JSON-LD notations with little effort
drawback: the @type is a full URL (Linked Data schema) and it takes a non zero amount of bytes. If you have many many links, it can take some space

Consider the links a structured value (as opposed to literal). Typically, it will look like {@context: <mlink context>, mlink: <hash>}.

advantage: it's a structured value, we can put other data in it and understood by JSON-LD
drawback: the @context, if repeated, can take some space in the serialized data as well

We can get away and not put @type or @context in every link, but that implies having a more general context that has to be understood. That means implementing a full JSON-LD processor (or restrict further what is put in the context, but it starts to get complicated). But perhaps that's still the best option we have.

@davidar please don't use my handle to document your code :p thank you!

@bar could you say hi to @foo for us sometime?

we need something for making sense of the { mlink:  } construction, and that's at the native level.

I would suggest registering a cbor tag specifically for that purpose, which would minimise overhead at the wire level.

Everything we write MUST have a 1:1 mapping to JSON. That is a strict requirement.

I never suggested otherwise. CBOR tags can be mapped 1:1 to whatever json escaping scheme you want.

it's about creating something extremely easy to work with across platforms and across physical computers.

:+1: That's all I've been trying to do here. IMO cbor tags would be easier to work with than escaping keys at the wire level

It is linked data, even if it isn't "Linked Data".

Hence the confusion ;)
PS: @foo @bar @baz :p

?

@bar sorry, couldn't help myself. For the record, I escaped your handle properly in my comment, it was @mildred who pinged you by quoting me ;)

Crosslinking with https://github.com/ipfs/specs/pull/37

I think this issue has been resolved elsewhere, so closing


Features/gpu support

Description
Full GPU Support, it is now possible to either run ARTIST on cpu or gpu.
Fixes #88
Type of change

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Question about your commit suggestions:
You suggested to change device: torch.device = "cpu" to device: Union[torch.device, str] = "gpu" everywhere, and to use  device = torch.device(device) to transform the string to an actual device. Will this work with the string "gpu"? Is it possible that we should instead use the string "cuda"?

Question about your commit suggestions: You suggested to change device: torch.device = "cpu" to device: Union[torch.device, str] = "gpu" everywhere, and to use  device = torch.device(device) to transform the string to an actual device. Will this work with the string "gpu"? Is it possible that we should instead use the string "cuda"?

Yes, you are completely right, this should of course be "cuda" and not "gpu"! Sorry for that.

In the commit suggestions, you asked why I always used "cuda:3". I understand that it should be just "cuda" in the main branch, because that makes the most sense and looks clean. However when i am computing on dlr machines i need to specify which gpu i am using for example "cuda:3" becuase other people might have reserved some gpu. "cuda" defaults to "cuda:0", I think. Will this be problematic at some point? Do we need to account for the case that in the main or some other entry point like the tests we specify "cuda:3" but since the default in all functions will be "cuda:0" we have two different devices afterall? Or will this never happen because we guarantee that if a device is specified in the beginning it will always be passed down to everything?

In the commit suggestions, you asked why I always used "cuda:3". I understand that it should be just "cuda" in the main branch, because that makes the most sense and looks clean. However when i am computing on dlr machines i need to specify which gpu i am using for example "cuda:3" becuase other people might have reserved some gpu. "cuda" defaults to "cuda:0", I think. Will this be problematic at some point? Do we need to account for the case that in the main or some other entry point like the tests we specify "cuda:3" but since the default in all functions will be "cuda:0" we have two different devices afterall? Or will this never happen because we guarantee that if a device is specified in the beginning it will always be passed down to everything?

Makes sense and I suspected something like that to be the reason. Currently, the GPU-based tests are not run in the Github test workflow anyway, right? Because torch.cuda.is_available() is always False right now. This will only become relevant once Markus set up the CI/CD pipeline using GPU resources from KIT and then it will probably change anyway.
I still think it should just be "cuda" in the main branch and I do not know an elegant reason for this right now. Maybe you can just introduce a global parameter for your tests where you set the device as you need it, run your tests locally, make sure everythings works as expected, and then change back to "cuda"  before pushing so that you do not have to change it everywhere at least:
For running your tests locally:
cuda = "cuda:3"

@pytest.fixture(params=["cpu", cuda] if torch.cuda.is_available() else ["cpu"])

Once everything runs as expected, before pushing:
cuda = "cuda"

@pytest.fixture(params=["cpu", cuda] if torch.cuda.is_available() else ["cpu"])

Another solution could be to just use "cuda" and set the GPU to use via the environment variable CUDA_VISIBLE_DEVICES, like:
CUDA_VISIBLE_DEVICES=3 python script.py  # Run script with GPU 3 only visible
Not sure whether this would work on the DLR system. But I think this could be a good solution.
Maybe, @Markus-Goetz or @kalebphipps know a better solution.
However, I do not think that we run into conflicts "cuda:0" vs. "cuda:3" as we pass down the device specified in the main by the user explicitly (at least, that is what we should do).

In your code, you consistently added a parameter device of type torch.device to function / method signatures whenever needed. However, the specified default value "cpu" was of type string and not torch.device. While this might work technically, I find it slightly inconsistent (I am kind of surprised that mypy is okay with that :D). I suggest to enable passing both str and torch.device-type values for device via: device: Union[torch.device, str] in the function signatures and converting the passed device to a torch.device in the beginning of each function / method (converts a string to a torch.device while nothing should happen for a torch.device. If a function contains only one occurrence of device, this conversion can also be done directly. I tried to make corresponding commit suggestions whenever possible. Note that this change involves importing Union from the typing package in all affected module files.

I imported Union in all affected files.
The device is now  device: Union[torch.device, str] everywhere.
In the beginning of each function device is converted from string to torch.device

I talked to @Markus-Goetz and @kalebphipps and they would prefer "gpu" as the default instead of "cpu" (different from what I told you last week -- sorry!). I tried to change this via commit suggestions in all signatures and docstrings.

The default is now cuda (in the code as well as in the docstrings)

Please once more check the docstrings and add device if it's not there (should only be one or two cases).

I think I found and added all missing docstrings

Please write full docstrings including "Parameters" section for test functions (this has rather low priority but I nevertheless would be happy to have that)

The docstrings in the tests are now complete

@Markus-Goetz suggested to have a utility function set_torch_device to set the device globally.

I dont really understand what that would look like.

Let's discuss @Markus-Goetz's suggestion regarding set_torch_device in our meeting tomorrow. I am also not sure whether we really need something like this right now as we pass the device explicitly from the most top-level main script down to all classes and functions everywhere.
Other than that, the code looks very good to me and I think we can merge this tomorrow after our meeting 🙂.


Refactor osdctl account servicequotas

Refactors osdctl account servicequotas describe to remove dependence on k8sclusterresourcefactory.
This PR also removes the update.go file, as there was no way to run it currently, as well as common.go as the coveredRegion map was never used for its values, only its keys.
This command now only works with a cluster ID. If a version of this command is required with an AWS Account ID, rather than tied to a cluster, this should be done in a separate PR.

/lgtm So clean !


Network issues

I've noticed that if my network goes wonky or I lose signal, the songs get all messed up.  They often stop halfway through.  I've tried the following:

Create a NetworkHelper to detect connectivity
When the onLoadError fires, I use the NetworkHelper to determine if the phone is online.
If it isn't online, I put the thread to sleep for 5 seconds and then try player.setPlayWhenReady(true)

This doesn't really seem to be that effective.  Wondering what your thoughts are around this?
Also noticed that you guys are working in a much newer branch -- any plans to merge that into master?

Sorry for late response. I would like to invite @rikochet to reply Android questions

And yes we are working on the media-caching branch that can do audio caching.
e.g. caching the latest 50 songs streamed so users don't need to download again.
We are using this branch on our app indiecast.fm right now. We will merge it soon.

@victor36max thank you for a very nice library. I was wondering what the current state of media-chaching (and the other caching branches) is?  Are they ready to be used, are you still planning to merge them?


add example notebook for mongodb

Add example notebook for StorageInterface functionality

Pieter, when do you think this notebook is done and can be reviewed? Currently there is some parameter mismatch with qilib.

Pieter, when do you think this notebook is done and can be reviewed? Currently there is some parameter mismatch with qilib.

@QFer When there has been a new release of qilib on pypi.


Check for stacks property on token

Check for stacks property on token before trying to access it.
Error before:

Error after:

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Need allow empty schema diff when schema version increased.

Enhancement
Before, update schema version and set schema diff in the same transaction. So after schema version grow up, we can get the schema diff at same time.
For now, We don't allow When schema version grow up, schema diff can not be read.
But now, update schema version and set schema diff not in the same transaction.

TIDB can only make sure the schema diff from lastest schema version - 1 is valuable.

If the schema diff from lastest schema version - 1 is empty, Then it means may failed in TIDB, in TiFlash, we can skip it.
If the schema diff from lastest schema version - 1 is empty, Then Tiflash should direct apply the schema diff.

TIDB can't make sure the schema diff with  lastest schema version is not empty.

**So we need to allow empty schema diff when schema version increased. **

related to the feature concurrent DDL https://github.com/pingcap/tidb/issues/32031


Hostpath provisioner CrashLoopBackOff: exec format error

I have a multi node microk8s cluster with a mixture of x86 and arm64 nodes. My host path provisioner pod is in a crash loop due to an exec format error, its currently scheduled on a arm64 node.
Looking at the image on the pod it's using this:
cdkbot/hostpath-provisioner-amd64:1.0.0

Is it expected that the host path provisioner should work in a mixed arch cluster?
~ microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: <ip 1>:19001 <ip 2>:19001 <ip 3>:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    rbac                 # Role-Based Access Control for authorisation
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    dashboard-ingress    # Ingress definition for Kubernetes dashboard
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    host-access          # Allow Pods connecting to Host services smoothly
    inaccel              # Simplifying FPGA management in Kubernetes
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    kata                 # Kata Containers is a secure runtime with lightweight VMS
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    openebs              # OpenEBS is the open-source storage solution for Kubernetes
    openfaas             # OpenFaaS serverless framework
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    registry             # Private image registry exposed on localhost:32000
    traefik              # traefik Ingress controller for external access

Hi @bryopsida, try the following:
microk8s.kubectl set image -n kube-system   deployment.apps/hostpath-provisioner  hostpath-provisioner=cdkbot/hostpath-provisioner:1.1.0

We recently pushed out a new multi-ach image (cdkbot/hostpath-provisioner:1.1.0) for the hostpath storage addon.

Hi @bryopsida, try the following:
microk8s.kubectl set image -n kube-system   deployment.apps/hostpath-provisioner  hostpath-provisioner=cdkbot/hostpath-provisioner:1.1.0

We recently pushed out a new multi-ach image (cdkbot/hostpath-provisioner:1.1.0) for the hostpath storage addon.

That fixed it thank you! Tested by patching this on the deployment to force it on one of the arm64 nodes.
nodeSelector:
  kubernetes.io/arch: arm64


Enhancement done in the Contact page with borders

Fixes Issue🛠
#967
Closes #
Description👨💻
Enhancement is done in the Contact page by adding border-bottom CSS that improves UI of the website.
Checklist✅

[✅] My code follows the style guidelines of this project
[✅] I have performed a self-review of my own code
[✅] I have commented my code, particularly in hard-to-understand areas
[✅] I have added demonstration in the form of GIF/video file
[✅] I am an Open Source Contributor

Screenshots/GIF📷

@agrwal697
Mention your issue number.

close issue number #967 Enhance the Contact page with borders.

Enhancement done in the Login/Register page with borders in every text item, its have better UI.
closes issue no. #967
with the label and level of GSSOC.


Implement desktop client with Electron

I would like to implement the desktop client with electron under GSOC`16.
Please let me know of further steps.

The Zulip desktop app is a thin wrapper around the main Zulip web app (only ~2500 LOC that's not 3rd-party code). The goal of that project would be to replace the QT+webkit desktop app with an electron app implementing the same functionality; if that's completed with time the spare, there's improvements to the app that we could add onto the project (e.g. making the desktop app support multiple teams at the same time nicely).  So the core application logic of the Zulip webapp we get to keep, but we'd be rewriting the "desktop app" codebase.
If you’re excited about working on this project via GSOC, the best thing you can do now is get involved in the Zulip development community, by getting a Zulip development environment setup and submitting pull requests to fix some bugs; see https://github.com/zulip/zulip#how-to-get-involved-with-contributing-to-zulip for how to navigate our issue tracker; you'll need to understand how the web application works to effectively do this, so I'd recommend targeting some web bugs.
(Also, you'll want to put together a proposal following these guidelines:
https://summerofcode.withgoogle.com/organizations/5197525313126400/)
Since the Electron project would be a change to the Zulip desktop app, not the Zulip server GitHub project, I'm closing this GitHub issue; we can discuss this project further on<EMAIL_ADDRESS>or the zulip-desktop bug tracker.


HTML to Markdown: extra newline characters around un-converted tags

I'm encountering an issue where two newlines (i.e. markdown paragraphs) are inserted after  elements. The same happens if it is a
 or a made-up <pqr> tag.
Here's a very simple test case:
const showdown = require('showdown');

const converter = new showdown.Converter();

const sampleText = '
A test case to see if I can reproduce the extra newline problem.
';

const ret = converter.makeMarkdown(sampleText);

console.log(ret);

Where I'd expect the result to be:
A test case to see if I can reproducethe extra newline problem.\n\n
I get:
A test case\n\n to see if I can reproduce\n\n the extra newline problem\n\n.
I don't expect this is the desired behavior... Is there an option I'm missing that would solve this problem? Or is it a bug?
UPDATE:
I seem to have fixed the problem for myself by overriding showdown.makeMarkdown.node, where rather than
default:
  txt = node.outerHTML + '\n\n';
  break;

I have:
  default:
    txt = node.outerHTML;
    break;
}

Thanks. I will look into it.

@SyntaxRules This is still an issue, and I can't figure out how to override it like @andosteinmetz did.


Feature/Multiprocessing

Enable multiprocessing support for Language.pipe()
Move general tests to single file test_spacy_udpipe.py

LGTM, but please update this CR after the first one is pushed to make sure I don't miss anything, then I'll review again.

Ofc :)


March towards > 90% unit test coverage.

I'm not a huge believer in full 100% coverage because at some point you start contorting badly for diminishing returns.  That said, we're no were near that point.  We need to go back through the code and start getting our coverage up.

Migrated to trello.


Expose action to listener

Today if you want to know if something changed you have to compare your previous state to the new state and in that case you might need to compare different values with one another.
With this PR we now also expose the action to the listener, making it easier to do something when an action was executed.
store.subscribe((action) => {
  if (action.type === 'SOMETHING') {
     // do something else
  }
});

@sandrinodimattia Was discussed tons of times. https://github.com/reactjs/redux/pull/1697

Whoops, thanks for pointing that out @TrySound ☺
Closing...


Get Mailbox to support canceled tasks gracefully.

Description
Without this when CancellationTokens were in use or the OperationCancelledException was explicitly thrown by an actor, it was causing that actor's Mailbox to freeze forever and never continue processing the new messages.
This technically fixes issues inside the consul cluster provider as well, because of the consuldotnet library using HttpClient and that throwing TaskCancelledException in certain cases, such as Consul agent being restarted.
Purpose
This pull request is a:

[X] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[X] I have added tests that prove my fix is effective or that my feature works
[X] I have added necessary documentation (if appropriate)

LGTM! Nice catch :)


Here's my chapter two exercise post.

:clock8:

The spaces in your filename will bork your post.  Other than that looks good!

I took out the spaces and subbed in dashes. @eah13 Should be good to go.


Keras 3 Featurespace running into error when using PyTorch backed

I am trying to use Keras 3 FeatureSpace on a tf.data dataset. It works fine with the tensorflow backend.
However, I am encountering the below error when I try to run the below example using a Pytorch backed : https://keras.io/examples/structured_data/structured_data_classification_with_feature_space/
The only change I am doing is changing the backend to Pytorch using: os.environ["KERAS_BACKEND"] = "torch"
The error is occurring at the line where the feature space is mapped to the dataset.
  File "/home/dercybersecurity/datapreprocessing/structured_data_classification_with_feature_space.py", line 306, in <module>
    preprocessed_train_ds = train_ds.map(
                            ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2280, in map
    return map_op._map_v2(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/map_op.py", line 40, in _map_v2
    return _ParallelMapDataset(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/map_op.py", line 148, in __init__
    self._map_func = structured_function.StructuredFunctionWrapper(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/structured_function.py", line 265, in __init__
    self._function = fn_factory()
                     ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1227, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1197, in _get_concrete_function_garbage_collected
    self._initialize(args, kwargs, add_initializers_to=initializers)
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 695, in _initialize
    self._concrete_variable_creation_fn = tracing_compilation.trace_function(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 178, in trace_function
    concrete_function = _maybe_define_function(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 283, in _maybe_define_function
    concrete_function = _create_concrete_function(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 310, in _create_concrete_function
    traced_func_graph = func_graph_module.func_graph_from_py_func(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 1059, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 598, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/structured_function.py", line 231, in wrapped_fn
    ret = wrapper_helper(*args)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/data/ops/structured_function.py", line 161, in wrapper_helper
    ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 693, in wrapper
    raise e.ag_error_metadata.to_exception(e)
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 690, in wrapper
    return converted_call(f, args, kwargs, options=options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 439, in converted_call
    result = converted_f(*effective_args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/__autograph_generated_filej1ue84y6.py", line 5, in <lambda>
    tf__lam = lambda x, y: ag__.with_function_scope(lambda lscope: (ag__.converted_call(feature_space, (x,), None, lscope), y), 'lscope', ag__.STD)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/core/function_wrappers.py", line 113, in with_function_scope
    return thunk(scope)
           ^^^^^^^^^^^^
  File "/tmp/__autograph_generated_filej1ue84y6.py", line 5, in <lambda>
    tf__lam = lambda x, y: ag__.with_function_scope(lambda lscope: (ag__.converted_call(feature_space, (x,), None, lscope), y), 'lscope', ag__.STD)
                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 377, in converted_call
    return _call_unconverted(f, args, kwargs, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 460, in _call_unconverted
    return f(*args)
           ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 723, in __call__
    preprocessed_data = self._preprocess_features(data)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 555, in _preprocess_features
    return {
           ^
  File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 556, in <dictcomp>
    name: self.preprocessors[name](features[name])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 123, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.11/site-packages/tree/__init__.py", line 435, in map_structure
    [func(*args) for args in zip(*map(flatten, structures))])
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tree/__init__.py", line 435, in <listcomp>
    [func(*args) for args in zip(*map(flatten, structures))])
     ^^^^^^^^^^^
NotImplementedError: in user code:

    File "/home/dercybersecurity/datapreprocessing/structured_data_classification_with_feature_space.py", line 307, in None  *
        lambda x, y: (feature_space(x), y)
    File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 723, in __call__  **
        preprocessed_data = self._preprocess_features(data)
    File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 555, in _preprocess_features
        return {
    File "/usr/local/lib/python3.11/site-packages/keras/src/layers/preprocessing/feature_space.py", line 556, in <dictcomp>
        name: self.preprocessors[name](features[name])
    File "/usr/local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 123, in error_handler
        raise e.with_traceback(filtered_tb) from None
    File "/usr/local/lib/python3.11/site-packages/tree/__init__.py", line 435, in map_structure
        [func(*args) for args in zip(*map(flatten, structures))])
    File "/usr/local/lib/python3.11/site-packages/tree/__init__.py", line 435, in <listcomp>
        [func(*args) for args in zip(*map(flatten, structures))])

    NotImplementedError: Cannot convert a symbolic tf.Tensor (ExpandDims_3:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported.

Hi,
Basically this error is happening due to the map function, where you're trying to pass symbolic tf.Tensor to feature_space of pytorch backend, which is not allowed when tf.Tensor tries to evaluate with the torch backend during map function.
You can modify the logiic by converting tf.data to numpy datatype using as_numpy_iterator() and then apply the feature_space.

@sachinprasadhs Do you mean converting tf.data to numpy iterator using train_ds = train_ds.as_numpy_iterator() and then applying map on that?
However, I am unable to apply the map function to the numpy iterator since I get the following error. Could you suggest how to apply the featurespace on the numpy iterator?
preprocessed_train_ds = train_ds.map(lambda x, y: (feature_space(x.as_numpy_iterator()), y), num_parallel_calls=tf.data.AUTOTUNE
                            ^^^^^^^^^^^^
AttributeError: 'NumpyIterator' object has no attribute 'map'

After modifying everything, I was able to run many majority of the code and now it is stuck at prediction.
Here is the modified Gist for reference

Thank you for the new code. I am including it below for reference.
x = []
y =[]
for i, j in train_ds.as_numpy_iterator():
    x.append(feature_space(i))
    y.append(j)
x = keras.ops.convert_to_tensor(x)
y = keras.ops.convert_to_tensor(y)
preprocessed_train_ds = tf.data.Dataset.from_tensors((x,y)).prefetch(tf.data.AUTOTUNE)

The solution seems to require iterating through the entire tf.data dataset and load it into memory. This may be impractical if the dataset is very large. In my actual application, I am streaming from disk using tf.data.make_csv_dataset
Could you please suggest an alternate solution?

AFAIK, symbolic tf.Tensor and pytorch backend is not allowed.

Hi @sibyjackgrove, you cannot use a non-TF featurespace in tf.data. We recommend either switching to the TF backend or not using tf.data.  Thanks!
To keep using the PyTorch backend, you can use a TorchDataLoader or a Keras PyDataset.

It is unfortunate that such a useful feature like Keras FeatureSpace doesn't have multi-backed support. I couldn't find anything in the documentation to suggest that Keras FeatureSpace doesn't work with PyTorch backend. I think this should be made explicit so that users don't utilize FeatureSpace thinking that it is mult-backend.

As suggested in the above comment, you can use it with like below using torch backend.

To keep using the PyTorch backend, you can use a TorchDataLoader or a Keras PyDataset.

or you can use .as_numpy_iterator():

@sachinprasadhs Could you please share an example for using Keras PyDataset with FeatureSpace? I could only find limited documentation on PyDataset. I used tf.data dataset since it has the most documentation and is very flexible.
Also, if I use Jax backend will it work with tf.data and FeatureSpace?


URL with percent-encoded characters breaks router

Using v0.13.2. We have the following route declarations:
<Route name="RootRoute" handler={App} path="/">
  <Route name="SSORoute" handler={SSO} path="sso" />
  [removed for conciseness]
</Route>

This matches http://hostname/sso?sso=bm9uY2U9NzJkOWZmM2NjMWNkMzM1OTliYWY4MTJiNGQyOTczN2QmcmV0dXJu. However, when I attempt to navigate to http://hostname/sso?sso=bm9uY2U9NzJkOWZmM2NjMWNkMzM1OTliYWY4MTJiNGQyOTczN2QmcmV0dXJu%0AX3Nzb191cmw9aHR0cCUzQSUyRiUyRmNvbW11bml0eS5ncm92ZWxhYnMuaW8l%0AMkZzZXNzaW9uJTJGc3NvX2xvZ2lu%0A&sig=5f8a827b0b4e6d56aa3473db01f45fb35ca165b742722bda7a043d477e5951d6 (the full URL, which includes percent-encoded characters) I get a white page. No errors reported on the client and server. I can't find any page that's being rendered as a result of going to that URL. Is this possibly related to https://github.com/rackt/react-router/issues/650?

I'm getting this too.

It's the linefeed character %0A. Sidestepping the issue at the moment but stripping them out of the url before redirecting here.

Did some experimenting in our app. It looks like the issue could be fixed in this method, in Match, by adding the call to replace():
static findMatch(routes, path) {
  path = path.replace(/\n/g,"");
  var pathname = PathUtils.withoutQuery(path);
  var query = PathUtils.extractQuery(path);
  var match = null;

  for (var i = 0, len = routes.length; match == null && i < len; ++i)
    match = deepSearch(routes[i], pathname, query);

  return match;
}

Thoughts?

Whoops, that doesn't work, because it appears that it results in the queries being modified.

Ok, different idea. Replace line breaks with the %0A characters, as shown below. I feel like this is not the right place to do this, or the right way to be doing this, but I will say that it works for my application (as far as I can tell).
value: function findMatch(routes, path) {
  path = path.replace(/\n/g,"%0A");
  var pathname = PathUtils.withoutQuery(path);
  var query = PathUtils.extractQuery(path);
  var match = null;

  for (var i = 0, len = routes.length; match == null && i < len; ++i) match = deepSearch(routes[i], pathname, query);

  return match;
}

I think I've traced the issue. The problem is with PathUtils.queryMatcher:
var queryMatcher = /\?(.*)$/;

. doesn't match newlines in regex, so it should be:
var queryMatcher = /\?(.|\n)*/;

Any updates on this?

would love to verify if this is still an issue on master (1.0 API has changed a bit), can you verify, and if its still a problem, a failing test would be awesome

Trying to investigate, but not sure how long the refactor to get rid of Router.run() will take.

@cmnjcs We're wrapping up the rewrite, getting ready to ship 1.0 very soon. The code on master is fairly stable right now. I don't anticipate any major changes between here and 1.0.

Sorry for the lack of clarity--when I said refactor, I meant of our application. The examples in the docs and the README (as a heads up, the examples are different) don't work in our application, so trying to restore functionality so I can test if the bug has been fixed.

This was fixed in #1166

Thanks! :+1:


Move to gogpt-3.5-turbo for REPL (and CLI?)

To support chats (what I call sessions). Reference: https://platform.openai.com/docs/guides/chat/chat-vs-completions

Working on this atm as it brings ChatGPT-like experience to REPL.


tuxedo-drivers: 3.2.14 -> 4.5.0

Description of changes
This updates the tuxedo-keyboard package to 4.5.0, and renames the package to its new upstream name.
While updating the nixos module, I also removed parts of the option description that were already removed in the 3.2.0 release.
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[X] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I tested it on my Tuxedo Pulse Gen1 with the 6.6 Linux Kernel, but i would appreciate feedback for newer Tuxedo Models.
From the changelog that would be at least Pulse Gen4, Sirius Gen2, Pulse Gen3 and Stellaris Gen6.

Nice work!
There is also this open pull request that intends to replace tuxedo-keyboard with tuxedo-drivers: #293017. I guess you can compare your implementation with theirs?
Either way, once this has been merged, would it make sense to enable the package by default in nixos-hardware?

I can confirm that this works on my Aura Gen1, using tuxedo-rs

I cherry-picked this onto nixos-24.05 and successfully used it for the past ~4 weeks with tuxedo-rs on a Pulse 14 Gen 4.

Can this be merged? Currently can't use tuxedo-keyboard in linux > 6.9 due to this https://github.com/NixOS/nixpkgs/issues/329383 (the patch from issue should probably be added too).

As was too distracted to actively work on this, i am closing this PR.
For those that still wants this merged, look at either #336633 (my preference) or #293017 that do the same thing.


New aging functions

Pi Config

Current $\Pi_c$ is the coarsened approach. We just look at whether or not the states are the same.
🚀 The non-coarsened approach is the probability of not leaving the state at all. We need this feature.

Pi Basin

Current $\Pi_b$ is the non-coarsened approach. We look at whether or not we leave the basin at all.
🚀 We have nothing to do except fix the existing problem! When we do the inherent structure at some point, we might have to though.

See #23 for the "other problem"


Fix missing command in test-cmd.sh

This fixes an unintended success in a test case.

LGTM

@kazegusuri can you please rebase?

rebased. PTAL


SELECTEDMEASURENAME and SELECTEDMEASURE wrong name shown (includes additional S)

Links contain wrong function name in the scrollbar visible on the left side of the screen for the functions SELECTEDMEASURENAME and SELECTEDMEASURE, i.e. there is letter S after SELECTED.
https://docs.microsoft.com/en-us/dax/selectedmeasure-function-dax
https://docs.microsoft.com/en-us/dax/selectedmeasurename-function-dax
Visible on all browsers.
Regards, Marko.

@HurikanLorenzo - Thanks for pointing this out. There were fifteen instances of that typo! it's fixed now.
#please-close


Fix build on macOS Mojave

Fixed #1898
Thanks to https://github.com/cryfs/cryfs/commit/08ed96680876c960d5aab0aabebb25206f0bda06

BTW, can't export LD_LIBRARY_PATH="/usr/local/opt/libomp/lib:$LD_LIBRARY_PATH" fix the issue on Mojave?

No, it doesn't fix the issue.
❯ LD_LIBRARY_PATH="/usr/local/opt/libomp/lib:$LD_LIBRARY_PATH" cmake ..
-- The C compiler identification is AppleClang <IP_ADDRESS>01145
-- The CXX compiler identification is AppleClang <IP_ADDRESS>01145
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /usr/local/Cellar/cmake/3.13.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.13.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.13.2/share/cmake/Modules/FindOpenMP.cmake:473 (find_package_handle_standard_args)
  CMakeLists.txt:80 (find_package)

-- Configuring incomplete, errors occurred!
See also "/Users/henry/.ghq/github.com/henry0312/LightGBM/build/CMakeFiles/CMakeOutput.log".

And what's the root of the issue on Mojave?

I don't know yet.

Homebrew has changed the libomp installation path or what?

No, homebrew seems to change nothing.

UPD: Naive question: can't the same result be achieved by
find_package(OpenMP REQUIRED PATHS /opt/local/lib/libomp /usr/local/opt/libomp)

PATHS affects paths of searching FindOpenMP.cmake, not libraries of libomp.


pleural cavity

Please add:
abnormal pleural cavity morphology as a parent term of "pleural effusion [MP:0010334]", "hemothorax [MP:0005243]", and "pneumothorax [MP:0010822]"
DEF: any structural anomaly of the potential space between the two pleurae (visceral and parietal) of the lungs; the pleural cavity, with its associated pleurae, aids optimal functioning of the lungs during breathing; it normally contains a thin film of serous fluid (“pleural fluid”) which lubricates the opposed pleural membranes so that they can glide smoothly against one another during ventilation.
Syn: abnormal intrapleural space morphology
Syn: abnormal pleural space morphology
From J:30559, J:54641
Thanks
Anna
Reported by: anna_anagnos
Original Ticket: obo/mammalian-phenotype-requests/1650

Already completed.


File Download does not work on Firefox

This code will not work for Firefox at least 38+. Probably lower.

Resolved by PR #2


net/haproxy: service setup/crash

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
The HAProxy service does not start at boot, and does not restart when crashed.
To Reproduce
Enable the service and reboot the device.
Expected behavior
When the service is enabled it should auto start, and auto restart if crashed.
Describe alternatives you considered
I am currently starting it up every time by hand, I tried to enable/disable the service, no difference. Now I monitor the service from an external host, and restart when needed.
Environment
OPNsense 24.1_1-amd64
DEC840

The HAProxy service does not start at boot

Any error message available? If not, a HAProxy config export might help to find the root cause of this issue.
A wild guess: one of your HAProxy "Public Services" is using a WAN IP. If your WAN connection is not setup, then HAProxy fails to start. This may be the case on boot. Simple fix: do not use a WAN IP in HAProxy, but instead use a local IP and add a NAT rule to forward traffic from your WAN IP to your local IP.

and does not restart when crashed

That's by design. There is no service supervisor in use, hence there is no automatic restart available.
In my personal experience it is extremely rare that the HAProxy service "crashes" (in the sense that it stops due to a software bug). So there should not be a need to automatically restart HAProxy. It would be a good idea to find out what is causing the service to malfunction.

I have not located related logs yet, but to be honest did not look too deep yet either, nor know right now where should I look for it. I tried few times, and after I had to start it up every time I rebooted, I felt that it's not normal and should be reported, but I do admit I have not spent enough time to debug it myself, I tried to disable/enable again, checked quickly the logs and that's about it, I hope this report will at least be useful for others with similar issue, even if it will not result in a bugfix. All that aside ...
The WAN interface specifically isn't used however you might be up to something with that guess, as there is one interface which might be blocking startup and that's related to my SD-WAN, so while the NAT workaround might work, knowing how easy this would be with systemd, just setup dependencies, I am having a hard time accepting that it's too hard in BSD, and there for I need apply a workaround rather then a solution, like just have service startup order enforced.
As far as I read the documentation something like this might be a suitable solution.
Although I see how the workaround remove the dependency and make the service more resilient, and I guess if this would be done as part of the config in a background, probably would not even see it as a workaround, but the UX for the NAT-ing just sub-optimal as is.
Related to the frequency of crashes, my experience is different, it happens quite frequently up to the point that I setup a monit rule for it as it was frustrating to deal with.

There is no service supervisor in use, hence there is no automatic restart available.

This was not just new but shocking to me, I guess I need to dive in and read up on the subject. I kind of expected self healing states.

I have not located related logs yet, but to be honest did not look too deep yet either, nor know right now where should I look for it.

You should see messages on the console during boot that should tell us why HAProxy is unable to start.

as there is one interface which might be blocking startup and that's related to my SD-WAN

I don't know anything about SD-WAN on OPNsense, so there's not much I can contribute. Having a look at the logs might reveal how and when it is started, which in turn might help to find a solution.

Not much in the logs.
2024-02-19T13:10:39	Notice	kernel	<118>/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy
2024-02-19T13:10:39	Notice	kernel	<118>Starting haproxy.
2024-02-19T13:10:39	Notice	root	/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

However I can see that the ZeroTier starts after HAProxy, so that supports the non-available interface idea, but without having an option to set the rcorder I don't see how would I fix that apart from NAT magic, also unsure why there is no more details in the log.
Maybe under the service setting , there could be an advance option for rc order

On a separated note, turning off the "introduction pages" makes no impact, no amount of reload / restart applied it.

However I can see that the ZeroTier starts after HAProxy, so that supports the non-available interface idea

Thanks for the confirmation. Let me check how the ZeroTier plugin handles service startup...

Let me check how the ZeroTier plugin handles service startup...

In theory the services should start in the correct order. However, there's a bug or rather an unexpected behaviour that causes the services to start in an incorrect order. I've proposed a fix in https://github.com/opnsense/core/pull/7259.

The wrong service start order during system boot can be fixed by applying the following patch:
# opnsense-patch edf756e

Regarding the crashes: we definitely need more information. Some things to check when it crashes again:

Most recent HAProxy log entries (either from the GUI or from /var/log/haproxy/latest.log)
Search the system log for entries related to HAProxy
Is the page Services: HAProxy: Statistics still accessible and shows stats data?
Output of service haproxy status
Output of ps auxww | grep haproxy
Output of ls -l /var/run/haproxy.*
Output of sockstat -l | grep haproxy (feel free to redact IP information)

Please collect this information when HAProxy is unavailable for you and before starting/restarting HAProxy.

Awesome , thanks @fraenki  and I guess fichtner from the merge, I will apply the patch and do a restart to test the changes outside of office hours tonight, if that works will close the issue.
About the crashes, I will now disable the auto-restart I hacked in via n8n, and will keep an eye on it, to keep things clear I will report that in a separated issue once I have data to show for.

I can confirm that the patch works, I applied the patch then reboot and haproxy started up fine.
How will the system update handle the presence of the patch ? Will I have to remove it before upgrade ( when the patch made it mainline ) or will it be just handled without removing it ? ( because now I would like to keep it in place for stability )

After every OPNsense upgrade you'll have to re-apply the patch for the time being.
There's a chance that this modification has unwanted side-effects, so it'll linger in the development version for some time, but will eventually be release in a stable version in the future. Please watch the OPNsense release notes, they'll contain more information when it's released in a stable version.


emscripten: revision bump from 1.35.4 to 1.36.4, was tested with comp…

Motivation for this change
xml.js would not compile with an older version. the 1.36.4 was tested by compiling xml.js from https://github.com/kripken/xml.js/commit/95addcf845fa8393f3d3b1a0f747404a1a51ec2c
see also:
https://github.com/NixOS/nixpkgs/issues/15636
requires:
export EMCONFIGURE_JS=2 to make tests pass
@matthewbauer i know you are using it. maybe you want to say something about this before i hit merge.

Looks good to me and I don't think anything has changed in emscrioten to be a problem.

It looks like there's been some mix up with the checksums on emscripten.


[9.x] Add handlerStats method for Http Client

Resubmit - method only.
Previous to this PR, retrieving handlerStats from the Http Client response was not intuitive or documented. The withOptions method fails with an error using Guzzle 'on_stats', and $response->transferStats->getHandlerStats() is not documented. To get handler stats required reverse engineering the Response.php class to some extent to figure out that getHandlerStats() can be called from the $response->transferStats property .
This PR addresses the issue, by mimicking behaviour of the 'effectiveUri()' method from the Response.php class, but the PR implements a new method calling the Guzzle getHandlerStats() method direct and returning handlerStats from the $this->transferStats property/object.

I don't think master is the right target for this? Why can't this go to 8.x?

I agree it seems this could go to 8.x branch.


ItemTransformContainingSemicolon_InTaskHost test failure

Mysterious failure in the test ItemTransformContainingSemicolon_InTaskHost

It appears that this failure came about because TaskHostNodes were able to connect to other OutOfProcNodes—their handshakes came to the same value. To make this more confusing, it had to do with the file version hash sometimes having a bit that masked the bit used to indicate if a node was a task host or not, so this only appeared if the file version hash happened to have a 1 in a specific position. Additionally, it only happened if a TaskHost tried to connect to a non-TaskHost node in the first place, so this consistently succeeded in some cases and normally but not always failed in other cases.


can't update playSound on Android

Trying to update the playSound-Property of a channel (without channelShowBadge) throws a java.lang.ArrayIndexOutOfBoundsException, because of:
https://github.com/rafaelsetragni/awesome_notifications/blob/5f8bc11ffdbdd20bd5c280ce37030a3e5e28634f/android/src/main/java/me/carda/awesome_notifications/awesome_notifications_core/managers/ChannelManager.java#L118

Yes, you're right. The correct would be:
channelModel.playSound = androidChannel.getSound() != null;

I copied and paste the line above and forgot to change it. This fix will be available at next beta version.

Hi! Sorry for taking so long to respond to your issue on GitHub. I was focused on releasing the new 0.7.0 version of awesome_notifications and awesome_notifications_fcm and i didn't had time enough to do both. But now I can answer all your questions.
So now i'm asking you to recreate this topic using the new issue template. There's a lot of missing informations that i need to understand your problem, and all the instructions are already in the new issue template. Also, remember to check beforehand if your issue was posted by another user.
So I will automatically close all previous issues so far.
Sorry for the inconvenience and i will be waiting for your new issue request.
Thank you so much for your support!


Running setup.py directly is now deprecated

% pytest  # generates these warnings:
tests/install/test_getstash.py::GetstashTests::test_install_setup
  /Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()

tests/install/test_getstash.py::GetstashTests::test_install_setup
  /Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` and ``easy_install``.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.

          See https://github.com/pypa/setuptools/issues/917 for details.
          ********************************************************************************

  !!
    self.initialize_options()

Hi
I have the same issue in installing 'setup.py' in matalab.
Could you solve this problem?


Reduce render samples?

Great little tool, thank you!
When rendering it defaults to 510 tiles, when I want to sample only 64. Is that possible?

Hello rgbaman, thank you for the kind comments! You can change the samples in the blender file and reupload it, or you can add the following code (the line with scene.cycles.sample) at the cell which enables GPU rendering. Example:
"print(cprefs)\n"+\
"scene.cycles.samples = 64\n"+\
"for compute_device_type in ('CUDA', 'OPENCL', 'NONE'):\n"+\
(other code)


[pentest] Return alerts as array

As for the pen. testing evaluating an array instead of individual variables for the triggered alerts is easier, this commit switches the format of how the alerts are returned to the host.

Thanks for your review @pamaury! Not sure how to avoid the duplicated code. Each function inside, for example, ibex_fi.c is an independent test that gets executed standalone. Each of these tests need to return different test results, only the err_status and the alerts field stay the same. For the test framework initiating the fault injections it is important that only a single uJSON message gets transmitted back from the host.


fixed flask example

fixed the two following errors when running the flask example:
TypeError: __init__() got an unexpected keyword argument 'scope'

and
ValueError: urls must start with a leading slash

This example was copy pasted from the README, could youd apply the same changes there so that I can merge this immediately

I already did


allow using json tag value as a fallback

Refer to https://github.com/gorilla/schema/issues/184. We can borrow the idea to httpin as:
adding an Annotation to allow using the value of json tag as an implicit query directive for decoding.
For example, the following struct
type ListUsersInput struct {
	IsMember bool  `in:"query=is_member" json:"is_member"`
	AgeRange []int `in:"query=age_range" json:"age_range"`
}

can be rewritten to
type ListUsersInput struct {
	httpin.UseJSONTagAsQueryArgs // annotation: auto-inject struct tags `in:"query={value of json tag}"`
	IsMember bool  `json:"is_member"`
	AgeRange []int `json:"age_range"`
}

Since httpin's struct tag is more complicated than gorilla/schema's. Bringing this feature can also bring burden on comprehension of how httpin works. Since it will generate implicit struct tags to drive httpin's algorithm.