Proposal to allow defining schema format other than default one (AsyncAPI Schema)

Introduction
Currently we can define a scheme in a different format than the default (AsyncAPI Schema) only in the message's payload field (https://github.com/asyncapi/spec/issues/528). E.g. defining the scheme in a different format for the message's headers field, or for the fields in bindings (see issue -> https://github.com/asyncapi/avro-schema-parser/issues/67), or in extensions the user cannot define.
Proposal
My proposal is mainly based on extending Scheme Object to union type:

with form of current one
and the second one is with schema and schemaFormat fields so the following examples will be compatible with each other:

components:
  schemas:

    schemaWithFormat:
      schemaFormat: 'application/vnd.aai.asyncapi;version=2.1.0'
      schema:
        type: object
        required:
        - name
        properties:
          name:
            type: string

    plainSchema:
      type: object
      required:
      - name
      properties:
        name:
          type: string

In addition, we should define a field on the root of the document defaultSchemaFormat - similar to defaultContentType. Any schema that does not have the schemaFormat field defined will be treated as with the format indicated by defaultSchemaFormat.
asyncapi: 3.0.0
info: ...

defaultSchemaFormat: 'application/vnd.apache.avro;version=1.9.0'

components:
  schemas:
    mySchema: # infer `schemaFormat` from `defaultSchemaFormat`
      type: record
      doc: User information
      fields:
        - name: displayName
          type: string

This solution also allows us to define in easier way schemas that have a string format (GraphQL types, Protobuf or XSD):
components:
  schemas:
    # Schema described by protobuf
    mySchema:
      schemaFormat: ...protobuf
      schema: |
        message Person {
          required string name = 1;
          required int32 id = 2;
          optional string email = 3;
        }

NOTE: Currently (in 2.0.0 and 2.1.0 version) you have option to define that format, but only in the message's payload.

How will this work with references? Very simply, imagine two files - the main one and the one with models - e.g with protobuf
# models.yaml
components:
  schemas:
    protobuf:
      schemaFormat: ...protobuf
      schema: |
        message Person {
          required string name = 1;
          required int32 id = 2;
          optional string email = 3;
        }

# asyncapi.yaml
asyncapi: 3.0.0
info: ...

channels:
  avroExample:
    publish:
      message:
        payload:
          $ref: '#components/schemas/avroSchema'
  protobufExample:
    publish:
      message:
        payload:
          $ref: './models.yaml#/components/schemas/protobuf'
  protobufExampleWithSchemaFormat:
    publish:
      message:
        payload:
          schemaFormat: ...protobuf
          schema:
            $ref: './models.yaml#/components/schemas/protobuf/schema'

components:
  schemas:
    avroSchema:
      schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
      schema:
        type: record
        doc: User information
        fields:
          - name: displayName
            type: string

Additionally, the use of custom formats in binding and extension will also be possible (used example from the https://github.com/asyncapi/avro-schema-parser/issues/67 issue):
asyncapi: '3.0.0'
info: ...

channels:
  example:
    subscribe:
      summary: Get updated products
      message:
        $ref: "#/components/messages/exampleMessage"

components:
  messages:
    exampleMessage:
      payload:
        schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
        schema:
          $ref: "SampleRecord.avsc"
      bindings:
        kafka:
          key:
            schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
            schema:
              $ref: "SampleRecord.avsc"

x-extension:
  needSchema:
    schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
    schema:
      $ref: "SampleRecord.avsc"

# SampleRecord.avsc
{
  "namespace": "sample",
  "type": "record",
  "name": "SampleRecord",
  "version": 1,
  "fields": [
    {"name": "NoUnionField", "type": { "type": "string",  "avro.java.string": "String" }, "doc": "Any doc"},
    {"name": "UnionField", "type": ["null", { "type": "string",  "avro.java.string": "String" }], "doc": "any doc"}
  ]
}

Parser implementation
Are we facing a breaking change in the parser and the corresponding Schema model? And this is surprising because no. Why? Every function in a Schema instance, first will check if we are dealing with a schema with schema and schemaFormat fields or not and do the appropriate fallback. I know this is an implementation detail, but there is a simple example:
class Schema extends Base {
  ...

  /**
   * @returns {number}
   */
  maximum() {
    return this.retrieveValue('maximum');
  }

  /**
   * @param {string} key - Name of field.
   */
  retrieveValue(key) {
    // we have case with `schema` and `schemaFormat` field
    if (this._json.schemaFormat && this._json.schema) {
      return this._json.schema[key];
    }
    return this._json[key];
  }
}

Obviously this will need to be optimized :)
Notes

The schemaFormat field at message level will be deprecated.
Added to the spec new field in the root - defaultSchemaFormat.
It may happen that one of the custom formats treats schemaFormat and schema as keywords - then we can use $schemaFormat, but I don't think it's necessary.
The parser should look for all the places where schemas are used in this second option (with schemaFormat) and transform them.
The proposal doesn't solve the problem of making references for deep values in the custom schema format - issue https://github.com/asyncapi/spec/issues/216 - please see below update

Update

The above proposal was not intended to support References/Pointer for non-JSON data structures but after reading Jesse's comment (thanks @jessemenning), I realized that my proposal could also give the ability to support references (aka pointers) to objects in non-JSON/YAML schemas. In addition to the schema and schemaFormat fields, we should also give the option of defining a schemaRef field, which exactly means reference, but for the schema defined in schema. This is a crucial difference from $ref in JSON/YAML. To illustrate the solution I will use an examples.

Avro example

Please forgive me, I am not an expert (or even I can say that I do not know it at all) in Avro and based on the information I found on the Internet, references to nested objects are provided by dot char with using namespaces.

// avro schema - avro.avsc file
{
 "type": "record",
 "name": "Pet",
 "namespace": "com.intergral.example.avro",
 "fields": [
   {
     "name": "name",
     "type": "string"
   },
   {
     "name": "toys",
     "type": {
       "type": "array",
       "items": {
         "type": "record",
         "name": "Toy",
         "namespace": "com.intergral.example.avro",
         "fields": [
           {
             "name": "name",
             "type": "string"
           },
           {
             "name": "price",
             "type": "long"
           }
         ]
       },
       "java-class": "java.util.List"
     }
   }
 ]
}

To retrieving Toy record using JSON references we should do something like this:
# asyncapi.yaml
asyncapi: 3.0.0
info: ...

channels:
  avroExample:
    publish:
      message:
        schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
        payload:
          $ref: './avro.avsc#/fields/1/type/items'

when using new schemaRef field we will make this:
# asyncapi.yaml
asyncapi: 3.0.0
info: ...

channels:
  avroExample:
    publish:
      message:
        payload:
          schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
          schema:
            $ref: "./avro.avsc"
          schemaRef: "com.intergral.example.avro.Toy"

Protobuf example
package my.org;

enum MyEnum {
  UNKNOWN = 0;
  STARTED = 1;
  RUNNING = 2;
}

message Outer {
  message Inner {
      int test = 1;
  }
  MyEnum enum_field = 9;
}

with using new schemaRef field:
# asyncapi.yaml
asyncapi: 3.0.0
info: ...

channels:
  protoExample:
    publish:
      message:
        payload:
          schemaFormat: ...protobuf
          schema:
            $ref: "./example.proto"
          schemaRef: "Outer.Inner" # resolve

Remarks:

$ref works as it did for JSON/YAML schemas

Often people don't know, but $ref can have reference to files that are not JSON/YAML. Then it treats these values (files) as a regular string (which is also a JSON derivative).

the responsibility of resolving references/pointer is moved to the custom parsers (Please note that currently we can define schemas other than JOSN only in messages, but in my proposal we can define them everywhere) - custom parser after parsing the schema should also extract the given reference pointed by schemaRef field (follows with syntax for given schema format) and "inject" it to the schema object, so (based on avro example) the resolved schema will be:
asyncapi: 3.0.0
info: ...

channels:
  avroExample:
    publish:
      message:
        payload:
          type: object
          properties:
            name:
              type: string
            price:
              type: string
              format: long

Any feedback are more than welcome :)

I like it. It is really a pain that you can only use AsyncAPI schema in components. Real pain for people using Avro and other formats.

This solution also allows us to define schemas that have a string format (GraphQL types or Protobuf)

I didn't really get this, how is this proposal solving this thing? It could be done with the current document structure too, right? Not sure if this is not adding unneeded complexity to the proposal, or I'm not getting something

Are we facing a breaking change in the parser and the corresponding Schema model

just keep in mind this is still a spec-breaking change. Cool that on the parser side we won't complicate things

@derberg Thanks for comment!

I didn't really get this, how is this proposal solving this thing? It could be done with the current document structure too, right? Not sure if this is not adding unneeded complexity to the proposal, or I'm not getting something

At the moment you can only define a custom format in the payload massag. Instead of the schema and schemaFormat fields, we could define the schemaFormat field at the schema level itself, like this:
components:
  schemas:

    schemaWithFormat:
      schemaFormat: 'application/vnd.aai.asyncapi;version=2.1.0'
      type: object
      required:
      - name
      properties:
        name:
          type: string

So it can only works with schema that can be written with JSON/YAML. The problem is with "string" (with own SDL) formats like GraphQL or Protobuf. You cannot make this (because you concatenate JSON/YAML field with string):
components:
  schemas:

    schemaWithFormat: |
      schemaFormat: ...protobuf
      message Person {
          required string name = 1;
          required int32 id = 2;
          optional string email = 3;
        }

I originally described this problem here -> https://github.com/asyncapi/spec/issues/528#issuecomment-822772695
So as you can see, adding possibility to define schema with two ways, with and without schema and schemaFormat fields, solve this problem.

just keep in mind this is still a spec-breaking change. Cool that on the parser side we won't complicate things

Yeah, it's a spec-breaking change but in

Are we facing a breaking change in the parser and the corresponding Schema model

sentence I had in mind breaking-change on the parser side.

Sorry but I still don't get how enabling support for GraphQL and Protobuf is better:
payload:
      schemaFormat: ...protobuf
      schema: |
        message Person {
          required string name = 1;
          required int32 id = 2;
          optional string email = 3;
        }

vs
schemaFormat: ...protobuf
payload:
      schema: |
        message Person {
          required string name = 1;
          required int32 id = 2;
          optional string email = 3;
        }

just want to make sure it relates really to this proposal as the only way or it is actually doable already, and just a side effect, for clarity of the proposal

@magicmatatjahu now I got it, thanks for your patience. You were referring only explicitly again to schemas from components. This is why I was confused, cause we could already do it for schemas in message 😄 Sorry for the confusion, and missing it from the examples. Please just update description, add a kind of note where you mention graphql and protobuf that atm this would be possible but only for schemas (payload) under the message but not under components directly.
Great proposal! I think this approach could also be suggested in question asked by Fran, on how to use kind and $ref in the server (haven't check the proposal yet, only saw a dedicated message in slack),  just have servers["myserver"].server.$ref and servers["myserver"].kind -> but yeah, completely different topic, brain dump from my side, let us not pollute this proposal with it :D

@derberg No worries :) You had patience for me at previous job, I have patience for you now :trollface: I updated description for Protobuf example.

Great proposal! I think this approach could also be suggested in question asked by Fran, on how to use kind and $ref in the server (haven't check the proposal yet, only saw a dedicated message in slack), just have servers["myserver"].server.$ref and servers["myserver"].kind -> but yeah, completely different topic, brain dump from my side, let us not pollute this proposal with it :D

Yeah, it's one of the solution, but I also gave another (as comment) to solve that problem in different way, probably easier :)

@derberg I extended proposal to use references to nested non-JSON schema objects - please see Update section. You should be interested in it :)

@magicmatatjahu one question:
In the example you wrote about retrieving a schema from avro using schemaRef, the value you wrote is:
schemaRef: "com.intergral.example.avro.Toy"

Wouldn't be:
schemaRef: "com.intergral.example.avro.Toys.Toy"

Note the Toys level before Toy.
If it isnt, would you mind clarifying to me, please.
Thank you!

@jessemenning Thanks for comment!

I want to have entire .xsd imported into AsyncAPI, as a string:

components:
  messages:
    UserSignedUp:
      payload:
        schemaParse: false
        schema:
          $ref: ./some_xsd.xsd
      contentType: "application/xml"

Here we can add a schemaParse field that indicates that the parser should not parse the schema. just import it as a string without any additional operations.

I just want a pointer from AsyncAPI to a schema registry/file, not bringing in the whole thing (maybe because it's huge) components:

Here we can use extension - as we talked about on the slack - or use the $remoteRef field, which would be a new field:
components:
  messages:
    UserSignedUp:
      payload:
        x-remote-ref: "https://example.com/myschema.xsd"
      contentType: "application/xml"

components:
  messages:
    UserSignedUp:
      payload:
        $remoteRef: "https://example.com/myschema.xsd"
      contentType: "application/xml"

I remember I wrote you on slack that using x-remote-ref (or x-payload-remote-ref) in a schema would be a bad idea, but now I don't see such problems, because in JSON Schema we can add additional keywords that don't affect validation. However, if there will be problems - e.g. with parsing things - we could use a construction like this:
components:
  messages:
    UserSignedUp:
      payload:
        schemaParse: false
        schema:
          $remoteRef: "https://example.com/myschema.xsd"
          # or
          x-remote-ref: "https://example.com/myschema.xsd"
      contentType: "application/xml"

I hope you understand :)

Provide a pointer to a particular element:

components:
  messages:
    UserSignedUp:
      payload:
        schemaParse: false
        schemaRef: "/PurchaseOrder"
        schema:
          $remoteRef: "https://example.com/myschema.xsd"
          # or
          x-remote-ref: "https://example.com/myschema.xsd"
      contentType: "application/xml"

Pointer to a particular element if importing the schema:

components:
  messages:
    UserSignedUp:
      payload:
        schemaRef: "/PurchaseOrder"
        schema:
          $ref: ./some_xsd.xsd
      contentType: "application/xml"

I remember I proposed $pointer in our slack conversation but I finally decided to use schemaRef field.
If you have questions, please ask! :)

I think it's a great proposal, and it also considered various scenarios. I am looking forward to see this proposal be adopted in next version of asyncapi.

this proposal would also allow the definition of mixed schemas, such as the one for the payload of the UserSignedUp message in the following example (which currently is not possible):
components:
  schemas:
    UserSchema:
      #$ref: user.yaml
      $ref: user.raml
    TimestampSchema:
      type: object
      properties:
        timestamp:
          type: string
          format: date-time
  messages:
    SignupUser:
      contentType: application/json
      schemaFormat: application/raml+yaml;version=1.0
      payload:
        $ref: "#/components/schemas/UserSchema"
    UserSignedUp:
      contentType: application/json
      schemaFormat: ??? no unique value possible here ???
      payload:
        allOf:
          - $ref: "#/components/schemas/TimestampSchema"
          - $ref: "#/components/schemas/UserSchema"

Based on this discussion, I am a bit scared that tooling will have a hard time follow an expected behavior, created a separate issue as it is only partly related to this: https://github.com/asyncapi/spec/issues/656

@GeraldLoeffler By my proposal we should be able to do:
components:
  schemas:
    UserSchema:
      schemaFormat: application/raml+yaml;version=1.0
      schema:
        $ref: 'user.raml'
    TimestampSchema:
      type: object
      properties:
        timestamp:
          type: string
          format: date-time
  messages:
    SignupUser:
      contentType: application/json
      payload:
        $ref: "#/components/schemas/UserSchema"
    UserSignedUp:
      contentType: application/json
      payload:
        allOf:
          - $ref: "#/components/schemas/TimestampSchema"
          - $ref: "#/components/schemas/UserSchema"

Thanks for great example, because because while writing this proposal I didn't look at nested custom schemas at all, but you can see it would be possible to support it 😅

Similar issue https://github.com/asyncapi/spec/issues/694

Still relevant.

Been watching this issue for sometime.  AsyncAPI spec + protobuf schemas + kafka.  Is this possible?  Ideally we could define the schema as protobuf, provide mock message examples in json format, and have tools understand to serialize/deserialize messages as protobuf.

@magicmatatjahua as the current referencing standard does not enable us to reference non-JSON data, I am proposing a new RFC that we design, see https://github.com/asyncapi/spec/pull/825. This should define exactly how non-JSON + JSON should work with references. The proposal encapsulates your proposal for using schemaFormat. The only change you need to do is to use Schema Reference and Linking Object for schema instead of the regular Reference Object 🙂
I have yet to map all your suggestions, along side the other issues, but I will take a look at that in the upcoming days.

I created a PR https://github.com/asyncapi/spec/pull/797 some time ago in which I added the possibility to define schemas in other formats in components.schemas section, but this has a lot of inaccuracies and some things do not work as they should (we should think about them). I will close PR for the moment, but we can open/change it again if we decide what to do next:

should other schemas be defined in components.schemas, or e.g. using an extension in components like components.x-avro-schemas?
how to make references to them in message's payload?
should we remove schemaFormat from message object and move to the schema object?
should we add possibility to define channel's parameters in different format than JSON Schema?
should we make it possible to define nested schemas, e.g. avro schema inside json-schema? (here we know 99% that we shouldn't to not create complexity in the spec).

cc @fmvilas @jonaslagoni @smoya @dalelane @char0n @derberg

should other schemas be defined in components.schemas, or e.g. using an extension in components like components.x-avro-schemas?

I think that can actually be an interesting approach. Now that work is happening on the extension catalog, this may be a good option to test how many people would actually be interested in this. What do you think, @derberg?

how to make references to them in message's payload?

It feels to me that we should probably "invent" another referencing mechanism just for AsyncAPI. Adapting $ref will only create more confusion and waiting for JSON Schema to actually implement what we need seems unrealistic to me, to be honest.

should we remove schemaFormat from message object and move to the schema object?

Why would you do this? 🤔

should we add possibility to define channel's parameters in different format than JSON Schema?

Categorically no 😄 Let's keep things simple.

should we make it possible to define nested schemas, e.g. avro schema inside json-schema? (here we know 99% that we shouldn't to not create complexity in the spec).

Definitely no. I'd say it's 100% clear haha! At least for me 😄

I think that can actually be an interesting approach. Now that work is happening on the extension catalog, this may be a good option to test how many people would actually be interested in this. What do you think, @derberg?

well anything can be tested with extension catalog 😄

Categorically no 😄 Let's keep things simple.

yes please 😅

should we make it possible to define nested schemas, e.g. avro schema inside json-schema? (here we know 99% that we shouldn't to not create complexity in the spec).

no, god forbid 😄

IMHO the only scope for the proposal should be to enable below to work:
asyncapi: 2.0.0
info:
  title: Example with Avro
  version: 0.1.0
channels:
  example:
    publish:
      message:
        schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
        payload:
          $ref: '#components/schemas/mySchema'
components:
  schemas:
    mySchema:
      type: record
      doc: User information
      fields:
        - name: displayName
          type: string

tl;dr current problem is that the above fails because validation do not understand mySchema is created with Avro

should we remove schemaFormat from message object and move to the schema object?

@fmvilas it is proposed so schemaFormat can be specified in the components

I know this issue for long and I can't believe it is a first time I think about it but...
Isn't it actually simply a bug of our parser? 😄
If parser has info schemaFormat: 'application/vnd.apache.avro;version=1.9.0' then before we try to dereference $ref: '#components/schemas/mySchema' (and get error) we should just make sure parser first converts mySchema into JSON Schema.
Or it is Monday and I missed something?

@fmvilas ping, especially last part of the message
@magicmatatjahu have also a look please

I think that only "good" solution now is to add the components.x-avro-schemas and components.x-raml-schemas components and in future maybe add them as normal sections in components. WDYT?

Yeah, that was my point on the last call. How many formats do we know that people will want to use in advance? Avro, Protobuf, XSD, maybe RAML data types, etc. Adding them to separate sections under components (either as supported extensions or without the x-) is not perfect but it solves the problem for the majority of people. And it's semantically clear what to expect in each of the sections.

So as you can see, referencing between JSON Schemas will be "broken". It's not a good solution.

I think we have discussed it already in other areas, and it is not a problem. Users must be aware of what they reference. In current AsyncAPI document you can already make error references anyway. You can reference external document where you have AsyncAPI Schema, and this schema can reference another file where you have JSON Schema.

I think that only "good" solution now is to add the components.x-avro-schemas and components.x-raml-schemas components and in future maybe add them as normal sections in components. WDYT?

Imho idea of extensions is to explore how we can do different things to find answers where some property should be placed when added to AsyncAPI spec, for example.
components.x-avro-schemas and components.x-raml-schemas feels like duck-taping

This case is not an experiment unless we really think that long term, we will have different objects for different schemas like, avro-schemas, json-schema-schemas etc. and what about the format of schema anyway, we don't want to end up with json-schema-schemas-draft-07, I hope 😅
Currently Schema Object is basically AsyncAPI schema only.
Currently, the spec says that component.schemas is:
Map[string, Schema Object | Reference Object]

Why not enable the idea from @magicmatatjahu that is intuitive as schemaFormat is used.
We have a chance to do it, as 3.0 is around the corner.
Schema Object should become an object that has 2 fields:

schemaFormat (moved from Message object) that is not required and defaults to AsyncAPI Schema
schema that is any | Reference Object (like payload at the moment...although it is strange we say payload is any and then talk about $ref as alternative solution in definition of Schema Object) and defaults to new AsyncAPI Schema Object (that is the same as current Schema Object)

Then in Message Object:

schemaFormat is no longer available
payload is no longer any but Schema Object

Then in components.schema we have:
Map[string, Schema Object]

Advantages

a consistent (exactly the same) way of defining schemas in message or in components.schemas
we get a dedicated Schema Object for all the schemas that can be easily extended in future minor releases if we, for example, want to add support for different referencing mechanisms, so schema for example becomes any | Reference Object | XSD Reference Object

Summary: extensions would be good for minor releases, but we have unique opportunity to solve it now for 3.0
Thoughts?

I do not know if you will be able to approve it in 2 weeks as I had a similar (but with stricter validation on components.schemas based on schemaFormat field) idea for half a year and hardly anyone was interested in it, but good luck 😄

times change, and RFC stage 2 doesn't mean it has to be accepted

@derberg I'm a little bit lost with your description. Would you mind adding an example of an AsyncAPI document illustrating what you mean? 🙏

sure, sorry, should have done it in previous comment

example uses 2.x spec but yeah this will be done for 3.x as it is breaking change

channels:
  example:
    publish:
      message:
        payload:
          $ref: '#components/schemas/myAsyncAPISchema'
    subscribe:
      message:
        #schemaFormat: 'application/vnd.apache.avro;version=1.9.0' -> no longer here
        payload:
          schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
          schema:
            type: record
            doc: User information
            fields:
              - name: displayName
                type: string
components:
  schemas:
    myAsyncAPISchema:
      schema:
        type: record
        doc: User information
        fields:
          - name: displayName
            type: string

both payloads are Schema Object, and so components.schemas.myAsyncAPISchema
schema in Schema Object is any | Reference Object | AsyncAPI Schema Object

example.publish.message.payload is Reference Object
example.subscribe.message.payload is Schema Object where schema is any
components.schemas.myAsyncAPISchema is Schema Object where schema is AsyncAPI Schema Object

I like this suggestion but I have a few remarks:

I'd call it format instead of schemaFormat. You know, it's already inside schema so we don't need to repeat it.
I'd add defaultSchemaFormat at the root of the document which will default to our current Schema Object: application/vnd.aai.asyncapi;version=3.0.0. This way people don't have to repeat schema or schemaFormat all over the document.
I'd simplify Channel Parameters' schema property so it's not a Schema Object but something like the Server Variable Object with the addition of the location property. There's an issue for that: https://github.com/asyncapi/spec/issues/583.

Doing it this way, I think we can safely release it in v3.

@fmvilas I added proposal about last point https://github.com/asyncapi/spec/issues/583

@fmvilas

I'd add defaultSchemaFormat at the root of the document which will default to our current Schema Object: application/vnd.aai.asyncapi;version=3.0.0. This way people don't have to repeat schema or schemaFormat all over the document

As we discussed in the meeting, we should clarify it that new root keyword strictly force using the given format, or people will be able to redeclare the format for given schema (like we do for defaultContentType). I'm in favour of possibility of redeclare.

Yeah, a default schema format. Not a unique schema format for the whole document. Just a default so we don't have to repeat every time.

After the last spec 3.0 meeting, it seems like there is an agreement on the approach, and we just need to land on a champion.
@magicmatatjahu seems like it's up you whether you want to champion it still or let @derberg do it 🙂

@derberg I can handle that, but give me some time, ok? :)

@magicmatatjahu can you more or less estimate 😄

If there is still no champion it for 3.0 I would like to.
I am mostly interested in proto3 support.
There for i see proposed "$remoteRef" as essential. Because for proto, avro, xsd you have always external code generator.
Having non json-schema based schemas Inlinen is cool but me (i guess many other too) will use schemas to generate code, because this is an easy way to contribute to, that the actual message is matching the schema.

Sorry guys. Now I don't have a time for champion it. Feel free to pick it up.

As i saied: I want to be the champion (for #622, #216, #881) . and already try to push it, as hard as i can.
I created a pr: #910 for this issue.

I was thinking twice about "AsyncApi Schema object"
If i remove this again that would allow:

having avro schema for header
having avro schema for parameters

And the complexity will grow:
messageId: userSignup
name: UserSignup
title: User signup
summary: Action to sign a user up.
description: A longer description
contentType: application/json
tags:
  - name: user
  - name: signup
  - name: register
headers:
  schemaFormat: application/vnd.aai.asyncapi;version=3.0.0
  schema:
    type: object
    properties:
      correlationId:
  	    description: Correlation ID set by application
  	    type: string
      applicationInstanceId:
  	    description: Unique identifier for a given instance of the publishing application
  	    type: string
payload:
  schemaFormat: application/vnd.aai.asyncapi;version=2.2.0
  schema:
    type: object
    properties:
      schema:
        user:
          $ref: "#/components/asyncApiSchemas/userCreate"
        signup:
          $ref: "#/components/asyncApiSchemas/signup"

Do we really want that?

Is it a problem with headers? is there no use case for using avro?
In the case of parameters, we discussed that on one call and conclusion what that it is not a problem because we anyway need to change parameters and not allow JSON Schema there but we should go server variables direction

I see no use case for avro based header. Headers are always (i know none products where it would be different) serialized by the messaging solution. Having support for avro schema is that the message payload is avro serialized and you dont want a schema mapping.
I would vote for having custom schema only for payload.

i see no usecase for exanple avro ind header oder parameters
will keep schemas simpler (one level less) and easier to read

I see no use case for avro based header. Headers are always (i know none products where it would be different) serialized by the messaging solution.

Hard for me to clarify really as before only AsyncAPI Schema was used anyway
@dalelane can you have a look, Avro schema for headers?
@fmvilas on last 3.0 meeting https://youtu.be/O4TQWBEaXy0?t=92 mentioned he knows people doing it

last resort, we can still reuse new Schema object for payload and headers, but in headers we would specify that only AsyncAPI schema is allowed, and then we can expand with 3.1, 3.2, 3.3 etc

I created a Draft PR with the initial changes on the JSON Schema files https://github.com/asyncapi/spec-json-schemas/pull/370
cc @GreenRover

Changes in Parser-JS and Parser-API will be also added

Recap of discussion from Slack:
@fmvilas:

Kafka has for header Map<String, byte[]> The byte array section can also be a Avro schema.
We should add a paragraph in the spec explaining mixing schemas should not be supported by implementations.

@GreenRover @derberg I can't find any other further discussion about  allowing other schema formats into the headers finally. Did we drop that finally?

tbh I do not remember, but definitely it got in -> https://github.com/asyncapi/spec/pull/910/files so new multi format is in the headers too

Ouch, for some reason I skipped when reading the messageObject spec doc 😮💨 . Thanks for pointing that out!

I created a new issue to discuss about the headers shape now that we allow multi format schemas https://github.com/asyncapi/spec/issues/948
cc @GreenRover @derberg

This one is only missing the JS parser implementation. Anyone volunteering?


Make sure xgboost.XGBClassifier and xgboost.XGBRegressor can be persisted

To increase reach, we should make sure the sklearn compatible xgboost estimators can be persisted.
This issue is to:

check if we can persist them out of the box
maybe add a test and to the CI to make sure they can be persisted
if needed, figure out what we'd need to implement on our side, or on the xgb side, to make this happen

Okay, I did a quick check with the 3 big ones:

XGBoost fails.
Catboost fails.
LightGBM: actually works for classifier and regressor!

Obviously, there are so many options with these estimators that a quick test can't cover all of them, so there might be cases where different results will occur.
Here is the code I used for testing LightGBM. Similar code was used for the others.
# python -m pip install lightgbm
from lightgbm.sklearn import LGBMClassifier, LGBMRegressor
import numpy as np
from skops import io

X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
y = np.dot(X, np.array([1, 2])) + 3

regr = LGBMRegressor().fit(X,y)
lr = io.loads(io.dumps(regr), trusted=[
    'collections.defaultdict', 'lightgbm.basic.Booster', 'lightgbm.sklearn.LGBMRegressor'])
lr.predict(X)

clf = LGBMClassifier().fit(X,y)
lc = io.loads(io.dumps(clf), trusted=[
    'collections.defaultdict', 'lightgbm.basic.Booster', 'lightgbm.sklearn.LGBMClassifier',
    'numpy.int64', 'sklearn.preprocessing._label.LabelEncoder'])
lc.predict_proba(X)


sneak spy fixes

A mixture of fixes for problems found by sneak spy in the budhud discord:

[x] (General) Add additional padding to scoreboard K/D/A/damage/assists for chinese characters
[x] (General) Increase text length for scoreboard map names
[x] (MvM) Fix respec overlap on mvm scoreboard
[x] (MvM) Fix sapper/medic ability overlap
[x] (MvM) Fix halloween spell overlap
[ ] (MvM) Increase floating health character limit to 6
[x] (General) Increase health bounding box in specator tournament panels
[x] (General) Fix overlap between spy disguised class and scoreboard

Floating health may be a bit of a doozy to implement, will have to take a closer look at it. I'm not sure if I can increase the character limit without it affecting standard, non-floating TargetIDs.


Consider changing the return type of the add method in all binary trees and graph data structures to boolean.

Describe the solution you'd like
Consider changing the return type of the add method in all binary trees and graph data structures to boolean.

This issue has been resolved in version 1.49.6


client: Prevent Out-Of-Range panic in task.Metrics()

Signed-off-by: Kenfe-Mickael Laventure<EMAIL_ADDRESS>

is this caused by a race?

Codecov Report

Merging #1559 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1559   +/-   ##
=======================================
  Coverage   42.36%   42.36%
=======================================
  Files          24       24
  Lines        3368     3368
=======================================
  Hits         1427     1427
  Misses       1612     1612
  Partials      329      329

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd1f89b...af2226c. Read the comment docs.

Yes, Docker regularly calls the metric endpoint from a separate go routine.
On Mon, Sep 25, 2017, 4:00 PM Michael Crosby<EMAIL_ADDRESS>wrote:

is this caused by a race?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/containerd/containerd/pull/1559#issuecomment-332037801,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APUifkYQXFNJTG7se4VSGJ1wNf3foUsjks5smDB4gaJpZM4PjcXr
.

--
Sent from my phone.

LGTM


Feature/groomer favorites

Description
The new "saved groomer"  feature give users (pet owners) the ability to save their favorite groomers. My goal for this feature is to display the list of favorite groomers on the pet owners dashboard. Once displayed, the user can then schedule appointments right from their dashboard.
There are a few more things needed to fully complete this new feature such as connecting the frontend to the backend. Also, we will need to add a frontend component.
This feature is part of the "Add a "Saved Groomer" component (database)" trello card linked here: https://trello.com/c/OiRJt9Fs
Type

[ ] This is a bugfix.
[x] This is a new feature.
[ ] This is part of a feature.

Organization

[x] Have you linked this pull request to a Trello card?
[x] Have you named the branch in the format of feature/describe_feature_or_change OR bug/describe_bug_this_is_fixing

Review

[x] Is this a work in progress?
[x] Is this ready to be reviewed?
[x] Have you added two reviewers to this pull request?

Great thank you. Everything looks good.


[BUG] serializeResponse failed at AbstractResponse.toStruct(version)  but without any exception thrown

Describe the bug
when kafka client sends request to kop for API version, before kop send response back to client, it failed to serialize Response at method call response.toStruct(version) and request close, without any exception been thrown, no clue for me to figure this out.
To Reproduce
Steps to reproduce the behavior:

Setup pulsar broker with kop

try to produce message to broker

the client debug log tells that the client was fetching API version, but could not get corresponding response

timeout and disconnected (timeout = 60s)

See error
client log
2022-02-09 14:50:43:425 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Completed connection to node -1. Fetching API versions.
2022-02-09 14:50:43:425 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating API versions fetch from node -1.
2022-02-09 14:51:13:434 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Disconnecting from node -1 due to request timeout.

Expected behavior
client succeed to get api version and metadata, then produce message to pulsar successfully
Additional context
kafka client version : 2.6.0
kop version: <IP_ADDRESS>
pulsar version: 2.9.1
JRE: OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
linux: Linux version 5.10.26-20

Could you upload the broker's log (debug level)?

It's better to try to reproduce the bug with standalone mode, and give more details. Then we can try to reproduce it locally.
For example,

Run KoP standalone with following configs.

kkk=vvv
# ...

Run Kafka 2.6.0 command line tool:

 # ...

Upload the standalone logs.

@BewareMyPower I have figured out the reason why this happened. A few days ago, to enable log4j2(pulsar broker) connecting Kafka topic for logging purpose, I put a kafka-clients-2.0.0.jar into pulsar/lib/ directory,  when I remove that kafka-client jar file, kop works fine. There must be some class loading conflict problem there! Do you know what I can do to solve this ?

Currently KoP depends on kafka-clients-2.0.0.jar for some common classes and KoP is loaded by broker dynamically, including uncompressing the NAR file and loading the classes.
IMO, It's better not to add other JARs to lib directory. This incompatibility issue is hard to test in CI.

@BewareMyPower OK, thanks a lot, maybe my problem get solved, let me close this issue.


Adding input file option

#1

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Use TDX quote provider to attest and verify.

go-tpm-tools uses go-tdx-guest's device driver to attest and verify quotes but with the recently upstreamed Unified ABI patch from Intel fetches attestation quotes via ConfigFS subsystem.  go-tdx-guest has incorporated this change and now fetches quote via go-configfs-tsm library instead of device driver's IOCTLs.  Added TDX quote provider in attest module which would be used instead of TDX device driver, and updated unit tests.

/gcbrun

/gcbrun


[Feature] WebSocket over HTTP/2 (h2+h2c)

Expected behavior
Currently, WebSockets only work over HTTP/1.1 in Netty, not on HTTP/2.
Actual behavior
We should make WebSocket work with HTTP/2 also.
Netty version
4.1

RFC 8441 is still in "proposed" status in 2023.
It's an interesting idea to support old ws-using clients, and I was contemplating it for Aleph, but how many people want this? HTTP/2 eliminated much of the value of Websockets, and offers better socket utilization via multiplexing, so I suspect 99% of those who would have used have just gone to H2 streaming.


LSTM Autoencoder

Hello everyone and happy new year
I am trying to create an LSTM Autoencoder as shown on the image bellow.

The encoder consumes the input "the cat sat",
and creates a vector depicted as the big red arrow.
The decoder takes this vector and tries to reconstruct the sequence
given the position in the sentence.
I would like to save this vector (big red arrow) to use it on another model.
The code i have wrote so far is the following:
from keras.layers import containers
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, AutoEncoder
import numpy as np
from keras.layers.recurrent import LSTM

train_x = [
    [[ 1 ,3 ],[ 1 ,3 ]],
    [[ 2 ,4 ],[ 2 ,4 ]],
    [[ 3 ,5 ],[ 3 ,5 ]]
]
train_x = np.array(train_x)

encoder = containers.Sequential([ LSTM(output_dim=5, input_dim = 2, activation='tanh' , return_sequences=True) ])
decoder = containers.Sequential([ LSTM(output_dim=2, input_dim = 5, activation='tanh', return_sequences=True) ])
autoencoder = Sequential()
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False))
autoencoder.compile(loss='mean_squared_error', optimizer='sgd')
autoencoder.fit(train_x,train_x, nb_epoch=10)

It is not clear to me if the code above does what i ask for.
If i do not use return_sequences=True it yields an error.
Should i use a graph model to do exactly what i ask for?
Thank you in advance for your help.

What you post does do what you figure describes.
You don't need an AutoEncoder layer to achieve this. You could simply do:
m = Sequential()
m.add(LSTM(output_dim=5, input_dim = 2, return_sequences=True))
m.add(LSTM(output_dim=5, input_dim = 2, return_sequences=True))

Also don't train RNNs with SGD. Use RMSprop instead.

Thank you very much.
I assume that then i can save the weights of the 1st Lstm's final state weigths using
m.layers[0].get_weights()

In my experience Adam does better than RMSprop

Hello again
I would like to get the outputs of the first layer of the following model

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import containers
from keras.layers.core import Dense, AutoEncoder, TimeDistributedDense, Activation
from keras.optimizers import RMSprop
from keras.utils import np_utils
from keras.layers.recurrent import LSTM

import numpy as np

LSTM_size_1 = 5

data = [
    [ [1 ,3] ],
    [ [2 ,4] ],
    [ [3 ,5 ] ],
    [ [4 ,6 ] ],
    [ [5 ,7 ] ],
    [ [6 ,8 ] ],
    [ [7 ,9 ] ],
    [ [8 ,10 ] ],
    [ [9 ,11 ] ]
]

data = np.array(data)

in_dim = data.shape[-1]
m = Sequential()
m.add(LSTM(LSTM_size_1, input_dim=in_dim, return_sequences=True))
m.add(LSTM(in_dim, return_sequences=True))
m.add(Activation('linear'))
m.compile(loss='mse', optimizer='RMSprop')
m.fit(data,data, nb_epoch=2)

when i type
m.layers[0].get_output()

i get
DimShuffle{1,0,2}.0

Is there a way to get the final output as a numpy array instead of the weights?
I believe the output is something like this :
O_t = σ ( W_o  *  X_t  +  U_o  *  h_t-1  +  V_o  *  C_t  +  b_o )

Thank you in advance

Hi,
you can save the weights of the first LSTM, create a separate model with only one LSTM layer and set the weights of this LSTM to your saved weights. After that you can use predict method to get the output of the first LSTM.

Truthfully this is not what i want.
I do not want to use the trained LSTM
as an input to another Neural Net
I want to use the output of the LSTM as an embedding.
So i do not want 4 matrices ( the trained weights of the LSTM )
but 1 matrix with dimensionality n*1 where n is the number on nodes in the LSTM.
This matrix is the output of the 1st LSTM which was used as input to the
second LSTM as shown with the red arrow in the picture

If you feed the same data to this new network with one LSTM layer you will get exactly what you want as the result of the predictions. You can save these results and use it anywhere you want.

If there are 100 instances there will be 100 autoencoders.
I want an autoencoder to overtrain on a specific instance and extract an embedding.
Think of it as compressing all the information for a text in a vector of size 10.
I want to use these 100 embeddings as input on another network.  (size: 100 X 10)
I cannot connect all LSTMs at the same time and feed the original data once more.
Neither could i connect one lstm at a time.
I just want the output of the 1st layer to numpy array.
How can i get it ?

http://keras.io/faq/#how-can-i-visualize-the-output-of-an-intermediate-layer

Thank you.
This is what i was searching for!

@dpappas , I am also facing the same issue. I tried the above link and I get the attribute error
'LSTM' object has no attribute 'initial weights'
Could you please post the exactly snippet what you did.

@fchollet, using "return_sequences=True" does NOT produce what is described in the figure!
This will cause the "decoder" LSTM layer to use the output vector of the encoder at each time step instead of only the single final vector after processing the whole sequence (as shown in the figure). Or am I mistaken here?

@GUR9000  I think you are right. At every time step, the decoder needs to take the output from last step of decoder rather than the output of the encoder.

I want to build a LSTM autoencoder.
My data looks like this, with shape (1200, 10, 5), which is (training_size, timesteps, input_dim):
array([[[0, 0, 0, 1, 0],
        [0, 0, 0, 0, 1],
        ...
        [0, 1, 0, 0, 0]],
        [[0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
        ...
        [0, 0, 0, 0, 0]]])

Code:
from keras.layers.recurrent import LSTM
from keras.models import Sequential

model = Sequential()

encoded = LSTM(3, input_shape=(10,5))
model.add(encoded)

decoded = LSTM((10,5), return_sequences=True)
model.add(decoded)

But for the decoded step it returns ValueError: Input 0 is incompatible with layer lstm_5: expected ndim=3, found ndim=2.
Thank you in advance.

@ScientiaEtVeritas
Your encoded LSTM returns only the last output of the LSTM
you need the encoded line to
encoded = LSTM(3, input_shape=(10,5), return_sequences=True)
Finally your decoded LSTM needs a proper number of nodes for the lstm.
You give a tuple as size
You need to change this to
decoded = LSTM( 10 , return_sequences=True )
or change the number 10 to the size you want.
I suggest you read the documentation and some explanation on LSTMs
[(http://colah.github.io/posts/2015-08-Understanding-LSTMs/)]

@dpappas : Thank you for your answer.
But for clarification and as mentioned before, using return_sequences=True is not what is shown in the picture. My goal is actually a single, final vector that represents the whole sequence as good as possible. I don't think this is the case when using return_sequences=True.
There is another issue I found that actually describes why https://github.com/fchollet/keras/issues/5138

@ScientiaEtVeritas
You are wright.
I have not managed to do that with keras.
You could do it with tensorflow using seq2seq.
Maybe in keras you could do it with the step funtion of the lstm or a callback function to use the output of the decoder from the previous timestep as input to the new timestep

@dpappas @ScientiaEtVeritas
Perhaps this has the desired effect:

model.add(LSTM(3), input_shape=(timesteps, input_dim))
model.add(RepeatVector(timesteps))
model.add(LSTM(10, return_sequences=True))


Allow developers to invite newcomers

Closes https://github.com/coala/corobo/issues/322
Reviewers Checklist

[ ] Appropriate logging is done.
[ ] Appropriate error responses.
[ ] Handle every possible exception.
[ ] Make sure there is a docstring in the command functions. Hint: Lookout for
botcmd and re_botcmd decorators.
[ ] See that 100% coverage is there.
[ ] See to it that mocking is not done where it is not necessary.

Need to update the tests, and also .. only allow invites to people "if they are a member of the room"

@meetmangukiya Can you help me with the tests?

@Vamshi99 @meetmangukiya this is kinda urgent. We could disable auto invite and let our developers help manage the load of inviting with a revised onboarding process.

Can we test this using IRC backend instead ?


QUESTION: Does this work with external drives?

We're trying to allow users of our angular frontend to browser files on the server, even files on external harddrives connectes to the server via usb.
We use Django.  Will the Django bridge app by default take into account those external drives?

anybody find a solution for this problem ?


Publish helm chart using @helm/chart-releaser-action

Issue #294,:
Fixes #294
Description of changes:
Ref: https://github.com/helm/charts-repo-actions-demo
Add @helm/chart-testing-action - To lint chart on PRs
Add @helm/chart-releaser-action - To publish Helm release on gh-pages
Works well on my Fork:
https://github.com/mcbenjemaa/opendistro-build
https://mcbenjemaa.github.io/opendistro-build/index.yaml
Test Results:
Updated README.md
Note: If this PR is related to Helm, please also update the README for related documentation changes. Thanks.
https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/master/helm/README.md
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on signing off your commits, please check here.

/cc @DerekHeldtWerle @kalvinnchau

Hi @mcbenjemaa we have moved to OpenSearch (opensearch.org)
And we just merged the 1st version of Helm there https://github.com/opensearch-project/opensearch-devops/pull/4
Could you close this PR and migrate there? Thanks.

Hi @mcbenjemaa we have moved to OpenSearch (opensearch.org)
And we just merged the 1st version of Helm there opensearch-project/opensearch-devops#4
Could you close this PR and migrate there? Thanks.

Do you think it would be worth to add a note about the move to the top of the README. If I haven't checked what's the status of the issue, I would be still using this repo.


issues when login to the GitHub in vscode which with startup parameters

Type: Bug
When starting vscode with "using startup parameters(--user-data-dir,--extensions-dir) to specify extensions and personal file storage locations", vscode will call the browser(edge in my case) for authorization when the extension requires login to GitHub, and the browser will start a new vscode instance which uses the default storage location, which makes it impossible to login to the GitHub account when starting vscode with "Use startup parameters to specify the extension and personal file storage location"
VS Code version: Code 1.76.2 (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:55:54.936Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: No

System Info

Item
Value

CPUs
12th Gen Intel(R) Core(TM) i5-12450H (12 x 2496)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
undefined

Memory (System)
15.75GB (9.88GB free)

Process Argv
--user-data-dir D:\ProgramFiles\VScode\setting --extensions-dir D:\ProgramFiles\VScode\extensions --crash-reporter-id 3f5e2d10-3231-4b24-a88d-89138bcc6824

Screen Reader
no

VM
0%

Extensions (5)

Extension
Author (truncated)
Version

copilot
Git
1.78.9758

copilot-labs
Git
0.12.791

copilot-nightly
Git
1.78.10605

vscode-language-pack-zh-hans
MS-
1.73.11020948

python
ms-
2023.3.10481011

(1 theme extensions excluded)

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

This issue also occurs in the latest version(1.77.0, 7f329fe6c66b0f86ae1574c2911b681ad5a45d63)

Yeah those startup parameters are really debug parameters and don't really work with how we store secrets.


Add Wansmer/langmapper.nvim

Checklist:

[x] The plugin is specifically built for Neovim, or if it's a colorscheme, it supports treesitter syntax.
[x] The lines end with a .. This is to conform to awesome-list linting and requirements.
[x] The title of the pull request is Add/Update/Remove `username/repo`  (notice the backticks around `username/repo`) when adding a new plugin.
[ ] The description doesn't mention that it's a Neovim plugin, it's obvious from the rest of the document. No mentions of the word plugin unless it's related to something else.
[x] The description doesn't contain emojis.
[x] Neovim is spelled as Neovim (not nvim, NeoVim or neovim), Vim is spelled as Vim (capitalized), Lua is spelled as Lua (capitalized), Tree-sitter is spelled as Tree-sitter.
[x] Acronyms should be fully capitalized, for example LSP, TS, YAML, etc.

Thanks for the PR! The changes you've made don't adhere to the checklist, I've deselected the item(s) that need to be fixed.

Thanks for the PR! The changes you've made don't adhere to the checklist, I've deselected the item(s) that need to be fixed.

I thought that “No mentions of the word plugin unless it's related to something else” is suitable for my description.
Rephrased description with 7c97292


Add support for List of datatype

It would be good to have support for lists of datatypes. For example, here I could map a proposal which has a list of tags :
entity Proposal {
    title String,
    tags List<String>
}

This could generate the following entity :
@Entity
public class Proposal {
   private String title;

   @ElementCollection
   @CollectionTable(name="tag_proposals", joinColumns=@JoinColumn(name="proposal_id"))
   @Column(name="tag")
   List<String> tags;

And why not allowing also sets :
entity Proposal {
    title String,
    rakings Set<Integer>,
    tags List<String>
}

That'd be interesting indeed! Again, all the heavy work would have to happen in the generator.
cc @jhipster/developers

I also had many people asking me about this feature so I tink might be worth to add in next major version

@agoncal I'll try some things this week-end

better to discuss together with jhipster/generator-jhipster#6275.

Closing and referencing this issue on Enriching the JDL grammar #6275

this issue and the referenced issue #6275 are both closed. but I don't find any discussion on this feature in #6275 either.
is this feature rejected?


Add Slack Signup Info

Can we add instructions and links to how to get connected to our channel on the Chicago Tech Diversity slack workspace.

Currently a person must go to http://www.chitechdiversity.com/, scroll down to the "Join us on Slack" link, put in their email and then once they have access to the workspace they can join our channel which is #org-chi-women-devs


Update version

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I think some old environment are failing to "update" because qt is at version 5.15.2 but their packages requires some migrations that are only available with 5.15.3 (which is why I was really adamant about holding off on 5.15.3 during the initial rollout).
I'm not sure if there is a way we can tell conda to forget about the qt, and replace with qt-main. But that seems hard.
Instead, I think we should make it an error to depend on qt in any new recipe, that way people get the correct pinnings.

Could add a lint to conda-smithy

???

Sorry for the lack of clarity, was responding to this last line

Instead, I think we should make it an error to depend on qt in any new recipe, that way people get the correct pinnings.

We have a number of lints starting here. For example this pip hint could be a good starting point for a qt related lint (or hint)


#17: Proper scaling factor in zoom slider

About Index page, seems some css bring to parent component instead of own component, no major changes. I think, it's better to maintain clean code on top level component. Attached both code structure as screenshot. What do you think @kirillt ? Also, we can hide zoom bar for gallery view by a simple login with galleryStore.galleryView store.

There is not really a need to maintain "clean-code" at top level component. Later, we want to remove the explicit check
class:hidden={!$galleryStore.galleryView}
and handle it with sveltekit rounding, but we can do that in a sepeate pr
In sveltekit, usually any CSS styles that are related to the page layout, (meaning margins, paddings, flex ect) should be in the layout component. This allows for the components of the page to be more pure and more reusable.

@alvinosh now it feels working to me, thanks.
@prokawsar does it look good enough for you? I have no idea about JS and Svelte code, but if something looks suboptimal but not critical we can merge and raise this question on dev sync.


Ford: get VIN without engine

The EPS and ABS know the VIN, and we lose engine without the OBD port.
Properly handling this fixes https://github.com/commaai/openpilot/pull/31298. Currently, we query the functional address 0x7DF to get the VIN and take the first response we get that looks valid, but we are ignoring the ABS and EPS which also respond with first frames, and likely put them into a bad state for up to 500ms, which is why adding another query (or our existing OBD query) seemed to fix the issue.

whoops


Ch 01 [new issue]

name: Aakash Deka
github_user_name: aks6-cpu

https://github.com/aks6-cpu/oss101.git


Lightweight system for webhooks

Created a lightweight system to provide an external service notification of a create / new publish event. Built primarily with Slack in mind, it is flexible enough to provide that type of notification for any similarity simple web hook service.
(This is a second PR for the same functionality as the first PR wasn't auto-mergable.)

Awesome, I tried it out and it works great. Merged Thanks


Installation error (syntax error near unexpected token)

OS: Ubuntu 15.04
BASH: 4.3.30(1)-release (x86_64-pc-linux-gnu)
Steps to reproduce:
simon@Lappis:~$ git clone --depth=1 https://github.com/Bash-it/bash-it.git ~/.bash_it
Cloning into '/home/simon/.bash_it'...
remote: Counting objects: 221, done.
remote: Compressing objects: 100% (178/178), done.
remote: Total 221 (delta 4), reused 162 (delta 1), pack-reused 0
Receiving objects: 100% (221/221), 157.38 KiB | 0 bytes/s, done.
Resolving deltas: 100% (4/4), done.
Checking connectivity... done.
simon@Lappis:~$ ~/.bash_it/install.sh
Your original .bashrc has been backed up to .bashrc.bak
Copied the template .bashrc into ~/.bashrc, edit this file to customize bash-it
/home/simon/.bash_it/install.sh: line 124: syntax error near unexpected token `fi'
/home/simon/.bash_it/install.sh: line 124: `fi'

Can you try this again (using a fresh clone)? I had accidentally pushed an incomplete change, and it looks like you cloned just in the middle of that.

Thanks!


It sucks!!

actually it is not working on my site and tried in local too and tried in two of my other sites.
Don't know what is the issue is>>>>

Very sorry. It is abandoned because I had no time to work on.


to many  package request  when using private registry

Bug report
Packages affected

[ ] sandpack-client
[x] sandpack-react

Description of the problem
I use private registry to load a dependency, but network show pretty much package request which cause slow render of preview.
the registry proxy I use is Verdaccio, the same to the docs example.
after requested the lib (the file is xxx-1.0.30.tgz), codesandbox starts to request the lib's dependency and send lots of package.json request.
I'm not sure that this situation occur is because the lib build isn't right or some Verdaccio proxy config lost.
I raise it here because the request initiator is codesandbox(the image show), sorry for the mistake if it's not proper to submit to sandpack

What were you doing when the problem occurred?
What steps can we take to reproduce the problem?

Link to sandbox: link (optional)
Your Environment

Software
Name/Version

Sandpack-client version

Sandpack-react version
^1.15.5

Browser
google chrome(107.0.5304.87)

Operating System
macOS 13.0

Hey, would you mind sharing your Sandpack configuration (without any sensitive data)? Like the NPM registry configuration and sandbox package json
Thanks

thanks for your reponse, here is my configuration

thanks for your reponse, here is my configuration
 <Sandpack
        theme={atomDark}
        template="react-ts"
        customSetup={{
          dependencies: {
            '@registry/mydependency': 'xxx',
          },
          npmRegistries: [
            {
              enabledScopes: ['@registry'],
              limitToScopes: true,
              registryUrl: 'https://npm.xxx.site',
            },
          ],
        }}
        files={{
          '/App.tsx': code,
        }}
        options={{
          visibleFiles: [
            '/App.tsx',
            'tsconfig.json',
            '/index.tsx',
            '/styles.css',
            '/public/index.html',
            '/package.json',
          ],
          showRefreshButton: true,
          showTabs: true,
          closableTabs: true,
          showLineNumbers: true, // default - true
          showInlineErrors: true, // default - false
          // wrapContent: true, // default - false
          editorHeight: '100vh', // default - 300
          editorWidthPercentage: 50, // default - 50
        }}
      />

As for the parent components, I used SandpackProvider to wrap the sandpack component for the useSandpack api
  <SandpackProvider
      options={{ bundlerURL: 'https://sandbox-bundler.netlify.app', skipEval: true }}
      template="react"
    >
      <Playground />
    </SandpackProvider>

there is one more question I didn't mention, the request still send from sandcodebox even though I config bundlerURL

@danilowoz Also, the xxx-1.0.30.tgz tarball contains bundled resources

but sandpack requests dependencies from https://cdn.jsdelivr.net/npm instead of using the es resource directly,
the tarball package.json looks like this:
  "sideEffects": [
    "dist/*",
    "es/**/style/*",
    "lib/**/style/*",
    "*.less"
  ],
  "main": "lib/index.js",
  "module": "es/index.js",
  "unpkg": "dist/sds.min.js",
  "typings": "lib/index.d.ts",
  "publishConfig": {
    "registry": "http://npm.xxx.site/"
  },

@danilowoz Any update?  Is it possible that I need to configure https instead of http for the private packge?
  "publishConfig": {
-    "registry": "http://npm.xxx.site/"
+    "registry": "https://npm.xxx.site/"
  },


aggressive purging for free large allocation spans

It is very difficult to do this well, so it requires a lot of thought.

It can use an extended header when the size of the free span is at least 2 * sizeof(struct large) with an intrusive doubly-linked list. Purged free spans could be preferred over dirty ones by ordering the tree by (size, state, addr) instead of (size, addr). The oldest dirty spans would be purged first. It needs a solid heuristic for determining when to purge, likely a combination of a minimum amount combined with a ratio.


v<IP_ADDRESS> crashes on Raspberry Pi

I have installed Denarius wallet as described here: https://snapcraft.io/install/denarius/raspbian
Raspberry Pi 3 seems to be OK, Denarius is able to start, but after several hours of synchronization process I see that it is closed. Since I start it via console, I see there a "segmentation fault" message. I don't know what to find in logs, though I was not able to find anything related to this issiue so far.

It could be an issue with RAM as the Pi 3 only has 1GB of RAM, we recommend at least 2GB of RAM to run Denarius.
You can try to grab the Pi chaindata from https://denarii.cloud and then extract the zip into your  Denarius data directory and attempt to start it again.


"Context not yet fully formed" exceptions

From time to time, I can see The context is not yet fully formed. exceptions being thrown when trying to access ServerContext.ContextUserName property.
The property is being accessed (for logging purposes) immediately after AcceptToken is called and successful.
Interestingly, the next thing after that logging call is an ImpersonateClient() call which always works.
Is there any hidden race condition/concurrency I'm perhaps not aware of here?
Here's an illustration of what my code is doing:
var status= ctx.AcceptToken(negotiateToken), out var serverToken);
LogTokenAccepted(ctx) // ctx.ContextUserName is being accessed here
using (ctx.ImpersonateClient()) { ... } // This never fails

That exception occurs when the server has not yet seen an AcceptToken call that results in a status of SecurityStatus.OK and you try to perform some action that requires a fully initialized context. That means that the authentication cycle is not complete. Check the return value from AcceptToken, and do not use the context before it returns SecurityStatus.OK.
This snippit from Context.cs shows how the exception occurs when init isn't done:
        public string ContextUserName
        {
            get
            {
                CheckLifecycle();  // <------------------
                return QueryContextString( ContextQueryAttrib.Names );
            }
        }

        protected void Initialize( DateTime expiry )
        {
            this.Expiry = expiry;
            this.Initialized = true;    // <------------------
        }

        private void CheckLifecycle()
        {
            if( this.Initialized == false )    // <------------------
            {
                throw new InvalidOperationException( "The context is not yet fully formed." );
            }
            else if( this.Disposed )
            {
                throw new ObjectDisposedException( "Context" );
            }
        }

This snippit from ServerContext.cs shows when init is called, which would disable the exception:
        public SecurityStatus AcceptToken( byte[] clientToken, out byte[] nextToken )
        {
             ...
                    if( this.ContextHandle.IsInvalid )
                    {
                        status = ContextNativeMethods.AcceptSecurityContext_1( ... );
                    }
                    else
                    {
                        status = ContextNativeMethods.AcceptSecurityContext_2( ... );
                    }
                }
            }

            if( status == SecurityStatus.OK )
            {
                nextToken = null;

                base.Initialize( rawExpiry.ToDateTime() ); // <-------------
...

I am accepting Kerberos tokens only, in which case AcceptToken results in SecurityStatus.OK on the first call. This is why I expected to not see this exception, but I figured out there were some unusual clients triggering this by sending NTLM tokens.
Thanks for the help :)


[enhancement]: Add new agent capability that allows repositories to be checked out below work directory level on self hosted agents

Describe your feature request here
This is a feature request for a new agent capability that would allow self hosted agents to check out repositories below the work directory level (so for example below "/_work", instead of just below the build pipeline directory level for example "/_work/1"), so large repositories can be shared between build pipelines without taking up space for each and every pipeline.
This feature should be disabled by default as pipelines could otherwise interfere with each other on public agents or users on public agents could change configuration files below the work directory, which would pose a security risk.
It could for example be activated by passing a flag to the configuration command in the command line when configuring the agent.
I would have already implemented this feature as described and have already created a pull request to the master branch of this repository: https://github.com/microsoft/azure-pipelines-agent/pull/3475
The pull request needs to be reviewed and accepted.
A more thorough description of how the feature was implemented can be found in the pull request.
PS: I'm reopening/readding this issue since https://github.com/microsoft/azure-pipelines-agent/issues/3951 (and previously https://github.com/microsoft/azure-pipelines-agent/issues/3483 ) were wrongfully closed by a stale-bot, it's not really stale, it just hasn't been processed yet

A couple notes:
1 - There's few things more maddening to me in the tech world than asking a question and someone without the context telling you "you're doing it wrong", but I'm going to risk doing that same thing. Have you considered other approaches to dealing w/ large repos? Shallow clones, only checking out a specific branch, etc?
If you're not familiar or in case it helps someone else:
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
2 - Your feature request specifies a particular approach to the problem, perhaps it should be more like "...that allows repositories to be checked out to a shared directory" or possibly even more generically rather than assuming /work/. My understanding of how the build process works is that there are a lot of assumptions built into how the agents do their work, directory structure, etc, and what you're trying to do probably violates those assumptions, and may require more changes than you think.

A couple notes:
1 - There's few things more maddening to me in the tech world than asking a question and someone without the context telling you "you're doing it wrong", but I'm going to risk doing that same thing. Have you considered other approaches to dealing w/ large repos? Shallow clones, only checking out a specific branch, etc?
If you're not familiar or in case it helps someone else: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
2 - Your feature request specifies a particular approach to the problem, perhaps it should be more like "...that allows repositories to be checked out to a shared directory" or possibly even more generically rather than assuming /work/. My understanding of how the build process works is that there are a lot of assumptions built into how the agents do their work, directory structure, etc, and what you're trying to do probably violates those assumptions, and may require more changes than you think.
Edit: I wonder if adding agent support for WorkTrees would help?

Thank you for your considerations but not only am I familiar with the agent I also already did the needed changes, tested them successfully and the pull request is active. I've considered all scenarios and assumptions of how the agent works and how it uses directories ;)
Please see the linked Pull Request for more details. You can test those changes too if you like.


E: Package 'glmark2-es2' has no installation candidate

Got out my pine A64+ for the first time since the kickstarter ended.
I booted stretch-minimal-pine64-bspkernel-0.7.30-140.img.xz and logged in via ssh. Works.
I wanted to try out the desktop. Is this the prefered way to install it? /usr/local/sbin/install_desktop.sh lxde
I'm getting the errors
E: Package 'glmark2-es2' has no installation candidate
E: Package 'chromium-browser' has no installation candidate
`root@pine64:~# /usr/local/sbin/install_desktop.sh lxde
Hit:1 http://security.debian.org stretch/updates InRelease
Hit:2 http://ppa.launchpad.net/longsleep/ubuntu-pine64-flavour-makers/ubuntu xen                                                                             ial InRelease
Ign:3 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Hit:4 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:5 http://ppa.launchpad.net/ayufan/pine64-ppa/ubuntu xenial InRelease
Get:6 http://deb.ayufan.eu/orgs/ayufan-pine64/releases  InRelease [1327 B]
Hit:7 http://cdn-fastly.deb.debian.org/debian stretch Release
Fetched 1327 B in 2s (586 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
26 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package chromium-browser is not available, but is referred to by another package                                                                             .
This may mean that the package is missing, has been obsoleted, or
is only available from another source
Package glmark2-es2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'glmark2-es2' has no installation candidate
E: Package 'chromium-browser' has no installation candidate
root@pine64:~#
`

If I get a desktop installed is HDMI still a problem? See;  hdmi not functional in prerelease #55

I received these same errors. When these are commented out of the /install_desktop.sh file, error #71 occurs.

I put this up after installing mine.  It really helps to have clear instructions around a problem for newbies.
https://amoghdesai.com/technology/how-to-resolve-errors-and-install-desktop-environment-on-pine64-a/
hope ayufan will update the script to fix this issue in next release.

Thanks @desaiamogh
FWIW I've moved over to Armbian


Urage For Help

I just want to reproduce the exact reported metrics by testing the model;
it always shows the : AttributeError: 'NoneType' object has no attribute 'shape'
so maybe the code can't find the test image,so could tell me what's wrong with the below  ;
I have download the dataset tusimple, there have two folder:test_set、train_set;
so i put the 0531 folder in the dataset folder(the 0531 file is from the test_set\clips\0531);
the detail is like that ：
├─cfgs
├─figures
├─datasets
│  └─tusimple
│      └─label_data_0531.json
│      └─0531
│    ──                └─1492626253262712112
│    ── ──                       └─1.jpg
│    ── ──                     └─2.jpg
│    ── ──                       └─......jpg
│    ── ──                     └─20.jpg
│   ──           └─1492626271917313999
│    ──          └─......
│   ──           └─1495489446318620134
├─experiments
├─lib
│  └─datasets
└─utils
└─test.py
└─train.py
└─config.yaml
and the root in config.yaml is that root: "datasets/tusimple"

You should have a directory structure like that:
datasets/tusimple:
clips  label_data_0313.json  label_data_0531.json  label_data_0601.json  readme.md

datasets/tusimple-test:
clips readme.md  test_label.json  test_tasks_0627.json  test.txt

For the train split, if using this structure, the root in config.yaml should be datasets/tusimple, for the test split, datasets/tusimple-test.

I'M trying with your advice, hope it can success


Add ability to adjust startup disk threshold

During testing it's handy to adjust the disk threshold as test machines can be pretty small.

Closed by #5


Improvements for NPC Sheets - Spirits - BlackSpiralDancer - Fomori

Would it be possible to have NPC sheets suitable for dedicated options for spirits, a simplified NPC sheet to generate BSDs that takes gifts and rage into account, and an NPC sheet suitable for Fomori?
Currently, the generated NPCs only take into account the statistics for creating humans and ghouls. An NPC sheet for generating vampires is also needed.
Grazie mille
Creos

This is already planned in the works! Worry not, it's going to be gotten to sooner or later.


ARROW-6705: [Rust] [DataFusion] README has invalid github URL

Thanks! Unfortunately I can't merge due to a bug in the merge tool: https://issues.apache.org/jira/browse/ARROW-6706


Feature: native coin-or solver

Addresses #45
Description
This PR provides a new solver to call CoinOR CBC trough the C API.
Implementation
The new struct NativeCBCSolver implements SolverTrait and uses the rust bindings in https://github.com/KardinalAI/coin_cbc/. A different NativeSolverTrait should be considered to extend the functionality in the future.

It compiles just fine on linux 5.9.1 but that does not seem to be the case on Windows. It needs more testing.

Thank you for your contribution. Can you find out where the problem is coming from? Compatibility with OS is quite important I think

It seems like the cbc_coin bindings are looking for CbcSolver.lib, since that's the name of the library in Linux systems. However, looking at the file that you use in appveyor, it seems like the library is called libCbcSolver.lib for Windows.
If I am correct, it would be a matter of conditional compilation. I'll open an issue at https://github.com/KardinalAI/coin_cbc/.
I don't have access right now to a Windows system (I will have it next week maybe).

The difference between this and the other backends is, that this has compile time dependencies.
So making it an optimal feature would be a good idea.
The feature could be only available on Linux, which would solve this problem.

@EdorianDark Maybe we can do that if there's no way to make it compile in Windows (which I think it's just me lacking the knowledge to do it). However, it is not a solution but a workaround, and I am sure that some people would like to get it working on Windows. Like I said, I could get my hands on a Windows computer this coming week to see if I can make it work.

Ok, thanks. Let me know if we can fix this issue. In last resort, we can imagine providing your implementation as an extension of the core lib. But it can be better to merge it here.

I've merge this PR and #51 but wait for a fix before pushing into production for the next version

All good. I keep only linux tests for the CI. Thanks


Proof format

What exactly is the format of the data field in the json proof file generated after creating a proof?
I would like to be able to extract values. My assumption was that it looks like a Vec, which should be decoded with String::from_utf8() but I do not have any luck with this.
A pointer on what to do or some documentation about the proof structure would super helpful if this exists, I could not find it on the page.

Adding onto this, it would also be great to know how to extract a proof from file path to use in verification as well. I could not find any tool for this in the sdk.

After doing some digging I found out that the bytearray under the data field does not encode properly only if the arguments being passed are longer than 332 characters. I will try to look more into this and see if there is an issue with my elf or if this potentially is a bug.


Missing dependency on package database

I am seeing the following while trying to build GHC 8f468fe05cf81ca516b3a8cf66792f842dbfcf40 with hadrian 68446abeab92087492baba6c746ab94c3bb7c2bb,
<command line>: unknown package: rts
shakeArgsWith   0.000s    0%
Function shake  0.220s    0%
Database read   0.006s    0%
With database   0.027s    0%
Running rules  47.492s   99%  =========================
Total          47.745s  100%
Error when running Shake build system:
* _build/stage1/rts/libHSrtsthr_l_-ghc8.3.20170926.so
* _build/stage1/rts/cmm/Updates.dyn_o
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-stage1 -Wall -hisuf dyn_hi -osuf dyn_o -hcsuf dyn_hc -fPIC -dynamic -i -i_build/stage1/rts -i_build/stage1/rts/build/autogen -Iincludes -I_build/generated -I_build/stage1/rts -I_build/generated -optc-I_build/generated -optc-fno-stack-protector -odir _build/stage1/rts -hidir _build/stage1/rts -stubdir _build/stage1/rts -c rts/Updates.cmm -o _build/stage1/rts/cmm/Updates.dyn_o -O2 -H32m -Werror -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -Irts -Wno-deprecated-flags
Exit code: 1
Stderr:
<command line>: unknown package: rts

)

Looks like this is related to the dynamic way.
@izgzhen Any ideas?

@bgamari Thanks for reporting. Was it the default command build -j?

Andrey Mokhov<EMAIL_ADDRESS>writes:

@bgamari Thanks for reporting. Was it the default command build -j?

Yep.

I can't reproduce that with newer GHC on my Linux machine. I will try build the exact GHC version later.

Two differences that I noticed between the Make build system and the command line produced by Shake,

Shake doesn't pass -DCOMPILING_RTS
Shake doesn't pass -this-unit-id rts

Unfortunately neither of these seem to make a difference.

I can't reproduce with GHC 8f468f either ...

For the record, the inplace package database also appears to be empty; I haven't confirmed but I suspect this is the issue. I berlieve you need to register the rts package before attempting to build it.

I had thought that perhaps this was just a race between the package registration and build. However, manually running hadrian inplace/lib/package.conf.d/.stamp and then rerunning hadrian doesn't appear to work either.

Oh dear; it seems I was just pulling from the wrong remote. Many apologies for the noise.

Never mind, I was pulling from the right remote; in the above comment I was just looking at the wrong GHC tree. I'll leave a fresh build running overnight as it's possible this issue was just due to stale files in my GHC tree.
Clearly it's time to go to sleep.

@bgamari This is just a missing dependency, unrelated to the dynamic way. Here is how to reproduce:

Do full build.
Delete inplace\lib\package.conf.d
Delete _build\stage1\rts\cmm\Updates.o
Run Hadrian with target _build\stage1\rts\cmm\Updates.o.

I'll try to fix this tonight.

Well, it seems that the build finished. I suspect the problem was simply stale files from the old build system left in the tree. Sorry again for the noise!

@bgamari Let me reopen this under a revised title, since I think there is a bug in Hadrian related to a missing dependency on the package database.

The above commit should fix the missing dependency, so I'm now closing this.

Unfortunately I still see a variant of this while cross-compiling. See #431.


Dual network card CentOS7.9 cell error

install cell module 1.3.0
report error details
try installing from online reciprocity...
dependency packages installed
**configure default network bridge fail: no ethernet interface available**

reinstall OS as same situation
$# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p8p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 70:b5:e8:4a:ca:3e brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global noprefixroute p8p1
       valid_lft forever preferred_lft forever
    inet6 fe80::72b5:e8ff:fe4a:ca3e/64 scope link
       valid_lft forever preferred_lft forever
3: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:e0:4c:82:6a:df brd ff:ff:ff:ff:ff:ff

net-twork card p1p1 disabled

p8p1 is normal

p8p1 is normal

#https://github.com/project-nano/installer/issues/2

#https://github.com/project-nano/installer/issues/2

the installer only browse interface with prefix "m", em1, em2, etc~

the installer only browse interface with prefix "m", em1, em2, etc~


Please fix the azure-pipeline.yaml

Hi, Mike. Thank you for this app and DevOps Courses both CloudSkills and INE.
FYI... The azure-pipeline.yaml file has an Ubuntu 16.04 . When I run it into Azure DevOps CI, the job stop, indicating that the image is not available. I look into the current ubuntu images in Azure DevOps for CI Jobs and the only two images available are 18.04 and 20.04. I imagine this change could improve the .yaml file and be usefull for the INE labs and DevOps Foundation Course. Thank you very much

!

use "ubuntu-latest" instead of specifying the image


Desert Lava Super Rod Fishing doesn't yield items(and only 2/3 pokemon) [Feature] Bug explanation

Describe the bug
After about 100 casts with a super rod(about 75% of the durability of a rod), fishing in the desert for omega shards, only Magcargo(all "!" were this) and Torkoal(all "!!" were this) were found. No items(ground gem, fire stone, smooth rock, fire stone shard should all be obtainable at relatively high rates) and no "!!!"s at all, meaning no omega shards. There is also supposed to be magmar in the pokemon pool..
I suspect that item fishing from lava is somehow disabled through config.
To Reproduce
Steps to reproduce the behavior:

Go to Lava in "Desert" biome
Fish 50+ times with a Super Rod
Crys in slugs and tortoises

Expected behavior
-Magcargo, torkoal and magmar available
-Gems, stones, and shards available
-Omega shards available
Screenshots
No buggy visuals. Just mechanical.
Username:
Cindermaji
Server name (all applicable):
Orion
Logs:
Some log lines from the fishing:
https://pastebin.com/N2cQddwp
Additional context
I may not have casted enough times to truly find a "!!!", but the "!" and "!!"s should be things other than pokemon.
Pixelmon fishing rods are also not supposed to lose durability, according to a mod who was online.

Fixed, closing


DEADLINE_EXCEEDED not throwing an error and hangs indefinitely

Describe the bug
I'm calling a gRPC service written in Java, and when passing an invalid argument to the service, the service will hang until the deadline. On the backend for the client I'm using connect-node. The channel is configured like so:
const DYNAMO_CLIENT = createPromiseClient(
  DynamoService,
  createGrpcTransport({
    baseUrl: process.env.BACKEND_URL,
    httpVersion: '2',
    useBinaryFormat: true,
  }),
);

I'm then calling a unary method on the service like so:
try {
    const response = await DYNAMO_CLIENT.listTargetServices(
      {
        targetServiceDiscoveryName: decodeURIComponent(target as string),
      },
      { timeoutMs: 10000 },
    );
    console.log(response);
  } catch (e) {
    console.error('Error listing target services', e);
  }

With this setup the call above just hangs indefinitely. I can see the server is responding with a DEADLINE_EXCEEDED as when calling this same method with grpcurl like so:
grpcurl \
  --plaintext \
  -d '{ "target_service_discovery_name": "shasdfasdf" }' \
  -max-time 10 \
  -H 'service: dynamo' \
  localhost:5990 \
  richieahb.dynamo.v1.DynamoService/ListTargetServices

I get a response after 10 seconds:
ERROR:
  Code: DeadlineExceeded
  Message: context deadline exceeded

I've also added breakpoints into the Java and tested locally and can see it's calling onError in the StreamObserver used for sending the responses to the client.
To Reproduce
If you encountered an error message, please copy and paste it verbatim.
If the bug is specific to an RPC or payload, please provide a reduced
example.
Environment (please complete the following information):

~@bufbuild/connect-web version: (for example, 0.1.0)~
@bufbuild/connect-node version: (for example, 0.8.5)
Frontend framework and version: (for example<EMAIL_ADDRESS>(although this call is from the server)
Node.js version: (for example, 18.15.0)
Browser and version: (for example, Google Chrome 111.0.5563.146)

If your problem is specific to bundling, please also provide the following information:

~Bundler and version: (for example, webpack@5.74.0)~
~Bundler plugins and version: (for example compression-webpack-plugin@10.0.0)~
~Bundler configuration file: ~

Additional context
N/A

I have a suspicion it may be the same as https://github.com/bufbuild/connect-es/issues/463 but not sure. It looks like the stream is closing on the server but this isn't getting picked up for whatever reason.

Thank you for the report, Richard. Yes, this looks very similar like #463, and may be the same root cause. We understand that this is a blocking issue and have this very high on our list.

Thanks for the response. We've currently got around this by fixing the issue that was causing the timeout on the server, but not the most sustainable solution! Happy to try help with a reproducible example next week if needs be but I think all the info is there to reproduce it.

Apologies but I've not had time to make a repro case for this but thought I'd share some more details around the specific call this seems to happen on.
Specifically, the case where I've noticed this failing has been when the Node promise client has called the server (say server A), which has then called another server (B). In my specific case, server A was unable to reach server B, and the call timed out before A ever maned to reach B (although I'm not sure whether or not the same would have happened if A would have opened a connection with B, but B just failed to respond within the deadline).
I still plan to try and make a repro but not sure when I'll be able to find time so thought I'd share.

Thank you for the details @RichieAHB. I am not sure that server B is relevant, as the Node.js client is only directly talking to server A.
The important question is what exactly this server is doing on the wire. It could respond with a gRPC error status in HTTP trailers, HTTP headers, or close the HTTP/2 stream with one of the error codes of the RST_STREAM frame. In the latest release v0.9.0, we make sure that all three situations are handled properly, and we fixed an issue in https://github.com/bufbuild/connect-es/pull/619 that may be related.
Could you give it a try with the latest release?

@RichieAHB, we'll need a reproducible case to continue here. v0.10.0 adds support for basic keep-alive, which would help in case the connection dies before the deadline is reached.
Regardless of keep-alive, v0.9.0 also added client-side support for deadlines - the call is canceled on the client when the deadline is reached, regardless of what the server does, so you should never see the request hanging past the deadline in any case.
Closing this, but please let us know if you have any issues with the latest release.


Windows: define ComputeSwiftDirectory for Windows

Assume that the Windows layout is relatively similar to that on Linux.
lldb.dll is located in the bin directory rather than lib and the
ComputeSupportExeDirectory provides a FileSpec that is off by a
level.  Using ComputeSupportExeDirectory with ..\lib\swift gives us
the actual location of the resource dir when lldb is installed into a
toolchain (as per the MSIs from
https://compnerd.visualstudio.com/windows-swift).

CC: @dcci @kusmour
This at least permits the repl to startup on Windows; there is still an issue with IRGen that needs to be investigated.

@kusmour took care of this I believe and this should no longer be needed


Now displaying confirmation message when customer signs and submits order

Jira ticket for this change
Summary
Acceptance criteria has been met in terms of displaying the plain text message in the details column. I believe the customer information being displayed will be handled by this pull request once merged.
Setup to Run Your Code

💻 You will need to use three separate terminals to test this locally.
Terminal 1
Start the Server locally.
make server_run

Terminal 2
Start the Customer UI locally.
make client_run

Terminal 3
Start the office client locally.
make office_client_run

Verification Steps for Author
These are to be checked by the author.

[x] Have the Jira acceptance criteria been met for this change?

Verification Steps for Reviewers
These are to be checked by a reviewer.

Create a new customer and go through the entire move process ensuring you sign and submit.
Login as a service counselor and visit the move history tab.
The event should read "Submitted move" and the details should read "Received customer signature".

Frontend

[ ] User facing changes have been reviewed by design.
[x] There are no new console errors in the browser devtools
[x] There are no new console errors in the test output

Screenshots

Warnings

:warning:
Please add the JIRA issue key to the PR title (e.g. MB-123)

  Generated by :no_entry_sign: dangerJS against 013ce34b01ef80ef4d0555c0790acfd42c5a6764


Switch start and end date if end date is before start date

As there is no validation in contentful if the startTime is after the endTime then we should switch them around as to not break elements in the UI.
I have also moved the locale into a separate module. (Probably should have done this in a separate PR but thought some housekeeping wouldn't go amiss).
Fixes: #73
Pre-flight check-list

[x] Documentation (N/A)
[x] Unit tests

Pre-merge check-list

[ ] Tester approved

TestPlan

[ ] Create/change an event in contentful with the start date before the end date.
[ ] App does not break and start and end date should be switched.

So the app isn't crashing when an event from Contentful has reversed start/end times, however this event's times aren't being corrected, instead the times are being flipped for all the other events (with correct start/end times):

"Swapped Date Test Event" is the only event in the list that has reversed dates.
Tested on a Galaxy S8

I can pick this up. My bad sozza

@declanslevin this is fixed now :)

closing in favour of https://github.com/redbadger/pride-london-app/pull/106


Flesh out swarm key concepts

Fixes #2048
cc/ @nishanttotla

@nishanttotla I addressed your feedback. Thanks!


Issue/6025 document discovery handler

Description

[x] add documentation including examples, see the design for more details.

part of https://github.com/inmanta/inmanta-core/issues/6025
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[x] Attached issue to pull request
[x] Changelog entry
[x] Type annotations are present
[x] Code is clear and sufficiently documented
[x] No (preventable) type errors (check using make mypy or make mypy-diff)
[x] Sufficient test cases (reproduces the bug/tests the requested feature)
[x] Correct, in line with design
[ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )
[ ] If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see test-fixes for more info)

Processing this pull request

Merged into branches master in 3bad712effae1704702b1e943b81ee2404125285


Web workers

Don't look at this yet.
There are several heavyweight operations performed on the client in the main JS thread that we'd like to offload onto web workers.
Constraints:

Each worker must compile down to a single .js file that can be loaded from the same origin.
The workers are used by core-frontend so must be delivered with that package or otherwise made accessible to it.
Each worker should have as few dependencies as possible and tree-shaken, to minimize script size.
Workers should definitely avoid doing dumb stuff like trying to access IModelApp or the DOM.
Workers should be written in TypeScript.

There are a variety of ways to meet these constraints. Here, I am building the worker code together with the rest of core-frontend, then web-packing each worker's output into a single .js file and copying it to lib/public/scripts. This simplifies many things. However, it makes it rather too easy to accidentally do dumb stuff like accessing IModelApp indirectly (e.g., by calling some function that calls some other function that uses IModelApp.renderSystem or IModelApp.tileAdmin).
It is not the goal of this PR to parallelize operations like tile decoding. We're not trying to decode multiple tiles simultaneously in different workers, because that will increase memory usage. We're just trying to parallelize these operations relative to the main thread.

Split off into #5532; worker support to be introduced subsequently.

Superseded by #5548.