Service broker can declare supported schema for parameters field

Service broker will provide schema information for both service instances and bindings for each plan. The schema information is for both parameter inputs and output properties. Currently the proposal is that the schema format will be JSON schema v4.
Proposal: https://docs.google.com/document/d/1-IKI-PwPnhtbK0su1UzWA_UoyaCjL8S2d6ky3Xmuwvg/edit?usp=sharing

This issue seems more appropriate for the Kubernetes service-catalog project. The openservicebrokerapi org is not meant for development of platform controllers.

@shalako the intention of this issue is for the parameter schema PR. I've renamed to make it more appropriate.

Proposal doc: https://docs.google.com/document/d/1-IKI-PwPnhtbK0su1UzWA_UoyaCjL8S2d6ky3Xmuwvg/edit?usp=sharing

@avade I know you're planning to draft a proposal for the CF community for this feature. You may want to consider Brendan's existing google doc; link to it and dedup or duplicate the content in yours? I expect going forward we'll have to have a design proposal for the API and another for CF changes.

I had already lifted much of the content form Brendan's doc :)
Brendan are you going to update the proposed API spec part of this doc or
sharing more as an example?
If this is going to be "the" proosal doc for this feature, can I get rights
to suggest edits, please?

Hey Alex,
The API spec part was more an example section, but I'm happy to make this
more the canonical doc if you'd like. I believe that you can suggest edits
if you have comment rights by switching to suggest mode. If not I can give
you edit rights.
B.
On Tue, Nov 29, 2016 at 11:40 AM, Alex Ley<EMAIL_ADDRESS>wrote:

I had already lifted much of the content form Brendan's doc :)
Brendan are you going to update the proposed API spec part of this doc or
sharing more as an example?
If this is going to be "the" proosal doc for this feature, can I get rights
to suggest edits, please?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/openservicebrokerapi/servicebroker/issues/59#issuecomment-263675904,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKwU8r_gzpQAj5w7o34U5k1hEWU42AgOks5rDH-fgaJpZM4JqRle
.

I have had a read through the comments and found it difficult to work out the state of the discussion.
Perhaps @bmelville / @gberche-orange could provide a summary and what needs to happen before we can accept the proposal and get started on a Cloud Foundry implementation.

To my understanding, the core specification needs a decision on where schemas should be attached at :

object level (service instance, service binding) to the expense of restricting some fields only to some verbs: create/update/delete/actions)
method level (provision, bind, update, delete, actions) to the expense of duplicating field definitions for the ones common between verbs, such as create and update

The JSON schema extensions (x-serviceBroker-updatable, x-servicebroker-supported-in-verbs, or x-servicebroker-confidential, x-servicebroker-displayType, x-servicebroker-service-instance-required-types, x-servicebroker-service-instance-required-tags ) seem less mature and needs more explorations of the use-cases to support.
The core specifications (a.k.a. without schema extension) could a good candidate for a 2.12 feature with an experimental status. This would enable to start getting feedback from prototype implementations in CF such as:

the CF CLI (distributed as a plugin) that would perform

arbitrary params validation
arbitrary params prompting (even if this is not currently the direction adopted by the CLI team  /CC @dkoper)
arbitrary params completions

generic service brokers (similar to pivotal's on-demand-broker) leveraging the schema to validate arbitrary params
CC validating the schema

Based on feedbacks from such prototype, decisions could be taken on

whether to remove the experimental status on core schema specs or amend them
which schema extensions to add to fullfill the missing use-cases

I think that's a great overview from Guillaume. The remaining work item is
the decision on where to hang schemas, and there will likely be more to
flush out, especially around the new Broker Actions proposal.
I think we have landed on method-level schemas, but it seems like there is
definitely room for a lot of exploration and understanding of use-cases.
On Thu, Dec 15, 2016 at 1:30 PM, Guillaume Berche<EMAIL_ADDRESS>wrote:

To my understanding, the core specification needs a decision on where
schemas should be attached at :

object level (service instance, service binding) to the expense of
restricting some fields only to some verbs: create/update/delete/actions)
method level (provision, bind, update, delete, actions) to the
expense of duplicating field definitions for the ones common between verbs,
such as create and update

The JSON schema extensions (x-serviceBroker-updatable,
x-servicebroker-supported-in-verbs, or x-servicebroker-confidential,
x-servicebroker-displayType, x-servicebroker-service-
instance-required-types, x-servicebroker-service-instance-required-tags )
seem less mature and needs more explorations of the use-cases to support.
The core specifications (a.k.a. without schema extension) could a good
candidate for a 2.12 feature with an experimental status. This would enable
to start getting feedback from prototype implementations in CF such as:

the CF CLI (distributed as a plugin) that would perform

arbitrary params validation
arbitrary params prompting (even if this is not currently the
direction adopted by the CLI team /CC @dkoper
https://github.com/dkoper)
arbitrary params completions

generic service brokers (similar to pivotal's on-demand-broker)
leveraging the schema to validate arbitrary params
CC validating the schema

Based on feedbacks from such prototype, decisions could be taken on

whether to remove the experimental status on core schema specs or
amend them
which schema extensions to add to fullfill the missing use-cases

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openservicebrokerapi/servicebroker/issues/59#issuecomment-267449723,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKwU8qnDJGQQq3Orkr_Cn82QpyPd6jX9ks5rIbFsgaJpZM4JqRle
.

Hey folks,
I have updated the proposal with the method-based schema approach, and have also added a section on extensions (outside the scope of this but worth discussing a bit since there have been so many questions).
Please take another look when you have a chance.
B.

I've read the new proposal. Looks good.

The "plan-attached schemasets" concepts were reviewed during the Feb 02, 2017 design session. The proposal has been updated with details, comparison table updated, along with other proposal archived for reference.
Feedback is welcome.
Next steps one feedback is collected would be:

Validate this proposal by implementation in CF platform
update the dependent proposals (e.g. #114, #116 and possibly #115) to validate extensibility support.

Hi all,
As discussed in the calls earlier this week, we've made some provisional spec changes for two of the proposals:

Proposal 5: Service-level schema-sets
Proposal document
Provisional spec

Proposal 2: Method-attached schemas
Proposal document
Provisional spec

Feedback welcome. We discussed creating spike implementations for both of the above proposals in Cloud Foundry as a good next step.

Personally I would have preferred to keep the actual schemas out of the catalog, either using a dedicated endpoint or referencing URLs for each schema, but if the majority of the group prefers an all-in-one approach then I would vote for option #2, method attached schemas. Although they increase redundancy at least it is obvious which schema applies when, which contributes to simplicity. Also, if in future different osb platforms are reflected by different catalog responses (requested with an osb platform specific query), then the need for extending the more generic applies-when clause from proposal #1 is less relevant.

I spoke with the PM for the Cloud Controller, which is the client of the broker in CF. He can work with either proposal. We discussed they're likely to story the schemas with plans, even if the broker exposes them at the service level. So if a schema has no applies_when.service_plan_id, then Cloud Controller will save the schema for all plans of the service. CF clients would then get schemas only for requested plans, and either parse them by applies_when or by nested object/method.
We'll send these proposals to our CLI and web client teams for feedback.

Also, if in future different osb platforms are reflected by different catalog responses (requested with an osb platform specific query), then the need for extending the more generic applies-when clause from proposal #1 is less relevant.

@petereberlein @shalako
Rethinking about last WG call discussion on having the catalog endpoint request convey the platform profile (e.g. in a context parameter): while this enables the broker response to restrict the schemas returned to the only requested platform (say CF), this is likely to make the broker federation use-cases more complex:
In the case of a federation of brokers (e.g. where a federating broker federates a number of federated brokers, supporting different platforms and possibly different OSB API versions), the federating broker likely needs to obtain all supported schemas by a federated broker at once, and would prefer avoiding to repetitively query the /catalog endpoint with all possible platforms in a context parameter or all possible X-Broker-API-Version header variations.
I still believe keeping room for future extensibility (without introducing breaking changes, such as with the applies-when construct) is useful and relevant, as we explore new use-cases and broader OSB API usage.
I hope that the "validation by implementation" phase in CF will provide additional useful feedbacks for both proposals #5 and #2.
Sorry I won't be able to make it to the weekly call today.

@gberche-orange I expect in that scenario, the federating broker would have to fetch all the schemas. A query param to filter the response would be an optional nice-to-have.
I have spoken with the team responsible for the Pivotal web UI for CF. They have expressed a preference for the simpler, more explicit declaration of schemas enabled by proposal 2.

I think that proposal 2 (method-level schemas) will be easier for platform providers to implement (they don't have to match schemas to plans, methods and operations) and easier for service broker authors to implement. Both proposals have capacity for future extensibility, so my preference would be to proceed with a test implementation of proposal 2.

I agree with what @mattmcneeney is saying and also propose we proceed with proposal 2.
I like the idea of extensibility applies-when could offer, but it feels like futureproofing

I'd also vote for the more explicit and straightforward method level schemas option:
+1

Both proposals have capacity for future extensibility

@mattmcneeney  Can you please detail how you envision the proposal 2 could offer extensibility for schemas that differ based on geographical locations (us-east vs us-west), OSB api version, platform profiles (CF vs k8s), or bind resources (e.g. credentials vs route service) ?
Besides, a minor wording clarifying fix would be helpful in proposal 2 example, see https://github.com/avade/servicebroker/commit/9ef94ce96ca65bfc9ce482d5ea5be0ad62643a84#commitcomment-21129675 )

I expect in that scenario, the federating broker would have to fetch all the schemas. A query param to filter the response would be an optional nice-to-have.

@shalako, I agree that in order to support the federation use-case (e.g. with different schemas per profiles K8S or CF), the federating broker would prefer to fetch all schemas and rely on a construct such as applies-when rather than a query param that would filter the response.

We discussed they're likely to story the schemas with plans, even if the broker exposes them at the service level. So if a schema has no applies_when.service_plan_id, then Cloud Controller will save the schema for all plans of the service. CF clients would then get schemas only for requested plans, and either parse them by applies_when or by nested object/method.
[...]
I have spoken with the team responsible for the Pivotal web UI for CF. They have expressed a preference for the simpler, more explicit declaration of schemas enabled by proposal 2.

@shalako I understand you considered in https://github.com/openservicebrokerapi/servicebroker/issues/59#issuecomment-282867764 to allow for future extensibility in the broker API through construct such as applies-when while optionally reducing initial complexity for clients that desire it by offering client UIs endpoints (CC endpoint for CF) that return a single schema based on filters expressed as query params (plan, object, method for now), i.e. clients don't have to handle the applies-when logic if they don't need to now.
This approach seems compatible with the feedback you received from the PCF webui team, and a fair balance of extensibility vs simplicity. Are you still considering this approach ?

@gberche-orange There are a number of ways proposal 2 could be developed to offer extensibility in the future (nested input parameters, request headers, similar applies-when logic, etc). Whilst it is obviously important to consider these future use cases, I think we should try to move forward with the simplest solution that fits the current use cases and then separately discuss what changes we would need to support more advanced use cases in the future.
To ensure this is what we are actually doing, do you have any services you are developing now that require different input parameters based on geographical locations that you could share with the group?
Thanks,
Matt

IBM is ok with proposal 2: Method-attached schemas.
With respect to extensibility, and in particular the geo-based parameters, two things come to mind. First, since I think we're talking about the geo of the platform making the request people may want to consider including this geo information in the "context" object and then perhaps the spec should including a "context" object on the GET /v2/catalog request. Then the Broker can customize the list appropriately.
Second, at what point should we look at the differences between services plans and say "that's too big of a difference, make it a new service" ? In the case of differing geos I'm not sure it crosses that line, but I'm mentioning this because as we think about variants between plans and how to represent them we always have the ultimate escape clause (create a new service) if people can't express the differences between plans sufficiently.

@mattmcneeney

There are a number of ways proposal 2 could be developed to offer extensibility in the future (nested input parameters, request headers, similar applies-when logic, etc).

I'm not sure to understand how the ways mentioned above would work without implying breaking changes for proposal 2.

"nested input parameters" seems problematic since in proposal 2 as the semantic of each field is determined by its depth in the JSON structure. Adding new depths would only enabling adding new mandatory ordered selectors, while in most cases optional selectors are needed

e.g. from
      "schemas": {
        "service_instances": {
          "create": {
            "parameters": {
              "$schema":

to the following after adding platform-profile and x-api-location dimensions
      "schemas": {
        "service_instances": {
          "create": {
            "CF": {
               "eu-west1": {
                  "parameters": {
                      "$schema":

"request headers" approach has similar problems as "Proposal 6: Schema endpoint" (forces platform apis to expose chatty apis, and does not support the broker aggregration use-case)
"similar applies-when logic" requires converting the "parameters" object into an array, which is a breaking change.

I think we should try to move forward with the simplest solution that fits the current use cases and then separately discuss what changes we would need to support more advanced use cases in the future.

To me this strategy fits well for iteratively building products where the cost of change is low (a.k.a impact of changes are controlled).
In the context of OSB, we are building an API (with a published change policy stating breaking changes will not be introduced), which is made available to multiple communities. Once OSB starts being adopted by multiple communities as we hope, I would expect the cost of changes to increase (with number of platforms, clients etc) and be harder to assess beforehand (because it might be harder to get feedback from a more widespread user base).
My efforts since late dec on this issue were trying to avoid that schemas get introduced with the simple input params use-case in mind, and fall short of supporting other use-cases that arise in the coming months.
I see some existing features in the spec that could have been handled by schemas (bindeable, plan_updateable)
I also understand similar abstractions have been introduced in the past in the service broker api to account for future extensibility, e.g. the bind_resource
I however might be wrong by:

suggesting potential use-cases that are not realistic or won't be needed anytime soon by OSB users
mis evaluating the cost of change of the schema part in the OSB
mis considering use of schemas w.r.t. dedicated focused spec additions for up coming features ("to a man with a hammer all problems appear as nails")

Thanks anyway for having taking time to consider my concerns and alternative proposals to try to address them.
I also agree with Shannon's point on WG weekly meeting on Feb 14th that some other features have a high priority and it might not be wise delaying them to solely focus on schemas.

To ensure this is what we are actually doing, do you have any services you are developing now that require different input parameters based on geographical locations that you could share with the group?

I updated the proposal document with more precise use-cases for additional selectors:

For the geo location, I imagine some service plans not being available to some locations (I recall @bmelville mentionned interest in this feature, maybe @vaikas-google could detail this), or features of services (such as HSM for backup encryptions) not being available in some locations. See examples 4a and 4b.
For api version selector, the db-dumper-service could make use of this selector. See example 6.
For service-plan-id update selector, I added example 8.

since I think we're talking about the geo of the platform making the request people may want to consider including this geo information in the "context" object

@duglin  I believe this approach falls down in the same limitation than an platform profile being passed in a context object to the catalog endpoint see https://github.com/openservicebrokerapi/servicebroker/issues/59#issuecomment-283046570 which would make the broker aggregation use-case hard to implement.

Thinking about using the schema to express which platforms are supported for certain inputs, If the OSBAPI returned the schema style @gberche-orange is advocating for, I do not think that we would expose non CF plans via the Cloud Controller API for CLI's / GUIs to consume. It would mean that Cloud Foundry want to interpret and filter the schemas/plans based on platform before presenting them in the marketplace.
Then this causes platforms to build logic to choose which plans to render in the marketplace and then clients of that marketplace also needing to interpret the applies when logic.
@duglin's idea to add context to the catalogue request makes more sense to me, as the broker is expected to only return relevant services and plans for the platform. Aggregating brokers would then be required to delegate requests rather than serving up static catalog content.
Interested to hear from @vaikas-google / @pmorie. I know @pmorie and @mattmcneeney need a decision soon to unblock their respective projects.

I have to be honest, I'm having a really hard time following all of this. I think it would help me if someone could clearly articulate the exact problem we're trying to solve. For example, I thought it was a fairly limited issue: need a way to discover the list of input parameters (and their schema) that can be specified on the instance and binding create() operations.
But now we seem to have turn into some rather complex filtering issue and I'm not clear what the requirements are for this - e.g. geo seems to be one.
If we want to solve the problem of certain plans, or even services, only being available based on certain criteria then I think this is a different problem from the "parameter discovery" one and I would prefer we address it in a different issue.  If, however, people think they are linked then can we get an updated problem statement so its more obvious what the exact issues are that we're trying to address?

Today on the call we agreed to proceed with the proposal known as "Method-attached schemas", or plan-level schemas.
Proposal document
Provisional spec

Quick question; how would everyone feel about adding the following to the spec (not necessarily in these words):
Service brokers MUST validate input parameters that are provided via calls to create, update or bind.
When looking into the validation of this work in Cloud Foundry, it would be ideal if service brokers performed validation on any input parameters to ensure they match the relevant schemas. There would be no easy way to enforce this, but it would encourage best practice so that brokers do not rely on platforms to perform validation (which may or may not exist). What are your thoughts?

We did that with our broker, and seems reasonable to ask brokers to validate the schema that they define.

Since this addition to the spec isn't directly related to input parameter schemas, I can create a PR for this minor change and we can continue the discussion there.

Would everyone be OK with me changing any references to "input parameters" in the spec and proposal to "configuration parameters"?
This would be more consistent to the existing CF experience, and with hindsight the word "input" is a bit meaningless.
$ cf create-service -h

NAME:
   create-service - Create a service instance

USAGE:
   cf create-service SERVICE PLAN SERVICE_INSTANCE [-c PARAMETERS_AS_JSON] [-t TAGS]

   Optionally provide service-specific configuration parameters in a valid JSON object in-line:

   cf create-service SERVICE PLAN SERVICE_INSTANCE -c '{"name":"value","name":"value"}'

re: wording change - ok with me

I'm OK with the wording change to 'configuration parameters'

For the MUST I'm OK with that as well. We should probably make it clear in the documentation that Controllers are not expected to do the validation so it's left to the broker. MUST is probably the best way to communicate that to the broker implementors.

I'm not following the idea of controllers vs brokers doing validation. Whoever receives a message is the one do the validation.
As for MUST vs SHOULD. The example I worry about is something like: a property is defined as an int  but someone passes in "1.0".   While it would be fine for a receiver of this message to reject it, I think its also ok for someone to be a bit forgiving and convert "1.0" to "1" and keep going - as long as the semantics of what the user intended can be unambiguously determined.  This is why I kind of prefer a SHOULD.

In reality I expect to see multiple points of validation. A CLI or Web UI could validate any parameters that are provided against the schema, followed by the platform, and finally the service broker. I don't think this is a bad thing.
I agree with @duglin that brokers should have some leeway to interpret a "1.0" as a "1", as that will provide a better, less finicky user experience. The important thing is that we add something to the spec (applying to all fields in every call) that guides when and how errors should be returned. A broker interpreting "1.0" as "1" is acceptable (in my opinion). A broker going further than that and interpreting "1.9" as "2" could be a confusing experience.
(This is also why fetching current values is going to be important for end users)

I've been assuming we could extend the json schema with additional fields as needed by platforms. Since the f2f its been made clear we can't, so there might be an issue.  We have a need to extend the schema with more info than what the json schema can support. For example, in bluemix we need to include a URL that the platform can perform a GET on to dynamically get the list values for the dropdown.  I'm beginning to think that perhaps we should not use JSON schema and develop our own way of expressing our requirements.
Here's the snippet of docs from Bluemix for reference:
Using service parameters to create customized input fields
Service parameters allow you to create customized input fields in the Bluemix service creation user interface. The values entered by the user will be sent to your service broker on the request to provision the service. The parameters attribute can be added as an array to your JSON, allowing you to create custom input fields of the following types: text, password, dropdown, checkbox, or radio. The parameters attribute should be added within the metadata entry under the JSON response of GET /v2/catalog.
Note: userDefinedService has been deprecated. Previously, when provisioning a service, custom service parameters could be added to your JSON by nesting them within the userDefinedService entry. Do not add these parameters under userDefinedService; instead, add them directly under metadata.
Service parameter examples
The following examples illustrate how service parameters can be used to customize a service:
Service parameter definitions

parameters
An array of key-value pairs that are defined by the service when the service instance is created.
"parameters": [
{
"name": "nodes",
"displayname": "Number of Data Nodes",

parameters.name
The key of the parameter.

parameters.displayname
The name of the parameter that is displayed.

parameters.type
The type of input field that is displayed by the Bluemix user interface. The value can be text, password, dropdown, checkbox, or radio.
Note: This value is required.
{
"name": "pwd",
"displayname": "Password",
"type": "password"
},

parameters.options
A JSON structure to specify available values for dropdown, checkbox and radio input type.

parameters.options.displayname
The name of the dropdown, checkbox, or radio button that you want to display in the UI.
parameters.options.value
The value of the parameter to be sent to the service broker when creating the service.
javascript
"options": [
{
"displayname": "HDFS",
"value": "hdfs"
},
{
"displayname": "YARN",
"value": "yarn"
},
]

parameters.value
The default value of the parameter. For checkbox input type, it can be an array of values.
javascript
["hfs","yarn"]

parameters.layout
Specifies the layout of checkbox or radio input types. When unspecified, the default layout is horizontal.

parameters.associations
A JSON structure to describe the interactions with price plans and/or other custom parameters
Note: This value is optional, and all subsequent values appended to parameters.associations. are optional.
"type": "text",
"associations": {
"plan": {
"showFor": ["apsMyserviceParametersDemo-entry","apsMyserviceParametersDemo-enterprise"]
}
}

parameters.associations.plan
A JSON structure to describe the interaction with price plans.
parameters.associations.plan.showFor -
parameters.associations.plan.showFor
An array of price plan IDs. If specified, the parameter will be shown in the UI when the corresponding price plan is selected.
parameters.associations.plan.optionsRefresh
When specified, the optionsUrl will be called when the associated price plan is selected.
parameters.associations.parameters
An array of JSON describing the interaction with other custom parameters.
parameters.associations.parameters.name
The name of the other custom parameter you want to interact with.
Note: Required when parameters.associations.parameters is in use.
parameters.associations.parameters.showFor
An array of selected values associated with the custom parameter you want to interact with. This parameter will show in the UI when one of those values is selected.
parameters.associations.parameters.optionsRefresh
When setting to true, the optionsUrl will be called when the associated parameter value is changed and this parameter is visible in the UI.

parameters.optionsUrl
The API that will be called by the Bluemix Console to get a list of options for dropdown, checkbox, or radio input type. The response will be a JSON that contains two fields:
options
An array of JSON that will return parameters.options; for more details see the entry for parameters.options above.
value
The new default value of the parameter. For more details see the entry for parameters.value above.
The Bluemix Console will pass the ace_config query parameter which contains the current org GUID, space GUID, the current value of all custom parameters and the current price plan ID. The bearer token will be propagated in the header.

parameters.invalidmessage
The message that appears when the content of the text box is invalid.

parameters.description
The description of the parameter that is displayed to help users with the value of the parameter.

parameters.required
A Boolean value that indicates whether the parameter must be entered in the Bluemix user interface.

parameters.pattern
Specifies a regular expression that the value is checked against.

parameters.placeholder
Specifies a short hint that describes the expected value.

parameters.readonly
A Boolean value that indicates whether the value of the parameter is displayed only and cannot be changed by users. The default value is false.

parameters.hidden
A Boolean value that indicates whether the key-value pair is hidden from users. The default value is false.
Adding service parameters to translated metadata
To specify custom service parameters in translated metadata, the i18n version should be placed in the metadata.i18n..parameters attribute.

See an example here:  https://gist.github.com/duglin/ee082145611f86a613d9ee95170f4ed9#file-parameters-L35

it just seems like this would be a lot easier for broker authors to create then to have to deal with json schema.

Thanks for posting this spec @duglin. We've had a good look through it and compared its capabilities to those of JSON schema. See our thoughts below!

There are two issues we want to solve with this spec change:

How can platforms and service brokers validate the configuration parameters provided to create/update an instance/binding?

How can client tools (UIs) provide a better user experience (through form generation, prompting for parameters, etc)?

The first problem is solved by having separate schemas for create and update on a service instance or binding. Standard JSON schema validation libraries can be used by both platforms and service brokers to quickly validate any configuration parameters that have been provided. We expect validation to happen at many levels from UIs through to service brokers to provide fast feedback.
The second problem is likely to require additional information that is not present in raw JSON schemas. This additional information can be used to describe, for example, if a field is "sensitive". This can easily be done by adding additional fields to the schema object (valid within the JSON schema specification) such as "sensitive: true". It is then the responsibility of client tools to understand these additional fields and interpret them appropriately. These additional fields are simply ignored by validation libraries.
During the F2F, we discussed how we could add additional GET endpoints (#159 - Fetching current input parameters values) could be used for pre-populating UI forms. During this discussion, we realised that if some service brokers expose parameters for create that are not allowed in an update, then it will be difficult for them to easily render this into a form. However, intelligent clients could work-around this using all of the information available (both schemas and a set of parameters).
Whilst defining our own standard for expressing requirements does allow us to add additional types (such as "password") and would allow service brokers to expose a single schema (using "forCreate" and "forUpdate" fields), it has some downsides:
- Service brokers will have to write their own validation code
- Platforms will have to write their own validation code
- The working group will have the overhead of defining and maintaining an additional standard
Many of the additional information in the BlueMix standard (such as "displayname") could easily be added to JSON schemas (without affecting validation) for platforms to use as they wish. Whilst the BlueMix standard is good for UI generation, it will require much more work for platforms and service broker authors as they will have to implement their own validation (and cannot use recommended validation libraries).
So, to summarise, we think that JSON schemas allow for platforms and brokers to quickly and easily validate input parameters, and shouldn't cause a problem for intelligent UIs who want to present the information intuitively. The overhead of having two schemas is reasonably small, and we expect many brokers to simply offer the same schema for both create and update. If we want to add another schema to describe non-parameter information in the future (such as "dashboard_url" or "credentials"), we can simply add another schema to do that (but we think this is outside the scope of this change).

@vaikas-google It would be great to learn how Google use JSON schemas for validation and UI generation to see if there is anything we have missed in our analysis above.
All thoughts very welcome. As said on yesterday's call, we'd love to get started on this ASAP so if you could all provide feedback today/tomorrow that would be very much appreciated 😄

@duglin Interesting! There are some cool use cases here.
Taking your gist we wrote a vanilla JSON schema and used an open source UI generator.
See the results here or here.This allows for full validation of the parameters and simple UI rendering. Using JSON schema allows simple client tooling to utilise these existing tools.
However as you rightly mentioned UIs might want to offer more than just what JSON schema supports in order to generate UIs correctly, e.g. sensitive, placeholder.
As Matt stated one option is to add additional keywords to JSON schema that would be ignored during validation but could be used for rendering. See a stripped down version of your gist with additional properties (placeholder, invalidmessage, optionsUrl, sensitive)
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Parameters",
  "type": "object",
  "properties": {
    "nodes": {
      "type": "string",
      "title": "Number of Data Nodes",
      "default": "1",
      "description": "Specify the number of data nodes required for your cluster",
      "placeholder": "please set me",
      "invalidmessage": "oops",
      "optionsUrl": "http://callmemaybe.com",
      "sensitive": true
    }
  }
}

The question then still remains if we want to define a set of keyword extensions as part of the specification?
Some UI generators completely separate JSONSchema and UI metadata. Both react-jsonschema-form and schemaform use an additional UI schema along with JSON schema. This provides the extra UI metadata on how forms should be rendered.
Can we provide a generic set keywords that brokers can write once and multiple clients on multiple platforms can use to generate UIs? Or is the nature of providing this additional UI decoration metadata implementation specific and outside of the scope of the spec and we leave it to the broker and platform authors to agree upon?
@Samze & @ablease

As far as our use of schemas and validation, I'm trying to find some
examples that I could share. They are internal so just want to make sure
that I can share them and what I can share...
On Wed, May 24, 2017 at 2:28 AM, Matt McNeeney<EMAIL_ADDRESS>wrote:

Thanks for posting this spec @duglin https://github.com/duglin. We've
had a good look through it and compared its capabilities to those of JSON
schema. See our thoughts below!
There are two issues we want to solve with this spec change:

How can platforms and service brokers validate the configuration
parameters provided to create/update an instance/binding?
2.
How can client tools (UIs) provide a better user experience (through
form generation, prompting for parameters, etc)?
The first problem is solved by having separate schemas for create and
update on a service instance or binding. Standard JSON schema validation
libraries can be used by both platforms and service brokers to quickly
validate any configuration parameters that have been provided. We expect
validation to happen at many levels from UIs through to service brokers to
provide fast feedback.
The second problem is likely to require additional information that is not
present in raw JSON schemas. This additional information can be used to
describe, for example, if a field is "sensitive". This can easily be done
by adding additional fields to the schema object (valid within the JSON
schema specification) such as "sensitive: true". It is then the
responsibility of client tools to understand these additional fields and
interpret them appropriately. These additional fields are simply ignored by
validation libraries.
During the F2F, we discussed how we could add additional GET endpoints (
#159 https://github.com/openservicebrokerapi/servicebroker/issues/159 -
Fetching current input parameters values) could be used for pre-populating
UI forms. During this discussion, we realised that if some service brokers
expose parameters for create that are not allowed in an update, then it
will be difficult for them to easily render this into a form. However,
intelligent clients could work-around this using all of the information
available (both schemas and a set of parameters).
Whilst defining our own standard for expressing requirements does allow us
to add additional types (such as "password") and would allow service
brokers to expose a single schema (using "forCreate" and "forUpdate"
fields), it has some downsides:

Service brokers will have to write their own validation code
Platforms will have to write their own validation code
The working group will have the overhead of defining and maintaining an
additional standard

Many of the additional information in the BlueMix standard (such as
"displayname") could easily be added to JSON schemas (without affecting
validation) for platforms to use as they wish. Whilst the BlueMix standard
is good for UI generation, it will require much more work for platforms and
service broker authors as they will have to implement their own validation
(and cannot use recommended validation libraries).
So, to summarise, we think that JSON schemas allow for platforms and
brokers to quickly and easily validate input parameters, and shouldn't
cause a problem for intelligent UIs who want to present the information
intuitively. The overhead of having two schemas is reasonably small, and we
expect many brokers to simply offer the same schema for both create and
update. If we want to add another schema to describe non-parameter
information in the future (such as "dashboard_url" or "credentials"), we
can simply add another schema to do that (but we think this is outside the
scope of this change).
@vaikas-google https://github.com/vaikas-google It would be great to
learn how Google use JSON schemas for validation and UI generation to see
if there is anything we have missed in our analysis above.
All thoughts very welcome. As said on yesterday's call, we'd love to get
started on this ASAP so if you could all provide feedback today/tomorrow
that would be very much appreciated 😄
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openservicebrokerapi/servicebroker/issues/59#issuecomment-303669523,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKwedOIBy9FSeMKuNp9zGzDfNBK3KEpiks5r8_g6gaJpZM4JqRle
.

We probably won't be able to discuss this on tonight's call since we're just discussing 2.12 features, so if we could get some feedback on this issue here, that'd be super helpful for us. Thank you!

My team (Bluemix Devops Services) has a controller that manages dev tools vs services but is otherwise very similar to CF and borrows many concepts like provisioning, binding, etc. We've been using JSON Schema for validation of parameters as well as a JSON Schema Form variant for UI for a few years now and by and large it has worked well for us.
Based on our experience we're in the camp that says JSON Schema is a good choice for input parameter validation. I think there is still need for broker-based custom validators but I believe that can be built as an extension of JSON Schema (and written up in a separate issue).
JSON Schema is relatively stable and there is an active community pushing a new round of (standardization)[http://json-schema.org/documentation.html]. For a UI binding there is still a lot of thinking going on in the Schema Form space. It is coalescing with efforts like JSON Schema Form however still are a number of libraries that make subtly different choices.
I would suggest that JSON Schema validation is mature enough and should be considered for addition. I think it would make sense to open a separate issue for the UI binding and track it independently. I'd also be very hesitant to add any new keywords like applies-when and instead be verbose and use JSON references to avoid repeating definitions.
I would also not add any new UI specific keywords to the schema at this point until we're sure this is a good idea. In our experience mixing UI and Schema is a bit of an anti-pattern and have found the structure of the schema and form are the same only simple cases where sections and choice are not involved.
What we ended up doing was adding one keyword form where we put all of our UI info but otherwise using JSON Schema as-is. Here's a simple example:
"parameters": {
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "username": {
            "title": "User name",
            "description": "Type the user name.",
            "type": "string"
        },
        "key": {
            "title": "Access key",
            "description": "Type your access key.",
            "type": "string"
        }
    },
    "required": ["username", "key"],
    "form": ["username", "key"]
}

Apologies if this sounded a bit preachy but I at least wanted to at least provide a few thoughts based on what we've learned. We're keen to work towards standardization in this space and share what we have here.

Thank you @skaegi for your detailed feedback. It's very helpful to hear from a team who have been using our proposed implementation for a few years and that your comments regarding its limitations seem to be consistent with the other feedback we have heard.
I think our plan for the first release of this will be supporting the core JSON schema keywords only, and that any possible extensions will be discussed in the future when we have strong and consistent use cases for them.

We use JSONSchema in several products, and in some of the infrastructure that supports the brokers we're building. All support draft v4, and typically use language-specific validation libraries that support draft v3 and draft v4.
None of the tooling we've found even talks about draft v6, so I would be very hesitant adopting it as the exclusive schema standard until language specific tooling is available.
Most validators support all supported drafts, and if we require that schemas specify their version, we could support any/all supported supported versions.
My proposal would be at the minimum to support v4 initially or both v4 and v6 to allow people to utilize this now.

#165 is merged - closing this issue! Well done all!

Huge :+1: for all the people who put in all the time to make this one a success.


Option to select imagePullSecret only for CP or DP

Describe the feature request
If I set imagePullSecret in the helm chart values here both Istiod (via its ServiceAccount) and all sidecars injected by it (default template) would use that group of secrets.
In some use cases, users need to decide whether to use this secrets in the CP or in the CP + DP. This is currently left to the user customizing the template.
Describe alternatives you've considered

Change injection template to check if the secret should not be injected like index .ObjectMeta.Annotations "sidecar.istio.io/ignore-imagePullSecret". This lets current behavior as default

Affected product area (please put an X in all that apply)
[ ] Ambient
[ ] Docs
[ ] Dual Stack
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

Yeah - this is a side effect of not consistently handling image property templating across all charts.
We support a global pull policy (fine) but don't have a local/scoped equivalent that isn't global for the same field.
Something like https://github.com/istio/istio/pull/51517/files#diff-cfe8054c550e36b443825fcdb9b37654018386b6bd3e38ea2d431f822f249e70R87 would fix this.


fix title length

fix #573

Ok it's not that easy
Very long titles will push vectorscope and spectrum out of window

You also have to set min-width: 0 on the div that wraps the h1 song title.


[CAMEL-20081]: Utilize the multicast processor in the dynamic router,…

Utilize the multicast processor in the dynamic router, rather than rolling an entirely new implementation.
Target

[ X ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ X ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ X ] I checked that each commit in the pull request has a meaningful subject line and body.

[ X ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

@davsclaus I build this and it isn't showing any uncommitted changes afterward.  I am running ./mvnw -Dquickly install.  Is there something that I am missing?

You can revert your change in camel-spring.xsd file so its not part of the commit

Thank you, @davsclaus and @orpiske for taking the time to look over the code and offer comments.  If I see any more comments, I'll be glad to make the changes for round 2.

Thank you, @davsclaus and @orpiske for taking the time to look over the code and offer comments. If I see any more comments, I'll be glad to make the changes for round 2.

Thanks for the changes @Steve973! I don't see anything big right now - other than what Claus has raised. I am going to test this manually on my CI and we can see how it goes.

Last build had 3 failures on my CI (which I consider pretty good).

MapJmsMessageTest.testTextMessage org.apache.camel.component.jms
JettyJmsShutdownInProgressTest.testShutdownInProgress org.apache.camel.itest.jetty
RocketMQRouteTest.testSimpleRoute org.apache.camel.component.rocketmq (knowk flaky - unlikely to be related to this)

I'll ran again with the recent changes.

This is a too big PR to do good review and there is too much nit picking already.
Can you add a section to the 4.2 upgrade guide doc about changes in this component so users should be mindful when they upgrade.

And can you include the files that was updated but not included in the PR (see the error)
atalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/dynamic-router.json

@Steve973 a heads up that we want to be ready tomorrow for the 4.2 release. Not sure if you have time to cleanup the last bits, and include the uncommited code, so the CI can do a build and test.

@Steve973 a heads up that we want to be ready tomorrow for the 4.2 release. Not sure if you have time to cleanup the last bits, and include the uncommited code, so the CI can do a build and test.

I think I finished up the last bits.  Do you want to go over this more thoroughly when you have some time?  Some of these things have been there since the initial introduction of the component, but I am quite willing to make changes as needed.

Last build had 3 failures on my CI (which I consider pretty good).

MapJmsMessageTest.testTextMessage org.apache.camel.component.jms
JettyJmsShutdownInProgressTest.testShutdownInProgress org.apache.camel.itest.jetty
RocketMQRouteTest.testSimpleRoute org.apache.camel.component.rocketmq (knowk flaky - unlikely to be related to this)

I'll ran again with the recent changes.

When you have a moment, @orpiske could you please let me know how my changes might have affected these tests?  I did not make any changes in the aforementioned modules.

Last build had 3 failures on my CI (which I consider pretty good).

MapJmsMessageTest.testTextMessage org.apache.camel.component.jms
JettyJmsShutdownInProgressTest.testShutdownInProgress org.apache.camel.itest.jetty
RocketMQRouteTest.testSimpleRoute org.apache.camel.component.rocketmq (knowk flaky - unlikely to be related to this)

I'll ran again with the recent changes.

When you have a moment, @orpiske could you please let me know how my changes might have affected these tests? I did not make any changes in the aforementioned modules.

Claus already merged, so it's not worth testing on my own CI. That said, you can watch the test results on the ASF CI.

this component is isolated and not used anywhere else at all

this component is isolated and not used anywhere else at all

Wouldn't this be tested by core? I was under the impression we had some tests for it on core.

Anyway, if it's isolated we should be fine!

this component is isolated and not used anywhere else at all

Wouldn't this be tested by core? I was under the impression we had some tests for it on core.

No its a regular camel component

Awesome. Thanks for the clarification @davsclaus!


Convert @angular/material imports to make them tree shakable

Please look at the material2 issue and specifically the comment from the link below:
https://github.com/angular/material2/issues/4137#issuecomment-336590835

This is a "must have" withing angular/v8+, I think. was partially done in #73 . will take a look at and merge #57 anyways.


ADD:add a go to top btn

Title :
Add a go to top button
Issue No. : #29
Code Stack : JavaScript,Html,Css
Close #29
Video/Screenshots (mandatory)

# Checklist:

[X] I have mentioned the issue number in my Pull Request.
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have gone through the  contributing.md file before contributing
Additional context (Mandatory )

Are you contributing under any Open-source programme?
JWOC

@rahat2134 Please go through the contributing.md file
Please don't make any type of PR until & unless you get assigned to any type of issue


Update catapis.md

Minor edits to common parameters, added a response example to _cat/allocation, put endpoints in alphabetical order
Issue #, if available:
Put endpoints in alphabetical order, made minor edits to common parameters table, changed title case, added response example to _cat/allocation
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I actually planned to add example responses for most of the endpoints. I
figured I would do them after the reordering so it didn't make us too
crazy.
It will help the user to know which headers are available for each
endpoint, because then they will have an idea of the content returned by
the system.
On Fri, Sep 6, 2019, 12:18 PM Andrew Etter<EMAIL_ADDRESS>wrote:

@aetter requested changes on this pull request.
Thanks a bunch. Some minor nitpicks: I think we should leave
_cat/allocation as-is and move the sample response to the top of the page.
In docs/elasticsearch/catapis.md
https://github.com/opendistro/for-elasticsearch-docs/pull/123#discussion_r321874551
:

GET _cat/allocation?v

+Example default response:

+```json

This is just plain text, not JSON.
------------------------------

In docs/elasticsearch/catapis.md
<https://github.com/opendistro/for-elasticsearch-docs/pull/123#discussion_r321874600>
:

>

## Allocation

Shows the allocation of disk space for indices and the number of shards on each node.

-

+Default request:

Remove.
------------------------------

In docs/elasticsearch/catapis.md
<https://github.com/opendistro/for-elasticsearch-docs/pull/123#discussion_r321875091>
:

>  ```json

GET _cat/allocation?v

+Example default response:
I like the idea of including at least one sample response on this page,
but I think I'd rather see it right at the top so that people get an idea
of what these APIs return. What do you think about moving this right below
the second paragraph with a little, "For example, a request to
_cat/allocation might return:"?
In docs/elasticsearch/catapis.md
https://github.com/opendistro/for-elasticsearch-docs/pull/123#discussion_r321875283
:

@@ -56,53 +56,68 @@ Typically, for any operation you can find out what headers are available using t

{:toc}

+## Aliases
Definitely appreciate the alphabetization. I wasn't sure what you'd done,
but it makes the page easier to scan.
In docs/elasticsearch/catapis.md
https://github.com/opendistro/for-elasticsearch-docs/pull/123#discussion_r321875388
:

@@ -35,13 +35,13 @@ To see what each column represents, use the ?v parameter:

GET _cat/<operation_name>?v

-To see all the available headers, you can use the `?help` parameter:

+To see all the available headers, use the `?help` parameter:

👍

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<https://github.com/opendistro/for-elasticsearch-docs/pull/123?email_source=notifications&email_token=AMNF6KKNPI5A7MVBLBC5PFTQIKUIPA5CNFSM4ITM4QLKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCD6YXIA#pullrequestreview-285051808>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMNF6KLIEQANKQHWNJLCHHDQIKUIPANCNFSM4ITM4QLA>
.

I actually think a response for each call is overkill. If they were something you might interact with programmatically, absolutely a sample response for each is important, but here, people just need to read them.

Including responses for each endpoint is standard API documentation. Format is basically:

API title
Description
Define params
Example endpoint call
Example response
Specific call/response if useful (like if for optional path parameters)

Each endpoint returns different possible headers. Without this content, the API documentation is painfully sparse. The user has only the vaguest idea of what will be returned, and they will not know what headers will be returned without either running the call themselves or running help on the call.
Including the responses will keep this documentation more consistent with other good API documentation and make it more user-friendly and helpful.
I do not agree with having the cat allocation response at the top of the page. It would cause more confusion than it would prevent.

Tonight I can either remove the example response or I can add more of them so that Allocation does not seem so weird.
Which would you prefer?

The reason that including a response is "standard" is because that response is typically JSON, and you need to know what's included in order to work with it programmatically. In this case, a human being is going to read the response, so including 19 sample responses just makes the page harder to scan for minimal additional value. Feel free to remove the response from _cat/allocation.

I updated my doc and pushed. Does that update this Pull Request, or do I need to do another step?

No other steps. Thanks!


podman build --cap-add/--cap-drop has no effect and value is not validated, contrary to buildah behavior

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Passing --cap-add or --cap-drop to podman build has no effect and its value is not validated. This is contrary to buildah, where it is validated and has an effect (see "additional information" below), so this does not appear to be a buildah bug.
For simplicity, only --cap-add is referred to here, but all applies also to --cap-drop.
Steps to reproduce the issue:

Create a Dockerfile like:
FROM fedora:31
RUN capsh --print && exit 1

Run podman build --cap-add=cap_sys_admin . (tested also: --cap-add=sys_admin and all caps variants of both).

Describe the results you received:
Note omission of cap_sys_admin.

Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap

Describe the results you expected:
Note presence of cap_sys_admin.

Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_sys_admin,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_sys_admin,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_sys_admin,cap_mknod,cap_audit_write,cap_setfcap

Additional information you deem important (e.g. issue happens only occasionally):

podman build seems to do no validation of the value of --cap-add, e.g. --cap-add=nonsense continues without error.
podman run performs validation of its --cap-add and, unlike buildah, permits omitting the cap_ prefix.
buildah does not have this problem, so this does not appear, on the surface, to be a buildah bug. E.g. running buildah run --cap-add=cap_sys_admin $(buildah from fedora:31) capsh --print includes cap_sys_admin. Further, passing --cap-add=cap_nonsense to buildah gives an error.
Since the buildah version bundled with my podman is a bit older, I also tested against buildah 1.14.0-dev (the only other version currently in my Fedora repos) and --cap-add is functioning in that version.

Output of buildah version:
Version:         1.14.6
Go Version:      go1.14
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.3.1
Git Commit:
Built:           Wed Dec 31 19:00:00 1969
OS/Arch:         linux/amd64

Output of podman version:
Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.14
OS/Arch:            linux/amd64

Output of podman info --debug:
debug:
  compiler: gc
  git commit: ""
  go version: go1.14
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  Distribution:
    distribution: fedora
    version: "32"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree:<PHONE_NUMBER>2
  MemTotal:<PHONE_NUMBER>4
  OCIRuntime:
    name: crun
    package: crun-0.13-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree:<PHONE_NUMBER>
  SwapTotal:<PHONE_NUMBER>
  arch: amd64
  cpus: 48
  eventlogger: journald
  hostname: <redacted>
  kernel: 5.6.2-300.fc32.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.3-6.0.dev.gita8414d1.fc32.x86_64
    Version: |-
      slirp4netns version 0.4.3+dev
      commit: a8414d1d1629f6f7a93b60b55e183a93d10d9a1c
  uptime: 5h 25m 38.29s (Approximately 0.21 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /home/<redacted>/.config/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.8-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 0.7.8
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  GraphRoot: /home/<redacted>/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 49
  RunRoot: /run/user/1000/containers
  VolumePath: /home/<redacted>/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):
podman-1.8.2-2.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical.

@amarshall I ran a quick test against the latest from upstream and it's working for me.  Can you easily clone from GitHub and then run it?  I think this may have been recently fixed.

I installed Podman from the COPR referenced from install docs which appears to be from latest git (1593d4c0) and can confirm --cap-add and --cap-drop work as expected.

However, unrelated to this issue, running that latest podman as root even for a simple run, e.g. sudo podman run -it --rm fedora fails with:
ERRO[0000] oci runtime "runc" does not support CGroups V2: use system migrate to mitigate
Error: this version of runc doesn't work on cgroups v2: OCI runtime error

Not sure why that is or if it warrants another issue (or if there is one). Running under my own account works.

Switch to crun for an OCI runtime.  runc does not yet support cgroup V2.
What OS are you running on?

@rhatdan I suspect it’s a packaging problem with the Podman from the aforementioned COPR repository, as it removes several dependencies (conmon, libvarlink-util, podman-plugins) compared with the Fedora packaged podman. I’m on Fedora 32.

Then please use the packages from the distro.

That’s what I do normally, but for the purposes of testing if this bug had been resolved in an unreleased version I used the development COPR repository listed in the Podman installation docs.
Anyway, testing on Podman 1.9.1 I can confirm this is resolved.


concrete_inheritance not working with multiple schema.xml files

concrete_inheritance apparently does not work across multiple schema files. Please see the schema snippets at the end of the post.
There are no errors when I rebuild the model, and all the classes have been generated as well, but if I save a new Agent object at runtime, a "Call to undefined method: setDescendantClass" exception is thrown. Propel should have generated this method for the User class but didn't.
I tried moving the agents table definition from schema2.xml into schema1.xml file and it works fine, however the idea is to have schema1.xml in a central framework, with applications extending the User class via their own schema file.
SCHEMA1.XML:

	<vendor type="mysql">
		<parameter name="Charset" value="utf8"/>
		<parameter name="Collate" value="utf8_unicode_ci"/>
	</vendor>
	<column name="id" type="integer" required="true" primaryKey="true" autoIncrement="true" />
	<column name="username" type="varchar" size="50" required="true" />

	...

	<behavior name="timestampable" />

SCHEMA2.XML:
<external-schema filename="schema1.xml" />

	<vendor type="mysql">
		<parameter name="Charset" value="utf8"/>
		<parameter name="Collate" value="utf8_unicode_ci"/>
	</vendor>

	<behavior name="concrete_inheritance">
		<parameter name="extends" value="core_users" />
	</behavior>

	...

	<behavior name="timestampable" />

Sure it is. You need to import the respective schema.
For example:
<database name="default" package="lib.model" external-schema="path/to/parent/schema.yml" defaultIdMethod="native">

</database>

I already have the external-schema import statement in my schema2.xml file (updated my original post) but that doesn't help.
Your solution of the external-schema definition inside the database tag gives me a schema validation error:
[PHP Error] DOMDocument::schemaValidate(): Element 'database', attribute 'external-schema': The attribute 'external-schema' is not allowed.

Let me check that. The example is from a 1.5.3 project, maybe it was changed until 1.6.

Yep, it's an element now :-)

I found the real reason now:
In my application, schema1.xml is actually called core-schema.xml and schema2.xml is called agent-schema.xml.
Since propel will build agent-schema.xml first (it's alphabetically), it generates an incomplete Agent class. So renaming the agent-schema.xml to something like d-schema.xml helps. It would be awesome if propel could figure out these dependencies on its own.


Update protobuf version

Update protobuf version to resolve security vulnerability.

https://relationalai.atlassian.net/browse/RAI-32950


Upgrade to Polkadot v0.9.40

https://github.com/paritytech/substrate/compare/polkadot-v0.9.39...polkadot-v0.9.40
https://github.com/paritytech/cumulus/compare/polkadot-v0.9.39...polkadot-v0.9.40
https://github.com/Phala-Network/phala-blockchain/pull/1221
In short: 0.9.40 is a minor upgrade, no visible change in this release

wPHA bug fixing
introduce inDEX pallet
Fix democracy incorrect configuration that enables the functionality of Child Bounties
Khala: Will do missing migrations, just like Phala v0.1.22
tokenomics parameters migration (budget limit)

We only lack disable Phala compute functionalities, but it's OK to send a separate PR, so I'm going to merge this


fix 1336 with corrected start and stop annotations

Should generate a corrected openapi.yaml. We should also look at testing the generated openapi.yaml with all endpoints?
closes: #1336

We should have a look at how we format date/time in Horreum. In TestService.listLabelValues we use strings as the type, but we accept either iso-like or epoch. I think the intention was to default to that, but it has not been communicated well.

TestService.listLabelValues supports different time formats for the before and after query parameters to make it easier for users to request data from Horreum. The issue this addresses is how we describe the data that Horreum provides.
When Horreum encodes an Instant into json it is currently encoded as a Long but the annotations told openapi to create an openapi.yaml that claims the values will be an ISO style string. The PR changes the annotations to match the data type Horreum is already using.
We can discuss if Long or an ISO String is more appropriate for integration with other tools but for now this PR is a bug fix for incorrect annotations.

This PR is missing the changes to openapi.yaml. Please can we ensure that if we change the openapi annotations, that the corresponding generated openapi.yaml is also checked in with the PR otherwise what we publish on the website is out of sync with the actual API


Hue-App 5.x introduced new features: create ioBroker-objects for security/alarm features & location settings

With Hue-App 5.x new security/alarm-features were introduced by Signify, these are:

Alarm deactivated / Active at home / Active in absence
Event-Timeline
Light-Alarm
Location-Settings (existed already in 4.x / which device is at home / not at home)

Can these settings/informations be read from the Hue-Bridge and displayed as objects in ioBroker for further processing?
Thank you!

Development of this Adapter seems to have stalled. Is there anything I can do to get this going again? Donation for the project maybe?


how to add selectors to images?

Let's say I have an image like so:
<img alt="Florida" src="images/works/im1s.jpg"
data-image="images/works/im1b.jpg"
data-description="Our trip to Florida" class="state florida" />

The plugin strips away my class tags, so I cannot apply conditional styling. How might I get around this?

Same problem here.. I'm adding it  again on the click events but I'ts not a solution


Syntactic sugar for vjp

As discussed on Slack, it is sometimes annoying to use reverse mode because you need autodiff_thunk whenever

the function output is a number and the cotangent is different from 1.0
the function output is anything but a number

A solution suggested by @wsmoses would be syntactic sugar for vjp (and maybe jvp but that is less necessary cause autodiff gets the job done every time). I'll try to hash out a prototype

Here's an MWE:
using Enzyme
using Enzyme.EnzymeCore: ReverseModeSplit

function vjp(
    rmode::ReverseModeSplit{ReturnPrimal},
    dresult,
    f::FA,
    ::Type{RA},
    args::Vararg{Annotation,N},
) where {ReturnPrimal,FA<:Annotation,RA<:Annotation,N}
    forward, reverse = autodiff_thunk(rmode, FA, RA, typeof.(args)...)
    tape, result, shadow_result = forward(f, args...)
    if RA <: Active
        dinputs = only(reverse(f, args..., dresult, tape))
    else
        shadow_result .+= dresult  # TODO: generalize beyond arrays
        dinputs = only(reverse(f, args..., tape))
    end
    if ReturnPrimal
        return (dinputs, result)
    else
        return (dinputs,)
    end
end

Tests
f(x, y) = x^2 + 10y  # scalar output
g(x, y) = [f(x, y)]  # vector output
x, y = 3.0, 4.0
dz = -5.0

z0 = f(x, y)
dx0 = dz * 2x
dy0 = dz * 10

julia> ((dx, dy), z) = vjp(ReverseSplitWithPrimal, dz, Const(f), Active, Active(x), Active(y))
((-30.0, -50.0), 49.0)

julia> z == z0 && dx == dx0 && dy == dy0
true

julia> ((dx, dy), vz) = vjp(
           ReverseSplitWithPrimal, [dz], Const(g), Duplicated, Active(x), Active(y)
       )
((-30.0, -50.0), [49.0])

julia> vz == [z0] && dx == dx0 && dy == dy0
true


Quote values with special or reserved characters

Quotes are required when the string contains special or reserved characters (:, {, }, [, ], ,, &, *, #, ?, |, -, <, >, =, !, %, @, and \).
See:
http://symfony.com/doc/current/components/yaml/yaml_format.html#strings

Before anything, note that I unchanged https://github.com/flarum/english/blob/master/locale/core.yml#L272
I don't know if you want to standardize double quotes (if so, I will replace single quotes with double quotes and add a backdash before each double quotation mark in the value) or let single quotes when double quotation marks are used in the value (to avoid the use of a backdash).
IMO we should standardize, but I'm waiting your choice for it.

I like the use of single quotes when the string contains double quotes. That's what we'd do it PHP too. IMO standardising really offers no benefit here.

Okay, that's fine, it will be unchanged. :)