[RFC] OpenSearch Catalog

Hi everyone,
We are seeking feedback for the proposal of the OpenSearch Catalog and have published the doc here in the repository.
It is also copied below, feel free to leave comments, suggestions, ideas, etc.

OpenSearch Catalog RFC
Introduction
The OpenSearch Catalog is designed to make it easier for developers and community to contribute, search and install artifacts like plugins, visualization dashboards, ingestion to visualization content packs (data pipeline configurations, normalization, ingestion, dashboards). The OpenSearch Catalog does this by providing a searchable catalog of community contributed OpenSearch artifacts and projects that users can browse, rate and download from the project website, OpenSearch CLI, and OpenSearch Dashboards. Users can install and update artifacts with a single click from OpenSearch Dashboards or by running an install/update command from OpenSearch CLI. When the artifacts have updates, they are highlighted in OpenSearch Dashboards to help users discover the latest versions of artifacts.
Problem
OpenSearch and OpenSearch Dashboards can be overwhelming to get started with as the depth of features and functionality requires expertise that takes time to understand and master. It is also difficult to discover additional functionality that is contained in plugins that are spread out through the ecosystem and find good examples to be reused for different parts of OpenSearch and OpenSearch Dashboards. The growing number of features in various plugins like Alerting, Index Management, Anomaly Detection, Observability among others makes OpenSearch a powerful search and log analytics platform. While these features add a lot of value to users and admins, they require a level of understanding of OpenSearch fundamental components like Data Ingestion pipelines, Data Normalization, Mappers,Indices, Shards, , Dashboards.
Solution
We are proposing to build an OpenSearch Catalog for OpenSearch and OpenSearch Dashboards to help users with sharing their artifacts like plugins, visualization dashboards, content packs, alerting monitors, index management policies, anomaly detection detectors, mappers among others for the benefit for the community along with helping users discover artifacts that can be installed with 1-click . The OpenSearch Catalog will provide contributors a place to publish their artifacts to see wide adoption from all users of OpenSearch and OpenSearch Dashboards and provide end users with a place to search, review, rate, and comment on artifacts.
OpenSearch Catalog

Developers should be able to publish their artifacts to OpenSearch catalog.
Multiple versions of a single artifact (plugin/dashboard) should be available to download in the event a user is using an older OpenSearch distribution.
Users should be able to install artifacts from OpenSearch Dashboards, CLI, and OpenSearch website.
Users should be able to rate and browse artifacts from OpenSearch Dashboards and OpenSearch website.
Artifact updates should be highlighted to cluster administrators via OpenSearch Dashboards and the CLI.

The goal is to reduce the friction from spinning up a new cluster to immediately deriving business value from being able to easily install multiple artifacts that have been created by the community.
OpenSearch Catalog components
The OpenSearch catalog will consist of the following components to support authentication, install, validation, search, and curation (ratings, reviews).
OpenSearch catalog website: The website will be the central place for developers to contribute and upload artifacts. It will support different functions of login, upload, browse, search, install, rate, review, comment, validate and curate the content.
OpenSearch Dashboards plugin: The OpenSearch Dashboards will have a UI plugin for users and admins of OpenSearch clusters to search and install a plugin or define an alerting monitor or index management policy.
Catalog plugin: The catalog plugin will be a back end plugin to maintain the metadata around installed artifacts (create/install, update, remove).
CLI: The OpenSearch CLI lets users to package and upload artifacts into the catalog.
Validation Framework: The validation framework will ensure the uploaded artifact is tested and compatible with different versions of OpenSearch.
Curation: The curation will support the workflow of content approval before being available in the catalog.
Artifacts
Artifacts are components of OpenSearch and OpenSearch Dashboards that allow users to immediately derive value without having to the understand complexity of installing and operating specific features on their clusters They also provide extra functionality (plugins) for the user to enhance the base OpenSearch and OpenSearch Dashboard distributions. The OpenSearch Catalog provides a searchable catalog of community contributed OpenSearch artifacts that users can browse, rate, and download from the project website, the OpenSearch CLI, and OpenSearch Dashboards.
Catalog allow users to:

Install new plugins that provide additional functionality
Add pre-built Dashboards with Visualizations for common data formats
Implement pre-defined complex queries and scripts with multiple log sources for common and advanced workloads
Add common index mappings and templates without having to build your own
Add common resources (e.g., monitors, policies) for installed plugins such as Alerting monitors and Index State Management policies.
And more!

For example authors can share their common Index State Management policies that can be useful for all users of OpenSearch. Such as a policy that sets replicas to 5 before rolling over the index once it reaches 7 days of age, transitioning to a warm state where it reduces replicas, sets the index to read-only, force merges the segments to improve search, and then deleting the index after 30 days of age. For a new user this would originally take a while to learn and setup on their own, but now they can quickly view examples and install them directly on their cluster to get started with.
{
  "policy": {
    "description": "ingesting logs",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          { "replica_count": { "number_of_replicas": 5 } },
          {
            "retry": { "count": 3, "backoff": "exponential", "delay": "1h" },
            "rollover": { "min_index_age": "7d" }
          }
        ],
        "transitions": [{ "state_name": "warm" }]
      },
      {
        "name": "warm",
        "actions": [
          { "read_only": {} },
          { "replica_count": { "number_of_replicas": 1 } },
          { "force_merge": { "max_num_segments": 1 } }
        ],
        "transitions": [{
            "state_name": "delete",
            "conditions": { "min_index_age": "30d" }
		}]
      },
      {
        "name": "delete",
        "actions": [{
	        "notification": {
		        "destination": { "slack": { "url": "<url>" } },
		        "message_template": { "source": "Deleting index {{ctx.index}}" }
	        }
        }, {
            "retry": { "count": 3, "backoff": "exponential", "delay": "1h" },
            "delete": {}
		}]
      }
    ]
  }
}

The artifacts require a Manifest.json file which is bundled together with the artifact data and media in a zip that is then published to the OpenSearch Catalog where it will go through the publishing lifecycle before being available for public use.
Manifest.json
The Manifest.json file is required in all artifacts zips. It contains all information required for an artifact to be published, validated, installed, and meta information to show on the catalog.

Property
Type
Mandatory
Description
Example

author
Object
No
The artifact's author
"author": { name": "Jane Smith" }

dependencies
Object
Yes
The dependencies of the artifact
"dependencies": { "opensearchDependency": "^1.2.0" }

description
String
Yes
The artifact's description
This is a description of my artifact

icons
Object
No
An object with keys to provided image names that will be used when showing an icon of the artifact
"icons": { "48": "icon.png", "96"<EMAIL_ADDRESS>}

id
String
Yes
The unique id/name of the artifact that will be used in the catalog URI
my-awesome-artifact

keywords
String[]
Yes
List of keywords used for searching on the catalog to help find the artifact
["awesome", "plugin", "business analytics"]

license
String
Yes
The license of the artifact
Apache 2.0

links
Object[]
No
List of links to be displayed on the artifacts page
[{ "name": "Project website", "url": "..." }, ...]

manifest_version
Number
Yes
The version of the manifest.json used
1

media
Object[]
No
Images and GIFs that are displayed on the catalog for the artifact
[{ "name" "Homepage", "description": "...", "path": "img/homepage.png" }, ...]

name
String
Yes
The artifacts name
My awesome artifact

path
String
Yes
The path to the artifact file
my-awesome-plugin.zip

type
String
Yes
The artifact's type
Index Mapping

variables
Object
No
Variable substitutions for the artifact which will look for <%variable%> in the supported artifact to replace with the users input.
"variables": { "index": { "type": "String" } }

version
String
Yes
The version of the artifact
1.2.0

author

Property
Type
Mandatory
Description
Example

email
String
No
Author's email
<EMAIL_ADDRESS>

name
String
No
Author's name
John Smith

url
String
No
Author's website
https://www.amazon.com

dependencies

Property
Type
Mandatory
Description
Example

opensearchDependency
String
Yes
The versions of OpenSearch this artifact supports and depends on
^1.2.0

artifact
Object[]
No
Other artifacts this artifact depends on
[{ "id": "artifact-id", "version": "1.3.0" }, ...]

artifacts

Property
Type
Mandatory
Description
Example

id
String
Yes
The id of the artifact
awesome-plugin

version
String
Yes
The version of the artifact
5.2.3

links

Property
Type
Mandatory
Description
Example

name
String
Yes
The name of the link
Project website

url
String
Yes
The url for the link
https://www.amazon.com

media

Property
Type
Mandatory
Description
Example

name
String
Yes
The name of the image
Homepage

description
String
Yes
The description of the image used for alt text for accessibility
Homepage view of awesome-plugin showing graph of business analytics

path
String
Yes
The path to the image
img/homepage.png

Artifact Types
This is not an exhaustive list of artifact types we plan to support, but some we see immediate benefit for the community.

Extension Type
Notes

Plugins
A plugin is a developed piece of software that is enhancing OpenSearch or OpenSearch Dashboards with specific functionality. It lives in a code repository and creates an artifact that can be shared.

Dashboards
A dashboard is a collection of Visualizations.

Visualizations
Single Visualizations of different types.

Queries
Queries can be time consuming to put together and are infact reusable when the data is formatted the same which is common.

Scripts (e.g. painless)
In spite of the name, painless scripts are incredibly painful for users to write. Having a place where users could share common patterns for scripts would benefit everyone.

Index Templates-Component Templates
The index template APIs follow a composable component template pattern where you create can different sub templates (component templates) and compose them together into a larger index template. These could be shared by providing common settings and mappings to be used for indices.

Index Mappings
When the data being ingested is of the same type/format then a common index mapping could be used for most users.

Index State Management Policies
ISM Policies are shareable for different use cases

Anomaly Detection Detectors
Anomaly detectors can be reused across clusters if the underlying data is in the same format

Alerting Monitors-Triggers
If users have the same format of data ingested and they want to alarm on it then a lot of them will want the same type of monitors and alarms/triggers.

Custom
We might not capture every type of artifact users want to be able to share. Instead of waiting for us to add support the custom type allows people to publish a custom type artifact that can still be shared with users

Tooling
We plan to provide tooling to support authors with building and publishing their artifacts and for users to download and manage their installed artifacts. To start with we will enhance the opensearch-cli with new APIs and commands that help bundle artifacts into the required format to be published. It will also provide functionality for helping the author get the artifact through the publishing lifecycle until it is officially published to the public.
// Create a new artifact locally called my-awesome-query
opensearch-cli artifacts create my-awesome-query --type query --profile prod

We plan to provide built-in support into OpenSearch Dashboards to easily export your built artifact types so they can be published such as visualizations and dashboards.
Lastly we will add support for a new Gradle plugin that plugin developers can add to their plugin for easily building the artifacts zip and publishing directly using Gradle to the catalog.
// Command will bundle the plugin, Manifest.json, and any required media files
// into an artifact zip to then be published to the catalog
./gradlew assemble artifact
...
./gradlew publishToOpenSearchCatalog

Publishing
For publishing we require users to create an account using one of the common social logins such as GitHub, Google, or Facebook. Once an account is created there will be an author portal on the OpenSearch Catalog to manage artifacts that are in the publishing process or already published.
The publishing process helps ensure the quality of artifacts on the catalog stay high and helps the author follow best practices. Authors can view and work through the publishing process directly from the OpenSearch Catalog or use the CLI.
When an artifact is first published it will be in the DRAFT stage where it will go through validations, security checks, and allow the author to make edits and review.
The validations of an artifact will verify that the artifact actually works for the supported versions listed by the author. This ensures that you don’t accidentally install an artifact that is broken for your version of OpenSearch.
We will also do security scans of all published artifacts periodically to help protect clusters from security exploits. This will happen during the publishing process and periodically after.
Once the author feels it is ready it will be submitted for review and enter the REVIEW stage where it will be officially reviewed.
If there are any comments from the review board that they feel are required then the author will be notified and the artifact will enter into a REVISION stage.
Once the comments are addressed and the review board signs off the artifact will enter the final PUBLISHED stage where it will become available to the public on the OpenSearch Catalog.
Discovering
The OpenSearch Catalog will be hosted as a web application that can be independently browsed through to discover new artifacts. Users can directly download artifacts from the OpenSearch Catalog and use the CLI tooling to install.
We also plan to provide an OpenSearch Dashboards plugin that will allow users to browse the OpenSearch Catalog directly from OpenSearch Dashboards. The eventual goal is to have this plugin provide a simplified installation process by introducing one-click installs directly from the UI. Users will be able to browse artifacts and click install to have it immediately installed in their cluster.
We plan to not require users to have an account to browse and download artifacts from the catalog. But, if users want to be able to review, rate, and comment on artifacts they must be logged in.
Installing and uninstalling
We are going to provide two mechanisms for users to install artifacts on to their OpenSearch clusters.
Outside the cluster environment and inside the cluster environment.
We will have a opensearch-cli tool which can interact with the OpenSearch catalog and the cluster on behalf of the user to list the artifacts available and install artifact on behalf of the user. This tool can be used as an ad hoc installer of artifacts.
// This will go and fetch the my-awesome-artifact from the catalog
// and start the installation process for it
opensearch-cli artifacts install my-awesome-artifact --profile prod`

Alternatively we will also provide an OpenSearch dashboard plugin that is part of the OpenSearch cluster which can provide users with ability to interact with the OpenSearch catalog to list down the artifacts. Along with this we will have a OpenSearch plugin that can install any applicable extension that the user desires on to the cluster . These two combined will provide a more streamlined user experience where users can see the list of artifacts installed on the cluster at a given moment, browse the catalog artifacts and install plugins from the artifact catalog by clicking a button.
For initial phase users won’t be able to install plugins through either of the channels, but just download the plugin from catalog.
Distributions
One challenge we want to tackle is providing support for companies and users that have their clusters deployed in such a way where they cannot directly connect to the OpenSearch Catalog over the internet. To do this we plan to provide distributions of the OpenSearch Catalog such that end users can directly host their own catalog that is essentially a mirror of the official OpenSearch Catalog. We plan to provide daily dumps of all artifacts split by type so they can be downloaded and hosted internally for their mirrored catalog to pull from.
The mirrored catalogs will have trimmed down functionally as we do not want to bifurcate ratings, reviews, comments, and accounts. It will provide the ability to browse, download, and install artifacts.
Providing Feedback
We seek comments and feedback on the proposed design for OpenSearch Catalog. Some specific questions we're seeking feedback on include:

What other types of artifacts would you like to see in the catalog?
Opt-in vs Opt-out? Would you prefer the OpenSearch Dashboards Catalog plugin to be turned on by default allowing users to browse artifacts from the online catalog or would you prefer to have it disabled by default and require an admin of the cluster to enable it before it can be used?
Would you expect OpenSearch, and in the future, elected community leaders to curate and approve artifacts before they are allowed to show up in the catalog?
Should we require all artifact's sources to be public so builds can be traced or built by the OpenSearch catalog itself? Such as requiring a GitHub repository that can be used to build the artifact.

@dbbaughe  I love this idea.

What other types of artifacts would you like to see in the catalog?

I don't have feedback on the additional artifacts.
I would see Plugins and Dashboards being the most useful.

Opt-in vs Opt-out? Would you prefer the OpenSearch Dashboards Catalog plugin to be turned on by default allowing users to browse artifacts from the online catalog or would you prefer to have it disabled by default and require an admin of the cluster to enable it before it can be used?

I would recommend that the most frequently used are browseable by default.  Part of being able to explore features like a plugin catalog is giving some examples. This helps disambiguate what is meant by a plugin catalog and encourages learning by trying.

Would you expect OpenSearch and in the future, elected community leaders to curate and approve artifacts before they are allowed to show up in the catalog?

Approved artifacts could be a category of artifacts. That would need a process and some tests to be meaningful.
For non-approved artifacts, I would recommend they are not limited but can be flagged for abuse.

Should we require all artifact sources to be public so builds can be traced or built by the OpenSearch catalog itself? Such as requiring a GitHub repository that can be used to build the artifact.

I like this idea. I would only suggest that it be open to other repositories and not limited to GitHub.


ARROW-12480: [Java][Dataset] FileSystemDataset: Support reading from a directory

@emkornfield - Are you able to have a review for this patch? Just a fundamental functionality to make Java dataset API able to read from files in folder.

Ping @kou @liyafan82 @kszucs - If we have other active Java code committers?

Sorry, not sure when I will be able to take a look (possibility of this weekend) but I'm generally behind on reviews.

I'm not familiar with Java. @kisizk Can you review this?

I'm not familiar with Java. @kiszk Can you review this?

Sure, I will do a review.

Looks good to me with a few comments.
I would like to get another review, for example, by @emkornfield or @liyafan82

Thanks everyone, merging!


Feature Request: par_enumerate()

Will there, by any chance, be a par_enumerate() in the future?

Isn't IndexedParallelIterator::enumerate what you need? Or is your initial parallel iterator not indexed?

Of course, you are right. I close this issue now.

not needed

not needed


can two trees share the same selection?

in other words, is it possible to compose selections?
const foo = new Baobab({
  a: 1,
  b: 2
})
const bar = new Baobab({
  a: foo.select('a'),
  c: 3
}) // TypeError: Cannot assign to read only property '_data' of [object Object]

// ideally...
bar.set('a', 4)
bar.get('a') // 4
foo.get('a') // 4

otherwise, i have to deal with some boilerplate:
const foo = new Baobab({
  a: 1,
  b: 2
})
const bar = new Baobab({
  a: foo.get('a'),
  c: 3
})
foo.select('a').on('update', ({ currentData }) => bar.set('a', currentData))
bar.select('a').on('update', ({ currentData }) => foo.set('a', currentData))

Hello @bcherny. What you describe is not currently possible. However, could you develop your use case here? I have trouble understanding why you would try to achieve with this binding of two different trees.

sure! i can start with my immediate use case, and then propose a few others:
i have an object that supports a set of filters. each filter has an id and a value, and is rendered as a separate component. the object needs knowledge of all its child filters so that it can filter down its result set, and each child component needs knowledge of its corresponding filter so it can render the current value and bind back changes:
const myObject = new Baobab({
  filters: {
    a: { id: 1, value: 2 },
    b: { id: 2, value: 'foo' }
  }
})
const myComponent = new Component({
  model: new Baobab({
    someInternalState: 42,
    moreInternalState: [],
    filter: myObject.select('filters').select('a')
  })
})

if myComponent's  filter was a proper lens into myObject.filters.a, i'd get binding for free, and myComponent and myObject wouldn't need to know about each other after this initial setup!
more use cases
what if i want to implement a linked list?
const c = new Baobab({
  value: 3,
  next: null
})
const b = new Baobab({
  value: 2,
  next: c
})
const a = new Baobab({
  value: 1,
  next: b
})

same goes for a tree, or any other data structure that has references to other nodes.

Why not just use a single instance of Baobab and chain monkeys?

cc @jacomyal

@bcherny, we are discussing the problem with  @jacomyal and the possibility to make both Tree and Cursor classes converge into a single one.

@Zache unless the monkey is returning a value from outside the tree, i'm not sure a monkey is what i'm after. i don't need a way to create computed tree nodes (i can already directly .set a node to the value of my leaf), but rather a way to compose them. in other words, if i have multiple trees in my application, i can already copy nodes from one tree to another; but it would be really useful to be able to specify relationships between the trees instead, so i can traverse these links across existing trees without having to copy data and sacrifice whatever cursors might already exist on my nodes.
@Yomguithereal it seems reasonable to combine trees and cursors into a single concept - they already share the same interface, right? but going further than that, it would be very cool if baobab trees knew how to deal with nodes of type Baobab.

@bcherny, sure but I am trying to see how it can be done and I struggle to find a performant way to do so.

Which operations are you concerned about? Maybe I can take a stab at it.

The main problem is that, for the time being, reading data is almost a zero-cost operation with the lib because it stores raw JS objects as they are. But considering data nodes as composites etc. much like the DOM, in fact, requires to build abstractions over data and probably need operations to be run on read to access data which can cost a lot of perf.
How do you see that? What's your opinion on how this should be built @bcherny?


slack.simpeg.xyz

Rather than using heroku to serve up the slack invite, we can use a permanent invite link from slack itself: https://join.slack.com/t/simpeg/shared_invite/enQtNjM2MDMxOTAwMjU4LWYzOTY2ZDk3ZTIyNzBjZWVjOTBiYjBlZTRlMjllMGE4YjYxOTZlNDIzNDJlYTQ0NGZlMmFlNDJjZmU4MGY0NjY
Heroku makes you pay for https, so lots of people aren't able to access it because their browser blocks it as unsafe.
@rowanc1: would you be willing to instead point the slack subdomain of simpeg.xyz here?

Hey @lheagy - I've been running into this issue with PyVista's slack workspace too (the join page is currently via the Heroku app). Do you have details on how to generate a link like that for a workspace?


Document new envoy-config make target

Add documentation on how to configure and use the envoy-config make target in doc/deploy.md. Couldn't find it was documented elsewhere either.

I wrote something related to this. It may need more details.


Developer docs for v3 package layout and annotations

We need to document in STYLE.md the package layout in v3, as well as the new UDPA migration annotations for devs.

See https://github.com/envoyproxy/envoy/pull/9549


Find and compare tools/services for creating easy dashboards for e.g. DORA and DPE metrics

Quick notes of some tools that Google surfaced:

https://www.okayhq.com/
https://www.geckoboard.com/
https://www.plecto.com/

┆Card is synchronized with this Github issue by Unito

Sofus Albertsen commented in Trello:

https://github.com/GoogleCloudPlatform/fourkeys

Steffen Petersen commented in Trello:

Maybe this is relevant: https://getdx.com


[Publishing] BigQueryToGCS pyspark

Multiple formats avro / parquet / orc etc
Incremental loads
Large data volumnes

Created the blog
https://medium.com/@surjitsh/moving-data-from-bigquery-to-gcs-using-gcp-dataproc-serverless-and-pyspark-f6481b86bcd1
Will close the issue once the blog is added to cloud community publication


Added python-boltons-pip

Added packages for OS distributions that have it. Let me know if that's better.


Try any failing tests up to 3 times to guard against network issues

Description
rapids-cmake has seen network issues when downloading files from GitHub. We will try tests up to 3 times when they fail to validate the issue isn't a poor network.
Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.
[x] The documentation is up to date with these changes.

I really want this fix since my PR #637 has been failing repeatedly due to network issues. I pushed a commit to fix a flag -- hope that's alright @robertmaynard.


422 error: Query too long - 21/20 tokens

Some of my users are experiencing 422 errors in production.
Here is an example of log received through Sentry.io :
{
  "body": {
    "message": "Query too long - 21/20 tokens"
  },
  "type": "HttpError",
  "message": "Query too long - 21/20 tokens",
  "request": {
    "body": null,
    "origin": "https://api.mapbox.com",
    "file": null,
    "_options": {
      "path": "/geocoding/v5/:mode/:query.json",
      "params": "[Object]",
      "method": "GET",
      "query": "[Object]"
    },
    "response": null,
    "id": 15,
    "headers": {},
    "client": {
      "origin": "https://api.mapbox.com",
      "accessToken": "[FILTERED]"
    },
    "params": {
      "query": "jade & Spa, Calle 55 Supermanzana II, Manzana 4 Lote I-0I, Puerto Morelos Benito Juárez, Puerto Morelos, Quintana Roo 77580, Mexic",
      "mode": "mapbox.places"
    },
    "aborted": false,
    "error": "[Circular ~]",
    "path": "/geocoding/v5/:mode/:query.json",
    "method": "GET",
    "query": {
      "country": "[undefined]",
      "proximity": "[Array]",
      "limit": 5,
      "language": "[Array]"
    },
    "emitter": {
      "_events": "[Object]",
      "_eventsCount": 0
    },
    "sent": true
  },
  "statusCode": 422
}

As seen in Mapbox documentation, we may have to filter words in query so it never sends more than 20 words: https://docs.mapbox.com/api/search/#geocoding-api-errors
Let me know if you have the same issue and if I can do anything.
Thanks

Hi @adesurirey, I've never come across this issue before. I think it would help to understand how your users are using the geocoder. Why is the query so long? Are the users copying and pasting it? Queries that long don't work with Mapbox's geocoder. I just tried it here https://docs.mapbox.com/mapbox-gl-js/example/mapbox-gl-geocoder/. If it's that long because they are editing a result they previously picked (they could hit the clear icon instead of doing this), you could use a combination of the inputValue and onResult props to truncate the value in the input field. Here's an example: https://codesandbox.io/s/react-map-gl-geocoder-shorten-query-result-example-k0jmq.
The normal workflow is typing a few characters and selecting one of the options. Typically the queries usually don't have more than a few tokens. I think most of my own queries are only one or two tokens.
My gut feeling is that your users probably don't even notice this error. I personally would ignore these Sentry errors unless you verify that it is a problem.

Hi @SamSamskies, can't know exactly but I think you're right about copy and pasting long address. I'll go with your solution. Thanks !


Proposal to improve the regex circuit

This is probably related to #16.
Rational
The regex circuit seems playing a key role in the content verification in zk email. I believe it is unique in the circuit market at this stage and would provide great value to the circom community if the regex circuit compilation can be encapsulated as a standalone circuit library.
To make the regex circuits usable for wider audience, there would be some changes needed as outlined in the following.
Features
Restructure counting for the matches
For example, assuming there are more than one match in the input message signal, then the output reveal signal would be like [0,0,0,0,1,1,1,1,0,2,2,2,2,0,0,0,3,3,3,0].
The sample reveal signal serves for two purposes.

Embed information for the counting.
Embed information for locations for nth matches.

With these two embedded information, the reveal array signal can be further processed to provide additional data points as outline in the next feature for different verification needs.
Additional data points
Given an input parameter for the nth match, it should output the following signals

start position
end position
matched value

For example, given the input is 2 and the original message is [a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t], the output signal values would be

start position: 9
end position: 13
matched value: [j,k,l,m]

Implementation

It is likely to consider together with #16. My current thought is to refactor the regex circuit compiler to include the counting restructure outlined in the feature above, and create a helper circuit for the features outlined in additional data points.

For the regex circom compilation #16, it currently uses a python script. If there is no particular reason for the preference over python language, I think it might be helpful to convert the script to JS as it is consistent with the parts of project for easier maintenance and testing.

Test coverage. The tests, similar to how the 0xparc-ecdsa does, would be very helpful for the development process. With the tests in place, the concerns can be isolated and imrove the efficiency of circuit compilation for testing. Overall, this would help improve the development process cycle in practice.

Looks good, we can discuss more tomorrow! We have code for start and end position (along with matched text) in email.circom, perhaps it can be moved here, outputs chosen can be toggled via template parameter).
Tests and restrurcturing would be awesome -- would love to hear what else you think could use some restructuring. Feel free to convert to JS! That wilk also help for in browser regex circuit generation if we ever decide to make that client side only.

I have done the JS conversion on my side and will refine the commands for regex compilation.
Let me know if you want to create a standalone repo for the regex circuit lib. I will push the code there.

Sounds great! I've added you to https://github.com/zk-email-verify/zk-regex/invitations.

Thanks for the arrangement. @Divide-By-0
I think now zk-regex got a skeleton and is ready to be integrated back into this repo. There are a couple of targets for the integration:

zk-regex now has a CLI to generate circuits for a downgraded regular expression(such as not yet supported character class, but will do in the near future). The README would be updated on how to use the new regex circuit compiler.

There will be changes to the regex circuit input and output signals to simplify the interaction with regex circuits in the email.circom. For example, these lines of code would be encapsulated in the regex circuits instead. The newly compiled regex circuits would include the input signals input_msg, nth_match (nth match would be optional), output signals start_position, end_position, revealed_text for nth_match(default 1st match). Overall, the email.circom code relating to regex verification would be simplified via the use of new regex circuits generated from zk-regex.

Tests will be set up to cover the email.circom circuit.

Meanwhile, I think it would be necessary to benchmark the email.circom circuit to see if there are any significant overheads introduced by the refactor. I am new to this area, happy to know if there are any ways to do the benchmark from your experiences.
Please feel free to let me know if any other items are also reasonable to be included in this batch of work.

In the latest commit, the zk-regex is now able to encapsulate the circuit better and simplify the email.circom circuit as mentioned in the 2 point above. I think with these circuit simplifications from the zk-regex, email.circom would be easier to be abstract for different use cases.
If you get a chance, please help check to see if the code is optimal.

Jern has a good suggestion for improvement:
So, the demo I and Anka built is based on the assumption that we try to match the subgroup from the minimized DFA (which is unique to each regex) However, since we label tag after it’s minimized, the tag is not 1-1 and can cause funny behavior in some cases.
Hence, to be more general, we need to start appending tag since NFA states. Now, if we literally use NFA with tags and put it in circom, it is fine tracking states, we don’t need registers. But it will take expo time due to NFA nature when extracting the tag. To collapse NFA to DFA and tracking the tag, most papers use register because it can collapse to a very succinct state machine since now you don’t need to care what index that the tag gets assigned to, you just keep the register number, and assign the index to it as the Algo reads the input.
That being said, the tagged DFA is definitely overkilled since it is literally tagged  between any alphabet in regex, not the subgroup. So, I’m weakening the assumption and care only the subgroup extracting with () stuffs, and make the version using register.
At first, I thought register in circom shouldn’t be bad (and if we do that, it can easily be applied to almost all versions of tagged stuffs), but it doesn’t seem so, so I will try modify it to be ok without it, so we can stick to the transition version that u and Sora already built (I think it should work using table stuffs) but either way, the naive minimized DFA will never work when we want subgroup extraction.


Add k8s top-level dir with kustomize deployment example

This adds a k8s folder which will hopefully in the future house best practices, patterns, and examples for deploying wasmCloud on k8s.
For folks like me that strongly prefer deployments with GitOps, the first step is often creating a kustomization. There are a few tricks when working with kustomize and CRD's, the helm charts that we depend on, and the order of operations for the deployment.
I started with this as an example in wasmCloud/wasmCloud but depending on the operator means it should be moved up a level. I considered adding it to the examples in the operator repo, but I want to extend this deployment to include a k8s secrets backend and likely other wasmcloud-contrib extensions.

I have to admit, I'm a little unclear on the benefits of deploying Helm charts with Kustomize, why not just skip Kustomize altogether and deploy them with Helm?
Is there some customization that you are not able to do with Helm alone, or is the goal to just have all of the rendered YAML in the repository?

All rendered yaml is the main thing. I like being able to deploy with overlays (dev vs prod), although in this demo I'm not yet taking advantage of it yet.


Fix friendbot 500 errors

Occasionally, friendbot responds with 500 errors. Some were fixed by #1511, but there are others that @accordeiro identified below. Let's fix those.

It seems like fixing #1511 got rid of most errors, but I've found two others that are still relatively frequent:
Error 1: invalid URL escape
It seems like providing an invalid URL causes a 500:
Example 1.1
Request
time=“2019-07-30T08:19:47.697Z” level=info msg=“finished request” bytes=444 duration=“53.107µs” method=GET path=“/?addr=%s” pid=15931 req=frontend-friendbot-001a/HlNPxBapp5-053399 status=500 subsys=http

Response
time=“2019-07-30T08:19:47.697Z” level=error msg=“invalid URL escape \”%s\”” pid=15931 stack=unknown

Error 2: result_xdr not populated
This will probably require some deeper digging.
Example 2.1
Request
time="2019-07-30T10:38:59.343Z" level=info msg="finished request" bytes=444 duration=30.690100736s method=GET path="/?addr=GCLO3KDICPIBYCDOZGBJYBVGKRD2L2G5DENJFJFBRD5BJOK24X5LD42M" pid=15931 req=frontend-friendbot-001a/HlNPxBapp5-054563 status=500 subsys=http

Response
time=“2019-07-30T10:38:59.343Z” level=error msg=“submitting tx to minion: submitting tx to horizon: error getting horizon error code: result_xdr not populated” pid=15931 stack=“[main.go:43 minion.go:52 asm_amd64.s:1333]”

Example 2.2
Request
time=“2019-07-30T05:18:50.886Z” level=info msg=“finished request” bytes=444 duration=30.404681807s method=GET path=“/?addr=GBHFUQUN5BGKU7G5I2FAFHLMR3VXXYFGMEZHWU2UJE4E2CHZOD7DMILN” pid=15931 req=frontend-friendbot-001a/HlNPxBapp5-051987 status=500 subsys=http

Response
time=“2019-07-30T05:18:50.886Z” level=error msg=“submitting tx to minion: submitting tx to horizon: error getting horizon error code: result_xdr not populated” pid=15931 stack=“[main.go:43 minion.go:52 asm_amd64.s:1333]”

@tomquisel I've created a PR (#1612) to address Error 1.
Error 2 seems trickier to reproduce since it seems to be an error between Friendbot <> Horizon, rather than between Client <> Friendbot – but I'll keep investigating.

Error 2 isn't occurring for me now in both examples above. Given this was related to Friendbot => Horizon is it possible something changed in Horizon that subsequently fixed these cases?
$ curl 'https://friendbot.stellar.org/?addr=GCLO3KDICPIBYCDOZGBJYBVGKRD2L2G5DENJFJFBRD5BJOK24X5LD42M'
{
  "_links": {
    "transaction": {
      "href": "https://horizon-testnet.stellar.org/transactions/59741f42e2d5fd92bf9b9367fe85751e6761699e509b0461cc510f50966f9099"
    }
  },
  "hash": "59741f42e2d5fd92bf9b9367fe85751e6761699e509b0461cc510f50966f9099",
  "ledger": 860917,
  "envelope_xdr": "AAAAAMBaBqzOwbbQH5aFMkY7zPrIMj6IA88VmwPzp4JllbxBAAGGoAAHBfUAAAC6AAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAEAAAAAEH3Rayw4M0iCLoEe96rPFNGYim8AVHJU0z4ebYZW4JwAAAAAAAAAAJbtqGgT0BwIbsmCnAamVEel6N0ZGpKkoYj6FLla5fqxAAAAF0h26AAAAAAAAAAAAmWVvEEAAABAuqnfhgJftmx6ckJubkvnFqG9vhjRHTfzlxMwHkkvoQI9nlmUEv42aabL/4FJRsmULHT6eNX5zHdZZO/dnmOfBoZW4JwAAABARMrKYdQfoHbeLzHChyn1H9Tae8Ynq9Dwfz5gazNxH9M6wwMzCcSD7Uq0LON2EPTuyjdZKkC5XG9LRFQYthe1BQ==",
  "result_xdr": "AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAA=",
  "result_meta_xdr": "AAAAAQAAAAIAAAADAA0i9QAAAAAAAAAAwFoGrM7BttAfloUyRjvM+sgyPogDzxWbA/OngmWVvEEAAAAAPAXlBwAHBfUAAAC5AAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAABAA0i9QAAAAAAAAAAwFoGrM7BttAfloUyRjvM+sgyPogDzxWbA/OngmWVvEEAAAAAPAXlBwAHBfUAAAC6AAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAABAAAAAwAAAAMADSLzAAAAAAAAAAAQfdFrLDgzSIIugR73qs8U0ZiKbwBUclTTPh5thlbgnADFbXwzG+UqAAAAwQAAAL8AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAEADSL1AAAAAAAAAAAQfdFrLDgzSIIugR73qs8U0ZiKbwBUclTTPh5thlbgnADFbWTqpP0qAAAAwQAAAL8AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAADSL1AAAAAAAAAACW7ahoE9AcCG7JgpwGplRHpejdGRqSpKGI+hS5WuX6sQAAABdIdugAAA0i9QAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA=="
}%

$ curl 'https://friendbot.stellar.org/?addr=GBHFUQUN5BGKU7G5I2FAFHLMR3VXXYFGMEZHWU2UJE4E2CHZOD7DMILN'
{
  "_links": {
    "transaction": {
      "href": "https://horizon-testnet.stellar.org/transactions/8dafa8b6d86d94deb7a0d89e3dc57bda31bacb99bb4a1e2fec18d34b5528c1ce"
    }
  },
  "hash": "8dafa8b6d86d94deb7a0d89e3dc57bda31bacb99bb4a1e2fec18d34b5528c1ce",
  "ledger": 860927,
  "envelope_xdr": "AAAAAPlCtJAvCuJZf+v2596U31MClKJtWWqv7e3JJ8D/kwf6AAGGoAAHBfUAAAC6AAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAEAAAAAEH3Rayw4M0iCLoEe96rPFNGYim8AVHJU0z4ebYZW4JwAAAAAAAAAAE5aQo3oTKp83UaKAp1sjut74KZhMntTVEk4TQj5cP42AAAAF0h26AAAAAAAAAAAAv+TB/oAAABA6PT0W2/SBuJ7EyTq6CVTLuay9i0IhZZZen2nuPQck5QuUrYd5kU/pRqKshESqvdDBiuDEC1tTPQfjltlTdKmAYZW4JwAAABAEA+pddQRmPEXr89uHLf5PvgqciKKKS0dFaYOi4HSWC28kXWv/wIYrwKwOJwBXwlQvsz8WgZ3wql3BnCU/5I1Dg==",
  "result_xdr": "AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAA=",
  "result_meta_xdr": "AAAAAQAAAAIAAAADAA0i/wAAAAAAAAAA+UK0kC8K4ll/6/bn3pTfUwKUom1Zaq/t7cknwP+TB/oAAAAAPAIzvwAHBfUAAAC5AAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAABAA0i/wAAAAAAAAAA+UK0kC8K4ll/6/bn3pTfUwKUom1Zaq/t7cknwP+TB/oAAAAAPAIzvwAHBfUAAAC6AAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAABAAAAAwAAAAMADSL+AAAAAAAAAAAQfdFrLDgzSIIugR73qs8U0ZiKbwBUclTTPh5thlbgnADFbR8RQEUqAAAAwQAAAL8AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAEADSL/AAAAAAAAAAAQfdFrLDgzSIIugR73qs8U0ZiKbwBUclTTPh5thlbgnADFbQfIyV0qAAAAwQAAAL8AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAADSL/AAAAAAAAAABOWkKN6EyqfN1GigKdbI7re+CmYTJ7U1RJOE0I+XD+NgAAABdIdugAAA0i/wAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA=="
}%


The random walker attacker is problematic during long attack sequences

The biggest problem with the random walker is when the attack has progressed a longer while, because at that point, the attack horizon is very big and the alternatives are many for the attacker. At that point, it is very difficult to predict what the attacker will do, and very few patterns will be visible, as a number of unrelated attack steps can succeed each other.
One option is to simply delete the oldest attack steps in the attack horizon, thus maintaining a reasonably-sized horizon.
A similar option would be to assign weights to attack steps, so that the most recently discovered ones are the most likely to be attacked. For instance, 50% of the probability mass could be assigned to the most recently discovered steps, 25% to the second-most recently discovered, etc.

I have now implemented an attacker, less_random, than prioritizes newly discovered actions.


Move codegen related decorations from functions to capabilities.

Decorations like __require_glsl_extension should really be defined on capabilities instead of individual functions.

Closing for now-- not going to be able to close this in the short time, because we can't rely on codegen logic here for a while.


Where can I find the data set you shared?

Thank you for sharing, your project is very interesting. I just read your paper carefully, and I want to use your data set to make some interesting projects (non-commercial). Where can I find the data set you shared?

Hi there,
Thanks for showing interest in our project. The datasets AM-2k and BG-2k can be accessed from both Google drive and baidu wangpan, the links are available on the repo homepage. PM-10 will be available after privacy-preserving from the project Privacy-Preserving Portrait Matting(https://github.com/JizhiziLi/P3M) very soon, you can check out the GitHub repo for updates. Cheers.

Okay, thank you very much for your reply. This is the first time I have received a reply from the author of a private message. I am very happy, thank you sincerely, and wish you a happy life and success in your work.
------------------ 原始邮件 ------------------
发件人:                                                                                "JizhiziLi/GFM"                                                                                @.>;
发送时间: 2021年11月12日(星期五) 下午4:36
@.>;
<EMAIL_ADDRESS>主题: Re: [JizhiziLi/GFM] Where can I find the data set you shared? (Issue #29)
Hi there,
Thanks for showing interest in our project. The datasets AM-2k and BG-2k can be accessed from both Google drive and baidu wangpan, the links are available on the repo homepage. PM-10 will be available after privacy-preserving from the project Privacy-Preserving Portrait Matting(https://github.com/JizhiziLi/P3M) very soon, you can check out the GitHub repo for updates. Cheers.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.


Implement of variant_rule

According to the document, function parse has 2 implementation. One of them is required to match the entire string.
The variant_rule should implement the ABNF rule variant = rule1 / rule2 / rule3 ....
When I combine parse and variant_rule, I expect it to match the entire string. However, if the preceding rules can partially match the string, based on the current implementation, it will return an error, and subsequent rules that may fully match will be ignored.
This is a test I conducted while implementing this set of rules.
// obs-FWS         =   1*WSP *(CRLF 1*WSP)
// FWS             =   ([*WSP CRLF] 1*WSP) /  obs-FWS
  constexpr auto wsp_chars = grammar::lut_chars(" \t");
  constexpr auto obs_fws_rule = grammar::tuple_rule(
      grammar::token_rule(wsp_chars),
      grammar::range_rule(grammar::literal_rule("\r\n"),
                          grammar::token_rule(wsp_chars)));

  /// Rule for FWS = ([*WSP CRLF] 1*WSP) / obs-FWS
  constexpr auto fws_rule = grammar::variant_rule(
      grammar::tuple_rule(
          grammar::optional_rule(grammar::tuple_rule(
              grammar::optional_rule(grammar::token_rule(wsp_chars)),
              grammar::literal_rule("\r\n"))),
          grammar::token_rule(wsp_chars)),
      obs_fws_rule);
  auto cv = parse("  \r\n  \r\n  ", variant_rule(fws_rule, obs_fws_rule));
  std::cout << cv.error() << std::endl;

here is the output
boost.url.grammar: 4

So, you are using the grammar algorithms! Great :)

Can you please show which part of RFC5322 you are trying to use the variant rule with?

I looked at the documentation for grammar and thought it was designed to deal with the ABNF syntax in rfc.

Yes but there is no code which looks for the longest match. You have to do that yourself, by writing a regular function. We could look into improving the variant rule, maybe. In most cases where variant rule is an attractive option there is usually a better performing solution in writing a custom function.

@A2ureStone, thanks for starting this discussion. At worst, the behavior of these functions should be better documented.
Yes. This is not a bug. Almost all basic grammar library implementations are greedy, including regex implementations. For instance:
const regex = /abc|abcd/;
const input = "abcd";
const result = regex.exec(input);
console.log("Match found:", result[0]);

will match abc. When dealing with alternatives, the order of alternatives affects the match because regular expressions are by default greedy.
The equivalent of regex.exec in C++ would be std::regex_search:
    std::regex regex("abc|abcd");
    std::string input = "abcd";
    std::smatch match;
    if (std::regex_search(input, match, regex)) {
        std::cout << "Match found: " << match.str(0) << std::endl;
    }

However, in some cases, such as std::regex_match, the implementation should backtrack: https://godbolt.org/z/zaqnve449 as it should backtrack when forced to match the end of the input with a $:
 /(abc|abcd)$/.exec('abcd');

In any case, the base case for grammar is always greedy. The same greedy logic applies to almost all parsing libraries (Boost.Spirit, Boost.Parse, PEGTL, ...) and grammars because the alternative is always worse: For every way that any rule could match any subset of the string, the algorithm would have to roll the tape to the end to check if it consumes the whole input, and otherwise, exponentially backtrack to all other alternatives.
The cost of backtracking is so high, and the use cases are so different that a parse variant that backtracks should be a new function. The API of rules also needs to be updated somehow so that rules can return all alternative matches it can produce so that the parent algorithm can iterate them. Because of this extra complexity, we usually want to write a grammar that works with greedy rules.
One easy way to do that (when it makes sense) is to list the rules that should match the longer string first in the variant. It's often the case that we have multiple rules but we know from context that some rules will always match longer strings than others.
In your case, the rules are:
obs-FWS         =   1*WSP *(CRLF 1*WSP)
FWS             =   ([*WSP CRLF] 1*WSP) /  obs-FWS

so I don't even think it makes sense to parse variant_rule(fws_rule, obs_fws_rule) at all because obs-FWS is already part of FWS. In any case, the first alternative is to use existing boost::urls::grammar rules and to change how the rules are combined so the algorithm is still greedy and correct. In the case of communications protocols, there should usually be a correct greedy algorithm to parse the input because the input is being received in a stream, so there's no concept of "end of message" at that level.
As a second option, you can also write your own non-greedy rule for this variant only, making it a rule that works with other greedy rules. The rule would inspect both paths and match the longest one. Or it would be a choice depending on the rules to come.
In a way, it's not impossible for url to implement a new longest_variant_rule, which would be an expensive variant of variant_rule that returns the longest match. However, it wouldn't be very useful since it's very common just to be able to fix everything by changing the order of the rules and not having this extra cost. Also because it doesn't completely solve your problem (more on that below).
I think Vinnie has a good point here:

In most cases where variant rule is an attractive option there is usually a better performing solution in writing a custom function.

Custom parsers are better than general parsers because there are alternatives like these optional or variant rules; custom parsers can use contextual information to eliminate alternatives in a way a general solution just can't. If you think about the other kinds of rule composition, a custom parser shouldn't be able to outperform a general parser because it's just iterating the string anyway.
For instance, contextual information is what solves the problem in the cases you mentioned, like "This example clearly demands a full match. If a string is a complete match relative-ref but only partially matches URI". This is not a problem because we know from context that these rules are not ambiguous so whatever fails to match would fail in both overloads of parse. Also note how your variant_rule is, on the other hand, ambiguous. It includes obs-FWS, which is also an alternative of FWS.
Also, matching the longest substring in variant_rule doesn't automatically solve your problem either. It's possible that matching the longest substring here means the rest of the input is now invalid for the following rules in a way that it may not happen in the shortest sequences. So, in any case, you'll need to backtrack unless you have contextual information that lets you know that the longest match will always solve the problem.
Another alternative is a backtracking rule of sorts to indicate that, unlike most rules, that variant rule should be considered an inflection point where all alternatives should roll the tape to the end. Still, the design of such a rule could become quite complex. One design is where it would take the alternatives AND the following rules so it can backtrack on the following rules. Another design is one where the rules would somehow return lists of matches instead of a match and the parent algorithm would need to be aware of that special logic.
Apart from that, if you can't find a greedy rule but you're writing your own parse_x algorithm for your rule, like parse_uri, etc., you can foresee that this rule might match prematurely and backtrack there. It's some work, but it's best to have and use the specific cases that might need to backtrack rather than rely on an implementation that tries to backtrack on all rules. And it can only "try" to backtrack because the library supports customs rules.

I'm glad you found a solution. You're correct: translating the grammar directly as documented doesn't always work well because rules can be implemented differently depending on what contextual information we have.


RunningOnValgrind symbol conflict

Describe the bug
When building gRPC Python which includes gRPC Core using Abseil, some builds end up failing with the following error.
In file included from third_party/abseil-cpp/absl/base/internal/spinlock.h:37:0,
                   from third_party/abseil-cpp/absl/strings/internal/cordz_info.h:24,
                   from third_party/abseil-cpp/absl/strings/cord.h:86,
                   from third_party/abseil-cpp/absl/status/internal/status_internal.h:20,
                   from third_party/abseil-cpp/absl/status/status.h:58,
                   from ./src/core/lib/gprpp/status_helper.h:26,
                   from ./src/core/lib/iomgr/error.h:33,
                   from src/python/grpcio/grpc/_cython/cygrpc.cpp:1850:
  third_party/abseil-cpp/absl/base/dynamic_annotations.h:456:28: error: 'int absl::base_internal::RunningOnValgrind()' conflicts with a previous declaration
   using absl::base_internal::RunningOnValgrind;
                              ^
  In file included from /usr/include/python3.4m/pyatomic.h:8:0,
                   from /usr/include/python3.4m/Python.h:53,
                   from src/python/grpcio/grpc/_cython/cygrpc.cpp:1219:
  /usr/include/python3.4m/dynamic_annotations.h:456:5: note: previous declaration 'int RunningOnValgrind()'
   int RunningOnValgrind(void);

This is because RunningOnValgrind symbol is defined in both Abseil's dynamic_annotation.h (link) and Python's dynamic_annotation.h (link). I guess gcc might be confused because Abseil's one looks like C++ and Python's one is C-style.
Steps to reproduce the bug
Building a source which includes both Abseil and Python include files which end up including dynamic_annotations.h.
What version of Abseil are you using?
The latest head
What operating system and version are you using
Linux Debian Jessie.
What compiler and version are you using?
gcc 4.9 (bulit-in in Debian Jessie.
What build system are you using?
Python build. FYI, a following command ran when emitting the error above
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.41.0.dev0" -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_linux -Ithird_party/re2 -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -Ithird_party/zlib -I/usr/include/python3.4m -I/tmp/tmp.XxN5JjXiya/include/python3.4m -c src/python/grpcio/grpc/_cython/cygrpc.cpp -o python_build/temp.linux-x86_64-3.4/src/python/grpcio/grpc/_cython/cygrpc.o -std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread

Additional context
Since gcc 4.9 is not supported anymore from Abseil, if this turns out to be a gcc 4.9 specific problem, it's okay to close this.

Thanks Derek, but should the following code block in dynamic_annotation.h be removed as well?
namespace absl {
ABSL_NAMESPACE_BEGIN
namespace base_internal {
ABSL_DEPRECATED(
    "Don't use this interface. It is misleading and is being deleted.")
ABSL_ATTRIBUTE_ALWAYS_INLINE inline int RunningOnValgrind() { return 0; }
ABSL_DEPRECATED(
    "Don't use this interface. It is misleading and is being deleted.")
ABSL_ATTRIBUTE_ALWAYS_INLINE inline double ValgrindSlowdown() { return 1.0; }
}  // namespace base_internal
ABSL_NAMESPACE_END
}  // namespace absl

using absl::base_internal::RunningOnValgrind;
using absl::base_internal::ValgrindSlowdown;

I don't see that code anywhere in https://github.com/abseil/abseil-cpp/blob/master/absl/base/internal/dynamic_annotations.h.

@derekmauro Oh my bad. I checked the internal source ;) Thanks for fixing this!


using test set as validation set

While doing a walk-forward validation to assess the performances of my model, I specified the test set as validation set in the following way:
    booster = xgb.train(params,
                dmat_train,
                evals=[(dmat_train, "train"), (dmat_test, "test")], verbose_eval = False,
                num_boost_round=num_boost_round)
    preds = booster.predict(dmat_test

)

thinking that it would have no effect and would be only for metrics monitoring during training.
However, when I do not do it:
    booster = xgb.train(params,
                dmat_train,
                evals=[(dmat_train, "train")], verbose_eval = False,
                num_boost_round=num_boost_round)
    preds = booster.predict(dmat_test)

performances are significantly worse.
I guess the second way of doing is the right one. But why? How does specifying the test set as cross-validation set actually leads to overfitting in XGBoost? Note that I do not do any hyper-parameter optimizations.

Could you please share the parameters that you are using?


Add another sample usage of a counter

Using counters to track the last successful time of a batch action is a wonderful way to decrease the noisiness of alerts.

That's a gauge, not a counter.

Thank you for your contribution. Closing this pull request due to lack of recent activity, but we'd be happy if you want to pick up the work again. In that case please make the requested changes, ensure all tests pass, and re-open the PR.


[global] intercept auth routes

This could be pretttty coool. Logging in without leaving my page

@jakemackie nice idea. i'm assuming this could be useful when (for example) sending out daily question emails, link goes straight to the questions and if the user's not logged in, show log in modal?


warning ....\package.json: No license field

Check that this is really a bug

[X] I confirm

Reproduction link

Bug description
After forking the project and doing its setup locally, after typing "yarn storybook" on the command line (the next step after setting up "yarn" was done successfully), it shows me this warning:
warning ..\..\package.json: No license field

The project runs successfully though.

Expected Behavior
No response
Actual Behavior

react-big-calendar version
1.12.0
React version
17.0.2
Platform/Target and Browser Versions
Microsoft Edge
Validations

[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
[X] Make sure this is a react-big-calendar issue and not an implementation issue

Would you like to open a PR for this bug?

[ ] I'm willing to open a PR

That is interesting. Must be a bug with Storybook, as "license": "MIT" is definitely in the package.json file


"Gingy keychain" not appearing as loot in raids when wishlisted

Using SPT-AKI 3.5.2 and the latest version of this trainer. Everything shows up when wishlisted in raids except the Gingy keychain. I've compelted over 100 raids and haven't come across it once.
Thanks for your help.

I just made a fresh profile and installed to test. The item shows in the tracklist in the console. Unfortunately I don't have a mod to allow me to drop a key chain in raid to see if it shows. If you issue a tracklist in the console do you see something similar to this?

Edit: I forgot the the SVM mod can disable that. I just tested and it tracks just as expected. Perhaps its just extremely bad luck =/

Technically this item is a container, so make sure you enabled the SearchInsideContainers option in the loot section. Else we'll just skip it.

I have SearchInsideContainers enabled and still haven't seen it. I'll try tracklist suggestion.

So I do see "Gingy (Wishlist)" when I open console and input tracklist but still have yet to come across it. I've noticed if items are in, let's say a Scav's inventory that's still alive, I won't see the item until that Scav is dead, at which point the loot message will pop as "X (in corpse)" or whatever. Is there a way to see loot that's wishlisted in alive Scav/PMC inventories?

Closing this one. We only search inside static containers (so only dead scav). That's because we cache the position several seconds for better performance. If we start tracking on live scavs, we'll see not-up-to-date values.


Background-position and background-size working in Outlook for windows desktop and yahoo

Based on: https://github.com/mjmlio/mjml/issues/1814
Background position and background size is now working with yahoo and outlook desktop.
I have only added background-posiiton-x and background-position-y, as it seemed to complicated to implement just background-position.
background-repeat: no-repeat will now also work for outlook, the only issue is that the image will be stretched if background-size is not set to either cover or contain.
Tests:
https://litmus.com/builder/9ebafc2
https://litmus.com/builder/fa7f3af
https://litmus.com/builder/06bf715
https://litmus.com/builder/2f46c4e
https://litmus.com/builder/ed180d2
https://litmus.com/builder/3fb5dbf
Edit:
I have added support for percentage values instead of just top, center, bottom, left and right.
Here are my tests for that
https://litmus.com/builder/4cdafaa
https://litmus.com/builder/ab592f7

We won't be able to ship it with 4.6 as it requires some testing and "might" change a bit the current behavior. That being say let's target this for a 4.7 / 5.0 with custom attributes first implementation

Ok we should take a look to get this merge cc @kmcb777

very nice improvement thanks

This is a wonderful improvement and solved a major issue I was having. Thank you!


[question] reorder dependencies in package.json

Hey,
I would like to create a Pull Request to reorder the devDependencies (alphabetically) in your package.json.
The reason behind this is that yarn automatically do that and it creates huge conflict when git fetching your repo into my private one.
Do you agree ? If so, I will create a PR ASAP.

@theomathieubhvr I saw that yarn is auto-sorting the deps using yarn upgrade but the main reason that the deps are not sorted, just grouped, was the ability to manage them by seeing in groups.
It's in my mind to find out a way how to make yarn to not sort them alphabetically. For now, let's better leave it :)

Ok !
Thanks for the explanation.


adding Binary Insertion Sort Algorithm C# implementation #254

Assigned Issue Number?
#254
Language Opted?
C#
Number of files added or updated?
1
Name of these files?
Algorithms => Sorting => Binary_Insertion_Sort => binaryInsertionSort.cs
Checklist

[x] I was assigned for this otherwise my PR will be rejected.
[x] I've read the contribution guidelines.
[x] I've written code by myself. No Plagiarism allowed
[x] I've referred the correct issue number.
[x] I've filled up this entire template correctly.

This algorithm has been tested with the following script that I wrote:
        int times = 1;

        bool passed = true;

        Random rnd = new Random();

        do

        {

            int len = (int)rnd.Next(10) * 100;

            double[] arr = new double[len];

            for (int index = 0; index < len; index++)

            {

                arr[index] = rnd.Next();

            }

            double[] clnArr = (double[])arr.Clone();

            Array.Sort(clnArr);

            double[] binarysortedarray = binaryInsertionSort(arr);

            for (int index = 0; index < arr.Length; index++)

            {

                passed = binarysortedarray[index] == clnArr[index];

                if (!passed)

                {

                    break;

                }

            }

            if (passed)

            {

                Console.WriteLine("Test case N {0:D} Passed ✅\n ------", times);

            }

            else

            {

                Console.WriteLine("----- Error on case {0:D} -----", times);

                Console.WriteLine("[{0}]", string.Join(", ", arr));

                Console.WriteLine("[{0}]", string.Join(", ", binarysortedarray));

                passed = false;

                break;

            }

            times++;

        } while (passed && times <= 1000000);

I tested the Algorithm over than 1000000 on multiple times with a Passed ✅ every time.
⋮
Test case N 999996 Passed ✅
 ------
Test case N 999997 Passed ✅
 ------
Test case N 999998 Passed ✅
 ------
Test case N 999999 Passed ✅
 ------
Test case N 1000000 Passed ✅
 ------

:star2: this repository and support.

You're doing great @OussamaSALAHOUELHADJ! Keep moving forward like this! Kudos to you!

You're doing great @OussamaSALAHOUELHADJ! Keep moving forward like this! Kudos to you!

Hey @hemanth-kotagiri, thank you for the learning/discovery opportunity ❤


migrate to python3

using the 2to3 tool

Converting the code to Python 3 is changing the results, and the only changes I made to the code was adding () to the print function and adding the list() function in some cases.
I think we should try to implement this at the end because most of the pruning algorithms that need to be included also have their implementations in python 3, so at the end, we can do all the conversions together at once.

did you try to convert manually or using the 2to3 tool?
i agree that the priority of this task is less than the priority of any
other task, but certainly adding braces to "print" should not change
anything as well (there is something else going on here)
On Tue, Jul 3, 2018 at 1:18 PM Shantanu Acharya<EMAIL_ADDRESS>wrote:

Converting the code to Python 3 is changing the results, and the only
changes I made to the code was adding () to the print function and adding
the list() function in some cases.
I think we should try to implement this at the end because most of the
pruning algorithms that need to be included also have their implementations
in python 3, so at the end, we can do all the conversions together at once.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/shan18/taxi/issues/3#issuecomment-402094311, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABY6vmPLUt2j7Xi76Bjwr23cocrW_7owks5uC0TzgaJpZM4UvLzN
.

I used the 2to3 tool. Maybe it has to do with the ordering of dictionaries in python 3 that you were telling me about that day.

i think that using 2to3 may be a problem. you need to apply very carefully
this tool and understand what it does. i suggest to start from scratch and
try converting files one by one with the tool trying to understand the
differences and accept them only if they are acceptable. otherwise, it may
also make sense just to convert everything manually without the help of any
tool.
On Tue, Jul 3, 2018 at 1:41 PM Shantanu Acharya<EMAIL_ADDRESS>wrote:

I used the 2to3 tool. Maybe it has to do with the ordering of dictionaries
in python 3 that you were telling me about that day.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/shan18/taxi/issues/3#issuecomment-402103177, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABY6vijerrekKAVIpX8t0p6zsH_-d87Iks5uC0pugaJpZM4UvLzN
.

Yes, doing it manually might be more helpful. Will try that.

you can also see what 2to3 really does - you can run it in a file by file
diff mode.
On Tue, Jul 3, 2018 at 3:38 PM Shantanu Acharya<EMAIL_ADDRESS>wrote:

Yes, doing it manually might be more helpful. Will try that.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/shan18/taxi/issues/3#issuecomment-402141544, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABY6vr5-AgV1JCaa7zx9ci6x_6p-y8Hrks5uC2XYgaJpZM4UvLzN
.


Reenable testing litgpt falcon 40b like

Disabled due to nvfuser internal assert
https://github.com/NVIDIA/Fuser/issues/3505

Lightning+NVIDIA had an interactive chat on 12/9 and decided that we can stop testing falcon 40b due to low utilization.

Thanks, I started talking with mixology about this. I think we might want to drop more than this; retitling.
https://pytorch-lightning.slack.com/archives/C04N128ARJ6/p1734009632309119?thread_ts=1733960395.508069&cid=C04N128ARJ6

Hello,
we will keep falcon-180b as it's the largest model.

Hello, removed falcon-40b from benchmarks - this issue can be closed now (unfortunately, seems like I have no permissions to do that myself).

Thank you, I'll remove it from the tests.


updates to loadHosts

Fixes function name in documentation.
Moves error check outside of the loop. It could never be executed
inside the loop, as .Scan() would have returned false.

thanks @ShawnMilo

You're welcome. :sunglasses:


FnCreateDevice takes Option?

FnCreateDevice takes an optional PhysicalDevice parameter:
https://github.com/sjb3d/spark/blob/master/spark/src/vk.rs#L43582-L43587
but this parameter is clearly specified to be valid in the spec.
Some other example like FnCreateSwapchainKHR also seems to have the same issue.

That is true, but I'd expect a user of the library to use one of the Device/Instance wrapper functions rather than the function pointer directly!  The wrapper functions do expose the parameter as non-optional:
    pub unsafe fn create_device(
        &self,
        physical_device: vk::PhysicalDevice,
        p_create_info: &vk::DeviceCreateInfo,
        p_allocator: Option<&vk::AllocationCallbacks>,
        version: vk::Version,
    ) -> result::Result<Device, LoaderError> {

Do you have a use case for using the function pointers directly where this important?  I'm hesitant to change this behaviour since I was regarding the function pointers as an implementation detail.

That is true, but I'd expect a user of the library to use one of the Device/Instance wrapper functions rather than the function pointer directly! The wrapper functions do expose the parameter as non-optional:
    pub unsafe fn create_device(
        &self,
        physical_device: vk::PhysicalDevice,
        p_create_info: &vk::DeviceCreateInfo,
        p_allocator: Option<&vk::AllocationCallbacks>,
        version: vk::Version,
    ) -> result::Result<Device, LoaderError> {

Do you have a use case for using the function pointers directly where this important? I'm hesitant to change this behaviour since I was regarding the function pointers as an implementation detail.

My use case is rather peculiar: developing a Vulkan layer.
A layer intercepts calls to functions including vkCreateDevice. For correctness I ensure their signature matches the function pointer definition by casting (e.g. my_create_device as FnCreateDevice).

Ah ok!
On reflection, I am still not sure if I should push this kind of spec detail into the function pointer types.  For example, for your use case, Vulkan applications written in other languages may call into your layer with a null handle even though the spec requires a non-null one, so I think the function pointer has to use types that cover all possible inputs, not only ones that are expected?
For the same reason, the callbacks use pointers (which can be null) with the wrappers making use of references (which cannot).

Vulkan applications written in other languages may call into your layer with a null handle even though the spec requires a non-null one

It's certainly possible for apps to violate the contract, although that's no different from other contract violation like passing freed or invalid pointers. The Vulkan spec allows undefined behavior if the VUIDs are violated.
That said, you're right that encoding API contracts into function pointers isn't a good idea. Function pointers are just a translation of the C API declarations and nothing more. Something like a validation layer might also want to receive function calls with violations without incurring UB immediately.
I'll close this for now.


Fix jwksURI base64 encoding

The JWKS structure consists of an array of JWK entries named "keys".

Key
Contents

n
The modulus (2048 bit) of the key - Base64URL encoded.

Whenever our authn authentication server fails and after restarting istio-ingressgateway, the following failures occur:

Istiod error log:
ADS:LDS: ACK ERROR istio-ingressgateway-8654477747-dzkmm.istio-system-1 Internal:Error adding/updating listener(s) <IP_ADDRESS>_8080: Provider 'origins-0' in jwt_authn config has invalid local jwks: Jwks RSA [n] or [e] field is missing or has a parse error
Istio-ingressgateway error log(restart istio-ingressgateway):
2022-06-08T13:20:27.514937Z	warn	Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 2 successful, 0 rejected; lds updates: 0 successful, 2 rejected

This bug causes envoy to fail to parse local_jwks properly, because the n dynamically generated by jwks before this is not base64 encoded.

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Hi @hexiaodai. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

@hexiaodai: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
2e71255212df09f97202db00b87e9e4e6e62cad8
link
true
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

what changed? was it always broken? envoy became more strict?

The branch has no chance to run in normal case, we need an integration test to cover it

Why is it not failing in the first push before ingress restart?

I guess during this time, there is no xds push

what changed? was it always broken? envoy became more strict?

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
...
spec:
  jwtRules:
  - forwardOriginalToken: true
    issuer: http://ip:port/auth
    jwksUri: http://ip:port/auth/xxx/certs

Whenever my jwksuri fails, restarting istio ingressgateway will fail to start normally.
yes.
The JWKS structure consists of an array of JWK entries named "keys". Previously, the N dynamically generated by jwks was not Base64 encoded.
| Key | Contents |
| :-- | :-- |
| n | The modulus (2048 bit) of the key - Base64URL encoded. |

Why is it not failing in the first push before ingress restart?

Because my authn authentication server works normally during the first push

@hexiaodai Can you add a release note, we need to backport this fix

@hzxuzhonghu How do I add a release note? I'm not familiar with the operation of github. 😂

https://github.com/istio/istio/tree/master/releasenotes

@hexiaodai Can you add a release note, we need to backport this fix

@hzxuzhonghu a release note has been added😂

auto cherry-pick seems not work...
/cherry-pick release-1.14
/cherry-pick release-1.13

@kebe7jun: new pull request created: #41082

In response to this:

auto cherry-pick seems not work...
/cherry-pick release-1.14
/cherry-pick release-1.13

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/cherry-pick release-1.13

@kebe7jun: new pull request created: #41083

In response to this:

/cherry-pick release-1.13

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Investigate transaction metadata readiness

Sync with @AlexBHarley to understand current state of transaction metadata. If it's ready for prod use, we can use it to signal the Finclusive ID of a user during a cUSD transfer to the Finclusive wallet. If it's not ready, we'll likely create a new API that Valora can call to pass this information out of band.

Checked with Alex, it won't be ready for prod use for sure. We either have to rely on tx comments or create a new API indeed.


Fix README for zsh

Very minor change.
When running the example with zsh, it tries to expand and fails, adding quotes fix it.

Yeah, but zsh user can do that by their own, right? Also, the expansion is optional and I normally disable it.

So I’m not so willing to include this.


Deferred Tasks

Convert to deferred task:

[ ] query download zip
[ ] queue generation

to fix #484

Also, score export should be on that list.
@alvinwan  We will have the deferred task write to GCS, right?


[features request] add the hability to set Jib registryMirrors

Requirement
hability to set registryMirrors in maven jib plugin to abel to download image from mirror.
Problem
actualy you need to set proxy setting if camel-k is not connected to internet, event that, is you pull to much on repository like docker hub you take pull rate error... usualy you use repository mirror like "distibution" or "gohabor" to avoid that...
you can easily do that on kubernetes, but when camel build they use docker less method in docker to do that and not k8s mecanisme...
Proposal
adding options in install phase that map file like maven settings (in form of config map) for jib json config file:
https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#global-jib-configuration
like https://camel.apache.org/camel-k/2.3.x/installation/advanced/maven.html#maven-settings
Open questions
No response

Thanks for the feature request. It looks like a good improvment.
We could probably leverage the map "PublishStrategyOptions" (--build-publish-strategy-option) available in the IntegrationPlatform CRD and use the value in https://github.com/apache/camel-k/blob/main/pkg/builder/jib.go. It was used in the past for Kaniko publish strategy configuration (the strategy has been removed).
Then it would be a matter of being able to use config (or any other name) publish property with some value [configmap|secret]:name[/key].
In the end Camel K would be installed with this platform configuration to be used.
Feel free to contribute to the feature if you can.

Probably we need to develop a generic feature that would allow the configuration of multiple JIB parameters. We'd need to make some analysis and thing the best approach to make it as much independent as possible.


When invalid value are set for workload-memory-default/workload-memory-maximum,they get set with only the numeric part of the string.

acorn version v0.5.0-rc3-18-gfbe46a2+fbe46a25
When invalid value are set for --workload-memory-default/workloadMemoryMaximum ,they get set with only the numeric part of the string.
acorn install --workload-memory-default=5c  ==> This results in workload-memory-default set to 5.
This the same case with workload-memory-maximum as well.
Expected behavior is to provide user with an error message.

Related to https://github.com/acorn-io/acorn/issues/1086 and more generally related to our use of cuelang. Putting in the aml rewrite bucket.


Consider supporting older Clojure versions

Perceived Problem
Babashka http-client relies on relatively recent Clojure fns:
Added in 1.11:

random-uuid

Added in 1.10

ex-cause
ex-message

This can make it less attractive as a dependency for libraries that would like to support less recent versions of Clojure. In my case this is Etaoin, which supports Clojure 1.9 and above.
Proposal

Replace the above newer fns with Clojure 1.9 compatible code (changes are minor).
Decide the minimum version of Clojure http-client wants to support (is 1.9 good?) and add test coverage (including on CI) for supported versions.

Next Steps
Happy to do this work if you are interested in this change.

Roughly 5 years ago I supported 1.9 and 1.10. Now 1.11 came out and it's almost 1.12 time.
Perhaps we can move the lower range to 1.10? Supporting the last two CLJ versions seems reasonable?

That seems reasonable to me. I can bump min version of Clojure for Etaoin to 1.10 as well.


Index embedding of MAISI

I tried to write the unconditional sampler of MAISI myself and found the body region embeddings confusing.
Since the body region index is currently obtained from a mask (get_body_region_index_from_mask), what should top_region_index_tensor and bottom_region_index_tensor be when doing unconditional sampling?
@guopengf @Can-Zhao

Hi, the diffusion unet is trained to condition the body region and spacing. As explained in this tutorial, "The body regions are formatted as 4-dimensional one-hot vectors: the head and neck region is represented by [1,0,0,0], the chest region by [0,1,0,0], the abdomen region by [0,0,1,0], and the lower body region (below the abdomen) by [0,0,0,1]. " The top/bottom region index roughly reflects where the CT scan cut-off is.  For example, when top_region_index_tensor is [1,0,0,0] and bottom_region_index_tensor is  [0,0,1,0], the generated image should cover from the head neck region to the abdomen region.

Hi, the diffusion unet is trained to condition the body region and spacing. As explained in this tutorial, "The body regions are formatted as 4-dimensional one-hot vectors: the head and neck region is represented by [1,0,0,0], the chest region by [0,1,0,0], the abdomen region by [0,0,1,0], and the lower body region (below the abdomen) by [0,0,0,1]. " The top/bottom region index roughly reflects where the CT scan cut-off is. For example, when top_region_index_tensor is [1,0,0,0] and bottom_region_index_tensor is [0,0,1,0], the generated image should cover from the head neck region to the abdomen region.

@dongyang0122 These seems to be important parameters to explain for users as they do not exist in standard diffusion model. I was wondering if you would like to expand it in tutorial?


Text animation added

Hi,
This is with reference to issue number #125
I have tried this animation with the text in text present title box,
I am not sure whether it deserves to be merged or not, its up to you but for at least once have a look on it..

@marizvi we can definitely use this for some other part of the website! But closing for now.


Empty resultset .rows

Maybe this is not an issue but more like a question but could please advise why resultset.rows  and Raw are returned as empty array/never called? I am sure the table contains data and running the same query from the vsql gives the expected results. Vertica runs locally on VM via VMWare player.
The code I try to use is:
var Vertica, connection;

Vertica = require('vertica');

connection = Vertica.connect({
    user: "...",
    password: '...',
    database: "...",
    host: '<IP_ADDRESS>'
}, function (err) {
    var query;
    if (err) {
        throw err;
    }
    query = connection.query("SELECT * FROM nypd_motor_vehicle_collisions LIMIT 5");
    query.on('fields', function (fields) {
        return console.log("Fields:", fields);
    });
    query.on('row', function (row) {
        return console.log('Row:', row);
    });
    query.on('end', function (status) {
        return console.log("Finished!", status);
    });
    query.on('error', function (err) {
        return console.error("Error!", err);
    });
    return connection.query("SELECT * FROM nypd_motor_vehicle_collisions LIMIT 5", function (err, resultset) {
        return console.log('Query', err, resultset.fields, resultset.rows, resultset.status);
    });
});

row event never gets triggered nor is the Query console.log getting resultset.rows as anything else but empty array.

Are there any errors thrown in console? Is fields event getting fired with fields list?
Try setting connection.debug=true to get more details

@sudhakar No error event triggered but fields event gets fired correctly and the fields returned are correct as expected. Enabling debug does not help. Same thing only little more info at the end. Hope it will help to guide me in the right direction.
<= CommandComplete { status: 'SELECT' }
Finished! SELECT
<= ReadyForQuery { transactionStatus: 84 }

Thats wired. Are you sure you are not exiting the process before the query is complete (some where by explicitly calling process.exit() )
Try narrowing it down by using only the buffered version & isolating it to a separate file. RowDescription should be followed by DataRow before CommandComplete. If not, most likely there arent any selectable rows in that DB.

Yeah, this looks to me like the database is simply returning an empty resultset. can you try with the query SELECT 1?

I have an issue similar to the one above using the "node-vertica" package. An empty resultset for every row was returned. Fields returned as expected as well as "end". I have tried using a "node-jdbc" package. And I get a similar behaviour.
HP Vertica DB/Analytics Platform Version:  7.1.x
Type: Community edition, without license
NodeJS version: 0.12.0
Node-vertica version: The latest one in the repo using "npm install vertica"
OS version: Ubuntu 14.04 LTS
Virtual machine version: Parallels 10
Host OS: OS X 10.9.5
Machine: Mac Book Pro [2.6 GHz Intel Core i7, 16 GB DDR3]
Using SELECT 1, I got the output below
Fields: [ { name: '?column?',
tableOID: 0,
tableFieldIndex: 0,
typeOID: 6,
type: 'integer',
size: 8,
modifier:<PHONE_NUMBER>,
formatCode: 0,
decoder: [Function] } ]
[ 1 ]
It actually returned the "1" as a value.
Advise?

Not sure. This looks like something on the server side to me - the client does exactly what you expect it to do based on the messages the server is sending. Maybe this is a permissions or settings thing?
The only thing I can think of is the Vertica version. This package is tested against Vertica 7.0; maybe something has changed for 7.1? Unfortunately I do not have access to Vertica 7.1 myself at the moment.

Thanks - You are right. I figured out it's a server side issue. Tested against another vertica database. It worked. Agreed, it  could be a settings or permissions thing.

Closing this. Please let me know if you find out the issue on the server's end.

Finally found a solution via this link: http://stackoverflow.com/questions/27164238/why-does-vsql-can-return-all-the-records-while-program-using-odbc-driver-cant
Before loading an sql data script with INSERT statements, I set the AUTOCOMMIT ON in the Vsql client.
Like so:
STEP 1a: \set AUTOCOMMIT on
STEP 1b: Press Enter
STEP 2a: [db]=>\i data.sql
STEP 2b: Press Enter

Yes, in Vertica you have to COMMIT in order for inserted/updated/deleted data to take effect. If you're using node-vertica to insert data, you can also do connection.query('COMMIT") after inserting data.


Proxy authentication not sended

I have checked the following:

[X] I use the newest version of bruno.
[X] I've searched existing issues and found nothing related to my issue.

Describe the bug
I'm using bruno to call an api over a squid proxy with basic auth.
when I use bruno I had an error that login ar not sended.
.bru file to reproduce the bug
No response
Screenshots/Live demo link

@rricordeau Thank you for reporting this issue.
Could you provide some additional details to help us investigate further?

Was this functionality previously working in Bruno and has now started occurring in a newer version?
Are you configuring the proxy settings manually (refer to the screenshot below), or are you relying on the system-level proxy settings?

It never worked on my computer
I configure with the interface in your screenshot

I'm using version 1.33.0 Golden Edition

@rricordeau, are you able to successfully hit the API endpoint using curl? If so, I would recommend trying the System Proxy Mode in Bruno.
Also, is this an issue you're encountering specifically in Bruno, or has it worked fine in other API clients you've used? Just trying to narrow down the cause here.

Yes, in curl, I set the proxy with http://login:password@myproxy:3128 and it work without any trouble
In bruno, I can use a proxy without proxy auth without any problem.


flamethrower_alt_fire_oil

Even though it's not listed, the direct damage dealt needs to benefit from life-steal effects (33%)

Afterburn is counting now, but not the direct damage from the oil.


Add dependencies to berksfile

To ensure that Berkshelf pulls down the required cookbooks, I think they need to be added to the Berksfile. Which version should be required may be up for debate, but I just pulled the latest ones and put a pessimistic restriction on them.

@DavidJFelix thanks for the contribution!
Cookbook dependency versions should really live in the chef-atom's metadata file, and not in the Berksfile. If you like to submit a PR for that I'd be willing to merge it.
Thanks,
Doug Ireton

@dougireton I think the omission in the metadata was an oversight on my part I can add it for this PR withno problem. I'm a little new to chef, so maybe you could help me out here: Is there any reason why the version shouldn't be in both the metadata file and the Berksfile? If it depends on a given version, wouldn't we want berks to get that version?

As long as you have the metadata line in the Berksfile, Berkshelf will pick up the version constraints from the metadata.rb.

@dougireton This should be everything we discussed. Let me know if you'd prefer it squashed.

@dougireton just a quick ping/reminder on this.

@DavidJFelix thanks for the reminder. Just a couple of minor comments above.

@dougireton I made the changes as requested.

Thanks for the contribution @DavidJFelix. 👍

👍
]


Fix interjections at the end

Experimental sparsehistogram work. Nothing to see here unless you are @Dieterbe or @codesome .
Sadly, this isn't fixing the problem completely, but it is definitely a bug.
Apologies for changing comments close to the code changes. My OCD forces me to start sentences with a capital letter and to end them with a period.

Failing tests are the known ones. Merging.


Cleaner TP

This cleans the implementation of TP to use a mixin class (TPModule). If ghstack was working, this would be 2/3, so inference.py can be ignored safely.

Update: the first PR has landed, and this has been manually re-based a la ghstack, all the diffs are relevant now


Select should have a value prop

In plain Preact you can choose which option of a select element should be selected like this:
<select value="2">
  <option value="1">Foo</option>
  <option value="2">Bar</option>
</select>

Unless I've missed something, then the Select component in this package only lets you set the selected option by either setting the selectedIndex prop of Select itself or using the selected prop of Select.Item.
I think it would make sense to let Select behave like a regular select in this regard, what do you think?

If I'm correct, this is a current limitation by mdc or does this work with plain html?

Currently selectedIndex prop is an equivalent to value prop

Well selectedIndex takes an index not a value

ah gotcha... sorry was attention was divided back then
Lets keep this for 2.0?


Consolidate validation errors across rows

If there are many of the same validation error in consectutive rows, consider concatenating so it shows the range. This might show up oddly on the UI though, so it could be consolidated in a different way or addressed with a larger validation UI change.

Closing, since PR #116 resolves this issue for now. We may improve further on visualizing duplicated errors later, #144.


Analyze and Improve performance

[ ] Analyze what is consuming most memory, I/O and CPU(in order of priority) in José
[ ] Improve it
[ ] Repeat and comment any discovered issues

Reports from pympler after some minutes of uptime:
http://gateway.glop.me/ipfs/Qmd4pD9FZ5j5Qzn8ArpHhg7cQdtocTMfDnrYNR5d41j5BG/shot-2017-02-02-01:49:51.png
I'll run it after some hours and then make another report

Some digging with prospector showed a lot of things that weren't top notch.
https://github.com/lkmnds/jose/compare/b198f32...2c66bb1 were the changes focused on prospector findings, mostly related to remove star imports and unused imports.
Also ext_load is now splitted across 4 more functions, so it becomes easier to understand and maintain later on( https://github.com/lkmnds/jose/commit/28e636589a9caa455d265cf3699911ba7986ae52 )

pympler report after #6 implementation
RAM usage is down 47.3% !!11!1!!!
https://p.fuwafuwa.moe/ekexxg.png