New Project Proposal - GenevaERS

*** Name of project** (must be unique within OMP)
GenevaERS, the Single-Pass Optimization Engine.
*** Project description** (what it does, why it is valuable, origin and history)
GenevaERS--the open source version of The IBM Scalable Architecture for Financial Reporting (SAFR)--is designed to deal with the problem of efficiently reporting on large volumes of transactional data. It is based on a set of software components with previously patented technology embedded, and it may be customized for use with new or existing data extraction and reporting applications.
GenevaERS is often used when large amounts of operational data reside in the z/OS® environment and complex reporting requirements create workloads that are difficult to manage through traditional techniques or tools.  GenevaERS:

Runs quickly and efficiently
Scales to deal with large volumes of detail data and table joins
Creates multiple outputs, such as cubes, reports, and data marts
Helps mitigate issues with reconciliation and restatement through a single view of the data
Removes the need for additional transformations of the source data, allowing a simplified yet flexible and responsive data model
Enables organizations to process data on the System z™ platform where it resides

The GenevaERS solution helps support key corporate decision-making. It has a proven track record of addressing real business problems, such as account activity analysis, product pricing, fraud detection, and multi-dimensional sales revenue analysis. It has also been used to enhance existing investments in large, packaged enterprise applications and extend their capabilities. It accomplishes this by executing highly tuned batch reporting workloads on IBM z/OS and System z.
*** Statement on alignment with OMP charter mission**
GenevaERS has a deep history on z/OS.  As a service offering (as opposed to a tool supported by license sales) built through collaboration with implementing partners, it has been developed in a collaborative way.  It fills a space neglected by other offerings.  The additional step of sponsorship by OMP will help with further adoption and development of the tool.
The closest technology might be Apache Spark; GenevaERS's native z/OS machine code generation provides much greater performance and distinct processing functionalities.
*** Sponsor from TAC** (sponsor helps mentor projects)
Enyu Wang, IBM, Program Director,  Ecosystem Business Development & Strategy
*** Preferred maturity level** (see project stages)
Incubation Stage
*** License and contribution guidelines** (refer to the OMP guidelines)
Apache 2.0
*** Source control** (GitHub by default)
TBD, likely GitHub.  Currently using Rational toolsets internally in IBM.
*** External dependencies** (including licenses)
No external licenses.  IBM certificate of originality in place, and all IBM approvals for contribution to open source secured.  Existing customers notified.  Current maintenance team all employed by IBM.
*** Initial committers** (how long working on project)

Global Financial System SME: Kip Twitchell, continuous involvement for nearly 30 years
Product Manager: James Hladyshewsky, manager for over 10 years, implementation consultant another 10 years
Lead architect Randall Ness, continuous involvement over 30 years

*** Infrastructure requests** (CI / OMP Cluster)
To be determined.  Source code currently maintained in Rational toolset internally within IBM.
*** Communication channels** (slack, irc, mailing lists)
Slack channel named "sharealedger"  https://sharealedger.slack.com.
Active GenevaERS Alumni e-mail lists will be converted.
*** Issue tracker** (GitHub by default)
To be determined, currently using Rational toolset internally in IBM.
*** Website**
https://genevaers.org Additional thought leadership on shared ledgers will be developed within the https://sharealedger.org domain.
*** Release methodology and mechanics**
To be determined.   Currently use a permission access website for executable download by existing customers
*** Social media accounts**
Related blogging/vlogging performed by @finsysvlogger, Kip Twitchell, on twitter, linked-in, and at ledgerlearning.com (or financialsystemseducation.com) and on YouTube at Conversations with Kip (https://www.youtube.com/kiptwitchellvlog) and Sharealedger-org (https://www.youtube.com/channel/UCbogHQcugFgZK1QNoovNRHw)
*** Community size and any existing sponsorship**
IBM maintenance and knowledgeable staff on the order of a dozen people.  Currently approximately 25 - 30 GenevaERS/SAFR staff in Diaspora.  Two existing customers have various teams supporting the product, and extensible code which might be contributed to the project.  Growing community of Sharealedger participants over the last three months.

The SAFR solution will continue to be offered by IBM as a commercial version of GenevaERS with the addition that it will be zIIP processor enabled.  It will only be available to customers who have a maintenance contract for the product with IBM.  There are no other planned differences in the open source contribution.

This was approved by the TAC on 2020-07-09.
@KipTwitchell - should I create new repo for the project? genevaers work?

Could use some counsel here.  I just looked at Zowe, and tell me if I"m wrong, but I see it as an organization, rather than just a single repro?  I wonder if, given the scale of the code base we have, we won't need more than one repro, and there is probably a little planning that needs to go into that, but we probably need a community repro immediately.  Thoughts?

It might be good to understand the scope of code coming into the project to better understand things. I know you said it's internal in the process of open sourcing - perhaps you could showcase it to me somehow? We'll want to do a license scan anyways before it comes into a project managed repo.

Working on onboarding now.


Add academicons

I'd like to add academicons to the template and need to change head.html for that. What's the best way to proceed? Can I create a new head.html, add the line for the academicons, and somehow link to your agency repo for everything else in the head.html?
Thank you providing this starter template in the first place!

Yes, that's definitely doable!
You'll need to override the theme's head.html file, which you can find here.
In your repo, create the _includes folder and create head.html inside of it. Then, I would suggest copying the contents of this file into head.html and adding whatever lines of code you need to add academicons support.
Let me know if you have any more questions!

Feel free to reopen this if you have more questions.


bug: CORS error with PostHog

Issue Description
pypacktrends.com/:1 Access to XMLHttpRequest at 'https://us-assets.i.posthog.com/array/phc_p0ITJzZ8QM1sBKYur3ugA5kqgemya2DMEpccVw5KmMO/config?ip=1&_=1734448136967&ver=1.201.1' from origin 'https://pypacktrends.com' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

us-assets.i.posthog.com/array/phc_p0ITJzZ8QM1sBKYur3ugA5kqgemya2DMEpccVw5KmMO/config?ip=1&_=1734448136967&ver=1.201.1:1

       Failed to load resource: net::ERR_FAILED

#110 fixed


GraphQL error on develop

Hey folks,
Hope everyone is safe home and virus-free 🤞🏻
I started a new CosmicJs project and when I type gatsby develop I get the following output :
success open and validate gatsby-configs - 0.044s
success load plugins - 0.516s
success onPreInit - 0.004s
success initialize cache - 0.014s
success copy gatsby files - 0.104s
success onPreBootstrap - 0.027s
success createSchemaCustomization - 0.008s
Starting to fetch data from Cosmic JS (posts)
Starting to fetch data from Cosmic JS (settings)

 ERROR

(node:6745) Warning: Label 'Fetch Cosmic JS data' already exists for console.time()

 ERROR #11321  PLUGIN

"gatsby-source-cosmicjs" threw an error while running the sourceNodes lifecycle:

Request failed with status code 404

  Error: Request failed with status code 404

  - createError.js:16 createError
    [tomthedev-gatsby-cosmic]/[axios]/lib/core/createError.js:16:15

  - settle.js:17 settle
    [tomthedev-gatsby-cosmic]/[axios]/lib/core/settle.js:17:12

  - http.js:237 IncomingMessage.handleStreamEnd
    [tomthedev-gatsby-cosmic]/[axios]/lib/adapters/http.js:237:11

  - task_queues.js:84 processTicksAndRejections
    internal/process/task_queues.js:84:21

warn The gatsby-source-cosmicjs plugin has generated no Gatsby nodes. Do you need it?
success source and transform nodes - 0.379s
warn Plugin `gatsby-source-filesystem` tried to define built-in Gatsby GraphQL type `File`
success building schema - 0.264s

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "allCosmicjsPosts" on type "Query".

If you don't expect "allCosmicjsPosts" to exist on the type "Query" it is most likely a typo.
However, if you expect "allCosmicjsPosts" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "allCosmicjsPosts" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: gatsby-node.js:13:10

[
  GraphQLError: Cannot query field "allCosmicjsPosts" on type "Query".
      at Object.Field (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/graphql/validation/rules/FieldsOnCorrectType.js:53:31)
      at Object.enter (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/graphql/language/visitor.js:324:29)
      at Object.enter (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/graphql/language/visitor.js:375:25)
      at visit (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/graphql/language/visitor.js:242:26)
      at validate (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/graphql/validation/validate.js:73:24)
      at GraphQLRunner.validate (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/query/graphql-runner.js:53:44)
      at GraphQLRunner.query (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/query/graphql-runner.js:78:25)
      at /Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/bootstrap/graphql-runner.js:21:37
      at /Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/gatsby-node.js:16:7
      at Promise._execute (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/bluebird/js/release/debuggability.js:313:9)
      at Promise._resolveFromExecutor (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/bluebird/js/release/promise.js:488:18)
      at new Promise (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/bluebird/js/release/promise.js:79:10)
      at Object.exports.createPages (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/gatsby-node.js:13:10)
      at runAPI (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/utils/api-runner-node.js:252:37)
      at Promise.catch.decorateEvent.pluginName (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/utils/api-runner-node.js:371:15)
      at Promise._execute (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/debuggability.js:384:9)
      at Promise._resolveFromExecutor (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:518:18)
      at new Promise (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:103:10)
      at /Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/dist/utils/api-runner-node.js:370:12
      at tryCatcher (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/util.js:16:23)
      at Object.gotValue (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/reduce.js:166:18)
      at Object.gotAccum (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/reduce.js:155:25)
      at Object.tryCatcher (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/util.js:16:23)
      at Promise._settlePromiseFromHandler (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:547:31)
      at Promise._settlePromise (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:604:18)
      at Promise._settlePromise0 (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:649:10)
      at Promise._settlePromises (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/promise.js:729:18)
      at _drainQueueStep (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/async.js:93:12)
      at _drainQueue (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/async.js:86:9)
      at Async._drainQueues (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/async.js:102:5)
      at Immediate.Async.drainQueues (/Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/node_modules/gatsby/node_modules/bluebird/js/release/async.js:15:14)
      at processImmediate (internal/timers.js:456:21) {
    message: 'Cannot query field "allCosmicjsPosts" on type "Query".',
    locations: [ [Object] ]
  }
]

 ERROR #11321  PLUGIN

"gatsby-node.js" threw an error while running the createPages lifecycle:

Cannot read property 'allCosmicjsPosts' of undefined

  34 |
  35 |         // Create blog posts pages.
> 36 |         const posts = result.data.allCosmicjsPosts.edges;
     |                                   ^
  37 |
  38 |         each(posts, (post, index) => {
  39 |           const next = index === posts.length - 1 ? null : posts[index + 1].node;

File: gatsby-node.js:36:35

  TypeError: Cannot read property 'allCosmicjsPosts' of undefined

  - gatsby-node.js:36
    /Users/thomas/Dev/Projects/portfolio-project/tomthedev-gatsby-cosmic/gatsby-node.js:36:35

failed createPages - 0.111s
success createPagesStatefully - 0.145s
success onPreExtractQueries - 0.002s
success update schema - 0.052s

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "allCosmicjsPosts" on type "Query".

If you don't expect "allCosmicjsPosts" to exist on the type "Query" it is most likely a typo.
However, if you expect "allCosmicjsPosts" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "allCosmicjsPosts" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: src/pages/index.js:54:5

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "cosmicjsSettings" on type "Query".

If you don't expect "cosmicjsSettings" to exist on the type "Query" it is most likely a typo.
However, if you expect "cosmicjsSettings" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "cosmicjsSettings" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: src/pages/index.js:66:5

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "cosmicjsSettings" on type "Query".

If you don't expect "cosmicjsSettings" to exist on the type "Query" it is most likely a typo.
However, if you expect "cosmicjsSettings" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "cosmicjsSettings" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: src/components/layout.js:17:9

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "cosmicjsPosts" on type "Query".

If you don't expect "cosmicjsPosts" to exist on the type "Query" it is most likely a typo.
However, if you expect "cosmicjsPosts" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "cosmicjsPosts" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: src/templates/blog-post.js:124:5

 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "cosmicjsSettings" on type "Query".

If you don't expect "cosmicjsSettings" to exist on the type "Query" it is most likely a typo.
However, if you expect "cosmicjsSettings" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "cosmicjsSettings" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions

File: src/templates/blog-post.js:135:5

failed extract queries from components - 0.301s
success write out requires - 0.044s
success write out redirect data - 0.002s
success onPostBootstrap - 0.002s

info bootstrap finished - 7.570 s

success run queries - 0.057s - 5/5 87.84/s

You can now view gatsby-blog-cosmicjs in the browser.

  http://localhost:8000/

View GraphiQL, an in-browser IDE, to explore your site's data and schema

  http://localhost:8000/___graphql

Note that the development build is not optimized.
To create a production build, use gatsby build

success Building development bundle - 3.243s

My package.json
{
  "name": "gatsby-blog-cosmicjs",
  "description": "Gatsby Blog Cosmic JS",
  "version": "2.0.0",
  "author": "Jazib Sawar<EMAIL_ADDRESS>  "bugs": {
    "url": "https://github.com/jazibsawar/gatsby-blog-cosmicjs/issues"
  },
  "dependencies": {
    "dotenv": "^8.2.0",
    "gatsby": "^2.19.44",
    "gatsby-plugin-google-analytics": "^2.1.16",
    "gatsby-plugin-offline": "^3.0.7",
    "gatsby-plugin-react-helmet": "^3.1.7",
    "gatsby-plugin-typography": "^2.3.7",
    "gatsby-source-cosmicjs": "^1.0.1",
    "gatsby-source-filesystem": "^2.1.24",
    "lodash": "^4.17.15",
    "react": "^16.9.0",
    "react-dom": "^16.9.0",
    "react-helmet": "^5.2.1",
    "react-typography": "^0.16.19",
    "typeface-merriweather": "0.0.72",
    "typeface-montserrat": "0.0.75",
    "typography": "^0.16.19",
    "typography-theme-wordpress-2016": "^0.16.19"
  },
  "devDependencies": {
    "eslint": "^6.4.0",
    "eslint-plugin-react": "^7.14.3",
    "prettier": "^1.18.2"
  },
  "homepage": "https://github.com/gatsbyjs/gatsby-starter-blog#readme",
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "main": "n/a",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/gatsbyjs/gatsby-starter-blog.git"
  },
  "scripts": {
    "start": "gatsby build; gatsby serve --port $PORT",
    "dev": "gatsby develop",
    "lint": "./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .",
    "test": "echo \"Error: no test specified\" && exit 1",
    "format": "prettier --trailing-comma es5 --no-semi --single-quote --write 'src/**/*.js' 'src/**/*.md'",
    "develop": "gatsby develop",
    "build": "gatsby build",
    "fix-semi": "eslint --quiet --ignore-pattern node_modules --ignore-pattern public --parser babel-eslint --no-eslintrc --rule '{\"semi\": [2, \"never\"], \"no-extra-semi\": [2]}' --fix gatsby-node.js"
  }
}

I have a .env file populated with GITHUB_TOKEN, COSMIC_BUCKET and COSMIC_READ_KEY
Any thoughts on what I'm doing wrong ?

@thomasfaller Hope you're safe and doing great. By default, gatsby doesn't support dotenv integration. To use it, follow instructions here https://www.gatsbyjs.org/docs/environment-variables/.
Or for development purposes, you can hardcode values in gatsby-config.js.


When using RTL langage, Text at top and bottom of bubble staying at the same place when the bubble is inverting side

When my device is in Left To Right (LTR) :

When I try in RTL I manage to invert the bubble by changing the incomingBubble and outgoing bubble but my texts under the bubble and at the bottom remained at the same place..

How can I taking these texts to the right place??

+1

Thanks @jerembab ! 👍
Possible issues:

the labels are using explicit .textAlignment
the labels are not using leading / trailing constraints (but left / right)

Hello everyone!
I'm sorry to inform the community that I'm officially deprecating this project. 😢 Please read my blog post for details:
http://www.jessesquires.com/blog/officially-deprecating-jsqmessagesviewcontroller/
Thus, I'm closing all issues and pull requests and making the necessary updates to formally deprecate the library. I'm sorry if this is unexpected or disappointing. Please know that this was an extremely difficult decision to make. I'd like to thank everyone here for contributing and making this project so great. It was a fun 4 years. 😊
Thanks for understanding,
— jsq


Have a count filter

That only returns the first n cards.
this is hard to do because it has to run after the cards are sorted.
It probably makes sense to just have it be a preted configurable filter that returns a no op, and then collection.sortedCards special cases it in the pipeline
Wasn't this captured in another issue before?

Useful for #310 and #317

Note that techncially cardsThatWillBeRemovedOnPendingFilterCommit will return the wrong things... but consistently on before and after, so it's fine

Rename this to limit

[ ] Document in wiki


Respect DOTNET_ROOT in testing

Summary
Allow using DOTNET_ROOT to specify where to find dotnet host executable for testing.

Release Notes Entry

Looks like the OSX build machines have DOTNET_ROOT already set so the apps under test are being redirected to the machine installation rather than the repo installation. I'll close this for now, make adjustments, and open a new PR.


Explain how to leverage image policy markers.

As a Flux 1 user, we did not have these abstractions, and no comments were needed.
This is better!
I was aware of neither kyaml nor any other approach to comment templating on Kubernetes,
This paragraph would have helped me, and I think it can help others.
Signed-off-by: Jim Van Fleet<EMAIL_ADDRESS>

Re-file of #627


Dramatic performance up in some case

Fixed "on" function

Codecov Report

Merging #51 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master      #51   +/-   ##
=======================================
  Coverage   91.47%   91.47%
=======================================
  Files          10       10
  Lines         129      129
=======================================
  Hits          118      118
  Misses         11       11

Impacted Files
Coverage Δ

lib/methods/on.js
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 687fa04...003a95f. Read the comment docs.

Cool! Do you have any performance numbers? I thought these caching optimizations were done automagically by the engine … I guess I must've read something inaccurate.

Caching result of query
No, querySelectorAll is not cached. The DOM tree could change between calls in loop. In this case, the execution of the loop will be violated (the arrays will be different).
Performance
On most systems this is not noticeable, but on one of my working machines it is visually noticeable (side effect on another code). Perhaps this is due to some characteristics of the equipment or system settings.
Also, it is shown on IE and EDGE.

Before:

After:


Better handle gestures to open/close side menu

I used the previous version of the code about gesture and just modify the accumulated distance of the gesture since the vertical touch started (gestureState.dy).
After some tests, I found out for a basic vertical swipe that the dy vary between 0 and 10 (or -10 depending on the way you move your finger) .
So I allow the absolute value of y to vary between 0 and 10.
I tried this change on listview in the main view and it seems to works even in the real IOS Device.

Looks cool, thanks! I haven't time for making test now, but code looks correct.


Styling Question: How to set a different font size in a popup window?

I'm using the, slightly modified, example Stylesheet and would like to reduce the font size in the clipboard popup. However, all my attempts are in vain and the font size is unchanged:
/* -- base styles -- */

* {
  font-family: FiraCode Nerd Font Med;
  font-size: 16px;
  border: none;
  border-radius: 0;
}

box,
menubar,
button {
  background-color: @color_bg;
  background-image: none;
}

button,
label {
  color: @color_text;
}

button:hover {
  background-color: @color_bg_dark;
}

.popup {
  border: 1px solid @color_border;
  padding: 1em;
}

/* -- clipboard -- */

.clipboard {
  margin-left: 5px;
  font-size: 8px;
}

.popup-clipboard {
  font-size: 8px;
}

.popup-clipboard .item {
  font-size: 8px;
  padding-bottom: 0.3em;
  border-bottom: 1px solid @color_border;
}

I'm not familiar with css at all so i guess i am missing something. Why do my font-size: entries have no effect?

Hey, CSS has an idea of 'specificity', which can be quite tricky especially with GTK. What this effectively means is that you're setting the font size at the item level, which is probably a Box. The label element in that box then provides its own size rule, which is more specific than your rule, and so gets overriden.
The solution to this usually is to target a more exact element. You might be able to get away with eg .popup-clipboard .item label to target all elements. You can use ironbar inspect to open the GTK inspector and view the elements to get a visual idea, or try a different element referenced on the wiki.
If you still can't get anywhere, let me know and I can have a play when I get time to get a working example out.

You were right, this works:
.popup-clipboard .item label {
  font-size: 12px;
}

Thank you for your quick help!
Maybe something like your explanation could be added to the documentation. It is surely obvious to someone who knows how to deal with css but for me it wasn't obvious, because the documentation didn't list it as a stylable element.

If i knew how to do a PR i would probably know how to write css as well. Right now, all i can do on GitHub is nagging someone with my problems while trying not to reveal my ignorance too much ;-)
But i'm going to learn it as soon as i have a use for git, i've already started to dabble with the rust track on exercism.org.

Best way to learn is to jump in and figure it out as you go :) Don't worry about how much you don't know, people are always willing to lend a hand so long as you're willing to learn.
I went to add some text and found there actually is some guidance on the styling guide already:
https://github.com/JakeStanger/ironbar/blob/master/docs/Styling guide.md

You can also target all GTK widgets of a certain type directly using their name. For example, label will select all labels, and button:hover will select the hover state on all buttons. These names are all lower case with no separator, so MenuBar -> menubar.


Integrate Event Stream API with the  Event Gateway service

Currently the event controller module  (server/controllers/event.controller.js) is using the Producer  defined in server/helpers/Producer.js .  Replace this with the EvenStreamProducer in api/EventStream.js

Will be closed by #18


Logging tracking issue

See this comment
For starters, let's reduce logs levels in code paths exercised by the CLI so that it is less noisy.

I agree with @dpc's rules of thumb for the logging levels.
Personally, I only want to see the cli-output/cli-error and no log lines in my terminal and much rather open another tmux pane/terminal and do tail -f <logfile> to follow the log.
For example
$ fedimint-cli reissue AQAAAAAAA...
2022-10-08T13:02:58.337688Z  INFO request{peer=0}: mint_client::api: web socket not connected, reconnecting
2022-10-08T13:02:58.337874Z  INFO request{peer=1}: mint_client::api: web socket not connected, reconnecting
2022-10-08T13:02:58.337940Z  INFO request{peer=2}: mint_client::api: web socket not connected, reconnecting
2022-10-08T13:02:58.338008Z  INFO request{peer=3}: mint_client::api: web socket not connected, reconnecting
2022-10-08T13:02:58.338341Z  INFO request{peer=0}: jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "<IP_ADDRESS>", host_header: "<IP_ADDRESS>:4021", _mode: Plain, path_and_query: "/" }
2022-10-08T13:02:58.338430Z  INFO request{peer=2}: jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "<IP_ADDRESS>", host_header: "<IP_ADDRESS>:4001", _mode: Plain, path_and_query: "/" }
2022-10-08T13:02:58.338474Z  INFO request{peer=1}: jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "<IP_ADDRESS>", host_header: "<IP_ADDRESS>:4031", _mode: Plain, path_and_query: "/" }
2022-10-08T13:02:58.338496Z  INFO request{peer=3}: jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "<IP_ADDRESS>", host_header: "<IP_ADDRESS>:4011", _mode: Plain, path_and_query: "/" }
2022-10-08T13:02:58.406067Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
2022-10-08T13:02:58.406179Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
2022-10-08T13:02:58.406243Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
2022-10-08T13:02:58.406298Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
{
  "id": {
    "txid": "f66a10f716d91368c045535a53e1fe8c7e39f36d468aca189fbfe7b6670d627f",
    "out_idx": 0
  }
}

While we could reduce the log level to debug for mint_client::api, the logs of jsonrpsee would still show up, which originate in the library (correct?).
Assuming that's correct, I would argue for (1) logging everything a logfile and only showing the cli-ouput and cli-error, which also contains an error message.
Alternatively, (2) only logging warnings/errors to the command line with a multi sink setup as dpc already suggested in his comment.
I definitely agree on using info as default log level. Regarding the specific log message that the web socket is not connected, I'd probably keep it at level info (but I think you can judge that better), though, I don't want to see it while using the cli, imho.
What do you think?

A log file would be rather uncommon for a client/one-shot CLI tool imo.

While we could reduce the log level to debug for mint_client::api, the logs of jsonrpsee would still show up, which originate in the library (correct?).

We can set different log level for different modules/crates by default so that during normal operation nothing is logged. We should also log to stderr instead of stdout imo so it never messes with the JSON output. This would also allow redirecting stderr to a log file in cases where this is wanted (e.g. when running with RUST_LOG=debug for debugging).
A good default log level might be error,mint_client=info,fedimint_cli=info after we fix our log levels internally.

ah yes, that would be unusual for a one-shot cli tool.
ok, sounds good :+1:

FYI, the next jsonrpsee release will downgrade those logs to debug.


Adding service - yii\base\Module::config() isn't called

Description
Running php craft make service states to add a method public static function config() to the module in order to register the service as a component, but that method isn't used anywhere by craft

Additional info

Craft version: <IP_ADDRESS>
PHP version: 8.1.12

Thanks for reporting that!
It was a bug to recommend adding a config() method there – that’s a plugin-specific feature. Modules are configured via config/app.php. So you should change this:
'modules' => [
    'mymodule' => modules\mymodule\Module::class,
],

to:
'modules' => [
    'mymodule' => [
        'class' => modules\mymodule\Module::class,
        'components' => [
            'myService' => modules\mymodule\services\MyService::class,
        ],
    ],
],

Generator 1.6.0 is out now with a fix for this. Going forward, services that are generated for a module will output this at the end:


FileUtils.make: don't bypass superenv

The intention here was to make sure that make or gmake is called as appropriate; as long as paths are set up right, superenv shims should call through to the Homebrew copy as appropriate. This should also work in stdenv (e.g., Tiger).
Fixes Homebrew/homebrew-portable#31.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you successfully run brew tests with your changes locally?

cc @xu-cheng

:+1:

I think you need to update https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/make#L4 to:
exec xcrun "$0 "$@"

I think this may not work. As calling gmake who is symlinked to make in superenv is equivalent  to calling normal make

I think you need to update https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/make#L4

Good catch, changed.

I think you need to update https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/make#L4

Good catch, changed.

Unfortunately, I think this is incorrect. As it will break normal formulae. If a formula is calling gmake and bsdmake on modern macOS, superenv will change it to make, since both gmake and bsdmake do not exist on macOS. However, this change will break such behaviour.

Hm, yeah. That's not great for existing things that call gmake - do we know if that's something we regularly encounter?

Hm okay, yes it does; redirecting just gmake breaks cmake test executables, which do call gmake.


Try to simplify some methods

Coverage remained the same at 100.0% when pulling fa2440867d2560b290a88d583abeb5b22195ffdd on kruton:simplify into 4e4775239c523886e7978ed32e980f788163fb2c on connectbot:master.


Cannot Create/Update Issue with upload

Last week I changed fro 2.x to 4.0.0 (because the IDs can now be set again).
Everything works fine, but I cannot create/update issues with uploads.
exceptions:
Newtonsoft.Json.JsonWriterException: 'Token PropertyName in state ArrayStart would result in an invalid JSON object. Path 'issue.uploads'.'
'Newtonsoft.Json.JsonWriterException: 'No token to close. Path ''.'
source code
IList<Upload> uploads = new List<Upload>();
foreach(var kvp in files_for_upload) {
   var file = kvp.Key;

   try {
        byte[] documentData = System.IO.File.ReadAllBytes(file);
        var attachment = Manager.UploadFile(documentData);
        attachment.FileName = kvp.Value;
        attachment.Description = "bla bla";
        attachment.ContentType = MimeTypeMap.GetMimeType(Path.GetExtension(file));
        uploads.Add(attachment);
    }
    catch(Exception ex) {
    }

    CurrentIssue.Uploads = uploads;
    CurrentIssue = Manager.CreateObject(CurrentIssue);
    // neither works:
    Manager.UpdateObject(CurrentIssue.Id.ToString(), CurrentIssue);

attachement:
Token: 1933.fb5304ba9ef91bd4481af06822c2667efc30e0fc4a4f89d8c67e0039b1ac1bc7
ContentType: "image/png"
Description and FileName "blablabla"

I just downloaded the actual sources and stepped into the source code.
The exception occurs inside:
 writer.WriteProperty(RedmineKeys.TOKEN, Token);
at:
        public static void WriteProperty(this JsonWriter jsonWriter, string tag, object value)
        {
            jsonWriter.WritePropertyName(tag); // << HERE
            jsonWriter.WriteValue(value);
        }

variables are: tag='token' and value='1939.fb5304ba9ef91bd4481af06822c2667efc30e0fc4a4f89d8c67e0039b1ac1bc7'

Git it. You have to add th Object-Declaration. Following to CustomFields I changed Upload.cs:
        public void WriteJson(JsonWriter writer)
        {
            writer.WriteStartObject(); // <<<<< new

            writer.WriteProperty(RedmineKeys.TOKEN, Token);
            writer.WriteProperty(RedmineKeys.CONTENT_TYPE, ContentType);
            writer.WriteProperty(RedmineKeys.FILE_NAME, FileName);
            writer.WriteProperty(RedmineKeys.DESCRIPTION, Description);

            writer.EndStartObject(); // <<<<< new
        }

Why do you call writer.WriteStart/EndObject() inside the object class and not in the outer loop?


Openstack LB SG not updating after port changes

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
If a port is added to an existing k8s LB service, the SG is not updated to reflect the change thereby traffic to the new port will not be allowed (unless you manually update the SG yourself).
How to reproduce it (as minimally and precisely as possible):
1.) Create new k8s service LB.
2.) Add another port to the service, check the SG if it add a new rule for the new port
You can also check the gist:
https://gist.github.com/infinitydon/7d27767c6650520a5395703be1b1cf58#gistcomment-2998106
Cloud-config:
[LoadBalancer]
lb-version = "v2"
subnet-id = "c1509fca-1872-40b9-a7b5-5cc01d611d6f"
floating-network-id = "8bb661f5-76b9-45f1-9ef9-eeffcd025fe4"
lb-method = "ROUND_ROBIN"
manage-security-groups = true

Environment:

openstack-cloud-controller-manager version: How to check this specifically? Cluster was installed with kubeadm
OS (e.g. from /etc/os-release): Ubuntu bionic
Kernel (e.g. uname -a): 4.15.0-55-generic
Install tools: kubeone (kubeadm v1.14.2)

N.B -- I tested this on other cloud providers like AWS and GCP, port changes are always reflected in the SG.

@infinitydon are you using Octavia or Neutron-LBaaS?

@lingxiankong I am using Neutron-LBaaS

Hi @infinitydon, please see my comments in https://github.com/kubernetes/cloud-provider-openstack/issues/628, currently we are lack of contributors who are using Neutron-LBaaS with openstack-cloud-controller-manager :-(

Can you confirm if Octavia is not affected with this issue? I currently don't have access to an octavia based deployment..

@infinitydon https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/provider-configuration.md#load-balancer-optional-parameters

When using Octavia, the worker nodes and the Octavia amphorae are usually in the same subnet, so it's sufficient to config the port security group rules manually for worker nodes, to allow the traffic coming from the the subnet IP range to the node port range(usually it is 30000-32767). As a result, manage-security-groups is not needed when using Octavia.

Ok.. But I will like to suggest that the feature should be looked at as this works very well with the public clouds, I have tested this on AWS, GCP, any update to a k8s LB service ports will trigger the appropriate changes in the security group that is associated with that LB.


[AppGateway] Max number of path-based rules per URL Map?

What is the maximum count of path-based rules that could be added to a single URL Map in Application Gateway configuration?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1cd57b6d-3c6f-e546-b3ea-f83892f4c23d
Version Independent ID: 27aecc7b-f107-96d4-b044-ae026bca2727
Content: Azure subscription limits and quotas
Content Source: articles/azure-subscription-service-limits.md
Service: billing
GitHub Login: @rothja
Microsoft Alias: byvinyal

@arkein Thanks for the Comment. We are actively investigating and will get back to you soon.

@rothja unable to find any reference to Pathbasedrule quota in any of the published docs. The only quota reference I see is for "loadbalancing rules" in the above doc. Is this the right quota?
https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#application-gateway-limits

I think this would be a question for @vhorne, who is the author on most off the Application Gateway content from what I can see. Vic, can you help out here? Thank you!

thanks @rothja
assigned to Vic for guidance.

@vhorne any update on this please?

@femsulu
The answer is 100, and the article is being updated.

@arkein FYI

#please-close


No support for optimum-habana pipeline() causes error during inference for PyTorch BERT finetuned model using dtype bf16

System Info
optimum-habana 1.5.0
docker version 1.9.0
pytorch version 1.13.1

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction
During the inference of bert (bert-large-uncased), finetuned on Financial PhraseBank dataset with bf16 data type, it results in an error.
The finetuning on Gaudi (HPU) is done with the help of optimum-habana library.
The transformers (4.28.1) and supporting libraries are installed as part of optimum-habana installation.
The finetuning works well for both data type (bf16 and fp32). The inference works well on fp32 data type. But when inference is done on bf16, it results in error.
The finetuning code is present here in finbert.py file.
import sys
import subprocess

subprocess.check_call([sys.executable, '-m', 'pip', 'install',
'numpy', ' pandas', ' scikit-learn', 'datasets', 'optimum.habana', '--user'])

import pandas as pd
import numpy as np
from transformers import AutoConfig, AutoTokenizer, AutoModelForSequenceClassification
from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from datasets import Dataset

def load_data():
    df = pd.read_csv(
        'FinancialPhraseBank-v1.0/Sentences_50Agree.txt',
        sep='@',
        names=['sentence', 'label'],
        encoding = "ISO-8859-1")
    df = df.dropna()
    df['label'] = df['label'].map({"neutral": 0, "positive": 1, "negative": 2})
    df.head()

    df_train, df_test, = train_test_split(df, stratify=df['label'], test_size=0.1, random_state=42)
    df_train, df_val = train_test_split(df_train, stratify=df_train['label'],test_size=0.1, random_state=42)

    dataset_train = Dataset.from_pandas(df_train, preserve_index=False)
    dataset_val = Dataset.from_pandas(df_val, preserve_index=False)
    dataset_test = Dataset.from_pandas(df_test, preserve_index=False)

    return dataset_train, dataset_val, dataset_test

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    return {'accuracy': accuracy_score(predictions, labels)}

def main():
    dataset_train, dataset_val, dataset_test = load_data()

    bert_model = AutoModelForSequenceClassification.from_pretrained('bert-large-uncased', num_labels=3)
    bert_tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')

    dataset_train = dataset_train.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
    dataset_val = dataset_val.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
    dataset_test = dataset_test.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length' , max_length=128), batched=True)

    dataset_train.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
    dataset_val.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
    dataset_test.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])

    args = GaudiTrainingArguments(
        output_dir='temp/',
        overwrite_output_dir=True,
        evaluation_strategy='epoch',
        save_strategy='no',
        logging_strategy='epoch',
        logging_dir='logs/',
        report_to='tensorboard',

        learning_rate=2e-5,
        per_device_train_batch_size=8,
        per_device_eval_batch_size=4,
        num_train_epochs=5,
        weight_decay=0.01,
        metric_for_best_model='accuracy',

        use_habana=True,                        # use Habana device
        use_lazy_mode=True,                     # use Gaudi lazy mode
        use_hpu_graphs=True,                    # set value for hpu_graphs
        gaudi_config_name='gaudi_config.json',  # load config file
    )

    trainer = GaudiTrainer(
        model=bert_model,                   # the instantiated 🤗 Transformers model to be trained
        args=args,                          # training arguments, defined above
        train_dataset=dataset_train,        # training dataset
        eval_dataset=dataset_val,           # evaluation dataset
        compute_metrics=compute_metrics
    )

    trainer.train()

if __name__ == '__main__':
    main()

It also needs a gaudi_config.json file which has details for bf16 dtype training.
The gaudi_config.json file is:
{
  "use_habana_mixed_precision": true,
  "hmp_is_verbose": false,
  "use_fused_adam": true,
  "use_fused_clip_norm": true,
  "hmp_bf16_ops": [
    "add",
    "addmm",
    "bmm",
    "div",
    "dropout",
    "gelu",
    "iadd",
    "linear",
    "layer_norm",
    "matmul",
    "mm",
    "rsub",
    "softmax",
    "truediv"
  ],
  "hmp_fp32_ops": [
    "embedding",
    "nll_loss",
    "log_softmax",
    "cross_entropy"
  ]
}

Note: Keep both finbert.py and gaudi_config.json files in same folder.
Run it with command:
export MASTER_ADDR="localhost"
export MASTER_PORT="12345"
mpirun -n 8 --bind-to core --map-by socket:PE=4 --rank-by core --report-bindings --allow-run-as-root python finbert.py
Note: It can also be finetuned on 1 card for debugging purpose.
After completing the finetuning on bf16 dtype, next while running the inference code either code-1 or code-2, it results in error.
Inference code-1:
from transformers import pipeline
device=torch.device('hpu')
pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device)
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))

Inference code-2:
from transformers import TextClassificationPipeline
pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
pipe.device=torch.device('hpu')
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))

Error seen after inference:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1146, in _LazyModule._get_module(self, module_name)
   1145 try:
-> 1146     return importlib.import_module("." + module_name, self.__name__)
   1147 except Exception as e:File /usr/lib/python3.8/importlib/__init__.py:127, in import_module(name, package)
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)File <frozen importlib._bootstrap>:1014, in _gcd_import(name, package, level)File <frozen importlib._bootstrap>:991, in _find_and_load(name, import_)File <frozen importlib._bootstrap>:975, in _find_and_load_unlocked(name, import_)File <frozen importlib._bootstrap>:671, in _load_unlocked(spec)File <frozen importlib._bootstrap_external>:848, in exec_module(self, module)File <frozen importlib._bootstrap>:219, in _call_with_frames_removed(f, *args, **kwds)File /usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py:56
     51 # Fused kernels
     52 # Use separate functions for each case because conditionals prevent kernel fusion.
     53 # TODO: Could have better fused kernels depending on scaling, dropout and head mask.
     54 #  Is it doable without writing 32 functions?
     55 @torch.jit.script
---> 56 def upcast_masked_softmax(
     57     x: torch.Tensor, mask: torch.Tensor, mask_value: torch.Tensor, scale: float, softmax_dtype: torch.dtype
     58 ):
     59     input_dtype = x.dtypeFile /usr/local/lib/python3.8/dist-packages/torch/jit/_script.py:1343, in script(obj, optimize, _frames_up, _rcb, example_inputs)
   1342     _rcb = _jit_internal.createResolutionCallbackFromClosure(obj)
-> 1343 fn = torch._C._jit_script_compile(
   1344     qualified_name, ast, _rcb, get_default_args(obj)
   1345 )
   1346 # Forward docstringsFile /usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py:863, in try_compile_fn(fn, loc)
    862 rcb = _jit_internal.createResolutionCallbackFromClosure(fn)
--> 863 return torch.jit.script(fn, _rcb=rcb)File /usr/local/lib/python3.8/dist-packages/torch/jit/_script.py:1343, in script(obj, optimize, _frames_up, _rcb, example_inputs)
   1342     _rcb = _jit_internal.createResolutionCallbackFromClosure(obj)
-> 1343 fn = torch._C._jit_script_compile(
   1344     qualified_name, ast, _rcb, get_default_args(obj)
   1345 )
   1346 # Forward docstringsRuntimeError:
Unknown type name 'DType':
  File "/usr/local/lib/python3.8/dist-packages/habana_frameworks/torch/hpex/hmp/utils.py", line 1811
def softmax(input: Tensor, dim: Optional[int] = None, _stacklevel: int = 3, dtype: Optional[DType] = None) -> Tensor:
                                                                                ~~~~~ <--- HERE
    r"""Applies a softmax function.
'softmax' is being compiled since it was called from 'upcast_masked_softmax'
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py", line 62
    x = x.to(softmax_dtype) * scale
    x = torch.where(mask, x, mask_value)
    x = torch.nn.functional.softmax(x, dim=-1).to(input_dtype)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    return x
The above exception was the direct cause of the following exception:RuntimeError                              Traceback (most recent call last)
Cell In[26], line 3
      1 from transformers import pipeline
      2 device=torch.device('hpu')
----> 3 pipe = pipeline("text-classification", model=trainer.model, tokenizer=bert_tokenizer, device=device)
      4 #pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
      5 #pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
      6 #pipe.device=torch.device('hpu')
      8 print(pipe("Alabama Takes From the Poor and Gives to the Rich"))File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/__init__.py:979, in pipeline(task, model, config, tokenizer, feature_extractor, image_processor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    976 if device is not None:
    977     kwargs["device"] = device
--> 979 return pipeline_class(model=model, framework=framework, task=task, **kwargs)File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/text_classification.py:85, in TextClassificationPipeline.__init__(self, **kwargs)
     82 def __init__(self, **kwargs):
     83     super().__init__(**kwargs)
---> 85     self.check_model_type(
     86         TF_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING
     87         if self.framework == "tf"
     88         else MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING
     89     )File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py:942, in Pipeline.check_model_type(self, supported_models)
    940 if not isinstance(supported_models, list):  # Create from a model mapping
    941     supported_models_names = []
--> 942     for config, model in supported_models.items():
    943         # Mapping can now contain tuples of models for the same configuration.
    944         if isinstance(model, tuple):
    945             supported_models_names.extend([_model.__name__ for _model in model])File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:644, in _LazyAutoMapping.items(self)
    643 def items(self):
--> 644     mapping_items = [
    645         (
    646             self._load_attr_from_module(key, self._config_mapping[key]),
    647             self._load_attr_from_module(key, self._model_mapping[key]),
    648         )
    649         for key in self._model_mapping.keys()
    650         if key in self._config_mapping.keys()
    651     ]
    652     return mapping_items + list(self._extra_content.items())File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:647, in <listcomp>(.0)
    643 def items(self):
    644     mapping_items = [
    645         (
    646             self._load_attr_from_module(key, self._config_mapping[key]),
--> 647             self._load_attr_from_module(key, self._model_mapping[key]),
    648         )
    649         for key in self._model_mapping.keys()
    650         if key in self._config_mapping.keys()
    651     ]
    652     return mapping_items + list(self._extra_content.items())File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:616, in _LazyAutoMapping._load_attr_from_module(self, model_type, attr)
    614 if module_name not in self._modules:
    615     self._modules[module_name] = importlib.import_module(f".{module_name}", "transformers.models")
--> 616 return getattribute_from_module(self._modules[module_name], attr)File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:561, in getattribute_from_module(module, attr)
    559 if isinstance(attr, tuple):
    560     return tuple(getattribute_from_module(module, a) for a in attr)
--> 561 if hasattr(module, attr):
    562     return getattr(module, attr)
    563 # Some of the mappings have entries model_type -> object of another model type. In that case we try to grab the
    564 # object at the top level.File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1136, in _LazyModule.__getattr__(self, name)
   1134     value = self._get_module(name)
   1135 elif name in self._class_to_module.keys():
-> 1136     module = self._get_module(self._class_to_module[name])
   1137     value = getattr(module, name)
   1138 else:File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1148, in _LazyModule._get_module(self, module_name)
   1146     return importlib.import_module("." + module_name, self.__name__)
   1147 except Exception as e:
-> 1148     raise RuntimeError(
   1149         f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
   1150         f" traceback):\n{e}"
   1151     ) from eRuntimeError: Failed to import transformers.models.gpt_bigcode.modeling_gpt_bigcode because of the following error (look up to see its traceback):Unknown type name 'DType':
  File "/usr/local/lib/python3.8/dist-packages/habana_frameworks/torch/hpex/hmp/utils.py", line 1811
def softmax(input: Tensor, dim: Optional[int] = None, _stacklevel: int = 3, dtype: Optional[DType] = None) -> Tensor:
                                                                                ~~~~~ <--- HERE
    r"""Applies a softmax function.
'softmax' is being compiled since it was called from 'upcast_masked_softmax'
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py", line 62
    x = x.to(softmax_dtype) * scale
    x = torch.where(mask, x, mask_value)
    x = torch.nn.functional.softmax(x, dim=-1).to(input_dtype)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    return x

Expected behavior
Inference is expected to work well in bf16 just like fp32 dtype.
Below shown output is expected
[{'label': 'neutral', 'score': 0.9094224572181702}]
[{'label': 'positive', 'score': 0.9752092957496643}]

Hi @hchauhan123! This is actually not a bug as Optimum Habana does not support Transformers' pipelines at the moment.
If you want to run inference on your test set, I recommend that you add trainer.evaluate() right after trainer.train() and this should work. Here is more information regarding running inference on Gaudi using the library.

Hi @regisss ,
So, I tried 2 methods just after the trainer.train() for inference.
(1) trainer.evaluate(eval_dataset=dataset_test) which gave me below overall summary
{'eval_loss': 0.90251624584198,
 'eval_accuracy': 0.8577319587628865,
 'eval_runtime': 5.853,
 'eval_samples_per_second': 82.864,
 'eval_steps_per_second': 20.844,
 'epoch': 5.0,
 'memory_allocated (GB)': 7.49,
 'max_memory_allocated (GB)': 9.99,
 'total_memory_available (GB)': 30.24}

(2) trainer.predict(dataset_test).metrics
Output summary:
{'test_loss': 0.90251624584198,
 'test_accuracy': 0.8536082474226804,
 'test_runtime': 1.2496,
 'test_samples_per_second': 388.11,
 'test_steps_per_second': 97.628}

This means at least here the model is able to do inference when finetuned above with bf16 dtype.
But I am unable to do what pipleine() helped me achieve. I am not sure how to provide a single sentence or input/ test data and get its visualization with label and score.
I understand here with trainer.evaluate() and trainer.predict() the summery is for whole test dataset.
dataset_test[1] does not help either to select only one test data entry.
Also, can you help me understand why would the above pipeline() work for fp32 dtype and not for fp16?

@hchauhan123 You're using a BERT model but the error refers to a GPTBigCode architecture, that's weird. I can tell you that I've observed the same error when using mixed-precision with GPTBigCode. This is something we are aware of and that we are going to investigate :slightly_smiling_face:
With BERT it should work though...
Could you try to add torch_dtype=torch.bfloat16 to your pipeline such as:
import torch
from transformers import pipeline
device=torch.device('hpu')
pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device, torch_dtype=torch.bfloat16)
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))

please?

@regisss I re-ran by adding torch_dtype=torch.bfloat16  and it still gives the same error as we saw earlier duing bf16 dtype.
RuntimeError: Failed to import transformers.models.gpt_bigcode.modeling_gpt_bigcode because of the following error
(look up to see its traceback):

Unknown type name 'DType':
  File "/usr/local/lib/python3.8/dist-packages/habana_frameworks/torch/hpex/hmp/utils.py", line 1813
def softmax(input: Tensor, dim: Optional[int] = None, _stacklevel: int = 3, dtype: Optional[DType] = None) ->
Tensor:
                                                                                ~~~~~ <--- HERE
    r"""Applies a softmax function.
'softmax' is being compiled since it was called from 'upcast_masked_softmax'
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py", line 62
    x = x.to(softmax_dtype) * scale
    x = torch.where(mask, x, mask_value)
    x = torch.nn.functional.softmax(x, dim=-1).to(input_dtype)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    return x

Also, I feel the error is dependent on version of transformer model installed. Just before doing the inference, I tried to install a different version of transformer (4.20.1) to override the transformer (4.28.1) which comes when installing optimum-habana, and the error was in different module. But again it is weird that why would it fail in some different model when I am using BERT. Again happens only during bf16.
Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1146, in _get_module
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers.models.ernie.modeling_ernie'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3505, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_79/1761011136.py", line 5, in <module>
    pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/text_classification.py", line 85, in __init__
    if isinstance(top_k, int) or top_k is None:
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py", line 942, in check_model_type
    raise NotImplementedError("postprocess not implemented")
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 644, in items
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 647, in <listcomp>
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 616, in _load_attr_from_module
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 561, in getattribute_from_module
    return self._extra_content[key]
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1136, in __getattr__
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1148, in _get_module
RuntimeError: Failed to import transformers.models.ernie.modeling_ernie because of the following error (look up to see its traceback):
No module named 'transformers.models.ernie.modeling_ernie'

During handling of the above exception, another exception occurred:

@regisss I re-ran by adding torch_dtype=torch.bfloat16 and it still gives the same error as we saw earlier duing bf16 dtype.

That's strange. Have you run this code snippet in a script where a GaudiTrainer object was already instantiated? If yes, could you try after commenting/removing the trainer instantiation?
I'm going to try to reproduce it on my side.

I am basically running this in a jupyter notebook. So yes, in my notebook, in a cell the GaudiTrainer object was already instantiated and then the finetuning was done. The inference code is just after that in another cell.

I see. Can you try running the following code snippet please?
import torch
from transformers import pipeline
from habana_frameworks.torch.hpex import hmp

device=torch.device('hpu')
pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device, torch_dtype=torch.bfloat16)

with hmp.disable_casts():
    print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
    print(pipe("Economists are predicting the highest rate of employment in 15 years"))

It gives the same error as posted above. No change.

Hmm. And this?
import torch
from habana_frameworks.torch.hpex import hmp

device=torch.device('hpu')

with hmp.disable_casts():
    from transformers import pipeline
    pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device, torch_dtype=torch.bfloat16)
    print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
    print(pipe("Economists are predicting the highest rate of employment in 15 years"))

I have the feeling that from transformers import pipeline is the culprit here. It imports several architectures, which would explain why you get errors which are not related to your model.

Again, the same error. Yes, that could be the case.
And what about if I use TextClassificationPipeLine (code-2 for inference). I believe since it is still sitting above pipleline() hence same issue occurs even there, right?

You can try, but I think a similar error will be raised, although with another imported architecture probably.
I'm going to see if I can reproduce it.

I meant, I have tried that too earlier and saw the same error even with TextClassificationPipeLine.

I meant, I have tried that too earlier and saw the same error even with TextClassificationPipeLine.

Yes I'm not surprised by this.
I managed to reproduce this error, I'm going to investigate it and will let you know when I find something.

Okay, so I managed to make it work with:
import torch
torch.jit._state.disable()

device=torch.device('hpu')

from transformers import pipeline
pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device)
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))

Could you try it?

Awesome. That works. Yes, the torch.jit seems not supported. So disabling that works.

Great :tada:
This solution is a bit hacky, but hopefully this should not be needed soon as we are going to use native PyTorch Autocast for managing mixed precision (the PR is open here: https://github.com/huggingface/optimum-habana/pull/226). So that should not interfere with pipelines.
Another remark if you would like to improve inference speed. This code snippet
import torch
torch.jit._state.disable()

from transformers import pipeline
from habana_frameworks.torch.hpex import hmp

device=torch.device('hpu')

with hmp.disable_casts():
    pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device, torch_dtype=torch.bfloat16)
    print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
    print(pipe("Economists are predicting the highest rate of employment in 15 years"))

will probably give better latency and throughput as the model is fully casted to bf16. Whereas the current way uses mixed precision (i.e. a mix of bf16 and fp32).
Let me know if we can close this issue!

Yes, please close the issue.


[3102] - Background mail notifications

Context

We have a new Sidekiq background container running, similar to geocoding.

Changes proposed in this pull request

Mail is sent out via a background task.

Checklist

[ ] Make sure all information from the Trello card is in here
[ ] Attach to Trello card
[ ] Rebased master
[ ] Cleaned commit history
[ ] Tested by running locally

 [M] Background mail notifications


[FEATURE] Letter Page size support

Hello,
Since many US companies use Letter page size. Is it possible to add page splitting for Letter page size?
Thanks

Not stale, hope to add this feature soon.

Is there a method to change the page size to Letter using CSS?

This has now been implemented in v3.6.8.


Error Handling - Measurement list only containing errors

Hi Team,
Ran into an issue when passing measurements containing errors to the chart component, wondering if you could advise.
Background:
I've removed hardcoded filtering out of measurements with API errors. Currently all measurements that have received a response from the API are sent directly to the charting component whether or not the API response object contains errors.
Previously I was filtering out measurements containing these API errors, determining error state by checking the following fields for values:
json.measurement_calculated_values.corrected_measurement_error,
json.measurement_calculated_values.chronological_measurement_error,
json.plottable_data.centile_data.chronological_decimal_age_data.observation_value_error,
json.plottable_data.centile_data.corrected_decimal_age_data.observation_value_error,
json.plottable_data.sds_data.chronological_decimal_age_data.observation_value_error,
json.plottable_data.sds_data.corrected_decimal_age_data.observation_value_error
The Issue:
After removing the previous filtering of measurements with errors:

When I pass a list of measurements to the chart component, if at least one of them contains an error, it results in the chart component putting errors in the console log.

When I pass a list of measurements to the chart component, if all of them contain errors, it results in the charting component breaking + putting errors in the console log. (see attached screenshot)

I've also included a measurement containing an error, one that would previously be getting filtered out of the measurements list.
Request:
Looking to clarify if this is working as intended, or if the chart component should be processing these errors internally.
In situations where all the patient's measurements have errors, we show a placeholder separate to the charting component. I'm wondering if this should be implemented internally within the chart component (i.e. the chart component shows its own placeholder message describing why it can't render the chart), or if the chart component could have some sort of 'error callback' that we can hook into for displaying a placeholder with an appropriate description.
Any advice you can provide is much appreciated!
Using Ver 6.1.15
Cheers, Dan

api-data.json

Thanks @dmc-cambric I noticed that errors what not pulling through to the tool tips recently so this is something I am grateful here to you noticing.
I have started on som fixes which will go into a new version 7 where we are looking at a refresh on some aspects of the charts. This will be a priority.

Just starting to look at this - thanks for raising
The <RCPCHChart/> component is wrapped in an <ErrorBoundary/> component (thank you to @chvanlennep for the innovation).
This retrieves errors that would prevent that chart from rendering and hopefully mostly are for situations when the server is unavailable or the data are unplottable. Most of that error handling though should have been addressed upstream.
Any errors to do with age calculation or measurements which would be plottable but maybe need a health warning will not be caught here, but should be signposted in the tooltips in the charts.
The message you have posted as an example is changing - in the next update we will be plotting preterm babies >= 22 weeks, even when reference data does not exist. But other errors, such as requesting weight in Turner and so on should be caught.

Have you found errors that you would prefer were handled or implemented differently?
Do you have any suggestions on how we might better handle errors?

These changes brought in v7.0.0 seem really handy. I don't have any immediate ideas around changes to error handling, although if the we (or clients) come up with any, will be happy to raise separately


Multiple Environments per run

Guess I get to be the one to request this. Being able to specify multiple environment per run as referenced in the missing RoundhousE features document would be a big help to us. We have a complex set of environment spread across our datacenter and Azure supporting multiple customers.  Being able to specify multiple environments would let us use the environment names like tags and be able to run scripts against any environment matching that tag. We have a lot of environments that are very similar except for a couple of settings.  It would be nice to be able to run a a script across all the UAT environments (we have 1 per customer) without having to name each UAT environment individually in the script. We have actually hit the windows file path limit a few times trying to get every environment named when using RoundhousE.

Currently, grate only supports a single environment per run, but you can create multiple runs, each per environment.
My initial idea is to use the script (bash, powershell) to trigger multiple runs depending on the input variable. See the example below.
My script
#!/bin/sh
# https://erikbra.github.io/grate/getting-started/
# https://erikbra.github.io/grate/configuration-options/
# https://github.com/erikbra/grate/tree/main/examples/k8s/multitenancy
# databasetype <mariadb | oracle | postgresql | sqlite | sqlserver
for i in $(echo ${Database__Env:-DEV} | tr "," "\n")
do
  echo "Migrating to environment $i"
  ./grate --connectionstring "$Database__ConnectionString" \
        --sqlfilesdirectory /db \
        --environment ${i:-DEV} \
        --databasetype ${Database__Type:-sqlserver} \
        --schema ${Database__Schema:-grate} \
        --silent \
        --version ${VERSION:-1.0.0-beta1} \
        --createdatabase ${Database__CreateDatabase:-true} \
        --transaction ${Database__Transaction:-true} || exit 1
done
echo "Migration database completed."

Grate Dockerfile build file ( I use docker, but you can choose which fit in your use case):
FROM ubuntu:jammy as grate-downloader
WORKDIR /app
ARG GRATE_VERSION=1.6.2
ADD https://github.com/erikbra/grate/releases/download/$GRATE_VERSION/grate-linux-musl-x64-self-contained-$GRATE_VERSION.zip /tmp/grate.zip
RUN apt update && apt install unzip -y
# extract grate to current folder /app
RUN unzip -d ./ /tmp/grate.zip

FROM alpine:3 as grate-migrator
WORKDIR /app
# COPY the sql script here or you can specify the mount volume at runtime
# COPY /sql /db

COPY --chmod=0755 --from=grate-downloader /app .
# migrate script
COPY --chmod=0755 migrate.sh .
# Add globalization support to the OS so .Net can use cultures
RUN apk add icu-libs
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

ENTRYPOINT [ "./migrate.sh" ]

Folder setup:

Run the migration script:
docker run -e Database__ConnectionString="connection_string_here" \
                   -e Database__CreateDatabase=true \
                   -v ./sql:/db \
                   -e Database__Env=DEV,TEST,CUSTOMER_A \
                   --network=host \
                   grate:latest

Grate output:

The main reason of choosing docker in my case:

Using grate as init container in k8s
Able to mount the migration script from storage (by using volume mount with migration script stored in blob storage, file share..etc without any concerns the security leakage)
Nicely support Argo-CD with GitOps, let's say you want to deploy the new version of your database, create the PR and then merge to the repository. Grab some coffee and everything just works.

This is a very elegant approach but I don't think it will work in our environment. At any given time I've got about 75, environments I'm managing all with data of various ages. The fact I can keep this running at all is a testament to how good roundhouse is., Being able to control the order script run in is a major issue for us. In the example above all the scripts for the Dev environment would always run before any of the scripts for Customer environment. I can see that causing issues for us.
Another lesser issue is the fact that roundhouse is by far the longest step in our deployment process, we have a lot of DBs so it can take up to 10 minutes. I'd like to avoid doubling or tripling that.
It would really be helpful to be able to specify multiple environments on the command line so the script order can be preserved and to avoid the overhead associated with multiple runs.

In the example above all the scripts for the Dev environment would always run before any of the scripts for Customer environment. I can see that causing issues for us.

You can control the order of environment in the command line mate.
-e Database__Env=DEV,TEST,CUSTOMER_A

with this command, you can specify the order, the script just splits by ',' and runs the environment from left to right.

Another lesser issue is the fact that roundhouse is by far the longest step in our deployment process, we have a lot of DBs so it can take up to 10 minutes. I'd like to avoid doubling or tripling that.

That is true, making the connection to every environment shouldn't be a good option, but by far I can see in my application, that not a deal breaker. My application is 20 years old with a thousand migration scripts but takes little time to migrate with grate. With RoundhousE, our migration takes 30 minutes or more, grate re-written in the way to have a good performance then. Give it a try and you will see how good grate is.

It would really be helpful to be able to specify multiple environments on the command line so the script order can be preserved and to avoid the overhead associated with multiple runs.

That is a good suggestion, let Erik (@erikbra) decide whether he wants grate to support that option. If yes, I'm happy to take the implementation.

Sorry for the late response, I have noticed grate seems to be faster. I'm working through some issues where scripts ran under roundhouse but are erroring in grate so I don't have good numbers yet.  Circling back around to the ordering of scripts being able to specify the order of environment doesn't completely solve the issue. Let's say I have these scripts
01.ENV.DEV.SQL
02.ENV.CUST.SQL
03.ENV.DEV.SQL
By specifying the environments I can get dev to run first the cust, so they would run in the order 01,03,02
or I could run cust then dev and get 02,01,03
but there is no way to get the scripts to run in 01,02,03 order.

I created a PR: #495  with a suggestion, would this solve your needs, @JackNWeems ?
https://github.com/erikbra/grate/pull/495/files#r1561531618
Would this also be able to solve your scenario without running grate multiple times, @hoangthanh28 ?

I'm ok with this PR, however we need to write more test cases.

01.ENV.DEV.SQL
02.ENV.CUST.SQL
03.ENV.DEV.SQL

This looks not straightforward, especially when the customer script interferes with another environment.
My views:

The script in the same environment must run sequentially.
Can run multiple environments.

What do you mean, that this is not straightforward, @hoangthanh28 ? The scripts will be run in the order they are named. So, if these three scripts are run with environments "DEV, CUST", they will be run in the order listed:
01.ENV.DEV.SQL
02.ENV.CUST.SQL
03.ENV.DEV.SQL
If run for "DEV", this order will be run:
01.ENV.DEV.SQL
03.ENV.DEV.SQL
If run for "CUST", only this will be run:
02.ENV.CUST.SQL
Would you expect it to be differently? If so, how?

Wasn't your problem, @JackNWeems , that the order would be wrong if you had to run grate twice, once first for environment "DEV", and one for environment "CUST"? But, now, when you can specify both environments at once, won't it be correct? Or did I misunderstand?

Solved by #495


adds install script to readme

The Problem:
we should have the install script in the readme
The Fix:
here's the install script in the readme
The Test:
Automation Overview:

Related Issue Link(s):
Release/Deployment notes:

Install script also works on CentOS! Not sure if ddev works on CentOS.


Nix improvements

This fixes #728 and generally makes the Nix option more pleasant to use.

Victory!


Release for sbt 1.0

Would be nice to see a release for sbt 1.0

It's already available with sbt-jni version 1.3.0-M1 (and source is in branch sbt-1). I haven't released a final version yet, since sbt-jni is affected by an upstream bug and I haven't been able to extensively test the workaround yet.

workaround fixed with an upgrade to sbt 1.0.1, used in version 1.3.0


Not working with Seamless Co-op & Randomizer

X-Post from https://github.com/soulsmods/ModEngine2/issues/58
Including previously posted info:
Hello @Cloudef , I've used your guide & build to attempt installing Randomizer + Seamless Co-Op under Arch w/ Proton Experimental. However, it results in a very similar issue to that described here. Seamless co-op launches fine, however Randomizer appears to not work (judging based on character creation screen) and the game will crash when attempting new character creation or loading a previous save. For the record, I've renamed elden_ring_seamless_coop.dll to seamless_coop.dll and I have the same content as the release notes in my modengine.toml. I have also tried Proton-GE and Proton 7 with the same results. Steam launch command is the same as described in release notes. Randomizer .exe was run with system wine and the application seemed to work fine. I also tried copying the randomizer directory from a Windows VM with no luck. I'll be attaching my logs (although everything seems to be working correctly by looking at them?) and my configs and directory structure. Thanks!
mod_loader_log.txt
modengine_2022-06-25.log
find.txt
config_eldenringrandomizer.toml.txt
mod_loader_config.ini.txt
modengine.toml.txt
(.txt extensions added so GH will let me upload them)
P.S. there is a slight typo in the release notes, modengine.toml is referenced as modengine2.toml under the "How to use with enemy / item randomizer?" heading.
Still not working with proton-v3 release. No longer crashes after character creation/load, but randomizer still isn't working. I would've x-posted this as an issue to your fork but I don't think issues are enabled on it.
md5sum modengine2.dll
15883ee3c55f1da1858f1677e7933508  modengine2.dll

md5sum eldenring.exe
36d03670099eb4037c5a9546db6cd7a7  eldenring.exe

It appears the US locale .exe is different from yours. For the record, the old (v2) modengine.2 dll:
md5sum modengine2.dll
2c65db279c481d62da31a329466a84ed  modengine2.dll

Your file layout looks correct. Can I have latest modengine_<date>.log file with the -v3?

Attaching a new modengine.log for v3:
modengine_2022-06-26.log
It appears all 3 dlls are getting hooked correctly.
Screenshots of file layout:

[2022-06-26 08:12:46.182] [modengine] [warning] Unable to resolve mod path

This is most likely the problem

modengine_2022-06-26.log
Sorry, last unable to resolve was a result of me attempting some debugging. Check last run in this log.

Okay, that looks like everything is fine. What randomizer settings did you use? Can you post the randomizer window.

Basically default setting. Randomizer exe run with system wine.

Ok everything seems fine, not sure why it does not work. The logic should be same to the vanilla modengine regarding to memory scanning. Does other mods work?

Also can you check if it's only items that don't get randomized. If enemies get randomized, it might be related to the hardcoded address used for all souls games. (Item randomization works with that)

Okay, yeah. Enemy randomization is working. Grafted Scion was replaced with a troll. The text strings associated with prompts and boss health appear to be broken too, reading ?MapText? and ?NpcName?. Just downloaded Reforged to see if it works fine.

Reforged works I think? I'm not sure what it changes, although the starting classes and Grafted Scion appear the same. It did break the UI though, with nothing appearing.

Okay, it might be related to this https://github.com/Cloudef/ModEngine2/blob/feat/overlay-scripting/src/modengine/ext/mod_loader/mod_loader_extension.cpp#L56 which is funny because normal modloader2 has this too. I can check if I can scan this address too later.

Yeah, checked the reforged change notes and the starting talismans aren't present, along with the mod's UI issues. Seems to be the same issue as is affecting randomizer.

@Lejaxak what proton are you use btw?

Proton Experimental

Ok, I'm using GE. Not sure what's the problem. Is your proton prefix clean? Anything else custom in your setup?

 I ended up doing an unarchived install of randomizer with UXM and launching through the seamless co-op launcher, which works just fine. I'll play around with the load delay later, and see if that changes anything. Thank you so much for the help with this.

Okay, had some time to experiment with it and this issue is resolved. Decreasing EML load delay to 2500 (down from 5000 default) resolves all the issues. Seems ER was loading before EML could hook the game correctly, like you guessed. Thank you again for all your help and this excellent modengine fork.


add list option for page size

Resolves #339

Coverage increased (+0.005%) to 92.318% when pulling e219a7ce41ea4cd6b182801adc0d10093c976cbd on AlanCoding:page_size into 6d99a154db316b77793267c73f7fd17462d3cf1e on ansible:v1.


Upgrade flutter 2.5.1

Should we update the required version of flutter?
What's the consequence of that?

This was not an issue


[libpng[apng]] Add msys installation to use awk

libpng[apng], the libpng library with the animated PNG patch, defines -DPNG_PREFIX=a as part of its portfile. However, this flag does not have any effect on the library and the symbols generated will not have the "a" prefix attached. This is because awk is required to run the header/source file updates. This [draft] PR intends to fix this by adding MSYS to the mix for apng-flavored builds, install [g]awk, and then adding MSYS to the PATH to pick up the awk dependency.
Describe the pull request

What does your PR fix? Attempts to fix #13837

Which triplets are supported/not supported? Have you updated the CI baseline?
No updates to the CI baseline. Pending checks to ensure that Linux/Mac will still work with this change.

Does your PR follow the maintainer guide? Yes - leaving this in draft as the issue is being discussed in the issue above + in this PR.

The baseline error will be fixed in PR #13942.

Thanks for the PR!


The embedded video is 404

The URL is https://www.mediastreamingmesh.io/assets/shared/mediastreamingmesh.mp4

we'll get this one fixed ASAP

fixed by PR #207
will upload a new video to YouTube shortly.


VotingDataOf<T> is a complicated tuple which should be a struct

Lets avoid tuples, and use a struct instead.
Originally posted by @shawntabrizi in https://github.com/paritytech/substrate/pull/9081#discussion_r655036353

would be good if we close this soon to keep the backlog clear

@kianenigma can I take a stab at this next week?

not relevant anymore.


How to limit search by file type?

Happy to update the docs with this info if anyone can point in the right direction.
Outside of Vim you can do rg sausages -tcss to search for sausages in any file type of 'css'. How do you pass the same file type limit argument in Vim?

For me that just works (on linux)
:Rg pattern -tcss and :Rg -tcss pattern work.
You have to specify pattern, as the script will only expand the current word when there are no arguments (if I understand correctly). You can manually expand the current word under the cursor on the vim command line with <c-r><c-w> (see :h c_<c-r>)


Watch this video and check current implementation of gateway

https://www.youtube.com/watch?v=puIJ1Mn9_LE

all done


Add kuksoolwonwoodbridge.co.uk website

Adds the Kuk Sool Won of Woodbridge website that was built with 11ty.

Thank you!


confirm before notebook installs a package

When I run a .ipynb file that I didn't create, I probably won't catch install commands that aren't in the first cell.  If there's a !pip or piplite.install command, please confirm that user wants to proceed with the installation.  Since I'm running a conda environment, pip can still install duplicative packages except from the pypi channel.

@DonJayamanne Would setting the workspace trust block a notebook from installing new packages?  Thanks.

setting the workspace trust block a notebook from installing new packages? Thanks.

if you don’t trust the workspace, then you won’t be able to execute


tfsuit: tfsuit: cannot execute binary file

Hello!
I'm trying to use tfsuit, but I'm getting the following error:
# with bash
tfsuit: tfsuit: cannot execute binary file

# with zsh
zsh: exec format error: ./tfsuit

I'm running WSL with Ubuntu 18.04.
tested with tfsuit 0.6.3

Thanks for the feedback, I'm checking why it didn't work for you.


DataSet.filters() / filter variable concept

We would want to have the option to declare specific "filter" variables, which would simply have one category (1 = "active" or something similar). These would get the recoded_filter property: Currently, such variables are only generated by DataSet.from_batch(additions={'full', 'filters'}) (using the inner private _manifest_filters() function). It would be good if we could have a proper method that generates such a variable from an arbitrary complex logic so it could be part of a Batch definition's variable list / could be used by the Batch class vice-versa, maybe similar to Batch.add_filter('FILTER_VAR_NAME').
@AndersFreund: I am adding you as we talked about this regarding the YG  / Crunch use case. Your ideas can be collected in this ticket.

done and merged


custom  Segmention

jetson Segmention examle is work but my custom model is faild
how to modify CLASS_NAMES and COLORS and postprocess parameters in main.cpp

CLASS_NAMES should be a vector of string which in equal to label id.
COLORS means the box color when visualize.

I dont know why examlpe can work but mustom model faild
i use jetson orin agx jetpack5.1.2
terminate called after throwing an instance of 'cv::Exception'
what():  OpenCV(4.5.4) /home/ubuntu/build_opencv/opencv/modules/core/src/matrix.cpp:1194: error: (-15:Bad number of channels) The total width is not divisible by the new number of channels in function 'reshape'
Aborted (core dumped)

I dont know why examlpe can work but mustom model faild i use jetson orin agx jetpack5.1.2
terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.5.4) /home/ubuntu/build_opencv/opencv/modules/core/src/matrix.cpp:1194: error: (-15:Bad number of channels) The total width is not divisible by the new number of channels in function 'reshape' Aborted (core dumped)

Do you modify the CLASS_NAMES (it's size should be the same as your model)

yes I use one class_names as my model

I dont know why examlpe can work but mustom model faild i use jetson orin agx jetpack5.1.2
terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.5.4) /home/ubuntu/build_opencv/opencv/modules/core/src/matrix.cpp:1194: error: (-15:Bad number of channels) The total width is not divisible by the new number of channels in function 'reshape' Aborted (core dumped)

I had the same problem as you when deploying to Jetson, did you solve it?


Update knockknock to 1.9.3

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Closes #45332.

https://www.virustotal.com/#/file/37f59fbeaa7e80fe92bd1a25bd6342d7b3f78778a248c307336e24cfebd3dbf8/details

sha1 from VirusTotal matches public sha1 https://objective-see.com/products/knockknock.html