Install error

Hi. When I try and install this I get the following error:
[~/Sites/sendit-hugo-template]$ npm i
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-ml9pkFGx5wczKPKgCJaLZOopVdI= integrity checksum failed when using sha1: wanted sha1-ml9pkFGx5wczKPKgCJaLZOopVdI= but got sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==. (1992 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-nLxoaXj+5l0WwA4rGcI3/Pbjg4k= integrity checksum failed when using sha1: wanted sha1-nLxoaXj+5l0WwA4rGcI3/Pbjg4k= but got sha512-BLq/cCO9two+lBgiTYNqD6GdtK8s4NpaWrl6/rCO9w0TUS8oJl7cmToOZfRYllKTISY6nt1U7jQ53brmKqY6BA==. (2040 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-RKrGW2lbAzmJaMOfNj/uXer98eo= integrity checksum failed when using sha1: wanted sha1-RKrGW2lbAzmJaMOfNj/uXer98eo= but got sha512-EV3L1+UQWGor21OmnvojK36mhg+TyIKDh3iFBKBohr5xeXIhNBcx8oWdgkTEEQ+BEFFYdLRuqMfd5L84N1V5Vg==. (1585 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-L19Fq5HjMhYjT9U62rZo607AmTs= integrity checksum failed when using sha1: wanted sha1-L19Fq5HjMhYjT9U62rZo607AmTs= but got sha512-Kx8hMakjX03tiGTLAIdJ+lL0htKnXjEZN6hk/tozf/WOuYGdZBJrZ+rCJRbVCugsjB3jMLn9746NsQIf5VjBMw==. (1733 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-IzTBjpx1n3vdVv3vfprj1YjmjtM= integrity checksum failed when using sha1: wanted sha1-IzTBjpx1n3vdVv3vfprj1YjmjtM= but got sha512-vavAMRXOgBVNF6nyEEmL3DBK19iRpDcoIwW+swQ+CbGiu7lju6t+JklA1MHweoWtadgt4ISVUsXLyDq34ddcwA==. (1736 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-2kL0l0DAtC2yypcoVxyxkMmO/qM= integrity checksum failed when using sha1: wanted sha1-2kL0l0DAtC2yypcoVxyxkMmO/qM= but got sha512-wpoSFAxys6b2a2wHZ1XpDSgD7N9iVjg29Ph9uV/uaP9Ex/KXlkTZTeddxDPSYQpgvzKLGJke2UU0AzoGCjNIvQ==. (1425 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-vjX1Qlvh9/bHRxhPmKeIy5lHfuA= integrity checksum failed when using sha1: wanted sha1-vjX1Qlvh9/bHRxhPmKeIy5lHfuA= but got sha512-aOIos8bujGN93/8Ox/jPLh7RwVnPEysynVFE+fQZyg6jKELEHwzgKdLRFHUgXJL6kylijVSBC4BvN9OmsB48Rw==. (2029 bytes)
WARN registry Using stale package data from https://registry.npmjs.org/ due to a request error during revalidation.
 Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      /Users/cjyabraham/.npm/_logs/2022-12-13T00_11_55_415Z-debug.log

Hi @cjyabraham 👋
It looks like this is being caused by an older version of NodeJS — I'll make a note to go through our templates and document what dependency versions are required. If you upgrade to Node >= 18 then this error should be resolved

Thanks. That helped me get further, however, now when I run npm run dev I'm getting stuck here:
[~/Sites/sendit-hugo-template]$ npm run dev

><EMAIL_ADDRESS>dev
> run-p hugo-server bookshop

><EMAIL_ADDRESS>bookshop
> npx @bookshop/browser@latest

><EMAIL_ADDRESS>hugo-server
> hugo server -D --watch

Start building sites …
hugo v0.108.0+extended darwin/amd64 BuildDate=unknown

                   | EN
-------------------+------
  Pages            |  27
  Paginator pages  |   0
  Non-page files   |   0
  Static files     | 100
  Processed images |   0
  Aliases          |   6
  Sitemaps         |   1
  Cleaned          |   0

Built in 176 ms
Watching for changes in /Users/cjyabraham/Sites/sendit-hugo-template/{assets,component-library,content,data,layouts,package.json,static}
Watching for config changes in /Users/cjyabraham/Sites/sendit-hugo-template/config.toml, /Users/cjyabraham/Sites/sendit-hugo-template/component-library/config.toml, /Users/cjyabraham/Sites/sendit-hugo-template/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at //localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop
📚 Looking for Bookshop component libraries...
📚 —— Loading Bookshop from ./component-library
node:internal/errors:484
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@bookshop/hugo-engine' imported from<EMAIL_ADDRESS>    at new NodeError (node:internal/errors:393:5)
    at packageResolve (node:internal/modules/esm/resolve:823:9)
    at moduleResolve (node:internal/modules/esm/resolve:872:20)
    at defaultResolve (node:internal/modules/esm/resolve:1087:11)
    at nextResolve (node:internal/modules/esm/loader:161:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:831:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:413:18)
    at ESMLoader.import (node:internal/modules/esm/loader:514:22)
    at importModuleDynamically (node:internal/modules/esm/translators:110:35)
    at importModuleDynamicallyCallback (node:internal/process/esm_loader:36:14) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v19.2.0
ERROR: "bookshop" exited with 1.

Ah, yes we need to tidy up some of the dependency versions there, sorry.
To fix this, change the @bookshop/* dependencies in your package.json to 3.3.3. i.e:
  "dependencies": {
    "@bookshop/browser": "^3.3.3",
    "@bookshop/generate": "^3.3.3",
    "@bookshop/hugo-engine": "^3.3.3"
  },

And re-running npm i
Since the npm script is running npx @bookshop/browser@latest it's causing a version mismatch there. You could also fix this by removing the @latest from that script command. I'll also make sure to get this fixed up in the template

Ok, I'm getting this now:
[~/Sites/sendit-hugo-template]$ npm run dev

><EMAIL_ADDRESS>dev
> run-p hugo-server bookshop

><EMAIL_ADDRESS>hugo-server
> hugo server -D --watch

><EMAIL_ADDRESS>bookshop
> npx @bookshop/browser@latest

Start building sites …
hugo v0.108.0+extended darwin/amd64 BuildDate=unknown

                   | EN
-------------------+------
  Pages            |  27
  Paginator pages  |   0
  Non-page files   |   0
  Static files     | 100
  Processed images |   0
  Aliases          |   6
  Sitemaps         |   1
  Cleaned          |   0

Built in 179 ms
Watching for changes in /Users/cjyabraham/Sites/sendit-hugo-template/{assets,component-library,content,data,layouts,package.json,static}
Watching for config changes in /Users/cjyabraham/Sites/sendit-hugo-template/config.toml, /Users/cjyabraham/Sites/sendit-hugo-template/component-library/config.toml, /Users/cjyabraham/Sites/sendit-hugo-template/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at //localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop
📚 Looking for Bookshop component libraries...
📚 —— Loading Bookshop from ./component-library
 > bookshop-import-file:components/global/faq/faq.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/global/faq/faq.bookshop.yaml
    1 │ import file from "./components/global/faq/faq.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/privacy/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/privacy/hero/hero.bookshop.yaml
    1 │ import file from "./components/privacy/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/about/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/hero/hero.bookshop.yaml
    1 │ import file from "./components/about/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/global/feature/feature.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/global/feature/feature.bookshop.yaml
    1 │ import file from "./components/global/feature/feature.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/global/header/header.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/global/header/header.bookshop.yaml
    1 │ import file from "./components/global/header/header.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/about/team/team.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/team/team.bookshop.yaml
    1 │ import file from "./components/about/team/team.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/global/testimonial/testimonial.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/global/testimonial/testimonial.bookshop.yaml
    1 │ import file from "./components/global/testimonial/testimonial.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/home/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/home/hero/hero.bookshop.yaml
    1 │ import file from "./components/home/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/about/video/video.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/video/video.bookshop.yaml
    1 │ import file from "./components/about/video/video.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/contact/form/form.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/contact/form/form.bookshop.yaml
    1 │ import file from "./components/contact/form/form.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/home/video/video.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/home/video/video.bookshop.yaml
    1 │ import file from "./components/home/video/video.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/pricing/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/pricing/hero/hero.bookshop.yaml
    1 │ import file from "./components/pricing/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/contact/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/contact/hero/hero.bookshop.yaml
    1 │ import file from "./components/contact/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/pricing/table/table.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/pricing/table/table.bookshop.yaml
    1 │ import file from "./components/pricing/table/table.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/global/counter/counter.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/global/counter/counter.bookshop.yaml
    1 │ import file from "./components/global/counter/counter.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 > bookshop-import-file:components/feature/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/feature/hero/hero.bookshop.yaml
    1 │ import file from "./components/feature/hero/hero.bookshop.yaml";
      ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:1493
  let error = new Error(`${text}${summary}`);
              ^

Error: Build failed with 16 errors:
bookshop-import-file:components/about/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/hero/hero.bookshop.yaml
bookshop-import-file:components/about/team/team.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/team/team.bookshop.yaml
bookshop-import-file:components/about/video/video.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/about/video/video.bookshop.yaml
bookshop-import-file:components/contact/form/form.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/contact/form/form.bookshop.yaml
bookshop-import-file:components/contact/hero/hero.bookshop.yaml__bookshop_file__:1:17: error: No loader is configured for ".yaml" files: component-library/components/contact/hero/hero.bookshop.yaml
...
    at failureErrorWithLog (/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:1493:15)
    at /Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:1151:28
    at runOnEndCallbacks (/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:941:63)
    at buildResponseToResult (/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:1149:7)
    at /Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:1258:14
    at /Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:629:9
    at handleIncomingPacket (/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/Users/cjyabraham/Sites/sendit-hugo-template/node_modules/esbuild/lib/main.js:596:7)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12) {
  errors: [
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/about/hero/hero.bookshop.yaml__bookshop_file__',
        length: 44,
        line: 1,
        lineText: 'import file from "./components/about/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/about/hero/hero.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/about/team/team.bookshop.yaml__bookshop_file__',
        length: 44,
        line: 1,
        lineText: 'import file from "./components/about/team/team.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/about/team/team.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/about/video/video.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/about/video/video.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/about/video/video.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/contact/form/form.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/contact/form/form.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/contact/form/form.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/contact/hero/hero.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/contact/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/contact/hero/hero.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/feature/hero/hero.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/feature/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/feature/hero/hero.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/global/counter/counter.bookshop.yaml__bookshop_file__',
        length: 51,
        line: 1,
        lineText: 'import file from "./components/global/counter/counter.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/global/counter/counter.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/global/faq/faq.bookshop.yaml__bookshop_file__',
        length: 43,
        line: 1,
        lineText: 'import file from "./components/global/faq/faq.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/global/faq/faq.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/global/feature/feature.bookshop.yaml__bookshop_file__',
        length: 51,
        line: 1,
        lineText: 'import file from "./components/global/feature/feature.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/global/feature/feature.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/global/header/header.bookshop.yaml__bookshop_file__',
        length: 49,
        line: 1,
        lineText: 'import file from "./components/global/header/header.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/global/header/header.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/global/testimonial/testimonial.bookshop.yaml__bookshop_file__',
        length: 59,
        line: 1,
        lineText: 'import file from "./components/global/testimonial/testimonial.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/global/testimonial/testimonial.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/home/hero/hero.bookshop.yaml__bookshop_file__',
        length: 43,
        line: 1,
        lineText: 'import file from "./components/home/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/home/hero/hero.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/home/video/video.bookshop.yaml__bookshop_file__',
        length: 45,
        line: 1,
        lineText: 'import file from "./components/home/video/video.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/home/video/video.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/pricing/hero/hero.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/pricing/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/pricing/hero/hero.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/pricing/table/table.bookshop.yaml__bookshop_file__',
        length: 48,
        line: 1,
        lineText: 'import file from "./components/pricing/table/table.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/pricing/table/table.bookshop.yaml'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'bookshop-import-file:components/privacy/hero/hero.bookshop.yaml__bookshop_file__',
        length: 46,
        line: 1,
        lineText: 'import file from "./components/privacy/hero/hero.bookshop.yaml";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".yaml" files: component-library/components/privacy/hero/hero.bookshop.yaml'
    }
  ],
  warnings: []
}

Node.js v19.2.0
ERROR: "bookshop" exited with 1.

Oh my, that one is new. I think that is a bug in one of the Bookshop libraries that I'll need to fix — sorry that you're hitting so many issues on this.
I'll look at getting that fixed, for now you could run npm start instead of npm run dev, which will skip running the component Browser locally and just run the Hugo site.

Ok, will do. Thanks.

Hi @bglw, are there any updates on a fix for the bookshop bug?

Not yet on the Bookshop side sorry! My main branch is blocked for a release with some larger upcoming changes for Bookshop Hugo, but it's preventing a release for this.
I have pushed a fix to this template repo though, so if you pull down those changes everything should be working. (The issue is the .yaml extension on component files, rather than .yml). You can see the set of changes here: #3
Let me know if that helps!

Ok. I pulled down the latest changes and am now getting the following. Am I missing the bookshop_bindings partial or something:
[~/Sites/sendit-hugo-template]$ npm run dev

><EMAIL_ADDRESS>dev
> run-p hugo-server bookshop

><EMAIL_ADDRESS>hugo-server
> hugo server -D --watch

><EMAIL_ADDRESS>bookshop
> npx @bookshop/browser@latest

Start building sites …
hugo v0.108.0+extended darwin/amd64 BuildDate=unknown
ERROR 2022/12/16 08:24:17 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2022/12/16 08:24:17 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2022/12/16 08:24:17 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2022/12/16 08:24:17 render of "home" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/list.html:1:23": execute of template failed: template: _default/list.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params.content_blocks`>: error calling partial: partial "bookshop_bindings" not found
Error: Error building site: failed to render pages: render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
Built in 225 ms
ERROR: "hugo-server" exited with 255.

Hi. I'm now not able to run the site even with npm start, which used to work previously. Did something change? I'm now getting this:
[~/Sites/sendit-hugo-template]$ npm start

><EMAIL_ADDRESS>start
> run-p hugo-server

><EMAIL_ADDRESS>hugo-server
> hugo server -D --watch

Start building sites …
hugo v0.108.0+extended darwin/amd64 BuildDate=unknown
ERROR 2023/01/07 16:32:53 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2023/01/07 16:32:53 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2023/01/07 16:32:53 render of "home" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/list.html:1:23": execute of template failed: template: _default/list.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params.content_blocks`>: error calling partial: partial "bookshop_bindings" not found
ERROR 2023/01/07 16:32:53 render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
Error: Error building site: failed to render pages: render of "page" failed: "/Users/cjyabraham/Sites/sendit-hugo-template/layouts/_default/single.html:1:23": execute of template failed: template: _default/single.html:1:23: executing "main" at <partial "bookshop_bindings" `.Params`>: error calling partial: partial "bookshop_bindings" not found
Built in 140 ms
ERROR: "hugo-server" exited with 255.

Hmm. I rebooted my machine and it worked. Seems there was some issue pulling in the hugo module or something...


Docker Image

Hi All, is it possible that you guys release a docker image for TaPAS? That would be extremely helpful for batch jobs at AWS and GCP.

Hello @OeslleLucena thanks for the interest, we'll take a look at this, in the meantime contributions are welcome as well.

HI @ebursztein, that's good to know! Thank you!

@eisenjulian
I tried to generate the docker image but I've got errors when installing tapas similar to issue #33
Any idea about how to fix that?
Please find below the logs for the errors.
Cloning into 'tapas'...
Collecting pip
  Downloading https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (1.5MB)
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-20.1.1
Obtaining file:///home/tapas
Collecting apache-beam[gcp]==2.20.0
  Downloading apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl (3.5 MB)
Collecting frozendict==1.2
  Downloading frozendict-1.2.tar.gz (2.6 kB)
Collecting pandas~=1.0.0
  Downloading pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl (10.1 MB)
Collecting scikit-learn~=0.22.1
  Downloading scikit_learn-0.22.2.post1-cp36-cp36m-manylinux1_x86_64.whl (7.1 MB)
Collecting tensorflow-probability==0.10.0
  Downloading tensorflow_probability-0.10.0-py2.py3-none-any.whl (3.5 MB)
Collecting tensorflow~=2.2.0
  Downloading tensorflow-2.2.0-cp36-cp36m-manylinux2010_x86_64.whl (516.2 MB)
Collecting tf-models-official~=2.2.0
  Downloading tf_models_official-2.2.1-py2.py3-none-any.whl (711 kB)
Collecting tf_slim~=1.1.0
  Downloading tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)
Collecting dataclasses~=0.7
  Downloading dataclasses-0.7-py3-none-any.whl (18 kB)
Collecting future<1.0.0,>=0.16.0
  Downloading future-0.18.2.tar.gz (829 kB)
Collecting pyarrow<0.17.0,>=0.15.1; python_version >= "3.0" or platform_system != "Windows"
  Downloading pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl (63.1 MB)
Collecting mock<3.0.0,>=1.0.1
  Downloading mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting httplib2<=0.12.0,>=0.8
  Downloading httplib2-0.12.0.tar.gz (218 kB)
Collecting numpy<2,>=1.14.3
  Downloading numpy-1.19.0-cp36-cp36m-manylinux2010_x86_64.whl (14.6 MB)
Collecting hdfs<3.0.0,>=2.1.0
  Downloading hdfs-2.5.8.tar.gz (41 kB)
Collecting pytz>=2018.3
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1; python_version >= "3.0"
  Downloading avro-python3-<IP_ADDRESS>.tar.gz (37 kB)
  WARNING: Requested avro-python3!=1.9.2,<1.10.0,>=1.8.1; python_version >= "3.0" from https://files.pythonhosted.org/packages/5a/80/acd1455bea0a9fcdc60a748a97dcbb3ff624726fb90987a0fc1c19e7a5a5/avro-python3-<IP_ADDRESS>.tar.gz#sha256=ca1e77a3da5ac98e8833588f71fb2e170b38e34787ee0e04920de0e9470b7d32 (from apache-beam[gcp]==2.20.0->tapas==0.0.1.dev0), but installing version file-.avro-VERSION.txt
Collecting pymongo<4.0.0,>=3.8.0
  Downloading pymongo-3.10.1-cp36-cp36m-manylinux2014_x86_64.whl (460 kB)
Collecting crcmod<2.0,>=1.7
  Downloading crcmod-1.7.tar.gz (89 kB)
Collecting oauth2client<4,>=2.0.1
  Downloading oauth2client-3.0.0.tar.gz (77 kB)
Collecting pydot<2,>=1.2.0
  Downloading pydot-1.4.1-py2.py3-none-any.whl (19 kB)
Collecting protobuf<4,>=3.5.0.post1
  Downloading protobuf-3.12.2-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB)
Collecting python-dateutil<3,>=2.8.0
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting grpcio<2,>=1.12.1
  Downloading grpcio-1.30.0-cp36-cp36m-manylinux2010_x86_64.whl (3.0 MB)
Collecting typing-extensions<3.8.0,>=3.7.0
  Downloading typing_extensions-<IP_ADDRESS>-py3-none-any.whl (22 kB)
Collecting fastavro<0.22,>=0.21.4
  Downloading fastavro-0.21.24-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB)
Collecting dill<0.3.2,>=<IP_ADDRESS>
  Downloading dill-<IP_ADDRESS>.tar.gz (151 kB)
Collecting google-cloud-datastore<1.8.0,>=1.7.1; extra == "gcp"
  Downloading google_cloud_datastore-1.7.4-py2.py3-none-any.whl (82 kB)
Collecting grpcio-gcp<1,>=0.2.2; extra == "gcp"
  Downloading grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-core<2,>=0.28.1; extra == "gcp"
  Downloading google_cloud_core-1.3.0-py2.py3-none-any.whl (26 kB)
Collecting google-cloud-videointelligence<1.14.0,>=1.8.0; extra == "gcp"
  Downloading google_cloud_videointelligence-1.13.0-py2.py3-none-any.whl (177 kB)
Collecting google-cloud-dlp<=0.13.0,>=0.12.0; extra == "gcp"
  Downloading google_cloud_dlp-0.13.0-py2.py3-none-any.whl (151 kB)
Collecting google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp"
  Downloading google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Collecting google-cloud-language<2,>=1.3.0; extra == "gcp"
  Downloading google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-vision<0.43.0,>=0.38.0; extra == "gcp"
  Downloading google_cloud_vision-0.42.0-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-bigquery<=1.24.0,>=1.6.0; extra == "gcp"
  Downloading google_cloud_bigquery-1.24.0-py2.py3-none-any.whl (165 kB)
Collecting google-cloud-pubsub<1.1.0,>=0.39.0; extra == "gcp"
  Downloading google_cloud_pubsub-1.0.2-py2.py3-none-any.whl (118 kB)
Collecting google-apitools<0.5.29,>=0.5.28; extra == "gcp"
  Downloading google-apitools-0.5.28.tar.gz (172 kB)
Collecting cachetools<4,>=3.1.0; extra == "gcp"
  Downloading cachetools-3.1.1-py2.py3-none-any.whl (11 kB)
Collecting google-cloud-spanner<1.14.0,>=1.13.0; extra == "gcp"
  Downloading google_cloud_spanner-1.13.0-py2.py3-none-any.whl (212 kB)
Collecting joblib>=0.11
  Downloading joblib-0.16.0-py3-none-any.whl (300 kB)
Collecting scipy>=0.17.0
  Downloading scipy-1.5.1-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
Collecting cloudpickle>=1.2.2
  Downloading cloudpickle-1.5.0-py3-none-any.whl (22 kB)
Collecting gast>=0.3.2
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Requirement already satisfied: six>=1.10.0 in /usr/lib/python3/dist-packages (from tensorflow-probability==0.10.0->tapas==0.0.1.dev0) (1.11.0)
Collecting decorator
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting absl-py>=0.7.0
  Downloading absl-py-0.9.0.tar.gz (104 kB)
Collecting h5py<2.11.0,>=2.10.0
  Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)
Collecting termcolor>=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting opt-einsum>=2.3.2
  Downloading opt_einsum-3.2.1-py3-none-any.whl (63 kB)
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting wrapt>=1.11.1
  Downloading wrapt-1.12.1.tar.gz (27 kB)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/lib/python3/dist-packages (from tensorflow~=2.2.0->tapas==0.0.1.dev0) (0.30.0)
Collecting tensorflow-estimator<2.3.0,>=2.2.0
  Downloading tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB)
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting tensorboard<2.3.0,>=2.2.0
  Downloading tensorboard-2.2.2-py3-none-any.whl (3.0 MB)
Collecting keras-preprocessing>=1.1.0
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting py-cpuinfo>=3.3.0
  Downloading py-cpuinfo-7.0.0.tar.gz (95 kB)
Collecting pyyaml
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
Collecting tensorflow-hub>=0.6.0
  Downloading tensorflow_hub-0.8.0-py2.py3-none-any.whl (101 kB)
Collecting tensorflow-datasets
  Downloading tensorflow_datasets-3.2.0-py3-none-any.whl (3.4 MB)
Collecting tensorflow-model-optimization>=0.2.1
  Downloading tensorflow_model_optimization-0.3.0-py2.py3-none-any.whl (165 kB)
Collecting matplotlib
  Downloading matplotlib-3.2.2-cp36-cp36m-manylinux1_x86_64.whl (12.4 MB)
Collecting opencv-python-headless
  Downloading opencv_python_headless-<IP_ADDRESS>-cp36-cp36m-manylinux2014_x86_64.whl (36.4 MB)
Collecting mlperf-compliance==0.0.10
  Downloading mlperf_compliance-0.0.10-py3-none-any.whl (24 kB)
Collecting tensorflow-addons
  Downloading tensorflow_addons-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl (1.0 MB)
Collecting gin-config
  Downloading gin_config-0.3.0-py3-none-any.whl (44 kB)
Collecting google-api-python-client>=1.6.7
  Downloading google_api_python_client-1.9.3-py3-none-any.whl (59 kB)
Collecting Cython
  Downloading Cython-0.29.21-cp36-cp36m-manylinux1_x86_64.whl (2.0 MB)
Collecting psutil>=5.4.3
  Downloading psutil-5.7.0.tar.gz (449 kB)
Collecting typing==<IP_ADDRESS>
  Downloading typing-<IP_ADDRESS>-py3-none-any.whl (25 kB)
Collecting sentencepiece
  Downloading sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)
Collecting kaggle>=1.3.9
  Downloading kaggle-1.5.6.tar.gz (58 kB)
Collecting Pillow
  Downloading Pillow-7.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
Collecting pbr>=0.11
  Downloading pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting docopt
  Downloading docopt-0.6.2.tar.gz (25 kB)
Collecting requests>=2.7.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting pyasn1-modules>=0.0.5
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
  Downloading rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyparsing>=2.1.4
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from protobuf<4,>=3.5.0.post1->apache-beam[gcp]==2.20.0->tapas==0.0.1.dev0) (39.0.1)
Collecting google-api-core[grpc]<2.0.0dev,>=1.6.0
  Downloading google_api_core-1.21.0-py2.py3-none-any.whl (90 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting google-resumable-media<0.6dev,>=0.5.0
  Downloading google_resumable_media-0.5.1-py2.py3-none-any.whl (38 kB)
Collecting google-auth<2.0dev,>=1.9.0
  Downloading google_auth-1.18.0-py2.py3-none-any.whl (90 kB)
Collecting fasteners>=0.14
  Downloading fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting werkzeug>=0.11.15
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.1-py2.py3-none-any.whl (18 kB)
Collecting markdown>=2.6.8
  Downloading Markdown-3.2.2-py3-none-any.whl (88 kB)
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
Collecting attrs>=18.1.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting promise
  Downloading promise-2.3.tar.gz (19 kB)
Collecting tensorflow-metadata
  Downloading tensorflow_metadata-0.22.2-py2.py3-none-any.whl (32 kB)
Collecting tqdm
  Downloading tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
Collecting dm-tree~=0.1.1
  Downloading dm_tree-0.1.5-cp36-cp36m-manylinux1_x86_64.whl (294 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.2.0-cp36-cp36m-manylinux1_x86_64.whl (88 kB)
Collecting cycler>=0.10
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting typeguard>=2.7
  Downloading typeguard-2.9.1-py3-none-any.whl (16 kB)
Collecting google-auth-httplib2>=0.0.3
  Downloading google_auth_httplib2-0.0.4-py2.py3-none-any.whl (9.1 kB)
Collecting uritemplate<4dev,>=3.0.0
  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting certifi
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting python-slugify
  Downloading python-slugify-4.0.1.tar.gz (11 kB)
Collecting urllib3<1.25,>=1.21.1
  Downloading urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.20.0->tapas==0.0.1.dev0) (2.6)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting monotonic>=0.1
  Downloading monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
Collecting text-unidecode>=1.3
  Downloading text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting zipp>=0.5
  Downloading zipp-3.1.0-py3-none-any.whl (4.9 kB)
Building wheels for collected packages: frozendict, future, httplib2, hdfs, avro-python3, crcmod, oauth2client, dill, google-apitools, absl-py, termcolor, wrapt, py-cpuinfo, pyyaml, psutil, kaggle, docopt, grpc-google-iam-v1, promise, python-slugify
  Building wheel for frozendict (setup.py): started
  Building wheel for frozendict (setup.py): finished with status 'done'
  Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3580 sha256=fe0e18fbeb4738e9eeb9f46b74689e7a61fd79a3b5213aaf87f9fa2214ccb3ba
  Stored in directory: /root/.cache/pip/wheels/c9/13/a1/b4f2255117a7dccdd6219408dce1d87446716b1bf77451cb97
  Building wheel for future (setup.py): started
  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=493275 sha256=8cf6913164b6eb5ece253e498678212d93ab05f0fb8c25a5f9011e4db9fd81b7
  Stored in directory: /root/.cache/pip/wheels/6e/9c/ed/4499c9865ac1002697793e0ae05ba6be33553d098f3347fb94
  Building wheel for httplib2 (setup.py): started
  Building wheel for httplib2 (setup.py): finished with status 'done'
  Created wheel for httplib2: filename=httplib2-0.12.0-py3-none-any.whl size=95116 sha256=c9129588c1d93621d3e4472b1829055bd036daf425277168a731918f562347d4
  Stored in directory: /root/.cache/pip/wheels/8f/42/db/47792ef8349ffb6fa2b96e203852f60681a351162b5515b45f
  Building wheel for hdfs (setup.py): started
  Building wheel for hdfs (setup.py): finished with status 'done'
  Created wheel for hdfs: filename=hdfs-2.5.8-py3-none-any.whl size=34733 sha256=7cd3d63724f3395697a985ee3504b207cb1acf8db1bc389ed71000793b32b4b4
  Stored in directory: /root/.cache/pip/wheels/3e/0c/c3/26ad975f80274d6bf73ed4d8facd055648f452428bc1623283
  Building wheel for avro-python3 (setup.py): started
  Building wheel for avro-python3 (setup.py): finished with status 'done'
  Created wheel for avro-python3: filename=avro_python3-file_.avro_VERSION.txt-py3-none-any.whl size=44890 sha256=aedd5b188459df8950f540a6fe3300b2054c7e7b7b6cf36f67546e9e5cd751fe
  Stored in directory: /root/.cache/pip/wheels/4e/08/0c/727bff8f20fedbdeb8a2c5214e460b214d41c10dc879cf6dac
  Building wheel for crcmod (setup.py): started
  Building wheel for crcmod (setup.py): finished with status 'done'
  Created wheel for crcmod: filename=crcmod-1.7-cp36-cp36m-linux_x86_64.whl size=37197 sha256=e015e52be20aa34adb30926c8f37b6a9145e7f705f415ce88555e6d2ed9b8f46
  Stored in directory: /root/.cache/pip/wheels/ac/bb/07/adfb4ffd0aaace2022ea25c082a7cdc688b10d30e86d6d2fde
  Building wheel for oauth2client (setup.py): started
  Building wheel for oauth2client (setup.py): finished with status 'done'
  Created wheel for oauth2client: filename=oauth2client-3.0.0-py3-none-any.whl size=107377 sha256=25f0e99226c832e7867ea6453352087043014ab8e9ea0dd30cef1693b27172e5
  Stored in directory: /root/.cache/pip/wheels/85/84/41/0db9b5f02fab88d266e64a52c5a468a3a70f6d331e75ec0e49
  Building wheel for dill (setup.py): started
  Building wheel for dill (setup.py): finished with status 'done'
  Created wheel for dill: filename=dill-<IP_ADDRESS>-py3-none-any.whl size=80833 sha256=4ececb8a8549538e458d7434c7fa8aa6a23880f05f2c9710a2b4efd4bd9ff378
  Stored in directory: /root/.cache/pip/wheels/09/84/74/d2b4feb9ac9488bc83c475cb2cbe8e8b7d9cea8320d32f3787
  Building wheel for google-apitools (setup.py): started
  Building wheel for google-apitools (setup.py): finished with status 'done'
  Created wheel for google-apitools: filename=google_apitools-0.5.28-py3-none-any.whl size=131643 sha256=edfd8c1459d1d12f9c69633d201eb62d6db94903a342aaa6c83a61e7455ac2b3
  Stored in directory: /root/.cache/pip/wheels/49/56/1c/73a513e437099b768ededdcb95106a58f5cdd048fb27ff640b
  Building wheel for absl-py (setup.py): started
  Building wheel for absl-py (setup.py): finished with status 'done'
  Created wheel for absl-py: filename=absl_py-0.9.0-py3-none-any.whl size=119396 sha256=fed19b1ecff172b5ac447d787c5d96c23da4dd5b108ad470b74c0a38690d6ae6
  Stored in directory: /root/.cache/pip/wheels/c3/af/84/3962a6af7b4ab336e951b7877dcfb758cf94548bb1771e0679
  Building wheel for termcolor (setup.py): started
  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=5679 sha256=d0317f1dc626dce91581e0ef6ed31438b60985b8f11d4acd60aef02968c16de4
  Stored in directory: /root/.cache/pip/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=69350 sha256=ae1fe13c5c4f1f871becedec3a67606d59a65ebd91c96bc46cc80286e24b784a
  Stored in directory: /root/.cache/pip/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63
  Building wheel for py-cpuinfo (setup.py): started
  Building wheel for py-cpuinfo (setup.py): finished with status 'done'
  Created wheel for py-cpuinfo: filename=py_cpuinfo-7.0.0-py3-none-any.whl size=20299 sha256=1ab842eeb035b82b50a210f828443e9e4ea42d6374ac9a3610dcf50ca16f2c2b
  Stored in directory: /root/.cache/pip/wheels/46/6d/cc/73a126dc2e09fe56fcec0a7386d255762611fbed1c86d3bbcc
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=45919 sha256=c3ff6a042df4955c75d72df1bdfd47ce9c21371415e3207bb1d7870fba2c79ae
  Stored in directory: /root/.cache/pip/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc
  Building wheel for psutil (setup.py): started
  Building wheel for psutil (setup.py): finished with status 'done'
  Created wheel for psutil: filename=psutil-5.7.0-cp36-cp36m-linux_x86_64.whl size=279827 sha256=841838e6feb3569d06a18c48868015ae0ee27360b55f681e1aaff890a7db8446
  Stored in directory: /root/.cache/pip/wheels/a1/d9/f2/b5620c01e9b3e858c6877b1045fda5b115cf7df6490f883382
  Building wheel for kaggle (setup.py): started
  Building wheel for kaggle (setup.py): finished with status 'done'
  Created wheel for kaggle: filename=kaggle-1.5.6-py3-none-any.whl size=73813 sha256=6cd19202645dcd630f55d7ea8afb8e4dd7abf35d8d177472551fb44a84fefb8b
  Stored in directory: /root/.cache/pip/wheels/01/3e/ff/77407ebac3ef71a79b9166a8382aecf88415a0bcbe3c095a01
  Building wheel for docopt (setup.py): started
  Building wheel for docopt (setup.py): finished with status 'done'
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=19852 sha256=cc0804ade908147b9a28df2c5e97ab43acb09b3776e62ba31f0f957343cbcdd6
  Stored in directory: /root/.cache/pip/wheels/3f/2a/fa/4d7a888e69774d5e6e855d190a8a51b357d77cc05eb1c097c9
  Building wheel for grpc-google-iam-v1 (setup.py): started
  Building wheel for grpc-google-iam-v1 (setup.py): finished with status 'done'
  Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py3-none-any.whl size=15418 sha256=b4681926cc5bc5928792b72aaf1f1796288ba1623a9af6b5478540dabad0e94c
  Stored in directory: /root/.cache/pip/wheels/76/65/cd/392da05e43270f143b6c5076ba88d39144abff586792593e7c
  Building wheel for promise (setup.py): started
  Building wheel for promise (setup.py): finished with status 'done'
  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=23948 sha256=f0a7e022866e1c079593b3a512539d53d65e643bd162c561b3e76e86d185402a
  Stored in directory: /root/.cache/pip/wheels/59/9a/1d/3f1afbbb5122d0410547bf9eb50955f4a7a98e53a6d8b99bd1
  Building wheel for python-slugify (setup.py): started
  Building wheel for python-slugify (setup.py): finished with status 'done'
  Created wheel for python-slugify: filename=python_slugify-4.0.1-py2.py3-none-any.whl size=7016 sha256=af64344f11e71991a0cc9e0cd2ffff79189af8303bc3b27da66a746ae15f07f4
  Stored in directory: /root/.cache/pip/wheels/72/e6/db/122611605e60148f54ee2abaca98b2bbeafc6e22486a867bad
Successfully built frozendict future httplib2 hdfs avro-python3 crcmod oauth2client dill google-apitools absl-py termcolor wrapt py-cpuinfo pyyaml psutil kaggle docopt grpc-google-iam-v1 promise python-slugify
ERROR: google-auth 1.18.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.
ERROR: google-cloud-bigquery 1.24.0 has requirement six<2.0.0dev,>=1.13.0, but you'll have six 1.11.0 which is incompatible.
ERROR: google-apitools 0.5.28 has requirement six>=1.12.0, but you'll have six 1.11.0 which is incompatible.
ERROR: apache-beam 2.20.0 has requirement avro-python3!=1.9.2,<1.10.0,>=1.8.1; python_version >= "3.0", but you'll have avro-python3 file-.avro-VERSION.txt which is incompatible.
ERROR: tensorboard 2.2.2 has requirement setuptools>=41.0.0, but you'll have setuptools 39.0.1 which is incompatible.
ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.5.1 which is incompatible.
ERROR: tensorflow 2.2.0 has requirement six>=1.12.0, but you'll have six 1.11.0 which is incompatible.
ERROR: tensorflow-hub 0.8.0 has requirement six>=1.12.0, but you'll have six 1.11.0 which is incompatible.
ERROR: dm-tree 0.1.5 has requirement six>=1.12.0, but you'll have six 1.11.0 which is incompatible.
ERROR: tf-models-official 2.2.1 has requirement oauth2client>=4.1.2, but you'll have oauth2client 3.0.0 which is incompatible.
Installing collected packages: future, numpy, pyarrow, pbr, mock, httplib2, docopt, certifi, urllib3, chardet, requests, hdfs, pytz, avro-python3, pymongo, crcmod, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, protobuf, python-dateutil, grpcio, typing-extensions, fastavro, dill, cachetools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpcio-gcp, google-cloud-videointelligence, google-cloud-dlp, grpc-google-iam-v1, google-cloud-bigtable, google-cloud-language, google-cloud-vision, google-resumable-media, google-cloud-bigquery, google-cloud-pubsub, monotonic, fasteners, google-apitools, google-cloud-spanner, apache-beam, frozendict, pandas, joblib, scipy, scikit-learn, cloudpickle, gast, decorator, tensorflow-probability, absl-py, h5py, termcolor, opt-einsum, google-pasta, wrapt, tensorflow-estimator, astunparse, werkzeug, oauthlib, requests-oauthlib, google-auth-oauthlib, zipp, importlib-metadata, markdown, tensorboard-plugin-wit, tensorboard, keras-preprocessing, tensorflow, py-cpuinfo, pyyaml, tensorflow-hub, attrs, promise, tensorflow-metadata, tqdm, tensorflow-datasets, dm-tree, tensorflow-model-optimization, kiwisolver, cycler, matplotlib, opencv-python-headless, mlperf-compliance, dataclasses, typeguard, tensorflow-addons, gin-config, google-auth-httplib2, uritemplate, google-api-python-client, Cython, psutil, typing, sentencepiece, text-unidecode, python-slugify, kaggle, Pillow, tf-models-official, tf-slim, tapas
  Running setup.py develop for tapas
Successfully installed Cython-0.29.21 Pillow-7.2.0 absl-py-0.9.0 apache-beam-2.20.0 astunparse-1.6.3 attrs-19.3.0 avro-python3-file-.avro-VERSION.txt cachetools-3.1.1 certifi-2020.6.20 chardet-3.0.4 cloudpickle-1.5.0 crcmod-1.7 cycler-0.10.0 dataclasses-0.7 decorator-4.4.2 dill-<IP_ADDRESS> dm-tree-0.1.5 docopt-0.6.2 fastavro-0.21.24 fasteners-0.15 frozendict-1.2 future-0.18.2 gast-0.3.3 gin-config-0.3.0 google-api-core-1.21.0 google-api-python-client-1.9.3 google-apitools-0.5.28 google-auth-1.18.0 google-auth-httplib2-0.0.4 google-auth-oauthlib-0.4.1 google-cloud-bigquery-1.24.0 google-cloud-bigtable-1.0.0 google-cloud-core-1.3.0 google-cloud-datastore-1.7.4 google-cloud-dlp-0.13.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.0.2 google-cloud-spanner-1.13.0 google-cloud-videointelligence-1.13.0 google-cloud-vision-0.42.0 google-pasta-0.2.0 google-resumable-media-0.5.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-1.30.0 grpcio-gcp-0.2.2 h5py-2.10.0 hdfs-2.5.8 httplib2-0.12.0 importlib-metadata-1.7.0 joblib-0.16.0 kaggle-1.5.6 keras-preprocessing-1.1.2 kiwisolver-1.2.0 markdown-3.2.2 matplotlib-3.2.2 mlperf-compliance-0.0.10 mock-2.0.0 monotonic-1.5 numpy-1.19.0 oauth2client-3.0.0 oauthlib-3.1.0 opencv-python-headless-<IP_ADDRESS> opt-einsum-3.2.1 pandas-1.0.5 pbr-5.4.5 promise-2.3 protobuf-3.12.2 psutil-5.7.0 py-cpuinfo-7.0.0 pyarrow-0.16.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.1 pymongo-3.10.1 pyparsing-2.4.7 python-dateutil-2.8.1 python-slugify-4.0.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-oauthlib-1.3.0 rsa-4.6 scikit-learn-0.22.2.post1 scipy-1.5.1 sentencepiece-0.1.91 tapas tensorboard-2.2.2 tensorboard-plugin-wit-1.7.0 tensorflow-2.2.0 tensorflow-addons-0.10.0 tensorflow-datasets-3.2.0 tensorflow-estimator-2.2.0 tensorflow-hub-0.8.0 tensorflow-metadata-0.22.2 tensorflow-model-optimization-0.3.0 tensorflow-probability-0.10.0 termcolor-1.1.0 text-unidecode-1.3 tf-models-official-2.2.1 tf-slim-1.1.0 tqdm-4.47.0 typeguard-2.9.1 typing-<IP_ADDRESS> typing-extensions-<IP_ADDRESS> uritemplate-3.0.1 urllib3-1.24.3 werkzeug-1.0.1 wrapt-1.12.1 zipp-3.1.0
Collecting tox
  Downloading tox-3.16.1-py2.py3-none-any.whl (137 kB)
Collecting py>=1.4.17
  Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0
  Downloading virtualenv-20.0.26-py2.py3-none-any.whl (4.9 MB)
Requirement already satisfied: importlib-metadata<2,>=0.12; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from tox) (1.7.0)
Collecting packaging>=14
  Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting toml>=0.9.4
  Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting pluggy>=0.12.0
  Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting six>=1.14.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting filelock>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting importlib-resources>=1.0; python_version < "3.7"
  Downloading importlib_resources-3.0.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->tox) (3.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging>=14->tox) (2.4.7)
ERROR: tf-models-official 2.2.1 has requirement oauth2client>=4.1.2, but you'll have oauth2client 3.0.0 which is incompatible.
ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.5.1 which is incompatible.
ERROR: tensorboard 2.2.2 has requirement setuptools>=41.0.0, but you'll have setuptools 39.0.1 which is incompatible.
ERROR: google-auth 1.18.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.
ERROR: apache-beam 2.20.0 has requirement avro-python3!=1.9.2,<1.10.0,>=1.8.1; python_version >= "3.0", but you'll have avro-python3 file-.avro-VERSION.txt which is incompatible.
Installing collected packages: py, distlib, appdirs, filelock, importlib-resources, six, virtualenv, packaging, toml, pluggy, tox
  Attempting uninstall: six
    Found existing installation: six 1.11.0
    Uninstalling six-1.11.0:
      Successfully uninstalled six-1.11.0
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 importlib-resources-3.0.0 packaging-20.4 pluggy-0.13.1 py-1.9.0 six-1.15.0 toml-0.10.1 tox-3.16.1 virtualenv-20.0.26
GLOB sdist-make: /home/tapas/setup.py
py36 create: /home/tapas/.tox/py36
py36 installdeps: -r/home/tapas/requirements.txt
py36 inst: /home/tapas/.tox/.tmp/package/1/tapas-0.0.1.dev0.zip
py36 installed: absl-py==0.9.0,apache-beam==2.20.0,astunparse==1.6.3,attrs==19.3.0,avro-python3===file-.avro-VERSION.txt,cachetools==3.1.1,certifi==2020.6.20,chardet==3.0.4,cloudpickle==1.5.0,crcmod==1.7,cycler==0.10.0,Cython==0.29.21,dataclasses==0.7,decorator==4.4.2,dill==<IP_ADDRESS>,dm-tree==0.1.5,docopt==0.6.2,fastavro==0.21.24,fasteners==0.15,frozendict==1.2,future==0.18.2,gast==0.3.3,gin-config==0.3.0,google-api-core==1.21.0,google-api-python-client==1.9.3,google-apitools==0.5.28,google-auth==1.18.0,google-auth-httplib2==0.0.4,google-auth-oauthlib==0.4.1,google-cloud-bigquery==1.24.0,google-cloud-bigtable==1.0.0,google-cloud-core==1.3.0,google-cloud-datastore==1.7.4,google-cloud-dlp==0.13.0,google-cloud-language==1.3.0,google-cloud-pubsub==1.0.2,google-cloud-spanner==1.13.0,google-cloud-videointelligence==1.13.0,google-cloud-vision==0.42.0,google-pasta==0.2.0,google-resumable-media==0.5.1,googleapis-common-protos==1.52.0,grpc-google-iam-v1==0.12.3,grpcio==1.30.0,grpcio-gcp==0.2.2,h5py==2.10.0,hdfs==2.5.8,httplib2==0.12.0,idna==2.10,importlib-metadata==1.7.0,joblib==0.16.0,kaggle==1.5.6,Keras-Preprocessing==1.1.2,kiwisolver==1.2.0,Markdown==3.2.2,matplotlib==3.2.2,mlperf-compliance==0.0.10,mock==2.0.0,monotonic==1.5,numpy==1.19.0,oauth2client==3.0.0,oauthlib==3.1.0,opencv-python-headless==<IP_ADDRESS>,opt-einsum==3.2.1,pandas==1.0.5,pbr==5.4.5,Pillow==7.2.0,promise==2.3,protobuf==3.12.2,psutil==5.7.0,py-cpuinfo==7.0.0,pyarrow==0.16.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pydot==1.4.1,pymongo==3.10.1,pyparsing==2.4.7,python-dateutil==2.8.1,python-slugify==4.0.1,pytz==2020.1,PyYAML==5.3.1,requests==2.24.0,requests-oauthlib==1.3.0,rsa==4.6,scikit-learn==0.22.2.post1,scipy==1.5.1,sentencepiece==0.1.91,six==1.15.0,tapas @ file:///home/tapas/.tox/.tmp/package/1/tapas-0.0.1.dev0.zip,tensorboard==2.2.2,tensorboard-plugin-wit==1.7.0,tensorflow==2.2.0,tensorflow-addons==0.10.0,tensorflow-datasets==3.2.0,tensorflow-estimator==2.2.0,tensorflow-hub==0.8.0,tensorflow-metadata==0.22.2,tensorflow-model-optimization==0.3.0,tensorflow-probability==0.10.0,termcolor==1.1.0,text-unidecode==1.3,tf-models-official==2.2.1,tf-slim==1.1.0,tqdm==4.47.0,typeguard==2.9.1,typing==<IP_ADDRESS>,typing-extensions==<IP_ADDRESS>,uritemplate==3.0.1,urllib3==1.24.3,Werkzeug==1.0.1,wrapt==1.12.1,zipp==3.1.0
py36 run-test-pre: PYTHONHASHSEED='2962058561'
py36 run-test: commands[0] | python -m unittest discover -p '*_test.py'
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/tapas/tapas/datasets/dataset.py:58: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From /home/tapas/tapas/datasets/dataset.py:73: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.
2020-07-13 11:58:09.847162: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-07-13 11:58:09.847223: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-07-13 11:58:09.847270: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
2020-07-13 11:58:09.847543: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-07-13 11:58:09.852386: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency:<PHONE_NUMBER> Hz
2020-07-13 11:58:09.852779: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f48e4000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-13 11:58:09.852795: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
.......s/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py:523: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
  tensor_proto.tensor_content = nparray.tostring()
..........sE.WARNING:tensorflow:From /home/tapas/tapas/models/bert/modeling.py:252: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From /home/tapas/tapas/models/bert/modeling.py:252: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
.sWARNING:tensorflow:From /usr/lib/python3.6/contextlib.py:60: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
WARNING:tensorflow:From /usr/lib/python3.6/contextlib.py:60: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
..s.........sEWARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model_test.py:65: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model_test.py:65: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model_test.py:68: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model_test.py:68: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f491014b048>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f491014b048>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model.py:192: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

WARNING:tensorflow:From /home/tapas/tapas/models/tapas_pretraining_model.py:192: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 7 vs previous value: 7. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 7 vs previous value: 7. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 10 vs previous value: 10. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 10 vs previous value: 10. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 12 vs previous value: 12. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 12 vs previous value: 12. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 14 vs previous value: 14. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 14 vs previous value: 14. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 16 vs previous value: 16. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 16 vs previous value: 16. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
.WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f48f8712400>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f48f8712400>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f49227ab8c8>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f49227ab8c8>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
.s.........WARNING:tensorflow:From /home/tapas/tapas/scripts/prediction_utils.py:41: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /home/tapas/tapas/scripts/prediction_utils.py:41: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
................................................................./home/tapas/.tox/py36/lib/python3.6/site-packages/apache_beam/io/filesystem.py:599: DeprecationWarning: Flags not at the start of the expression 'tapas\\/utils\\/testda' (truncated)
  re_pattern = re.compile(self.translate_pattern(pattern))
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
/home/tapas/.tox/py36/lib/python3.6/site-packages/apache_beam/io/filesystem.py:599: DeprecationWarning: Flags not at the start of the expression '\\/tmp\\/tmpfkm018bx\\/' (truncated)
  re_pattern = re.compile(self.translate_pattern(pattern))
.................................................
======================================================================
ERROR: tapas.experiments.prediction_utils_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tapas.experiments.prediction_utils_test
Traceback (most recent call last):
  File "/usr/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/tapas/tapas/experiments/prediction_utils_test.py", line 21, in <module>
    from tapas.experiments import prediction_utils
  File "/home/tapas/tapas/experiments/prediction_utils.py", line 25, in <module>
    from tapas.models import tapas_classifier_model
  File "/home/tapas/tapas/models/tapas_classifier_model.py", line 31, in <module>
    import tensorflow_probability as tfp
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/__init__.py", line 76, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/__init__.py", line 23, in <module>
    from tensorflow_probability.python import distributions
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/distributions/__init__.py", line 88, in <module>
    from tensorflow_probability.python.distributions.pixel_cnn import PixelCNN
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/distributions/pixel_cnn.py", line 37, in <module>
    from tensorflow_probability.python.layers import weight_norm
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/layers/__init__.py", line 31, in <module>
    from tensorflow_probability.python.layers.distribution_layer import CategoricalMixtureOfOneHotCategorical
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/layers/distribution_layer.py", line 28, in <module>
    from cloudpickle.cloudpickle import CloudPickler
ImportError: cannot import name 'CloudPickler'

======================================================================
ERROR: tapas.models.tapas_classifier_model_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tapas.models.tapas_classifier_model_test
Traceback (most recent call last):
  File "/usr/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/tapas/tapas/models/tapas_classifier_model_test.py", line 22, in <module>
    from tapas.models import tapas_classifier_model
  File "/home/tapas/tapas/models/tapas_classifier_model.py", line 31, in <module>
    import tensorflow_probability as tfp
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/__init__.py", line 76, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/__init__.py", line 23, in <module>
    from tensorflow_probability.python import distributions
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/distributions/__init__.py", line 88, in <module>
    from tensorflow_probability.python.distributions.pixel_cnn import PixelCNN
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/distributions/pixel_cnn.py", line 37, in <module>
    from tensorflow_probability.python.layers import weight_norm
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/layers/__init__.py", line 31, in <module>
    from tensorflow_probability.python.layers.distribution_layer import CategoricalMixtureOfOneHotCategorical
  File "/home/tapas/.tox/py36/lib/python3.6/site-packages/tensorflow_probability/python/layers/distribution_layer.py", line 28, in <module>
    from cloudpickle.cloudpickle import CloudPickler
ImportError: cannot import name 'CloudPickler'

----------------------------------------------------------------------
Ran 163 tests in 23.146s

FAILED (errors=2, skipped=6)
Running pipeline with direct runner this might take a long time!
ERROR: InvocationError for command /home/tapas/.tox/py36/bin/python -m unittest discover -p '*_test.py' (exited with code 1)
py37 create: /home/tapas/.tox/py37
ERROR: InterpreterNotFound: python3.7
___________________________________ summary ____________________________________
ERROR:   py36: commands failed
ERROR:  py37: InterpreterNotFound: python3.7
The command '/bin/sh -c cd /home     && git clone https://github.com/google-research/tapas.git     && cd tapas     && pip3 install --upgrade pip     && pip install -e .     && pip install tox     && tox' returned a non-zero code: 1

Apparently there was an issue with a tensorflow probability dependency, and the right fix at the moment is changing the version dependency to 0.10.1 as mentioned in https://github.com/tensorflow/probability/issues/991 We will release a new version soon, in the meantime you can update the requirements file directly.

@OeslleLucena I have tried to dockerize the tapas and have pushed my docker into the docker hub repo.  But the code is an adaptation of the TAPAS framework for just prediction using the SQA base model. Just FYI if that helps.

Any one post a link to Docker image for TAPAS


Use db cursors for reindex operations

Context: The FHIR server has a "reindex all" operation that exhaustively reindexes all resources of a certain resource type. This can be run when there are significant changes to indexing logic.
Before: The reindex process used a simple single SELECT query to get all resources.  That's impractical when there are 100k+ resources in a single table.
After: The reindex process uses a Postgres cursor to operate in smaller batches.

Coverage decreased (-0.0002%) to 93.931% when pulling 135664d68b7b014aea15c4a46287c1b290816bc1 on cody-cursors-for-reindex-all into a0c2fc921dd2d1c21503168b6c1909bbdc7fc7f8 on main.


Use go1.10.1 and go1.9.5

The latest patch versions include two important fixes for the Prometheus
project:

https://github.com/golang/go/issues/23995
https://github.com/golang/go/issues/24059

@brian-brazil @sdurrheimer
Needed for https://github.com/prometheus/prometheus/issues/3983 and https://github.com/prometheus/node_exporter/issues/870

@grobie You need to change 1.10/Makefile.COMMON and 1.9/Makefile.COMMON too.

Thanks, updated.


cp.roll cannot take array as shift argument

I was taking the output of cp.unravel_index, which returns an array, and feeding it to cp.roll, which shifts an array. This raises the error below, but works when the shift argument is a tuple. It applies to 1D shifts as well.
It looks like perhaps the numpy.broadcast(shift, axis) in the error below should be cupy.broadcast(shift, axis) instead?
>>> arr = cp.array([1,2,3,4]).reshape((2,2))
>>> shift = cp.array([1,1]) # shift (1,1) works
>>> cp.roll(arr, shift)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-158-005527abcbc1> in <module>
      1 arr = cp.array([1,2,3,4])
      2 shift = cp.array([1])
----> 3 cp.roll(arr, shift)

~\Miniconda3\envs\py\lib\site-packages\cupy\_manipulation\rearrange.py in roll(a, shift, axis)
     99     """
    100     if axis is None:
--> 101         return roll(a.ravel(), shift, 0).reshape(a.shape)
    102     else:
    103         axis = _reduction._get_axis(axis, a.ndim)[0]

~\Miniconda3\envs\py\lib\site-packages\cupy\_manipulation\rearrange.py in roll(a, shift, axis)
    103         axis = _reduction._get_axis(axis, a.ndim)[0]
    104
--> 105         broadcasted = numpy.broadcast(shift, axis)
    106         if broadcasted.nd > 1:
    107             raise ValueError(

cupy\core\core.pyx in cupy.core.core.ndarray.__array__()

TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array explicitly.

CuPy Version                 : 8.2.0
NumPy Version                : 1.19.4
SciPy Version                : 1.5.3
Cython Build Version         : 0.29.21
CUDA Root                    : C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1
CUDA Build Version           : 11010
CUDA Driver Version          : 11020
CUDA Runtime Version         : 11010
cuBLAS Version               : 11201
cuFFT Version                : 10300
cuRAND Version               : 10202
cuSOLVER Version             : (11, 0, 0)
cuSPARSE Version             : 11200
NVRTC Version                : (11, 1)
Thrust Version               : 100910
CUB Build Version            : 100910
cuDNN Build Version          : None
cuDNN Version                : None
NCCL Build Version           : None
NCCL Runtime Version         : None
cuTENSOR Version             : None
Device 0 Name                : GeForce GTX 1080 Ti
Device 0 Compute Capability  : 61

Closed via #4884


plot to nvme first and pool plotting

Added

Ability to pass -c param to Chia plots create
Be advised, this is a beta. It is based on the demo we saw from Mariano at the pool operators call. Might not be working correctly yet. closes #78
Ploto now checks for NVMEs to spawn the jobs on. If an NVME is found and defined as a TempDrive, it will be used first to spawn a plot on. closes #74

Changed

Added an additional check for OutDrives, if they have JobsinProgress.

Removed

ProgressBar for PlotoMover using Bits

I've looked around but wtf is the c-option in Ability to pass -c parameter to Chia plots create? :)

We will usa that param to create portable pool plots once they are available.

Testing showed nothing to break. Disclaimer: -c param is still untested.

are there portable pools released yet?

are there portable pools released yet?

Sadly not. We need to wait for chia devteam releasing it to mainnet. You may create portable plots on testnet. But those won't be usable on testnet (missing singleton).


Update: LB probing should be used to repair VM, in situations where VM fails to boot/stuck in booting state

[Enter feedback here]
Please include "LB probing should be used to repair VM, in situations where VM fails to boot/stuck in booting state."
Application health extension should be healthy, running and probing the endpoint for the repair to trigger. In OS hang/Stuck in boot scenario, extension is not running and app health doesn't go out of the VM,in this sceanrios, its treated as missing app health and repair won’t be triggered. When using Application health extension, application endpoint is considered not reachable  when app health extension probes on endpoint and that endpoint is not setup or unreachable.
Therefore in these scenarios, LB probe should be used to trigger a repair.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e0e152c2-665d-022e-9f54-c53e9607add8
Version Independent ID: ccbc724d-ea1f-eb6d-e47c-685c4fb95ca5
Content: Automatic instance repairs with Azure virtual machine scale sets - Azure Virtual Machine Scale Sets
Content Source: articles/virtual-machine-scale-sets/virtual-machine-scale-sets-automatic-instance-repairs.md
Service: virtual-machine-scale-sets
Sub-service: instance-protection
GitHub Login: @mamccrea
Microsoft Alias: mamccrea

@SauravB-MS
Thanks for your feedback! We will investigate and update as appropriate.

@SauravB-MS
Thanks for bringing this to our attention.
I'm going to assign this to the document author so they can take a look at it accordingly.
@mamccrea
Could you Please include "LB probing should be used to repair VM, in situations where VM fails to boot/stick in booting state."
So, it would be helpful to customers.

#reassign: @ju-shim

Thank you for bringing this documentation gap to our attention. I am double-checking this information with my team and we will add an appropriate message to the article in question.

Closing this issue out. Recent updates were made to this doc and others in question. The team will revisit the documentation again at a future date. Thank you.
#please-close


Please include a compatibility layer!

As rubyforge is down, it seems impossible to find ruby-debug-base19-0.11.26.gem. And thus impossible to install ruby-debug with ruby2.1.
The problem is that all windowed debuggers are written to userequire 'ruby-debug'.

Thanks for reporting your issue! This is one of my 7 active issues. Use that link to check how soon your issue will be answered. Don't forget to check your issue against this project's CONTRIBUTING.md. Cheers.


Added support for listing groups by user

Added get_user_groups base function and User::groups helper to fetch a list of groups for a given user. Please do lmk if anything needs changing/renaming/whatever else ^_^

Thanks for this! Sorry it took me so long.

No worries, thanks for writing the library!

Released in v0.8.0


integration: forward errors from gitlab project lookups

Small thing I ran into when creating an access token that didn't include API scope :grin:

Tugboat has finished building the preview for this pull request!
Link:

https://pr4289-xvn0hslxwhnriwllrxpy3ofj1xzyjzra.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/6012f03214dc587be7bd8133

Tugboat has finished building the preview for this pull request!
Link:

https://pr4289-xvn0hslxwhnriwllrxpy3ofj1xzyjzra.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/6012f03214dc587be7bd8133


UAT | Treasury Integration for Registered Post

Court Fees Payment for Registered post is still bypassed.

clicking on Pay Online should redirect to online payment gateway.
Currently, it is auto-collecting payment.
Payment Fees is reflected as 0
Clicking on Print Receipt does not generate a receipt

Hi @Taherabharmal About this by passing of etreasury has been done as per discussion due to the pending integration of sbi payment gateway . This issue will be sorted and aligned as soon as it is done.
cc:_ @Ramu-kandimalla  @anirudh-0

Hi @rajeshcherukumalli  made the changes related to payment amount please check.
cc- @Ramu-kandimalla

Issue still there in UAT env.

https://jam.dev/c/d4758fc0-4b98-4d37-9898-84cfcfb22d34

verified in QA[sol] env., Not working as expected.
https://jam.dev/c/1592ff9c-0c04-45c3-ba73-ab96162fe529

Hi @Taherabharmal  We have  ticket logged by Devaraj for SBI Epost integration  Refer


Resources using pssword should honor sensitive flag

In database cookbook I found one place where password is being printed in log message which will not be right if consumer is using sensitive flag which is supported by chef from 11.4.2 + versions [1].
In entire project there is one place in database_mysql_user LWPR for grant action [2] where password is part of sql statement and that is being printed in log.
All other places like create user, grant in other providers has been avoiding printing password.
I am planning on submitting PR to not log that message. For the consistency sake this seemed fine.
If that's not acceptable I can addd a check for sensitive flag and then log it.
[1]: https://tickets.opscode.com/browse/CHEF-5098
[2]: https://github.com/opscode-cookbooks/database/blob/master/libraries/provider_database_mysql_user.rb#L130

I think this can be closed now? #138 was merged, so this doesn't apply any more.


Isolating multi-tenant data via database schemas

Problem
Multi-tenant applications want to leverage per tenant schemas to implement data isolation.
Suggested solution

a shared model referencing a schema that holds tenant information
one or more tenant models where the schema name is dynamic (e.g. per tenant)
at runtime inject tenant information into a query

Alternatives
Initiate a new Prisma client per tenant as described here:  https://darioielardi.dev/schema-based-multi-tenancy-with-nestjs-and-prisma
Additional context
This issue was created as a sub issue to #1122

Hi @floelhoeffel ! Is there anything we can do to help the Prisma team with this issue?

Lack of this functionality prevents me from switching to Prisma. Please do it.
We need to specify schema in runtime based on who is making the request. Here is how knex.js does this https://knexjs.org/guide/query-builder.html#withschema

Hey @Dhalias and @prudnikov - thanks for the input! We are working to get multi-schema support (without the runtime component though, so won't help with multi-tenancy) into full preview with 4.7.0 for PostgreSQL.
Unfortunately, the multi-tenancy use cases adds a lot of complexity for Prisma schema. We would need to introduce a new class of "dynamic" models where the data model is shared among multiple schemas (aka tenants).
Depending on the number of tenants and models you have, would a workaround via @@map be useful?
model TenantAComment {
  id      Int    @id
	...
  @@schema("tenantA")
  @@map("comment")
}
model TenantBComment {
  id      Int    @id
	...
  @@schema("tenantB")
  @@map("comment")
}

and then something like this:
type TennantAComment = {
  text: string
}

type TennantBComment = {
  text: string
}

declare class PrismaClient {

  tennantAComment: {
    findMany: () => Promise<TennantAComment[]>
  }

  tennantBComment: {
    findMany: () => Promise<TennantBComment[]>
  }
}

type TennantId = 'tennantA' | 'tennantB'
type ModelName = 'Comment'

function moduleName<Id extends TennantId, Model extends ModelName>(id: Id, model: Model): `${Id}${Model}` {
  return `${id}${model}`
}

const prisma = new PrismaClient()

const comments = await prisma[moduleName('tennantA', 'Comment')].findMany()
//    ^?

declare const unknownTennant: TennantId

const comments2 = await prisma[moduleName(unknownTennant, 'Comment')].findMany()
//    ^?

Haven't tried running this though 😅

We accomplished this by creating one client per tenant, and then using a function to switch between clients for each of the tenants. We use mysql, so the multiSchema feature isn't currently supported. It would be really nice if I could include this info in the Prisma schema, and not have to manage 50 clients (and all of their associated database connections).
Given our use-case, it would be extremely cumbersome to use @@map as a workaround. I would rather have the ability to list the available tenants  in the datasource section of the schema. Maybe with the ability to set a default. Then when making database calls, we could either switch between them via an argument, or as a method that takes a tenantId, and returns a model client.
const comments = await prisma.comment.findMany({
  tenant: tenandId,
})
const alsoComments = await prisma.tenant(tenantId).comment.findMany()

As a note, this would also help in our scenario unrelated to multi-tenancy.  We have to stage periodic batch data changes from another system, and when we do there's a clean cutover to a completely fresh schema.
If we could pass the schema as a tenant or use some other dynamic schema provision setup, we'd be able to drive this via configuration instead of having to hack around it with a bunch of views to be updated (and by extension also having to clear caches manually).

Hey @Ustice I'm currently facing something very similar to this and was thinking of the same approach. Would love it if you can elaborate a bit more on how you provide one client per tenant. How is a prisma client generated and how does it get connected to a databse schema on the go? Also, do you use any form of microservice/caching of these clients as a separate service to support this or is that an overkill?
Thank you in advance 🙏

@tech-chieftain one approach on how to have a client per tenant is linked in the OP

https://darioielardi.dev/schema-based-multi-tenancy-with-nestjs-and-prisma

Thank you for sharing @nephix, I read this page when I first saw this issue. The proposed solutions could be migrated to frameworks other than Nestjs with some workarounds but I was curious to see if @Ustice had another solution in mind that would be beneficial.

I think it would be sufficient if prisma supported sharing the connection pool between multiple clients connecting to the same host and database. The main problem of not having multi-tenancy support is that for each tenant client a separate connection pool is created. If you have a lot of tenants, that becomes a problem very quickly.

An idea to solve this problem:
prisma.schema
`
model Account {
id        String   @id @default(uuid())
name      String   @unique
email     String   @unique
password  String
tenantId  String  @unique
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@multiTenancy(tenants: [tenentId])
@@schema("public")
}
model Products {
id    String @id @default(uuid())
name  String @unique
price Float
@@tenantSchema("_schema") // results in a schema <TENANT_ID>_schema
}
`
this will do it all, the tag "@@multiTenancy..." select the field which identify the correct schema.
on migrate schema, run each @@tenentSchema by @@multiTenancy and update it.
on client-side something like "prisma.tenant(id)..."

Just to add to some of the considerations here, CockroachDb allows seamlessly querying across multiple databases as well as multiple schemas.  We have a tiered kind of multi-tenancy, with each tenant served by one database, containing an arbitrary number of separated sites represented by one schema within that database.
It sounds unlikely that prisma will support our usecase in the near future, but it'd be nice to see a solution that enables all the features of the underlying tech.  As in the OP, a prisma client per database is an option if we just get arbitrary schema support, but this involves using massively more database connections than a raw cockroach solution would.

Hey @Ustice I'm currently facing something very similar to this and was thinking of the same approach. Would love it if you can elaborate a bit more on how you provide one client per tenant. How is a prisma client generated and how does it get connected to a databse schema on the go? Also, do you use any form of microservice/caching of these clients as a separate service to support this or is that an overkill?
Thank you in advance 🙏

@tech-chieftain  Sorry for the late reply. So the way that we handle it is that our Prisma schema is linked via environment variable to enable migrations. We essentially create a Prisma client per database. We don't do that all at once, but instead have a getClient function that basically has a Map that points a tenantId to a Prisma client, and if the client doesn't already exist, we create it, overriding the connection string in PrismaClient instantiation to point to . The prisma object that I referenced in my code above is not the PrismaClient, but another object that I use as a Proxy that controls some other things (I built it before prisma.$extends was available). The prisma.tenant function basically just calls that getClient function that I mentioned earlier. All of the types are the same, so that is really all you have to do.

A somewhat sad part of the way Prisma currently generates queries is each query in Postgres has a forced schema added to table specifiers. e.g. SELECT ... FROM "public"."<table>" which means you can't even use client extensions to set the search path before each query.
It could be as easy as removing explicit schema's in Postgres and letting the community deal with it via client extensions for custom use cases 😭

G  no h
GhhHg

As mentioned by robertjpayne, this use case is easily solvable with search_path if the Postgres schema is not forcibly included. Are any of the maintainers able to suggest how we might fork/patch this behavior away?

I've been poking around the code looking for the logic that defaults to qualifying table identifiers with schema names. @janpio any suggestions? I think a very simple patch can likely solve this issue, along with the use of client extensions.

@michaelbridge Hi! Is there any news about solving this issue?

I think maybe this is buried in a binary; I've heard nothing from the maintainers and moved on to a more transparent lib.

Any updates?
On my project, we are trying to generate more than 23 tables across different schemas, but it seems that prisma doesnt support that use case. Docs says that we should use multiSchema and map each table with @@map and @@schema, but our schema file already has 426 lines and duplicating this for each one of the users seems like a really, really bad nightmare.
I think that our problem could be solved if we could do something like that:
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["multiSchema"]
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
  schemas  = ["config", "user1", "user2"]
}

model Bills {
  id String @id @default(cuid())

  @@schema(["user1", "user2"])
}

Or, even better:
model Bills {
  id String @id @default(cuid())

  @@schema({
       except: ["config"]
   })
}

I think maybe this is buried in a binary; I've heard nothing from the maintainers and moved on to a more transparent lib.

What library do you use now?


add support for negative progress

ArrowHead now occasionally pushes negative progress for assignments.
Since so far this hadn't happened we used ulong for tracking and this started causing crashes

given this is a hotfix and I'm getting quite some reports about this breaking a lot of projects I'm going to be a terrible dicator and bypass approvals for this one


Refactor MiqExpression#_to_sql to use Arel for EQUAL expressions

This pulls out a tiny chunk of the #_to_sql method and refactors it to use Arel. In this clause, Arel replaces most of the work done by .operands2sqlvalue. The only thing it can't do is parse the field.
It's worth noting that we parse the field data (typically doing split("-"), etc..) in a number of different places throughout MiqExpression, and not always in the same way. There is a .parse_field method, but that's currently only used by one other method .get_col_type (which is a bit problematic in itself - it treats tags and fields as interchangeable, even though their structure is subtly different), and I'd rather have an object here with some behavior, so I don't continue to spread knowledge of the field's structure around as I go. That's why I created a Field object - eventually I'd like it to be the only place that fields are parsed.
This is part 1 of a series of PRs, where I plan to continue going through this method replacing with Arel. Hopefully at the end we can :scissors: :fire: some of the code that is doing what Arel is probably doing much better. I think trying to do it all at once would result in a big diff, so I'm aiming to do this in smaller refactorings to make it easier to review.
/cc @gtanzillo @jrafanie @yrudman
@miq-bot add-label core, refactoring
Part of https://trello.com/c/Vje3DjhW

@gtanzillo can you re-review? I changed to support only one association, and to raise a ParseError for a field that doesn't match.
Incidentally, I think rubocop is wrong about both those offenses

@gtanzillo please review

This looks great! :+1: :+1: