Integration into Laravel Vue Application with Laravel Mix

Been reading on adding / uploading source maps for easier debugging JavaScript/Vue. That besides PHP debugging . So found your documentation on using the webpack plugin https://docs.bugsnag.com/build-integrations/webpack/ .
Not clear however on the integration into webpack.mix.js . Miss that at https://github.com/bugsnag/webpack-bugsnag-plugins and in documentation https://docs.bugsnag.com/platforms/php/laravel/ .
If we use this in webpack.mix.js:
const mix = require('laravel-mix')
const { CleanWebpackPlugin } = require('clean-webpack-plugin')
const { BugsnagBuildReporterPlugin } = require('webpack-bugsnag-plugins')
…
if (mix.inProduction()) {
  mix.webpackConfig({
    plugins: [
      new CleanWebpackPlugin({
        cleanStaleWebpackAssets: false,
        protectWebpackAssets: true,
        cleanOnceBeforeBuildPatterns: ['js/components/*'],
      }),
      new BugsnagBuildReporterPlugin({
        apiKey: 'api-key-here',
        publicPath: 'https://site.com/public',
        overwrite: true,
    }),

    ],

    output: {
      chunkFilename: 'js/components/[name].js?id=[contenthash]',
    },
  })
} else {
  mix.webpackConfig({
    output: {
      chunkFilename: 'js/components/[name].js',
    },
  });
}

mix.options({
    processCssUrls: false
});

mix.sass('resources/sass/editor.scss', 'public/css')
    .sass('resources/sass/login.scss', 'public/css')
    .sass('resources/sass/preview.scss', 'public/css')
    .sass('resources/sass/published.scss', 'public/css')
    .sass('resources/sass/dashboard.scss', 'public/css')
    .js('resources/js/editor.js', 'public/js')
    .js('resources/js/preview.js', 'public/js')
    .js('resources/js/published-vendor.js', 'public/js')
    .js('resources/js/published-utils.js', 'public/js')
    .js('resources/js/published.js', 'public/js')
    .js('resources/js/dashboard.js', 'public/js')
    .vue()

if (mix.inProduction()) {
    mix.version();
}
....

What should normally be the publicPath here? Should it be https://site.com/public ? Or do I need to add a slash?
What if staging and production both use inProduction , but different domains(staging.domain.com and domain.com)?
Any recommended Laravel Mix template available?

NB Reading https://github.com/bugsnag/webpack-bugsnag-plugins/issues/11 some now as well, but not quite clear yet and would appreciate some feedback.

Testing
f (mix.inProduction()) {
  mix.webpackConfig({
    devtool: 'source-map',
    plugins: [
      new CleanWebpackPlugin({
        cleanStaleWebpackAssets: false,
        protectWebpackAssets: true,
        cleanOnceBeforeBuildPatterns: ['js/components/*'],
      }),
      new BugsnagSourceMapUploaderPlugin({
        apiKey: process.env.BUGSNAG_API_KEY,
        appVersion: process.env.BUGSNAG_APP_VERSION,
        publicPath: '*/js'
...

now

Hi @jasperf,

publicPath should be the path to your bundled assets as it appears in the stacktrace. If your app is running in the browser this is normally the base URL of your app, e.g. https://site.com/public/. This should have a trailing slash, but the plugin will add one if not present so it's not required.
You can use wildcards for the publicPath if your JS is being served at multiple URLs, e.g. https://*.site.com/public/
We don't have an example laravel mix template, but it shouldn't require any extra configuration beyond the standard webpack examples

@yousif-bugsnag Thanks for getting back on this. As I posted we are now using a wildcard as part of publicPath and we did see feedback in terminal on running npm run prod . Feedback that source maps were being uploaded.
Only we do not have new errors yet in which the source maps are loaded. Old errors still show that source maps do not load.  Is there a way to check uploaded sourcemaps besides their usage in error reports?

Also it seems now that this setup I shared is causing production request for source maps also which are not on the server.. and should not be. Any ideas how we can avoid that @yousif-bugsnag ?

We will by trying things with hidden-source-map https://webpack.js.org/configuration/devtool/#production

Using hidden-source-map got me errors
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-i.js?id=6bf8cf2a14c0ec3b"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-contact-module.js?id=a24b946c0ba7fdca"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/main.js"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-image-module.js?id=5bc33caddd1bcfe2"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-gridline-toggle.js?id=e44b906fa647e946"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-image-size.js?id=cc183720813d5b89"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-ruler.js?id=41d0116a857dd7a4"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/publish/smt-skype-call-module.js?id=60b7ffc31108f2a8"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-background-size.js?id=3b03341cbfc3d88e"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-four-sides-number-input.js?id=ea0594b57954b977"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-video-uploader-input.js?id=cddaee2d0f6f7058"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-menu-list.js?id=6c415e54a1a20703"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/editor/smt-generate-inputs-modal.js?id=f1326467b81d19d4"
<i> [BugsnagSourceMapUploaderPlugin] uploading sourcemap for "*/js/js/components/publish/smt-diagonal-line-module.js?id=9c2a74b58dd93f85"
[webpack-cli] NetworkError: HTTP status 409 received from upload API
    at /Users/jasper/code/site.com/site.com/node_modules/@bugsnag/source-maps/dist/Request.js:105:33
    at ConcatStream.<anonymous> (/Users/jasper/code/site.com/site.com/node_modules/concat-stream/index.js:37:43)
    at ConcatStream.emit (node:events:402:35)
    at finishMaybe (/Users/jasper/code/site.com/site.com/node_modules/readable-stream/lib/_stream_writable.js:624:14)
    at afterWrite (/Users/jasper/code/site.com/site.com/node_modules/readable-stream/lib/_stream_writable.js:470:3)
    at processTicksAndRejections (node:internal/process/task_queues:85:21) {
  isRetryable: false,
  code: 1,
  cause: null,
  responseText: 'duplicate source map file for */js/main.js version 1.11.8'

Never mind, I needed a new version.

I just had another error but was told no sourcemap was located, but they were uploaded. How to debug this?
Generate source maps to see the original source file and line numbers. No link to a source map was found in your minified JavaScript file. Either include the link when generating source maps or upload them to us. (Note that events are not reprocessed after source maps are uploaded.)

@jasperf By default, we look for uploaded-to-Bugsnag maps first, then if no matches are found there, we look for a //# sourceMappingURL (or X-SourceMap header). The error message you've quoted above suggests that:

No uploaded-to-Bugsnag source map was found that matches a minified frame in this stacktrace, the app version, or both.
No //# sourceMappingURL exists on this webpage, which figures as you're using hidden-source-map which removes this comment from your output bundle.

If you could drop us a link to an event that's failing to map but that you would expect it to map, we can take a look into that specific example and help explain why. You can message us at<EMAIL_ADDRESS>or drop the link here.

Will email de link to the even to you @xander-jones . Hope we can work out what is missing. Setup was kind of tricky for us and we really want to make it work so debugging becomes easier then it is without source maps.

Thanks for the link @jasperf! I'm going to close this issue as we are communicating via the support ticket, but for the benefit of future travellers:
Bugsnag uses both the frame URL and the app version to match uploaded source maps to events, so if you are specifying an appVersion in your webpack config you will need to ensure the same value is used for the appVersion config value when initialising Bugsnag in your JS app.

Also good to mention this which was mentioned by Bugsnag in email
So to summarize, you should:
- Specify an appVersion when initializing Bugsnag in your app
- Set the publicPath to '*' in your webpack config

Had to leave this be for a while but I am afraid I am still seeing

Missing JavaScript files:Provide JavaScript files for improved grouping and better stack traces
Ensure uploaded source maps match this version or make your JavaScript files accessible from our servers. (Note that events are not reprocessed after source maps are uploaded.)
--
[Ensure uploaded source maps match this version or make your JavaScript files accessible from our servers. (Note that events are not reprocessed after source maps are uploaded.)]()
--

and we do use
if (mix.inProduction()) {
  mix.webpackConfig({
    devtool: 'hidden-source-map',
    plugins: [
      new CleanWebpackPlugin({
        cleanStaleWebpackAssets: false,
        protectWebpackAssets: true,
        cleanOnceBeforeBuildPatterns: ['js/components/*'],
      }),
      new BugsnagSourceMapUploaderPlugin({
        apiKey: process.env.BUGSNAG_API_KEY,
        appVersion: process.env.BUGSNAG_APP_VERSION,
        publicPath: '*'
      })
    ],

and we update BUGSNAG_APP_VERSION=1.15.13 in .env every npm run prod` . And the files are uploaded.

Hey @jasperf at a glance it looks like source maps are being applied correctly to recent events in your project - is this still an issue you're experiencing? If please share a link again with<EMAIL_ADDRESS>and we can take a look.

I have zero JS error events currently. Guess we are deploying well to production.  But before that I never had sourcemaps loading properly for JS.  Will be testing some more with staging added as environment.

Saw the latest files built getting uploaded on npm run prod but still I see:
TypeError/editor
u.left.replace is not a function
12 minutes ago
Apr 7th, 12:11:08 GMT+7
production
unhandled

Missing JavaScript files:
Provide JavaScript files for improved grouping and better stack traces
View Details
Source maps guide
TypeError · u.left.replace is not a function
https://site.test/js/editor.js:2:141391 n.value
https://site.test/js/components/editor/module.js:1:20167 a.positionObject
https://site.test/js/editor.js:2:1889012 hn.get
https://site.test/js/editor.js:2:1890152 hn.evaluate
https://site.test/js/editor.js:2:1890917 a.positionObject
https://site.test/js/components/editor/module.js:1:23890 a.<anonymous>
https://site.test/js/editor.js:2:1896989 a.t._render
https://site.test/js/editor.js:2:1930681 a.r
https://site.test/js/editor.js:2:1889012 hn.get
https://site.test/js/editor.js:2:1888930 new hn

Hi @jasperf - I've had a look at your dashboard and it appears that your events are missing an app version, whereas your sourcemaps have an app version set. The app versions must match in order for source maps to be applied correctly. You can read about setting app version in our docs.
If you're still having trouble please feel free to write back into<EMAIL_ADDRESS>

@luke-belton our Laravel Vue app has the app version set in .env using BUGSNAG_APP_VERSION=1.19 which is loaded from webpack.mix.js:
const mix = require('laravel-mix')
const { CleanWebpackPlugin } = require('clean-webpack-plugin')
const { BugsnagSourceMapUploaderPlugin } = require('webpack-bugsnag-plugins')
....
if (mix.inProduction()) {
  mix.webpackConfig({
    devtool: 'hidden-source-map',
    plugins: [
      new CleanWebpackPlugin({
        cleanStaleWebpackAssets: false,
        protectWebpackAssets: true,
        cleanOnceBeforeBuildPatterns: ['js/components/*'],
      }),
      new BugsnagSourceMapUploaderPlugin({
        apiKey: process.env.BUGSNAG_API_KEY,
        appVersion: process.env.BUGSNAG_APP_VERSION,
        publicPath: '*'
      })
    ],

    output: {
      chunkFilename: 'js/components/[name].js?id=[contenthash]',
    },
  })
} else {
  mix.webpackConfig({
    output: {
      chunkFilename: 'js/components/[name].js',
    },
  });
}

And in main js files for app we use
import bugsnag from '@bugsnag/js'
import bugsnagVue from '@bugsnag/plugin-vue'

if (process.env.NODE_ENV == 'production' || process.env.NODE_ENV == 'staging') {
    let bugsnagClient = bugsnag({
        apiKey: 'key',
        interactionBreadcrumbsEnabled: false
    })
    bugsnagClient.use(bugsnagVue, Vue)
}

So what else are we missing here?

Hi @jasperf,
Setting the appVersion in the webpack plugin means that your source maps will be uploaded with an appVersion set, but  this is separate to the Bugsnag client. In order for events to be sent with an appVersion (which is necessary for matching with uploaded source maps) you will also need to set the appVersion config option when initialising Bugsnag in main.js.
You should also be calling Bugsnag.start() passing in the Vue plugin to initialise the client - please see our Vue integration guide for how to initialise Bugsnag in your Vue app: https://docs.bugsnag.com/platforms/javascript/vue/
import Bugsnag from '@bugsnag/js'
import BugsnagVue from '@bugsnag/plugin-vue'

if (process.env.NODE_ENV == 'production' || process.env.NODE_ENV == 'staging') {
    Bugsnag.start({
      apiKey: 'YOUR_API_KEY',
      appVersion: process.env.BUGSNAG_APP_VERSION
      plugins: [new BugsnagPluginVue()]
    })
}

@yousif-bugsnag Thanks a lot for explaining this in more detail. Seems I was missing Bugsnag.start() to initialize the client. this to allow the appVersion to be sent alone with events, not just in the uploaded source maps.
I will be testing the suggested changes asap and will get back on the results. Fingers crossed I will finally be able to debug  Vue and JavaScript with ease.

Hi @jasperf - apologies, the import statement above should have been:
import BugsnagPluginVue from '@bugsnag/plugin-vue'

I've edited our last comment to reflect that.
Are you using Vue 2 or Vue 3? The way you install the error handler will depend on which version of Vue you are using, as described in our docs: https://docs.bugsnag.com/platforms/javascript/vue/#basic-configuration.

@luke-belton Thanks for the edit. We are using Vue 2.

Version
import Bugsnag from '@bugsnag/js'
import BugsnagPluginVue from '@bugsnag/plugin-vue'

if (process.env.NODE_ENV == 'production' || process.env.NODE_ENV == 'staging') {
    const bugsnagClient = Bugsnag.start({
    apiKey: '32a21ca56ec7f76bfc82ee3e648db1ec',
    appVersion: process.env.BUGSNAG_APP_VERSION,
    plugins: [new BugsnagPluginVue()]
  })

  Vue.use(bugsnagClient) // // this is also important
}

did not cause errors on build. If I am still missing something please let me know.

Hi @jasperf,

@luke-belton Thanks for the edit. We are using Vue 2 so that would mean
const bugsnagVue = Bugsnag.getPlugin('vue')
bugsnagVue.installVueErrorHandler(Vue)

Yes that's the correct way to initialise Bugsnag in Vue 2. Note that this should be added after the call to Bugsnag.start()

@yousif-bugsnag thanks for getting back to me. Previously added code with different Vue.use cal still had the error that BugsnagPluginVue() was not a constructor.
When we  testted:
import Bugsnag from '@bugsnag/js'
import BugsnagPluginVue from '@bugsnag/plugin-vue'

if (process.env.NODE_ENV == 'production' || process.env.NODE_ENV == 'staging') {
  Bugsnag.start({
    apiKey: '32a21ca56ec7f76bfc82ee3e648db1ec',
    appVersion: process.env.BUGSNAG_APP_VERSION,
    plugins: [new BugsnagPluginVue()]
  })

const bugsnagVue = Bugsnag.getPlugin('vue')
bugsnagVue.installVueErrorHandler(Vue)

}

we still have the same error Uncaught TypeError: BugsnagPluginVue(...) is not a constructor

@jasperf
Please update to the latest version (7.16.*) for @bugsnag/js and @bugsnag/plugin-vue, everything will work fine.

Built locally now with latest versions but did get
bugsnag] installVueErrorHandler() was called unnecessarily
installVueErrorHandler @ file.js?id=cb0bf50710bd2e81dc36:2

Unfortunately I am still getting
Source mapping failed:Missing source map linkHide DetailsSource maps guideGenerate source maps to see the original source file and line numbers. No link to a source map was found in your minified JavaScript file. Either include the link when generating source maps or upload them to us. (Note that events are not reprocessed after source maps are uploaded.)
--
Missing source map link
Hide Details
[Source maps guide](https://docs.bugsnag.com/platforms/browsers/js/source-maps/)
Generate source maps to see the original source file and line numbers. No link to a source map was found in your minified JavaScript file. Either include the link when generating source maps or upload them to us. (Note that events are not reprocessed after source maps are uploaded.)

On npm run prod we do generate sourcemaps and these are uploaded.

Hi @jasperf - you should be able to use the hidden-source-map devtool option if you're uploading source maps to Bugsnag. We have guidance on uploading source maps in our docs: https://docs.bugsnag.com/platforms/javascript/source-maps/#uploading-source-maps-recommended.
If you're still having trouble, please can you write into<EMAIL_ADDRESS>referencing this GitHub issue and sharing links to events in your dashboard that aren't being mapped, as well as the command you're using to upload source maps. Thanks!

@luke-belton we are currently testing
appVersion: `'${process.env.BUGSNAG_APP_VERSION}'`,

in
if (process.env.NODE_ENV == 'production' || process.env.NODE_ENV == 'staging') {
  Bugsnag.start({
    apiKey: 'redacted',
    appVersion: `'${process.env.BUGSNAG_APP_VERSION}'`,
    plugins: [new BugsnagPluginVue()]
  })

// call installVueErrorHandler not needed
// const bugsnagVue = Bugsnag.getPlugin('vue')
// bugsnagVue.installVueErrorHandler(Vue)

}

to have appVersion uploaded as a string as well. Seems it was rejected by Bugsnag because it was not sent in as a string like the API key.

Seems process.env does not work. Not inside Bugsnag.start nor defined as const before the if statement. Hardcoded version as string for now. But not happy about it.

Are you using https://webpack.js.org/plugins/define-plugin/? If you want process.env.x to be available in the bundle at runtime you need to "export" it by using DefinePlugin

I think you should keep BugsnagSourceMapUploaderPlugin so that your source maps get uploaded for production builds

@djskinner True. Need to keep it for uploading with path.  Thank you for your feedback. Much appreciated. Will test all out soon. Hope it will all work once and for all.

In the end I realized Laravel Mix does have its own implementation of Define Plugin. But it blocks loading in .env values unless you prefix them with MIX_ like MIX_BUGSNAG_APP_VERSION=. Version loading now. Checked source code on staging. Just need to test on Bugsnag side.


Supporting pydantic<3

Closes https://github.com/huggingface/huggingface_hub/issues/1726
I manually and locally tested with pydantic v2 using Python 3.11.4:
> pip install -e .[testing]
> pip list | grep pydantic
pydantic                  2.4.2
pydantic_core             2.10.1
> pytest tests/test_inference*.py
...
83 passed, 6 skipped, 16 warnings in 5.73s

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Damn it didn't just work 🥲

Both PRs are now merged and I updated this PR to latest changes. Let's see if CI is happy now! :smile:


wrong Matrix Multiplication

Issue Description
When computing matrix multiplication, C = A * B. if A is a view and with shape of [N, 1], the result of C is wrong .
public class Data {
public static void main(String[] args) throws Exception {
INDArray a = Nd4j.zeros(3,3);
a.put(0,0, 1);
a.put(1,0, 1);
a.put(2,0, 1);
INDArray b = Nd4j.rand(1,5);
System.out.println(b);
INDArray c = a.get(NDArrayIndex.all(),NDArrayIndex.interval(0, 1)).mmul(b);
System.out.print(c);
}
}
here is the output :
[0.39, 0.63, 0.99, 0.15, 0.65]
[[0.39, 0.63, 0.99, 0.15, 0.65],
[0.00, 0.00, 0.00, 0.00, 0.00],
[0.00, 0.00, 0.00, 0.00, 0.00]]
Expected output:
[0.39, 0.63, 0.99, 0.15, 0.65]
[[0.39, 0.63, 0.99, 0.15, 0.65],
[0.39, 0.63, 0.99, 0.15, 0.65],
[0.39, 0.63, 0.99, 0.15, 0.65]]
Version Information
Nd4j 0.8.0,  Intel CPU,  Java 1.8.

Thanks for your report. We're going to focus discussion on #1854, which is the same issue.


⚠ Bio Links has degraded performance

In 5ec5eec, Bio Links (https://bio.rheinklang.events/links) experienced degraded performance:

HTTP code: 200
Response time: 1884 ms

Resolved: Bio Links performance has improved in e700861 after 21 minutes.


add github.com/inconshreveable/log15?

github.com/inconshreveable/log15 is used in github.com/revel/log15
github.com/inconshreveable/log15/term in root.go
github.com/inconshreveable/log15 in ext/handler.go
github.com/inconshreveable/log15 in ext/ext_test.go

We needed to modify the github.com/inconshreveable/log15/logger. Wasn't planning on making this a long term repository though. Is this an issue for you ?

no issue


[F6][Responsive Toggle] both menus appear prior to javascript loading

I copied the Responsive Toggle navigation example from the docs and noticed that you can see both the top-bar and the mobile menu for a split second. Once Foundation's javascript loads, one of the two menus disappears, depending on which breakpoint you're at, and it looks normal.
This is easily fixed by:
.title-bar {
	@include breakpoint(medium) {
		display: none;
	}
}

.top-bar {
	display: none;

	@include breakpoint(medium) {
		display: block;
	}
}

Is there a different/better way to go about this? And is it worth mentioning a fix for this in the docs?

Hm, we probably don't want to hardcode in those styles, because you could use either component for anything, at any screen size.
You could do this:
.no-js [data-responsive-toggle] {
  display: none;
}

This will hide the mobile thing if JavaScript is disabled, only showing the desktop thing (the menu itself). That means on desktop browsers there's no flash, but mobile browsers would get a flash. With the flash, you'd go from seeing the desktop thing only, to seeing the mobile thing only.

BTW this is also visible on [the documentation site for F6] (http://foundation.zurb.com/sites/docs/responsive-navigation.html).
when you scroll down and reload the page it will flash the hidden menu.
The mentioned no-js workaround works for now where I tried it, however it does not feel good.

without the no-js thing I fixed it for desktop by adding hide-for-medium to the title-bar div
For the top-bar it needs hiding by css (But only at < medium), I hid it with the no-js trick and that resulted in a flash of nothing then the menu appearing at deskop, and mobile working perfectly.
I need to write a sass rule like:
.no-js .top-bar {
 display: none;
}

but need to limit it to the mobile sizes, then all would be well, but I'm having trouble importing the breakpoint stuff into my web-app's scss file, foundation is in a vendor folder, and I think this rule should use breakpoints from F6S but live in my app. @importing "/blah/blah/_settings" and "..../_breakpoint" results in it complaining about being unable to find util and my lack of sass knowledge is too great to resulve that.

without the no-js thing I fixed it for desktop by adding hide-for-medium to the title-bar div
For the top-bar it needs hiding by css (But only at < medium), I hid it with the no-js trick and that resulted in a flash of nothing then the menu appearing at deskop, and mobile working perfectly.
I need to write a sass rule like:
.no-js .top-bar {
 display: none;
}

but need to limit it to the mobile sizes, then all would be well, but I'm having trouble importing the breakpoint stuff into my web-app's scss file, foundation is in a vendor folder, and I think this rule should use breakpoints from F6S but live in my app. @importing "/blah/blah/_settings" and "..../_breakpoint" results in it complaining about being unable to find util and my lack of sass knowledge is too great to resulve that.

To close the book on this old issue, per my comment above it's tricky to find a good way to build in CSS that prevents FOUC, since the way Responsive Toggle can be set up is somewhat flexible.
With that in mind, I instead added a section to the Responsive Toggle docs explaining that the FOUC will happen, along with code examples showing how to prevent it.

:+1:

BTW when running npm run build for a production release, the no-js thing is removed, I can not find the string 'no-js' in the resulting app.js. I just wondered why I just hit that annoying FOUC trouble again... now I have put it in the header of all pages, because I do not want to dive into why minifycss removes it and how to avoid that.

Ah, of course I meant the gulp uncss plugin. Turns out that excluding something from being stripped works like this: just put new RegExp('^\.no-.*') into your gulpfile on line 110 (if you are still using the orginal gulpfile that comes with the template.)

We configure UnCSS to scan every page in src/ ending in .html: https://github.com/zurb/foundation-zurb-template/blob/master/gulpfile.js#L106
But if it's still stripping .no-js, then like you said, you can explicitly whitelist it as well.

i dont see why this issue is closed, unless this just reappeared again. i have found no solution to get rid of the FOUC on a responsive menu and it looks terrible.


fixed node install failing on closure-util update

npm install

<EMAIL_ADDRESS>install /home/zvikara/dev/geocell/ol3
node tasks/install.js

<EMAIL_ADDRESS>postinstall /home/zvikara/dev/geocell/ol3
closure-util update

sh: 1: closure-util: not found
npm ERR! Linux 3.13.0-53-generic
npm ERR! argv "/usr/bin/iojs" "/usr/bin/npm" "install"
npm ERR! node v2.0.2
npm ERR! npm  v2.10.1
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR<EMAIL_ADDRESS>postinstall: closure-util update
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script 'closure-util update'.
npm ERR! This is most likely a problem with the openlayers package,
npm ERR! not with npm itself.

I don't think this change makes sense.  npm puts ./node_modules/.bin on the path before running scripts.  So if closure-util was successfully installed, it should be on your path during the postinstall script.  I can't reproduce the issue with a fresh install using<EMAIL_ADDRESS> Please write back if you are still able to successfully run npm install (I'm using Node 0.12.4, and haven't confirmed if there is an issue with io.js).


fix mac damaged

If you alter the pyinstaller file with mkdir or shutil it will make the final dmg damaged. @darodi

@darodi can confirm the new DMG works.

@axu2
There was a small problem during the rebase with setup.py
Thanks a lot for the recent PRs.

@darodi Are you referring to you adding this commit to fix it? https://github.com/ciromattia/kcc/commit/8e04ccde180f9ed68649b45c32866e0e189798aa
Seems like your squashing missed some of my original commits


 Then hide them as they are currently null fields

PR: https://github.com/equinor/lighthouse-client/pull/1710


Basic project setup & added some basic converters

Description of Change
Added a multi-targeting .csproj file targeting some of the more popular platforms. I also added a few basic converters that are useful as a starting point. Has no tests or sample just yet. Unit tests should probably be added and we can already think of a nice sample app.
Bugs Fixed

Fixed #119

API Changes
Added:

InvertedBooleanConverter
StringIsNullOrEmptyConverter
StringIsNotNullOrEmptyConverter

PR Checklist

[ ] Has tests (if omitted, state reason in description)
[ ] Has samples (if omitted, state reason in description)
[ ] Rebased on top of master at time of PR
[ ] Changes adhere to coding standard
[ ] Updated documentation

How do we want to handle converters that are basically the opposite of another converter? Just create a copy and do the inversion or could we come up with something more elegant through ConverterParameter or an additional property?
Do we want to support things like Apple Watch, Tizen etc.
Are we going for one big NuGet package or do we separate by type? (converters/effects etc.)

@sthewissen
How do we want to handle converters that are basically the opposite of another converter? Just create a copy and do the inversion or could we come up with something more elegant through ConverterParameter or an additional property? The first approach looks good to me because every converter will be independent of each other and we will avoid one additional step with the "unboxing" of converter's parameter.
Do we want to support things like Apple Watch, Tizen etc. Can we do it partially? I mean all cross-platform stuff can be used on those platforms.
Are we going for one big NuGet package or do we separate by type? (converters/effects etc.) Good question _________
Who will be the "Author" for this package on NuGet as it is a community effort with potentially a lot of authors? I guess Microsoft ?

Are we going for one big NuGet package or do we separate by type? (converters/effects etc.)

Multiple packages is somewhat cleaner as you only pull in what you want, on the other hand it brings a lot of overhead as well (both producer and consumer side). In the end @jamesmontemagno did merge all his tools into Xamarin.Essentials as well.

I'm inclined to agree with @bartlannoeye regarding single library. The simple single include in XAML pages makes things cleaner and the additional overhead from having packaged a few additional .cs files in the binary is insignificant to me.

@sthewissen what do you think about making a couple of converters more common?

StringIsNullOrEmptyConverter -> IsNullOrEmptyConverter
=> value == null || (value is string str && string.IsNullOrWhiteSpace(str));
StringIsNotNullOrEmptyConverter -> IsNotNullOrEmptyConverter

@sthewissen @jfversluis congratulations with the 1st one :)


test RAPL on AMD Zen2 or newer

QA-request
Description
Until now Azuki with her power-measuring was only tested on Intel-CPUs. Theoretically since the Zen2 architecture it should also work on AMD-CPUs. To be sure this still has to be tested.

will be part of https://github.com/kitsudaiki/Hanami-AI/issues/36


"Error: ArgumentError: x is either an observed nor a state variable." when run discrete_x = sol[x]

use the first code in the tutorial: https://docs.sciml.ai/MethodOfLines/stable/tutorials/heat/
is it a known issue?
or the code below.
using OrdinaryDiffEq, ModelingToolkit, MethodOfLines, DomainSets
# Method of Manufactured Solutions: exact solution
u_exact = (x,t) -> exp.(-t) * cos.(x)

# Parameters, variables, and derivatives
@parameters t x
@variables u(..)
Dt = Differential(t)
Dxx = Differential(x)^2

# 1D PDE and boundary conditions
eq  = Dt(u(t, x)) ~ Dxx(u(t, x))
bcs = [u(0, x) ~ cos(x),
        u(t, 0) ~ exp(-t),
        u(t, 1) ~ exp(-t) * cos(1)]

# Space and time domains
domains = [t ∈ Interval(0.0, 1.0),
           x ∈ Interval(0.0, 1.0)]

# PDE system
@named pdesys = PDESystem(eq, bcs, domains, [t, x], [u(t, x)])

# Method of lines discretization
dx = 0.1
order = 2
discretization = MOLFiniteDifference([x => dx], t)

# Convert the PDE problem into an ODE problem
prob = discretize(pdesys,discretization)

# Solve ODE problem
using OrdinaryDiffEq
sol = solve(prob, Tsit5(), saveat=0.2)

# Plot results and compare with exact solution
discrete_x = sol[x]

Are you using a recent DiffEqBase where sol is a PDESolution?

Are you using a recent DiffEqBase where sol is a PDESolution?

Hi Chris, thanks! I found that Atom restricted version of the packages. After removing Atom everything is solved


added scrollbar drag styling support

Added dragginCls param:
when user dragging scrollbar - this class adds to scrollbar, for best user experience and more native visualization while navigating in this manner.
(cause of without this solution when dragging scrollbar and move mouse out - it's no possible to show, that user interacts scrollbar, css :hover state and other pseudo classes not work)

https://github.com/Diokuz/baron/pull/100


teardown ansible error.

The Ansible teardown script encounters an issue when attempting to remove the "maas" package using the command specified in the "teardown.yaml" file. The command assumes that "maas" was installed via the "deb" package and attempts to remove it using the "apt" command. However, if "maas" was installed via the "snap" package, the command will fail because the regex pattern used in the command matches the "snap" package name.
On teardown.yaml for both roles/maas_rack_controller/teardown.yaml and maas_region_controller/teardown.yaml
# - name: Uninstall MAAS - deb
#   ansible.builtin.apt:
#     name: "maas*"
#     purge: true
#     state: absent

The tearown command uninstalls both snap and deb, irrespective of which method was used to install.
Have you encountered a breaking error with this method while using the playbooks? We can change it to use similar checks as in the install

I’ve encountered this as well. Marked as bug, will look into a fix asap

In theory we could parse "apt list " or "snap info " to determine the install method

I believe the wildcard  "maas*" breaks it.


Add support for Holden/Opel Astra BK

Longitudinal and lateral control tested and working in @knackerbrot's Astra. The Astra BK uses the same platform as the Volt, so most of it works identically. I've added its fingerprint (with ASCM unplugged), set the mass, wheelbase, etc, and skipped the stuff relating to the absent regen paddle.

Video: https://www.youtube.com/watch?v=vHFx5NM7gkY

looks great. Will merge after 0.5.6 gets released.

Thanks for reviewing :)

@AlexHill , pending conflicts resolution, this seems ready to be merged.

Thanks @rbiasini. I just tried the Github web-based conflict resolution tool for the first time and broke something - I'll rebase when I get home.

Ok, I think this is good to go now.


Build failed

Build: https://dev.azure.com/monacotools/a6d41577-0fa3-498e-af22-257312ff0545/_build/results?buildId=260926
Changes: https://github.com/Microsoft/vscode/compare/6bdee9d...639b71b

Did not see this for a while and seems random. Hence closing this.


Inconsistent Spelling and styles

Challenge Override Class Declarations with Inline Styles has an issue.
User Agent is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.96 Chrome/58.0.3029.96 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
Here is third paragraph's html :

Use an in-line style to try to make our h1 element white. Remember, in line styles look like this:

Which seems to me to be like :

Use an inline style to try to make our h1 element white. Remember, in line styles look like this:

@amritpalnagrame thanks for the issue. It appears you're pointing out that in-line has a hyphen in one sentence but not the other?

@erictleung yes, but it is also given styling of code.

@amritpalnagrame oh I see now. You're also comparing it to what is styled in the first paragraph, correct? Because style is stylized in the first paragraph but then has in-line style in the third paragraph?
If so, this was on purpose. in-line style is specific and we also add this kind of formatting to new words that we introduce. It should have different tags around it though. I do agree that it should be in-line style with a hyphen within it.

@erictleung Ok thanks


fix: Update to follow change in db_instance id

Description
A breaking change in the AWS provider changed the output of the upstream id attribute. Update the id module output to return the RDS DB instance identifier. This is an alternate solution to the one proposed in #495.
Motivation and Context
db_instance was updated by https://github.com/hashicorp/terraform-provider-aws/pull/31232 such that the id attribute now tracks the resource ID, and the identifier is now returned in a new attribute named identifier.
The resource ID is already available in instance_resource_id. This fix allows existing users to continue using this module with no changes and preserves the original function of the id attribute in this module.
Fixes the base issue reported in #495.
Breaking Changes
This should not represent a breaking change, as the referenced identifier attribute has
existed in db_instance since 2.33, which is well earlier than the current requirements for
this module.
How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[X] I have tested and validated these changes using one or more of the provided examples/* projects

[ ] I have executed pre-commit run -a on my pull request

This issue has been resolved in version 6.0.0 :tada:


Remove explicit type check when encoding

Current checks restricts to only a few types and raise errors with other types of Numeric. (e.g. BigDecimal)
It also seems like NUM2DBL will do some type-checking and raise TypeError https://webwallet.reddcoin.com/mri/source/object.c?v=2.3.8#3034

cc @BuonOmo

@BuonOmo  Thanks for checking! you're right I missed the free 🤦🏼
Yea this was meant to support more types with/without extra precision. Let me close this for now.


Support Laravel Herd

This PR adds Laravel Herd support, with fallback support to Valet.
The reason for prefering Herd over Valet, has to do with that Valet users are likely to upgrade to Herd, but Herd users aren't as likely to downgrade back to Valet. Simultaneously, when Herd is uninstalled, removing the folder that's being looked up here is part of the required steps, as per https://herd.laravel.com/docs/1/troubleshooting/uninstalling

Thanks, @claudiodekker.
I have also deprecated the valetTls config in favour of the more generic detectTls config option.
Updated docs: https://github.com/laravel/docs/pull/8925


[HW]

Name:
Email<EMAIL_ADDRESS>Linkedin Profile: <www.linkedin.com/in/aayushi-chaudhari-78212b106>
Here find the improvement needed in my homework.
Attach the homework screenshots below for both step II and step III:
Crud Document.docx

Hi Aayushi, thank you for sending this over. The steps look good from the exercises in the repo. The only thing I am missing are the 2 scenarios from step 3 of the homework steps.
3. Complete the following short courses: Cassandra Data Modeling and Cassandra Query Language. Take screenshots of the "Congratulations" page from each course.
Please provide those screenshots and we'll be good to go. :)

Ok, between this and your other submission we are good. I have all of the materials. Thank you.

Duplicate #97


Godot4: SteamNetworking fixes for options array size and memory leaks.

I was trying to work with the createListenSocketP2P for Steam's SteamNetworkingSockets and was running into an issue where my game would instantly crash whenever the method was called. After debugging it a little bit, I was able to narrow in on the length of the options array being passed was using the sizeof(options) to determine number of options passed. This sizeof returns an incorrect number of options, as that is the full Array object size, rather than the length of the internal Array. I swapped the sizeof to use Array's size() method and this fixed the crashes I was observing.
I also noticed that one of the GodotSteam network wrapper functions was correctly cleaning up the allocated Options array, but the other instances did not. I converted all of these non-conformant functions to match the correct one so that they all would correctly delete the memory allocated by convertOptionsArray(options).
I suspect these changes are also valid to backport to GodotSteam's support for Godot3, but I was only setup to test against Godot4.
Tested on a local Windows build of Godot latest master (5dccc940e73d39a1ac4f3d64ccc92373e6609add) and the latest of GodotSteam on the godot4 branch.
Tested with:
	var options= [];
	var result = Steam.createListenSocketP2P(0, options);

and
	var options = [[Steam.NETWORKING_CONFIG_TIMEOUT_INITIAL,1,15000]];
	var result = Steam.createListenSocketP2P(0, options);

Sidenote: The tuple of (STEAM_CONFIG_ENUM, ENUM_TYPE, VALUE) was a little confusing at first for passing an option. This also causes a crash in convertOptionsArray if the developer does not provide all three values. I'm unfamiliar with how Godot native modules should handle that situation, but I feel it would be best to confirm the tuple contains 3 items before referencing values out of it and if it doesn't, log out a runtime warning. I didn't fix this issue, but I am open for any suggestions to implement.

Hey there!  Thank you so much for fixing this!  Yeah, I keep forgetting the sizeof and size thing.  I can backport and test on 3.x; I need to patch some things in there.  And yes, convertOptionsArray, my super inelegant way of putting the array into a struct, I'm also not sure what the Godot way would be ideally.  I guess we'd just check to see if the array contained all three data points and it not, as you say, log a warning.
I'll get this added and do the backport.  I will think on the fix to convertOptionsArray but also feel free to tinker with it.  Thank you again!


Remove error summary override init function

Fixes https://github.com/alphagov/govuk-design-system/issues/1666
Note: this PR uses a pre-release of GOV.UK Frontend. The first commit will need to be deleted before merging.
What
Replaces the JavaScript override of the error summary init function with the new disableAutoFocus macro option instead.
Doing this means we have to introduce code.njk files for our error summary component examples, so that users viewing the HTML and Nunjucks code examples don't see the disableAutoFocus macro option/data attribute being set and accidentally copy that into their own service.
There should be no visual/behavioural changes as a result of this PR.

The visual examples on the error summary page (or anywhere else they're used on the Design System website - I couldn't find any other uses) shouldn't capture focus
The HTML/Nunjucks code examples shouldn't show the disableAutoFocus macro option or the data-disable-auto-focus attribute

Why
When we release v4.0.1 of GOV.UK Frontend, it'll include a change to allow disableAutoFocus to be passed to the error summary component. We don't want auto focus on the design system website - if we did have it enabled then if someone visited https://design-system.service.gov.uk/components/error-summary/ they would be autofocused on the examples.
At the moment we have a line of JavaScript code that overrides the error summary init function to disable the auto focus behaviour. With v4.0.1, we can replace this and use the disableAutoFocus macro option instead.

Oof, I hadn't twigged quite how many places we'd need to do this… that's quite a lot of 'duplicated' examples to maintain.
I feel like there must be a better way of doing this, but not sure what that is yet…
Maybe we should go with this and create an issue to try and find a better way? I still think this is a marginally better approach than monkey patching the init function.

@36degrees Yep, I think this is still better than what we've got now. I'd be happy to go with this (once 4.0.1 is released) and write up a separate card for how we approach separate code examples generally.

Written up an issue here https://github.com/alphagov/govuk-design-system/issues/2041


Is it support for iOS 6?

I can not find any API for iOS 7 and later, but the cocoapods said it is for iOS 7 and later.
Is it support for iOS 6?

Hi,
There are some iOS 7.0 specific API such as TextKit additions and new API in UIFont (UIFontDescriptor).
Regards,


Windows 10 "zoom flicker"

Hey there,
I don't know how to explain the issue I currently have, but on my Windows 10 Surface Book, the latest version of the application is buggier than the one before. When I hover some links, there's like a part of the application screen that zoom and it flickers on and off. It's disturbing.

uninstall completely, download the latest 2.0.2 release, reinstall - it should work properly :)

Can confirm @RemyFV is correct


Question: Lifetime for blob object in openblob

Thanks for your work on this library! It's exciting to see a library offer blob I/O support alongside the database/sql interface.
I'm working on porting PicoShare to use this library, and I have some questions about the blobio extension.
The ideal semantics for me are if I can open an io.ReadSeeker to a blob in my database and then later use it how I'd like (similar to how I'd open a file handle on my local filesystem).
Attempt 1: Saving a copy of blob outside the callback
I tried doing this:
func (d *DB) ReadEntry(id string) (io.ReadSeeker, error) {
	var blob *sqlite3.Blob
	_, err := d.db.Exec(`SELECT openblob('main', 'entries', 'contents', (SELECT rowid FROM entries WHERE id = :id), :writeMode, :callback)`,
		sql.Named("id", id),
		sql.Named("writeMode", false),
		sql.Named("callback",
			sqlite3.Pointer[blobio.OpenCallback](func(b *sqlite3.Blob, _ ...sqlite3.Value) error {
				log.Printf("start callback")
				blob = b
				log.Printf("end callback")
				return nil
			})))
	if err != nil {
		return nil, fmt.Errorf("error opening blob for id %s: %w", id, err)
	}

	if blob == nil {
		return nil, fmt.Errorf("no entry found for id %s", id)
	}

	if _, err := io.Copy(os.Stdout, blob); err != nil {
		panic(err)
	}

	return blob, nil
}

Which results in this error:
$ go run main.go
2024/09/06 08:43:05 start callback
2024/09/06 08:43:05 end callback
panic: sqlite3: bad parameter or other API misuse

goroutine 1 [running]:
main.(*DB).ReadEntry(0xc000596070, {0xc001d5fe40, 0x1c})
        /home/mike/code/nc-simple/main.go:84 +0x3b8
main.main()
        /home/mike/code/nc-simple/main.go:103 +0xc5
exit status 2

I assume it's because I'm not allowed to use the blob outside of the callback.
Attempt 2: Only read the blob within the callback
If I perform the copy within the callback, things work correctly:
func (d *DB) ReadEntry(id string) error {
	_, err := d.db.Exec(`SELECT openblob('main', 'entries', 'contents', (SELECT rowid FROM entries WHERE id = :id), :writeMode, :callback)`,
		sql.Named("id", id),
		sql.Named("writeMode", false),
		sql.Named("callback",
			sqlite3.Pointer[blobio.OpenCallback](func(b *sqlite3.Blob, _ ...sqlite3.Value) error {
				log.Printf("start callback")
				if _, err := io.Copy(os.Stdout, b); err != nil {
					panic(err)
				}
				fmt.Fprint(os.Stdout, "\n")
				log.Printf("end callback")
				return nil
			})))
	if err != nil {
		return fmt.Errorf("error opening blob for id %s: %w", id, err)
	}

	return nil
}

$ go run main.go
2024/09/06 09:03:18 start callback
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
2024/09/06 09:03:18 end callback

Full gist: https://gist.github.com/mtlynch/638e10675377515f15d9b086b929bb71
That said, only being able to interact with the blob from within the callback makes the semantics difficult for callers.
Context for what I'm doing
Ultimately, what I'm trying to do here is let HTTP clients write to and read files from the SQLite database by sending HTTP requests.
I currently do this with the mattn library and custom wrappers that expose an io.WriteCloser for writes and an io.ReadSeeker for reads.
The mattn library doesn't support blob I/O, so instead, my wrapper breaks up files into chunks of fixed size and then stores the chunks as distinct rows in a SQLite table.
Ultimately, what I'm hoping to do with your library is use the io.Reader interface from an HTTP multipart form POST and pass that to SELECT writeblob in the blobio extension of this library. And then I'd like to pass an io.ReadSeeker that I get from SELECT openblob and pass it to http.ServeContent for serving the data over HTTP.
The writing part seems like it should work fine, but I'm having trouble with the reading part.
Potential path
My best idea is to write my own wrapper around the SELECT openblob call that exposes the io.ReadSeeker interface but only calls openblob when a Read call occurs, kind of like this:
type wrapper struct {
  db *sql.DB
}

func (w Wrapper) Read(p []byte) (int, error) {
  var bytesRead int
_, err := w.db.Exec(`SELECT openblob('main', 'entries', 'contents', (SELECT rowid FROM entries WHERE id = :id), :writeMode, :callback)`,
		sql.Named("id", id),
		sql.Named("writeMode", false),
		sql.Named("callback",
			sqlite3.Pointer[blobio.OpenCallback](func(b *sqlite3.Blob, _ ...sqlite3.Value) error {
				n, err := b.Read(p)
                                bytesRead = n
				return err
			})))
  return bytesRead, err
}

func (w Wrapper) Seek(offset int64, whence int) (int64, error) {
  // TODO: Get this to affect the next Read call above
}

Does that make the most sense? Or is there a better way to achieve what I'm trying to do?

I'll have to read your entire issue with more care to give better suggestions, but the semantics of the openblob function is that you can only use the blob inside the callback.
In fact, for a single query, the callback may be called more than one once.
Consider this query:
SELECT openblob('main', 'entries', 'contents', (SELECT rowid FROM entries WHERE id = :id), :writeMode, :callback)

This could be written:
SELECT openblob('main', 'entries', 'contents', rowid, :writeMode, :callback)
  FROM entries
  WHERE id = :id

Now consider this modification:
SELECT openblob('main', 'entries', 'contents', rowid, :writeMode, :callback, id)
  FROM entries
  WHERE id GLOB 'id:2024-09-06T*'

If I understood your schema and id generation correctly, this would invoke your callback for all blobs created on 2024-09-06, and if you note the additional id argument to openblob, your callback gets not just the blob, but the id as an argument.
This is something we can't really do if the blob outlives the call. Also, to use this with database/sql you really can't have the blob outlive the call either. A sqlite3.Blob* is bound to a specific connection and database/sql is built around a pool. When your query ends, you release the connection you got from the pool, and by the time you want to play with the sqlite3.Blob*, your connection may be being used by someone else in some other goroutine.

@ncruces - Thanks, that's very helpful!
I'll keep playing around with my io.ReadSeeker idea and see if I can get that working in my app.

I'm not sure I understand why you can't start with your attempt 2, and use http.ServeContent instead of io.Copy?
Isn't http.ServeContent synchronous reading from the io.ReadSeeker and sending the necessary bytes?
I definitely would expect/want that to work (obviously, only a single blob should match your query).

I notice that http.ServeContent uses io.CopyN which is not the most efficient implementation:

can't use WriteTo
uses a really small buffer

This is unfortunate, but it means that, probably, sqlite3.Blob should be reworked to optimize Read a bit more.
The idea would be to allocate the internal buffer only once. This will save a malloc/free for every read. It's still far from optimal (WriteTo/ReadFrom are much better), but it's probably the best we can do under the circumstances.

I notice that http.ServeContent uses io.CopyN which is not the most efficient implementation:
* can't use `WriteTo`

* uses a really small buffer

Looking at io.CopyN it calls through io.Copy which does use WriteTo if the source implements is. So I think that should still be OK?

I'm not sure I understand why you can't start with your attempt 2, and use http.ServeContent instead of io.Copy?

Yeah, that should work.
The issue in my case is that my storage component shouldn't care about HTTP, and my HTTP handling code shouldn't care about SQLite. It's easy to keep the two components loosely coupled if I could grab an io.ReadSeeker out of the database and hand that to http.ServeContent, but it should be doable with a callback as well.
I'll either write a wrapper that abstracts the SQLite stuff behind an io.ReadSeeker or figure out a way for my storage code to accept a callback that calls http.ServeContent with the sqlite3.Blob.

Looking at io.CopyN it calls through io.Copy which does use WriteTo if the source implements it.

If I understand correctly since CopyN wraps the source in LimitedReader, which does not implement WriterTo, io.Copy won't use it even if the source passed to CopyN does implement it.

If I understand correctly since CopyN wraps the source in LimitedReader, which does not implement WriterTo, io.Copy won't use it even if the source passed to CopyN does implement it.

Yes, this is the reason. Then there's TCPConn.ReadFrom but that eventually gives up (if it can't splice or sendFile) and you're back at io.copyBuffer called with no buffer which uses a 32Kb buffer.
The reason I'm calling this out is that:

sqlite3.Blob.Read will malloc/sqlite3_blob_read/free each time which has overhead [^1]
despite what it looks like, sqlite3_blob_read is not random access

A large BLOB in SQLite is a linked list of pages, and even sequential sqlite3_blob_read don't remember where they were. So you'll be traversing the linked list every time. The 1Mb buffer I use in WriteTo/ReadFrom is to amortize this cost. It's still not the best idea to store very large BLOBs this way. But it's OK for smallish MB sized BLOBs and not terrible for tens of MBs.
[^1]: one of the main reasons this database/sql driver can beat other drivers is that it can make a single call to read/scan all columns in a row at once.

9d77322d5000eaab6fb827488f78319397cfcfb9 exposes sqlite3_realloc and fixes some memory management issues (some APIs expect memory from sqlite3_malloc and I was giving them malloc).
The only odd ball now is aligned_alloc which, given how it's used (memory is never actually freed), could be done through sqlite_malloc/sqlite3_realloc.

For reusing the same buffer for all calls to Read/Write/WriteTo/ReadFrom, 336a582062996fe3d3be4542633c2f88f6d4e550 should do it (assuming they are consistent in the buffer size used).

I'll leave this open while you work on https://github.com/mtlynch/picoshare/pull/567.
If/when you're closer to done, if you want me to tag these/other improvements, feel free to ping me.

Been thinking a bit more about this. Really curious what the performance will be like.
If we need to amortize reads better with larger buffers (because of io.LimitedReader), my bet is that the sqlite3.Blob should simply be wrapped in a bufio.Reader, like so:
type BufferedReadSeeker struct {
	bufio.Reader
	s io.ReadSeeker
}

func NewBufferedReadSeeker(r io.ReadSeeker, size int) *BufferedReadSeeker {
	return &BufferedReadSeeker{Reader: *bufio.NewReaderSize(r, size), s: r}
}

func (b *BufferedReadSeeker) Reset(r io.Reader) {
	b.s = r.(io.ReadSeeker)
	b.Reader.Reset(b.s)
}

func (b *BufferedReadSeeker) Seek(offset int64, whence int) (int64, error) {
	b.Reader.Reset(b.s)
	return b.s.Seek(offset, whence)
}

Thanks for all your work on this!
Also, just want to clarify that my donation was based on appreciation of past work, so I hope it didn't feel like there were strings attached or that I expected it to be enough to compensate you for further feature work here.
I've got a version of PicoShare's working with your driver, and it works locally, but I'm seeing strange errors in my end-to-end tests and when I deploy the code to a real cloud server. I'm going to try another test, but my schedule is a bit wacky for the next few weeks/months.
I tried uploading a 1.5 GB file to a PicoShare instance on fly.io with the existing code and one with the nc-sqlite branch I'm working on. Your driver was outperforming the mattn driver by about 30% in terms of upload speed to the server, but then when the upload reached 100%, it failed with "SQL logic error." I ran this test on September 8th, so I think I was pulling in c1915feb2e5b88fb3f4bb8fe5a394261a0ff0eff of your driver.
For a bit of additional context, for PicoShare's needs, I/O speed is not as important as memory overhead. In my current implementation where I break the file into chunks with the mattn driver, on a small 256 MB RAM cloud server, things work fine until files get to be about 1 GB, and then I start seeing memory exhaustion for files larger than that (https://github.com/mtlynch/picoshare/issues/355). Throwing a ton of server resources at it solves the problem, but I get a lot of reports from users about crashes or sluggish performance on large files.

OK so... First of all, congrats! Second, thanks for the donation! Third, I feel no pressure from the donation, and I also hope you don't feel any pressure from my eagerness: please, enjoy your family time!
The reason I've been this engaged is that I always envisioned the blob API being used for this exact use case... and you (accidentally?) volunteered as guinea pig.
Looking over your code, I see one potential issue: using last_insert_rowid() for writeblob (which I did on the example, but "fixed" in 356dd56e5f1d79c3dec914cb981e9cea362608f9). The issue there is that last_insert_rowid() is "per-connection" and depending on what your Store.ctx is the Execs might be run on a connection pool with multiple connections, and last_insert_rowid() may run on a different connection than the INSERT.

Thanks!
Okay, I tried deploying a fix to a 1x-shared-CPU-256MB-RAM fly.io server based on https://github.com/mtlynch/picoshare/pull/567/commits/dfbedf814e363ed6c1896511ed8cda50b4e9f701, which includes your last_insert_rowid fix.
The app uses about 83 MiB of 213 MiB of available RAM at idle, and I can upload small files fine.  Should I try rearchitecting it to break up the file into smaller blobs that span several rows rather than sticking files of arbitrary length into a single blob per file?

I would not expect this to cause any memory issues. Let me look into it.
I'm not sure I can replicate fly specifically without creating an account with them, but I have a small hosted VM I can try. It'll take me time though.

This is odd: app/store/sqlite/entries.go:172 is the INSERT that creates the blob, not the SELECT that writes it. It really shouldn't allocate that much memory.
Tangent:
Other than that it is failing in a way that is actually very interesting to me, so I'll look into this further. When the Go runtime can't allocate memory, that's usually a catastrophic event that can't be recovered from. So wazero was designed assuming this, because wasm memory is just a []byte that we append. I then extended wazero to to allow custom memory allocators, and use mmap to back that []byte. Not being able to mmap can be handled, but I'm panicking, so sbrk crashes instead of ENOMEM, and malloc and SQLite don't get a chance to handle it more gracefully. This could work better.
Other than that, my assumption is that as your VM is memory strapped, and since each connection lives under the illusion that it can allocate 4GB of memory, it tries to, and eventually fails. You could try something like this, to force each connection to live with 32MB of memory (and maybe SetMaxOpenConns if not doing so already):
sqlite3.RuntimeConfig = wazero.NewRuntimeConfig().
	WithMemoryLimitPages(512)

But this is more configuration than should be needed, and I'm not sure it works. I guess for memory constrained devices setting the soft/hard limits would be better, but are disabled in the current build.
This driver probably does consume more memory than mattn, and I wasted your time. 😔

I'm not sure I can replicate fly specifically without creating an account with them, but I have a small hosted VM I can try. It'll take me time though.

Were you able to repro? I realized I could repro a little easier using Docker's resource limits (docker run --memory 256m)
I made a branch that makes the repro easier:
https://github.com/mtlynch/picoshare/tree/deploy-ncruces
Script for spinning up the Docker container is ./dev-scripts/serve-docker. The problem is that I don't see the log messages about the crash like I do on fly.io. I just see the logs stop and the container dies when it tries to write the blob.

Other than that, my assumption is that as your VM is memory strapped, and since each connection lives under the illusion that it can allocate 4GB of memory, it tries to, and eventually fails.

Is 4 GB an assumption of SQLite itself or ncruces/go-sqlite3?

You could try something like this, to force each connection to live with 32MB of memory (and maybe SetMaxOpenConns if not doing so already):

I just tried this, but I continue seeing the crash under resource-capped Docker.

This driver probably does consume more memory than mattn, and I wasted your time. 😔

Aw, are you sure? I feel like there has to be some way of writing an arbitrarily large file to SQLite with only 256 MB of RAM.
256 MB isn't even a hard requirement. If I can tell users that arbitrarily large files will work as long as they have at least N GB of RAM, that would work too as long as N isn't something crazy.

Were you able to repro? I realized I could repro a little easier using Docker's resource limits (docker run --memory 256m)

No, I didn't have the time. I'll try though, as I'm interested in the wazero side of this too.

Is 4 GB an assumption of SQLite itself or ncruces/go-sqlite3?

It's a Wasm assumption. Each Wasm module is like a sandbox/VM, and my SQLite module is default configured to max out at the maximum possible Wasm memory (4G). Each connection lives in its own sandbox. But memory is allocated as needed, and an Exec with a zeroblob shouldn't need much.
What this makes hard is to inform SQLite of the memory restriction so it can adapt. Your machine has a lot of memory, the container has much fewer memory (and Go may or may not know this), Wasm thinks there's a lot of memory, and so does SQLite. When a malloc fails there's probably lots of cached memory that could be freed, and lots of GC that could free up memory, but it doesn't know, so it panics.
Beyond improving what happens on malloc failure (SQLite is supposed to be able to deal with that, if the failure is reported instead of crashing), I should probably expose the soft/hard limit APIs. It's also possible that tweaking the page cache size, and cache spill settings.
But it is a fact that this is less optimized for low memory than a Cgo driver. Maybe it makes sense to push for a better blob API on the Cgo driver: https://github.com/mattn/go-sqlite3/issues/239#issuecomment-1806762291

One thing with the container; Go is not cgroup-aware, so it doesn't know that it doesn't have the hosts' full memory at its disposal when running in a container.
There's an automemlimit package to help with that, or you can set GOMEMLIMIT to about 90% of the amount of memory assigned to the container.

It's a Wasm assumption. Each Wasm module is like a sandbox/VM, and my SQLite module is default configured to max out at the maximum possible Wasm memory (4G). Each connection lives in its own sandbox. But memory is allocated as needed, and an Exec with a zeroblob shouldn't need much.

If I made a PicoShare-specific fork of ncruces/go-sqlite3, would it be practical for me to cap the wasm memory to something smaller? Or is there any possibility of making this configurable or dynamic in the main implementation based on system RAM?

But it is a fact that this is less optimized for low memory than a Cgo driver. Maybe it makes sense to push for a better blob API on the Cgo driver: https://github.com/mattn/go-sqlite3/issues/239#issuecomment-1806762291

I don't think anyone's willing and able to do feature development on the mattn implementation. There hasn't been much activity there for a while.

Beyond improving what happens on malloc failure (SQLite is supposed to be able to deal with that, if the failure is reported instead of crashing), I should probably expose the soft/hard limit APIs. It's also possible that tweaking the page cache size, and cache spill settings.

Cool, I'd be interested in re-trying this if you end up implementing that. Your library seemed like a potential solution to this memory exhaustion issue I've had, but I don't need the fix urgently, as I'm able to use PicoShare fine for 99% of my needs without supporting 1 GB files.

I already uncovered and fixed a bug in the allocator in fdfaaa8cec75c240c41f83817a58521b59591523. I'll have to make a release shortly because this can cause data corruption (not silent data corruption, only in a OOM crash).
You can reduce memory usage today with:
sqlite3.RuntimeConfig = wazero.NewRuntimeConfig().
	WithMemoryLimitPages(512)

But then you're likely to hit the bug above. Each Wasm page is 64K, so that's 32MB. I've got a 150MB Blob inserted with as little as 4MB memory per connection, though.
The problem is that this limits SQLite to X MB, but SQLite doesn't know. The first time it finds out is when a malloc returns NULL. Then, depending on what's being allocated it may try to release memory and try again. Or it may give up. Ideally, you say the limit is X, and then tell SQLite "make do with X-N." This needs some additional APIs.
And if you use those APIs, maybe you don't need to limit Wasm memory after all, it can grow dynamically, and malloc is good enough about recycling memory.
It's not out of the question that I have some leak, somewhere, because really this shouldn't require that much memory. I should also look over your PRAGMAs to see if you're using more memory than necessary. Cloud VMs also often add an issue where you think you're using temporary files, but these are actually RAM that counts against your allotment.

Updated the above comment. Ideally I add these new APIs, and it's a matter of calling them, and getting to a reasonable low memory configuration (smaller page cache, and/or a soft memory limit). Then you can have a few connections (like 4) each keeping under 32Mb (with a soft limit of say 16 or 24Mb) with room to spare for Go and everything else.
For a low memory environment that should be plenty, and you can do that with a few PRAGMAs.

I already uncovered and fixed a bug in the allocator in fdfaaa8. I'll have to make a release shortly because this can cause data corruption (not silent data corruption, only in a OOM crash).

I tested with this version but see the same behavior, unfortunately.

Ideally I add these new APIs, and it's a matter of calling them, and getting to a reasonable low memory configuration (smaller page cache, and/or a soft memory limit). Then you can have a few connections (like 4) each keeping under 32Mb (with a soft limit of say 16 or 24Mb) with room to spare for Go and everything else.

Cool! If you do add these APIs, let me know, and I'm happy to test on PicoShare again.

Cool! If you do add these APIs, let me know, and I'm happy to test on PicoShare again.

I haven't tested much, but these should be in v0.18.3.
I would expect setting the soft_heap_limit PRAGMA to constrain memory usage.
I would also suggest looking into PRAGMAs: cache_size, cache_spill and temp_store.
I haven't got round to testing PicoShare proper, because my work machine is severely constrained, but I plan to. As I said, I was able to insert 100MB blobs into a database with writeblob with as little as 4MB of memory dedicated to each connection, so something else should be the problem (not necessarily PicoShare, but not the blobio extension in isolation).

I'm guessing, if the problem persists, it might be related to this comment in your source code: "We deliberately don't use a transaction here, as it bloats memory..."
With the blob API the code becomes transactional, because even if you don't do a transaction, inserting the (empty) blob is a single statement (so transactional) and updating it is, again, a single statement. It might actually be cheaper to wrap them in a transaction.
The transaction should spill to disk if it gets too big, though, and statement journals, which are stored as temporary files, so could end up in memory in a container, should not be involved.
So I'm still not sure what's happening here.

Okay, thanks! I tried again with 0.18.3 and unfortunately am seeing the same crash when I deploy to Fly / constrained Docker.
https://github.com/mtlynch/picoshare/tree/120278e89829335e8b95de4c8eea4eac5ef665d6

As I said, I was able to insert 100MB blobs into a database with writeblob with as little as 4MB of memory dedicated to each connection, so something else should be the problem (not necessarily PicoShare, but not the blobio extension in isolation).

Yeah, that makes me realize I should create a simpler test case. I'm thinking my next experiment will be:

Create a simple Go program that reads and writes 1000 MB to SQLite via the blobio extension
If that works, wrap it in a simple web app that allows the user to upload arbitrary files and read them back
If that works, figure out what in PicoShare is causing the difference from (2)

This was my attempt:
package main

import (
	"database/sql"
	"log"
	"os"
	"path/filepath"

	"github.com/ncruces/go-sqlite3"
	_ "github.com/ncruces/go-sqlite3/driver"
	_ "github.com/ncruces/go-sqlite3/embed"
	"github.com/ncruces/go-sqlite3/ext/blobio"
	"github.com/tetratelabs/wazero"
)

func main() {
	sqlite3.RuntimeConfig = wazero.NewRuntimeConfig().WithMemoryLimitPages(64)
	sqlite3.AutoExtension(blobio.Register)

	var err error
	db, err := sql.Open("sqlite3", "file:test.db?_pragma=journal_mode(WAL)")
	if err != nil {
		log.Panic(err)
	}
	defer db.Close()

	_, err = db.Exec(`CREATE TABLE IF NOT EXISTS files (name, data)`)
	if err != nil {
		log.Panic(err)
	}

	for _, path := range os.Args[1:] {
		func() {
			file, err := os.Open(path)
			if err != nil {
				log.Panic(err)
			}
			defer file.Close()

			info, err := file.Stat()
			if err != nil {
				log.Panic(err)
			}
			log.Println("size", info.Size())

			r, err := db.Exec(`INSERT INTO files (name, data) VALUES (?, ?)`,
				filepath.Base(path), sqlite3.ZeroBlob(info.Size()))
			if err != nil {
				log.Panic(err)
			}

			id, err := r.LastInsertId()
			if err != nil {
				log.Panic(err)
			}
			log.Println("id", id)

			_, err = db.Exec(`SELECT writeblob('main', 'files', 'data', ?, 0, ?)`,
				id, sqlite3.Pointer(file))
			if err != nil {
				log.Panic(err)
			}
		}()
	}
}

Cool, thanks! That's interesting. Using your simple app, I'm able to repro the crash even on a system with lots 24 GB of 32 GB of RAM free.

A bit of progress. It turns out that the OOM was from this line:
sqlite3.RuntimeConfig = wazero.NewRuntimeConfig().WithMemoryLimitPages(64)

I bumped it to 512, the OOM went away, even when I used Docker to limit RAM to 128M:
https://github.com/mtlynch/picoshare/tree/ncruces-simple-2
See: https://github.com/mtlynch/picoshare/blob/ncruces-simple-2/scripts/test-under-docker
The strange thing is that any pragma that Litestream recommends causes a crash, even though none of the pragmas seem like they'd bloat memory:
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA wal_autocheckpoint = 0;

I've tried applying only one pragma at a time, and the test program always crashes, and I've tried with none of them enabled and the test program always succeeds with a ~1 GB file and 128M of RAM. I thought it was just luck, but I've been able to reproduce it 10x with the pragmas all disabled and with them individually enabled.
Do you have any ideas why that could be happening?
The other issue I ran into was that if I tried a file larger than 1 GB, it failed with  SQLITE_TOOBIG. Is it possible to increase the limit through go-sqlite3 and the database/sql interface? I found the Limit API, but it looks like that's not available through the database/sql interface.

Interestingly setting PRAGMA wal_autocheckpoint = 0 alone causes OOM crash even when I run the application outside of Docker with 24 GB of RAM available:
https://github.com/mtlynch/picoshare/blob/ncruces-simple-3/scripts/test
~/code/picoshare (ncruces-simple-3)$ ./scripts/test
+ rm test.db test.db-shm test.db-wal
++ ./scripts/build
+ readonly BINARY_DIR=bin
+ BINARY_DIR=bin
+ mkdir -p bin
+ readonly BINARY_NAME=bin/test-ncruces
+ BINARY_NAME=bin/test-ncruces
+ go build -tags netgo -o bin/test-ncruces main.go
+ echo bin/test-ncruces
+ BINARY_NAME=bin/test-ncruces
+ go build -tags netgo -o bin/test-ncruces main.go
++ ./scripts/make-data-file
++ mktemp
+ TEST_FILE=/tmp/nix-shell.qzdm97/tmp.s6fSehu6Ao
+ readonly TEST_FILE
+ dd if=/dev/urandom of=/tmp/nix-shell.qzdm97/tmp.s6fSehu6Ao bs=1M count=952
952+0 records in
952+0 records out
998244352 bytes (998 MB, 952 MiB) copied, 2.40168 s, 416 MB/s
+ echo /tmp/nix-shell.qzdm97/tmp.s6fSehu6Ao
+ TEST_FILE=/tmp/nix-shell.qzdm97/tmp.s6fSehu6Ao
+ readonly TEST_FILE
+ ./bin/test-ncruces /tmp/nix-shell.qzdm97/tmp.s6fSehu6Ao
2024/09/14 11:03:38 starting up!
2024/09/14 11:03:39 size 998244352
2024/09/14 11:03:40 id 1
panic: sqlite3: out of memory (recovered by wazero)
wasm stack trace:
        env.go_shm_map(i32,i32,i32,i32,i32) i32
        sqlite3.wasm.walIndexPageRealloc(i32,i32,i32) i32
        sqlite3.wasm.walIndexAppend(i32,i32,i32) i32
        sqlite3.wasm.pagerWalFrames(i32,i32,i32,i32) i32
        sqlite3.wasm.pagerStress(i32,i32) i32
        sqlite3.wasm.getPageNormal(i32,i32,i32,i32) i32
        sqlite3.wasm.accessPayload(i32,i32,i32,i32,i32) i32
        sqlite3.wasm.sqlite3BtreePutData(i32,i32,i32,i32) i32
        sqlite3.wasm.blobReadWrite(i32,i32,i32,i32,i32) i32
        sqlite3.wasm.sqlite3_blob_write(i32,i32,i32,i32) i32 (recovered by wazero)
wasm stack trace:
        env.go_func(i32,i32,i32,i32)
        sqlite3.wasm.go_func_wrapper(i32,i32,i32)
        sqlite3.wasm.sqlite3VdbeExec(i32) i32
        sqlite3.wasm.sqlite3_step(i32) i32

goroutine 1 [running]:
github.com/ncruces/go-sqlite3.(*sqlite).call(0xc002ed2a88, {0x68af1e, 0xc?}, {0xc002c7d728?, 0x6e3f10?, 0xc002c7d668?})
        /home/mike/code/go/pkg/mod/github.com/ncruces/go-sqlite3@v0.18.3/sqlite.go:181 +0x1ad
github.com/ncruces/go-sqlite3.(*Stmt).Step(0xc003712000)
        /home/mike/code/go/pkg/mod/github.com/ncruces/go-sqlite3@v0.18.3/stmt.go:109 +0x5f
github.com/ncruces/go-sqlite3.(*Stmt).Exec(0xc003712000)
        /home/mike/code/go/pkg/mod/github.com/ncruces/go-sqlite3@v0.18.3/stmt.go:133 +0x1f
github.com/ncruces/go-sqlite3/driver.(*stmt).ExecContext(0xc003712030, {0x6e3f10, 0xa14c00}, {0xc003710050?, 0x0?, 0xc002c7d7e0?})
        /home/mike/code/go/pkg/mod/github.com/ncruces/go-sqlite3@v0.18.3/driver/driver.go:468 +0xd8
database/sql.ctxDriverStmtExec({0x6e3f10, 0xa14c00}, {0x6e3fb8, 0xc003712030}, {0xc003710050, 0x2, 0x2?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/ctxutil.go:65 +0xa3
database/sql.resultFromStatement({0x6e3f10, 0xa14c00}, {0x6e3bf8, 0xc0000a40f0}, 0xc002c7d9c8, {0xc002c7dd60, 0x2, 0x2})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:2670 +0x13a
database/sql.(*DB).execDC(0x9b5501?, {0x6e3f10, 0xa14c00}, 0xc0000ee000, 0x0?, {0x698e87, 0x32}, {0xc002c7dd60, 0x2, 0x2})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1722 +0x42c
database/sql.(*DB).exec(0xc0000b09c0, {0x6e3f10, 0xa14c00}, {0x698e87, 0x32}, {0xc002c7dd60, 0x2, 0x2}, 0x78?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1683 +0xd5
database/sql.(*DB).ExecContext.func1(0xc?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1662 +0x4f
database/sql.(*DB).retry(0x4c8f40?, 0xc002c7dc68)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1566 +0x42
database/sql.(*DB).ExecContext(0xc0000a6150?, {0x6e3f10?, 0xa14c00?}, {0x698e87?, 0xc0000ee000?}, {0xc002c7dd60?, 0x69b71b?, 0xf5?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1661 +0xc8
database/sql.(*DB).Exec(...)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/database/sql/sql.go:1675
main.main.func1({0x7fffffff6595, 0x24}, 0xc0000b09c0)
        /home/mike/code/picoshare/main.go:69 +0x49f
main.main()
        /home/mike/code/picoshare/main.go:76 +0x2e5

Yeah, I was about to say that. This is related to the shared memory index used by WAL. There might be a bug on my end (though looking at the code, it's not apparent, I'll look into it more).
So, the way this works is:

In WAL mode, a new transaction appends pages to the WAL.
Because there's simply appended, SQLite needs an index to quickly find page N in the WAL.
The size of this index is proportional to the number of changed pages in the WAL.
Because you're doing inserts, lots of pages are added (not updated) so the index grows.
This cleared when the WAL is check pointed, and pages are copied to the main database file.
But you've have disabled check points (litestream is supposed to handle them, but you're probably not running litestream).
So the WAL, and the index, grows without bound.
Compounding this, my driver loads one copy of the index per connection, instead of one copy for the entire process.

One thing that I've noticed improves the situation is to wrap the INSERT and SELECT writeblob into a transaction. The INSERT creates zeroed pages in the WAL, the SELECT updates them in place. So, half the size written to the WAL and indexed.
Still, I saw the WAL index using 256Kb × [number of connections] for 100MB files, so it shouldn't be that bad.
I really need to test your code under Linux to see if this is an OS issue.

Right, so a transaction roughly cuts the size of the index needed in half. Still, the index for your sample 512MB file, is around 2MB. I'm sure there's some space lost to fragmentation, but it's 2MB.
Never checkpointing the WAL though, makes this grow forever. So if you're doing PRAGMA wal_autocheckpoint = 0, you need to run Litestream. Not because of one insert, but definitely in a running service.

The other issue I ran into was that if I tried a file larger than 1 GB, it failed with  SQLITE_TOOBIG. Is it possible to increase the limit through go-sqlite3 and the database/sql interface? I found the Limit API, but it looks like that's not available through the database/sql interface.

I had missed this. The maximum size of a BLOB is 1,000,000,000. This can be raised at compile time to 2GB (adding SQLITE_MAX_LENGTH to sqlite_opt.h); you'd need to build a custom sqlite3.wasm, tough. The size can only be lowered at runtime, not raised. I could raise the default limit (although I'm weary of changing defaults), but I'm guessing for really large files, you might need to split them anyway. I'm sorry.

About APIs that are only available through the lower level API. If it's something you want to change for every new connection:
sqlite3.AutoExtension(func(c *sqlite3.Conn) error {
	// this runs whenever any database connection is opened
	return nil
})

Or:
db, err := driver.Open("file:test.db",
	func(c *sqlite3.Conn) error {
		// this runs whenever a connection to this database is opened
		return nil
	},
	func(c *sqlite3.Conn) error {
		// this runs before a connection to this database is closed
		return nil
	}
)

If it's something you want to do in a specific connection, at a specific moment:
conn, err := db.Conn(context.TODO())
if err != nil {
	log.Fatal(err)
}

err = conn.Raw(func(driverConn any) error {
	var c *sqlite3.Conn = driverConn.(driver.Conn).Raw()
	// we have a low level Conn, lets Backup
	return c.Backup("main", "backup.db")
})
if err != nil {
	log.Fatal(err)
}

So I finally ran this on Linux as well.
The WAL index takes around 2MB per 1GB of WAL file. So with a 1GB WAL, SQLite seems to manage fine with just 8MB: 2MB for the default page cache, 2MB for the WAL index, the other 4MB are "overhead", but I can't really fault it.
So the problem here seems to be:

huge transactions,
disabling auto checkpoints, and/or
Litestream not being timely about checkpointing the database.

When you put those 3 things together, yeah SQLite is going to consume unbounded memory, and my driver adds overhead by siloing each connection. [^1]
But WithMemoryLimitPages(256) (16MB per connection) should be OK for most apps running in low memory settings. I was able to write a 998MB BLOB with half as much as that.
Other than that I can only stress that if you disable auto checkpoints to make Litestream happy, you must run Litestream, and maybe tweak it to monitor the database more often: https://github.com/benbjohnson/litestream/issues/100. Wrapping each INSERT and SELECT writeblob pair in a transaction will also help (Litestream won't monitor in the middle of those and issue a checkpoint, like SQLite would).
[^1]: Little known fact: WAL is way more inefficient than rollback for writing this much new data to a database. But that's irrelevant since people want to use Litestream.

For BLOBs larger than SQLITE_MAX_LENGTH, I guess a CRUD library that handles chunking, etc, would be valuable.
I'm not sure I'm up to the task though.

Okay, thanks for digging so deeply into this!
I'll experiment more with my sample program and PicoShare and see if I can get this working by breaking the BLOBs into chunks and using transactions.

Other than that I can only stress that if you disable auto checkpoints to make Litestream happy, you must run Litestream, and maybe tweak it to monitor the database more often: https://github.com/benbjohnson/litestream/issues/100. Wrapping each INSERT and SELECT writeblob pair in a transaction will also help (Litestream won't monitor in the middle of those and issue a checkpoint, like SQLite would).

That should be okay for PicoShare. Because it's not like an app where lots of users are interacting simultaneously and you can lose important data if you throw away the last minute of database writes. In the case of PicoShare, the worst that's probably going to happen is that you lose the last file you uploaded, which probably happened anyway if PicoShare crashed. Or maybe you lose a minute of analytics, but that's not a big deal either.

For BLOBs larger than SQLITE_MAX_LENGTH, I guess a CRUD library that handles chunking, etc, would be valuable.
I'm not sure I'm up to the task though.

Okay, that's fine. I've already implemented this for the mattn driver, so it should be pretty straightforward to modify it for your driver.

That should be okay for PicoShare. Because it's not like an app where lots of users are interacting simultaneously and you can lose important data if you throw away the last minute of database writes.

What I meant was, if auto checkpointing is disabled, with two transactions (one to reserve space for the BLOB, the other to write bytes to it) the WAL grows faster, and the index needs more memory.
Because Litestream will not check at the "right" time, and checkpoint the WAL.
With auto checkpoints, if the BLOB is big enough, SQLite decides the WAL grew too much and checkpoints between the transactions.
I'll close this for now as the discussion long outlived the original question.
I still fixed an important bug, improved wazero, did some BLOB improvements, and gained a contributor, so thanks!
Feel free to open more issues.


Report The Author Warning

InteractiveBooks: 1.6.3
Server: git-Purpur-1957 (MC: 1.19.4)
For old versions working correctly
Example: git-Purpur-1948 (MC: 1.19.4)

It was a library not updated to Minecraft 1.19.4.


Program Block edit page has multiple forms. When one form is submitted, unsaved changes in other forms trigger a warning.

Is your feature request related to a problem? Please describe.
Program block edit page has multiple forms, but only one can be submitted without a page reload.
Describe the solution you'd like
If there are changes in multiple forms, changes in forms that aren't being submitted are caught and a warning is triggered to the user.
Alternatively, could it be one giant form?
Additional context
Add any other context or screenshots about the feature request here.
Done when
What conditions need to be met before this issue can be closed?

Related to #1050 and #995

[DUPLICATE #995]
Solution: every action on this page is an atomic form submission, so we'll make the block name/description edit atomic by using a modal that forces the admin to save it. Then everything on this view will behave the same.


Libdoc html in-browser switch to list one shortcut per line

Hi,
when reading the library documentation in a web browser, I would like to have a list of the "shortcuts" (keywords) one per line, because for me it seems easier to read.
With the attached pull request, the HTML page created by libdoc looks like this:

A click on "One per line" switches the list to one keyword per line:

The new optional "One per line" shortcuts list style takes more vertical place. However, it may be easier to spot a specific keyword than in the compact list. It is possible to switch back to the original style in the browser.
Tested for the libraries from Robot Framework source and for SeleniumLibrary in Google Chrome 84, Mozilla Firefox 79 and Internet Explorer 11.
Best Regards,
Rudolf

Interesting idea. Could you please submit an issue about this enhancement idea for tracking purposes? You could possibly also ask comments from the community on out Slack or mailing list.
Did you think about other alternatives for the widget controlling the behavior? Could it perhaps be some king of a toggle switch?

Codecov Report

Merging #3634 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3634      +/-   ##
==========================================
- Coverage   74.19%   74.17%   -0.01%
==========================================
  Files         199      199
  Lines       16477    16482       +5
  Branches     2657     2660       +3
==========================================
+ Hits        12223    12224       +1
- Misses       3821     3826       +5
+ Partials      433      432       -1

Impacted Files
Coverage Δ

src/robot/running/arguments/typeconverters.py
58.12% <0.00%> (-1.26%)
:arrow_down:

src/robot/running/testlibraries.py
75.95% <0.00%> (+0.32%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dd9a3ca...4501c9a. Read the comment docs.

Thanks for the feedback. I like that kind of switch and changed the code of this pull request to implement it.
I created an issue: https://github.com/robotframework/robotframework/issues/3635

Tested this locally and I like it! Some fine tuning could still be done, though:

The toggle styles could be enhanced. There could possibly be text on both sides (not sure what yet) and possibly the text and the toggle could have some light background.

Instead of rendering shortcuts twice and hiding one set based on the toggle position, it ought to be possible to render them once and then change · with
 depending on the toggle.

I'll experiment with both of these locally.

Maybe a text like:
"Compact list" - "Expanded list"
"Terse" - "Enlarged"
"Deflated" - "Inflated"
"Minimum height" - ?
"All in one line" - "One per line"
"Only one line" - "Multiple lines"
"Crammed" - "Easy to read"

I've been doing some fine tuning locally. The basic functionality is exactly the same, but these have been changed:

Both the slider and the text around it are a bit smaller.
Background color stays the same regardless the switch position.
Text is now List style: Compact [x ] Expanded but been trying also Compact list [x ] Expanded list.
Also tags have same functionality as shortcuts.
The separator between items is toggles between · and
 instead of rendering both lists and toggling which is shown.

I really like the original idea and think the small enhancements above make it even better. I still have an idea to see how easily the toggle position could be saved persistently in browsers local storage. Our reports have such functionality and if the code is easily reusable it would be nice addition to this feature as well. After checking that I'll commit the changes. I'll either update this PR and then merge it to master or merge the PR first and do changes then.

Probably easiest for me to just merge this PR and then do changes afterwards. I hope you @Rudolf-AT can review the changes once they are committed. And thanks for the PR!

I have to admit that I'm happy that you like this idea :)
One very minor thing:
if a libdoc html has only two or three keywords, meaning no switch, the shortcuts are shown in the expanded style if the persistet state is expanded.
Steps to reproduce:

load "BuiltIn.html"
set "expanded"
load "Screenshot.html" (has three keywords)
-> Shortcuts are shown expanded, but should be compact.
Maybe ignore. Or maybe change the ".length > 3" to ".length > 1" for shortcuts and tags. Easy, no surprises, why not show the switch for 2 keywords as well.

Everything else seems to be ok/as expected.

BuiltIn: in compact list: shortcut links to anchor
BuiltIn: in expanded list: shortcut links to anchor
SeleniumLibrary: all shortcuts are in compact list
SeleniumLibrary: all shortcuts are in expanded list
All shortcuts are shown if only three keywords exist
State is persistent: open html-1 and switch to expanded, open html-2 in another tab: expanded list is shown
Quick general check ok for: Firefox 80, Google Chrome 85, Internet Explorer 11
Quick check in browser console: no error

Thanks for review! Showing expanded list and no switch with less than four keywords is definitely a bug. I agree showing the toggle when there's more than one keyword is the best solution. Hiding it with 2-3 kws doesn't bring any real benefits and would require some extra logic.


Updated 'stylize_video.py'

I added an implementation of batch processing. You can treat in-path as a directory that will iterate through the different videos in the dir. Same applies with styles when given a style folder.

Looks good, thanks!


renderRightButton and renderLeftButton are not working

Version

react-native-router-flux v3.2.6
react-native v0.22

Expected behaviour
renderRightButton and renderLeftButton should be working for scenes that are not tabs
Actual behaviour
renderRightButton and renderLeftButton are only working for scenes that are tabs
Steps to reproduce

add renderRightButton={()=>} to loginModal scene in Example.js

  <Scene key="login" direction="vertical"  >
      <Scene key="loginModal" component={Login} title="Login" renderRightButton={()=><Right/>}/>
      <Scene key="loginModal2" hideNavBar={true} component={Login2} title="Login2" panHandlers={null} duration={1}/>
  </Scene>

run the example
right button is not rendered

turned out I just need to specify it at the parent scene like this
      <Scene key="login" direction="vertical"
                  renderLeftButton={ButtonRenderer.back}>
                  <Scene key="loginModal" component={Login} schema="modal" title="Login"
                    navigationBarStyle={styles.navBar}/>

I've got the exact same problem in 3.2.7 I just slightly modified the example to add a right button in the login and register screen of the example but the button is not rendered on top.. In fact during the transition I can see the button of the login screen below the main bar.

what's ButtonRenderer.back?


Showing the local code in the URL

Hi, first of all this package is great! Super easy to setup ! Thanks a lot :)
I'm just asking what is the best way to show the local in the url and get something like :
www.exemple.com/en/page
or
www.exemple.com/fr/page
Thanks !

It would be really nice to include this feature in the package :)

+1

+1

+1


Separate the TOC from the navigation bar

Starting a discussion.
I find the navigation bar too long and verbose, it's overwhelming. I think we either need to either:

Limit the depth of the navigation bar (i.e. via toc_depth: 2)
Move the table of contents out of the navigation bar over to the right side

I prefer the second, though I think we ought to do some styling to improve the aesthetics.

vs

I'm ok with moving the ToC to the right.


Content Update

Added a source of information to point users to use custom settings for Windows 10 devices in Intune.
Problem: https://github.com/MicrosoftDocs/windows-itpro-docs/issues/4482

Thanks @joinimran. Can you please also correct the spelling of Integrity?
@JohanFreelancer9 @nenonix for copy review please. @illfated feel free to share your thoughts if you have time. Thank you!

Nothing further to add from my side, but maybe @JohanFreelancer9 or @nenonix would like to suggest something more.

Ready to go live.

#sign-off

Thanks @Dansimp for merging this PR!


Add experimental softplus to LFQ to fix negative commit loss

Negative commit loss fix based on recommendation from the similar issue in LFQ

@avramdj hey Avram, thanks for this PR
have you tested this, does it work well with it turned on by default?
also, what is your interest in meshgpt? for personal use or for some startup / company?

@avramdj the negative commit loss isn't an issue btw, just kind of disconcerting to look at

@avramdj hey Avram, thanks for this PR
have you tested this, does it work well with it turned on by default?
also, what is your interest in meshgpt? for personal use or for some startup / company?

Yup I've tested it out on a private dataset and it fixes the issue. It's too early to tell but it looks like recon loss is pretty much the same.
But I wonder what the practical impacts of negative commit loss could be. Doesn't it affect training stability?

@avramdj nevermind, it turns out i use the x-transformers Decoder, so it was an easy add
no worries, i know you startups keep your cards close. just shoot me a Signal with your results, if you can

go make the holodeck happen :wink:


Upgrade to React Router v2

React Router v2 is now out and stable, so it's time to upgrade. This should not mean discontinued support for v1, but docs and example app should be updated to ensure v2 support.

Hi, I'm trying to use this plugin with react-router ^2.4.0.
I follow the example with react-router v1 and just changed like below
if (typeof document !== 'undefined') {   ReactDOM.render(     <Router routes={routes} history={browserHistory} />,     document.getElementById('root')   ); }
but I get Error like below
[01:13:51] [webpack-dev-server] http://localhost:8080/webpack-dev-server/index.html Warning: React.createElement: type should not be null, undefined, boolean, or number. It should be a string (for DOM elements) or a ReactClass (for composite components). /Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:38   return (0, _isUndefined2.default)(route.props.path) || (0, _isUndefined2.default)(route.props.component);
TypeError: Cannot read property 'path' of undefined     at hasNoComponent (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:38:48)     at getNestedPaths (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:71:7)     at /Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:66:12     at Array.map (native)     at getNestedPaths (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:65:51)     at getNestedPaths (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:75:24)     at getAllPaths (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/utils.js:79:37)     at Compiler.<anonymous> (/Users/Harim/.nvm/versions/node/v5.11.0/lib/node_modules/react-static-webpack-plugin/dist/index.js:96:40)     at Compiler.applyPluginsAsync (/Users/Harim/Develop/euplus/node_modules/tapable/lib/Tapable.js:71:13)     at Compiler.emitAssets (/Users/Harim/Develop/euplus/node_modules/webpack/lib/Compiler.js:226:7)
So I wonder If I can't use it just changing browserhistory part.

Thanks @iicdii. I haven't had a chance to upgrade this yet but it's still on my todo list

I've just downgrade react-router^1.0.3 and test in same way, I get same error just like before.
I think there is some problem with my project.

Hm, can you link me to the repo? I can figure out what's wrong if I'm able to reproduce.
The simple solution that might work is to run
rm -rf node_modules && npm install

😉

@iansinnott IMO you can close this issuem since in the package.json I see:  "react-router": "^2.4.0"

Ah thanks @scherler. Yup this should no longer be an issue


Camera control from the game

Issue
We need to control the camera (bounds, target, zoom) from the game codebase.
Fix
Added world's method to trigger client code events from the world lib.
Unbound the camera from the hero thang on game-dev levels (we don't hero there).

If I understand correctly, this means you can publish the "camera:zoom-to" command from the game database?
Looks great.
This method allows the game codebase to publish any event. Could this be abused? Why not attach a public camera method to the world with a zoom to method?

If I understand correctly, this means you can publish the "camera:zoom-to" command from the game database?

Yes.

This method allows the game codebase to publish any event. Could this be abused?

The same way as you can do it from browser dev console I think. I can exec for example "Backbone.Mediator.publish("camera:set-camera", {bounds: [{x: 0, y: 0}, {x: 240, y: 160}]})" from the dev console and get the same effect as publishGlobalEvent. So if it can be abused then we have more serious problems :-)

Why not attach a public camera method to the world with a zoomTo method?

This what I tried to do first, however world that we have in game codebase is not the same world that we have in Surface, and game world don't have direct access to surface and camera.
Plus this way we get more control then certain methods.

And does freeing camera from game dev set the camera to defaults?

Without "freeing" camera its zoomTo HeroPlaceholder on each frame, but in game dev levels HeroPlaceholder is gameReferee so when I'm trying to zoom to a thang on next frame the camera move to a corner (if it's zoom in)
I need to think about "abuse" hypotesis and check one idea, maybe it's not the best way. Thanks!

@AndrewJakubowicz
I've changed the publish event method and allowed only a camera event. I see how publishing any event can be abused. Thank you for that your point, you are right.

This is great! Thanks for answering my questions. I think there might be a little typo and then it's good to ship!


Automator: update<EMAIL_ADDRESS>in<EMAIL_ADDRESS>

Generated by Automator - 2024-05-03T01:46:55+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio_release-1.19
daf2bb5feb03611e0d370c74498e2bc40e6e14a4
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-basic-arm64

/test unit-tests-arm64