Webpack 2 + postcss-loader 1: TypeError: Cannot read property 'config' of null

Using Webpack 2 with postcss-loader 1.0 produces:
TypeError: Cannot read property 'config' of null
    at .../node_modules/postcss-load-config/index.js:50:22

Is my webpack.config.js syntax in order?

  module: {
    rules: [
      {
        test: /\.jsx?$/,
        include: paths.src,
        use: {
          loader: 'babel'
        }
      },
      {
        test: /\.s?css/,
        use: [
          {
            loader: 'style-loader',
            options: {
              sourceMap: true
            }
          },
          {
            loader: 'css-loader',
            options: {
              importLoaders: 1,
              modules: true,
              localIdentName: '[path]_[name]_[local]_[hash:base64:5]'
            }
          },
          {
            loader: 'postcss-loader',
            options: {
              sourceMap: true,
              plugins: () => []
          },
          {
            loader: 'sass-loader',
            options: {
              sourceMap: true
            }
          }
        ]
      }

Same problem:
import webpack from 'webpack'
import path from 'path'

import cssnext from 'postcss-cssnext'
import easyImport from 'postcss-easy-import'

export default {
    entry: path.join(process.cwd(), './components/App/index.jsx'),
    output: {
        path: path.join(process.cwd(), './public'),
        filename: 'main.js',
    },
    plugins: [
        new webpack.LoaderOptionsPlugin({
            postcss: () => [
                easyImport,
                cssnext({
                    browsers: ['last 2 versions', 'IE > 10'],
                }),
            ],
        }),
    ],

    module: {
        loaders: [{
            test: /\.jsx?$/,
            exclude: /node_modules/,
            loader: 'babel',
        }, {
            test: /\.css$/,
            loader:
                'css-loader?modules&localIdentName=[local]--[hash:base64:5]' +
                '!postcss-loader',
        }],
    },
}

It is just a warning, when you didn't provide any plugins to PostCSS.
Both your config had no plugins, just set plugins and warning will be removed 😊.
We fixed this issue, just waiting for postcss-load-config release.

Just a warning? can somebody confirm?
I tried to setup Webpack 2 + postcss-loader 1 and couldn't get it to work with plugins.
Sure this is not an error and not just a warning?

I also just ran into this problem. Using the recommended Webpack 2 configuration does not work. The plugins key of options is not being passed to the loader. It appears that this.query only does not keys that are functions. It works fine when I make a postcss.config.js file. Otherwise I get the very cryptic:
[TypeError: Cannot read property 'config' of null]

Here is my config file:
const path = require('path')
const HtmlWebpackPlugin = require('html-webpack-plugin')
const autoprefixer = require('autoprefixer')

module.exports = {
  devtool: 'eval-source-map',
  entry: {
    app: './src',
  },
  output: {
    path: path.join(__dirname, './build'),
    filename: "[name].[hash:6].js",
    chunkFilename: "chunk.[id].[hash:6].js"
  },
  module: {
    rules: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        loader: 'babel-loader',
      },
      {
        test: /\.css$/,
        use: [
          {
            loader: 'style-loader',
            options: {
              sourceMap: true
            }
          },
          {
            loader: 'css-loader',
            options: {
              importLoaders: 1,
              modules: true,
              localIdentName: '[name]_[local]_[hash:base64:5]'
            },
          },
        ],
      },
      {
        test: /\.scss$/,
        use: [
          {
            loader: 'style-loader',
            options: {
              sourceMap: true
            }
          },
          {
            loader: 'css-loader',
            options: {
              importLoaders: 1,
              modules: true,
              localIdentName: '[name]_[local]_[hash:base64:5]'
            },
          },
          {
            loader: 'postcss-loader',
            options: {
              sourceMap: true,
              plugins: () => [autoprefixer],
            },
          },
          {
            loader: 'sass-loader',
            options: {
              sourceMap: true,
            },
          },
        ],
      },
    ],
  },
  resolve: {
    extensions: ['.js', '.jsx'],
  },
  plugins: [
    new HtmlWebpackPlugin({
      template: 'src/index.html',
    }),
  ],
}

The exception is being thrown from loadConfig on line 60. loaderUtils.parseQuery includes sourceMap but not plugins.

I'm also getting the same error if I give the configuration using webpack's LoaderOptionsPlugin.
new webpack.LoaderOptionsPlugin({
      postcss: {
        sourceMap: true,
        plugins: () => [autoprefixer],
      }
    })

@michael-ciniawsky let’s release new version ;)

@lourd don’t use LoaderOptionsPlugin ;) postcss-loader supports webpack 2

@lourd did you try PostCSS Common Config? It is on the top in docs, because it is recommended way to use postcss-loader.

It works when I use a postcss.config.js or .postcssrc file in the top directory of my project, using these formats:
// js
module.exports = {
  plugins: [
    require('autoprefixer')
  ]
}

// rc
{
  "plugins": {
    "autoprefixer": {}
  }
}

It does not work when I export a function, as postcss-load-config would lead me to believe
module.exports = (ctx) => {
  plugins: [
    require('autoprefixer')(ctx.plugin)
  ]
}
// [TypeError: Cannot read property 'plugins' of undefined]

It also does not work when I put the config file somewhere other than the top level directory, such as in config/. If I'm going to have to do specify the postcss options separately from the webpack config, I would strongly prefer to at least keep it out of the main directory.

@ai released 👍
@lourd
module.exports = (ctx) => {
 return { // <-- ;)
    plugins: [
      require('autoprefixer')(ctx.plugin)
    ]
  }
}

Ah, right, thanks @michael-ciniawsky. I submitted a PR to fix the example on your README, too.
I'm confused, should the new version of postcss-load-config fix this problem of webpack 2 not passing the config as expected?

@lourd yep, yep I missed that als a few times myself 😋
You mean e.g that looking for the config in another dir? Should be resolved now, postcss-loader will look in the dir of the file for config and move upwards if none is found.

postcss-loader 1.1 is now releases and it doesn’t show this warning anymore.

Using webpack2 + postcss-loader 1.1, warning is now:

PostCSS Config could not be loaded. Please check your PostCSS Config.

How do I avoid this ?

@tomeraz This should be the correct warning, when loading fails, i8f you don't use postcss.config.js etc. at all postcss-load-confi8g shouldn't be executed? Can you show/describe your current setup ?

Here:

  module: {
    rules: [
      {
        test: /\.jsx?$/,
        include: paths.src,
        use: {
          loader: 'babel'
        }
      },
      {
        test: /\.s?css/,
        use: [
          {
            loader: 'style-loader',
            options: {
              sourceMap: true
            }
          },
          {
            loader: 'css-loader',
            options: {
              importLoaders: 1,
              modules: true,
              localIdentName: '[path]_[name]_[local]_[hash:base64:5]'
            }
          },
          {
            loader: 'postcss-loader',
            options: {
              sourceMap: true,
              plugins: () => []
          },
          {
            loader: 'sass-loader',
            options: {
              sourceMap: true
            }
          }
        ]
      }

Same "Config could not be loaded" error here. I don't have an external config file but I'm adding plugins inline:
// bunch of other loaders...
{
  loader: 'postcss-loader',
  options: {
    plugins: () => [
      autoprefixer({ browsers: ['last 3 versions'] })
    ],
  },
}

We shouldn't be using the query key insted of options in the config?
Docs for reference.

The options key is part of some new webpack2 changes.
module.loaders –> module.rules
loaders –> use
query –> options

Oh thanks for that. Great info in there. I will try to debug this tonight if I have some time.

hmm.. :) I will take a look at postcss-loader later, when query/options are set, postcss-loader shouldn't trigger postcss-load-config at all and no warning popping up either. Otherwise the warning has i8s right there. @tomeraz @albertogasparin does the CSS get transformed? Because normally when postcss-load-config gets executed without config plugins === [] options === {} and the PostCSS instance should run with this 'setup'.

@michael-ciniawsky yes, the CSS gets transformed.
Compilation works well, the error seems to have no effect on it.

@tomeraz kk, but it's still buggy, because has metioned before postcss-load-coni8g shouldn't be executed and that the inline options are applied is even wierder. Seems just to be a missing/falsy check in postcss-loader I patch it later, just ignore the warning for now :)

@tomeraz did you load the plugins through the webpack 2 config?

@gaastonsr haven't used any plugins

Right. Loading plugins with webpack2 doesn't work yet.
I took the time to investigate more why it's not working and found it's because of a bug when style-loader is in the loader chain.
webpack/webpack#3136 for reference.
Possible solutions:

Load plugins with a config file (e.g. postcss.config.js)
Use weback 2.1.0-beta22 and use the webpack 1 config style.

This is working!
Webpack 2.1.0-beta.25
new webpack.LoaderOptionsPlugin({
      test: /\.css$/,
      debug: true,
      options: {
        postcss: [
          precss(),
          autoprefixer({
            browsers: [
              'last 3 version',
              'ie >= 10',
            ],
          }),
          mqpacker(),
        ],
        context: path.join(__dirname, 'src'),
        output: {
          path: path.join(__dirname, 'dist'),
        },
      },
    }),

@marcelaraujo not working for me. Anything else you did other than
config.plugins.push(new webpack.LoaderOptionsPlugin({

I still get PostCSS Config could not be loaded. Please check your PostCSS Config.

Here's his full config file for reference
http://pastebin.com/Lmka3rju

But you will still get the PostCSS Config could not be loaded. Please check your PostCSS Config. warning.
The only way to get rid of that message at the moment is to create one config file for postcss.

Great, thank you @gaastonsr .

Solution @marcelaraujo mentioned works for me with webpack 2.1.0-beta.25. No PostCSS Config could not be loaded. Please check your PostCSS Config. warning.

Probably after this fix #3306 on Webpack, solves this issue.

PostCSS Config could not be loaded. Please check your PostCSS Config.
I am still seeing this in v2.1.0-beta.27, tried @marcelaraujo solution too with no luck.

I'm seeing the same warning in v2.1.0-beta.27 but only when style-loader is in my chain.
I also noticed that if I set plugins: to [] instead of () => []  everything works as expected without style-loader, but completely blows up the build when style-loader is in the chain and I get a more cryptic warning:
Module build failed: TypeError: Cannot read property 'postcss' of null
Which is weird, because I'd expect it to fail regardless of whether or not style-loader is in the chain.

@soluml same issue.

Using webpack v2.2.0-rc.1

@CrashyBang please show you PostCSS common config and webpack config.

PostCSS config:
{
    test: /\.scss$/,
    exclude: ['/node_modules/'],
    use: [{
        loader: 'style-loader',
        query: {
            plugins: [],
        },
    }, {
        loader: 'css-loader',
        query: {
            sourceMap: true,
            plugins: [],
        },
    }, {
        loader: 'postcss-loader',
        query: {
            plugins: () => ([
                require('autoprefixer')({
                    browsers: ['last 2 versions', 'ie > 8'],
                }),
            ]),
            // If you disable the style-loader
            // the below will work but styles
            // will not actually be compiled
            // plugins: [
            //     require('autoprefixer')({
            //         browsers: ['last 2 versions', 'ie > 8'],
            //     }),
            // ],
        }
    }, {
        loader: 'sass-loader',
        query: {
            sourceMap: true,
            plugins: [],
        },
    }]
},

Package details:
{
  "name": "hassle",
  "version": "0.0.0",
  "description": "Hassle an event organisation app",
  "main": "index.js",
  "scripts": {
    "dev": "NODE_ENV=development node build/server.js",
    "build": "NODE_ENV=production webpack --config build/webpack.prod.js",
    "test": "echo 'lets implement'"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/Shipwrecked/Hassle.git"
  },
  "author": "Otis Wright",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/Shipwrecked/Hassle/issues"
  },
  "homepage": "https://github.com/Shipwrecked/Hassle#readme",
  "dependencies": {},
  "devDependencies": {
    "autoprefixer": "^6.5.4",
    "babel-core": "^6.2",
    "babel-loader": "^6.2",
    "babel-preset-env": "^1.0",
    "css-loader": "^0.26",
    "debug": "git://github.com/visionmedia/debug#master",
    "file-loader": "^0.9",
    "node-sass": "^4.0",
    "postcss-loader": "^1.2.1",
    "rimraf": "^2.5",
    "sass-loader": "^4.1",
    "style-loader": "^0.13",
    "webpack": "^2.2.0-rc.1",
    "webpack-dev-server": "^2.2.0-rc.0"
  }
}

@CrashyBang please check postcss-loader docs. You need to put plugins into separated postcss.config.js config.

@ai do you know if there is a way to change the config file path?

@DylanPiercey
{
  loader: 'postcss-loader', options: { config: 'alternative/path/to/postcss.config.js' }
}

@michael-ciniawsky did we already release config option?

Yep 😛 #145 but without being able to handle/set context, release in 1.2.0. I can add docs for it in PR for ctx

“1 month ago” 😅
Yeap, we need a Config section in options docs.

For posterity: I just ran into this issue and the fix for me was downgrading file-loader from 0.11.2 to 0.11.1. I've seen nothing to indicate the degree or method of involvement.


Seems to be version issues with contentlayer

To Reproduce
fumadocs-core depends on remark 15.x, and there appears to be an issue with contentlayer and the latest version of remark and several remark and rehype dependencies as the whole unified framework goes through a major update. There's an issue here, but there are quite a few similar issues posted elsewhere.
Current vs. Expected behavior
The fumdadocs contentlayer template, without any modification produces the same error:
TypeError: Cannot read properties of undefined (reading 'inTable')

Provide environment information
Operating System:
- MacOs
- Apple silicon M1

Which area(s) are affected? (Select all that apply)
Core, MDX, Integrations (e.g Contentlayer, OpenAPI, Typescript DocsGen), Examples
Additional context
This seems to be an issue with remark 15+, and all unified dependencies that recently updated to support remark 15+. I've had a similar issue with the same app before moving to fumadocs (which is otherwise awesome by the way), and was able to resolve it by setting a strict version for remark, remark-math, rehype-mathjax, and particularly remark-gfm but currently the fumadocs package is overriding that with their 15.x remark dependency.

You have to override the dependency versions with overrides option in package.json
{
  "overrides": {
    "unified": "^11.0.4",
    "mdx-bundler": "^10.0.1"
  }
}

I have that. The issue appears for me even in the template generated through pnpm create fumadocs-app when using the contentlayer source. I was originally using the fumadocs-mdx package, but came to the same issue on any page that included math so I tried switching to using contentlayer as a source. I have those overrides in both the app that I'm trying to build and in the generated template, and both apps are producing the same error. The generated app hasn't been modified at all.

For pnpm, you need pnpm.overrides instead. Regarding the math problem, it sounds like a XY problem. You only need to follow the Math setup guide in the docs

Closing this as the Contentlayer adapter is now deprecated, you may consider Fumadocs MDX, Content Collections, MDX Remote or other content sources.

Closing this as the Contentlayer adapter is now deprecated, you may consider Fumadocs MDX, Content Collections, MDX Remote or other content sources.

No worries. It's a great library regardless. I'll try out the other options.


Map events are not $broadcast inside an $apply

Currently, if I hook into map events using $scope.$on('mapboxglMap:...' the handler is running outside of a digest cycle and so model changes inside of them will not be reflected through the rest of the app.
According to the Angular dev guide, $scope.$apply() should occur as close to the async event binding as possible.
Is there a reason not to do $rootScope.$apply($rootScope.$broadcast('mapboxglMap:' ...)) inside mapboxglEventsUtils.exposeMapEvents or in glControl when adding angular events for the control?
This is relevant particularly when listening to mapboxglDraw events.

The only problem may be with the mouse events, where each mouse move would trigger a digest cycle through $rootScope.apply, may be too much no?

Also i need to control and get access to data in controllers. can it be done with about mapInstance


Design the blogs page blog.html

Till now i have one blog received from Dan Schenker-: https://docs.google.com/document/d/14W-uGQFLrQySVddv-jzwPRu2vCVYpkRahCgOB34wApY/edit?usp=sharing
he has contributed in our repo
A template that i liked very much is attached
just surfing google for templates but we have to create from scratch

multiple contributors can work on it so please comment if u want to be assigned


Better support for unicode character width

When vty renders emojis it does not compute the right character width for them.
I know that, as discussed here, there are no guaranties for terminal emulator behaviour but it seems like most implementations now use 2 characters, as specified in later Unicode 9 standard.
It seems that the vendored version of wcwidth() in vty is old and does not include emojis. I could not find a reliable source for wcwidth.c that is maintained. There is some discussion here about using the system version of wcwidth that comes with glibc or not, but not conclusion.
I do agree with the comment here that we need to provide a working version for systems that don't offer wcwidth natively. Though I think vty could default to the system version when possible. This would mean that we don't need to maintain a separate implementation that will get out of date. That said glibc's wcwidth is known to have issues and to lag behind the Unicode standard but we have better guaranties that it will be updated, eventually.
There is a Haskell library that wraps around system wcwidth when possible: https://github.com/solidsnack/wcwidth that currently gives the correct result for at least Unicode 11.0/Emoji 11.0/2018 emojis (vty version 5.25.1, wcwidth version 0.0.2, glibc version 2.28):
> putStrLn "\x1F355"
🍕
> Graphics.Text.Width.wcwidth '\x1F355'
1
> Data.Char.WCWidth.wcwidth '\x1F355'
2
> putStrLn "\x1F973"
🥳
> Graphics.Text.Width.wcwidth '\x1F973'
1
> Data.Char.WCWidth.wcwidth '\x1F973'
2

I am not sure if we want to use that library as we potentially only want a thin wrapper around the C function.
Also I am not sure if we want to use wcwidth and count the total width in vty for a given string or use wcswidth instead?
Depending on responses I am happy to come up with a PR for this.

Thanks for filing this. I'm not exactly sure what the right path forward is, because in my experience, no approach works fully due to the issues you cited. For example, @glguy had a problem where an upgrade to Vty's wcswidth C code which ostensibly fixed Vty caused breakage for the terminal he was using because its width tables are stale (and are unlikely to get upgraded soon). The conclusion of that discussion suggested the only reasonable (but nonetheless annoying) solution that I know of: let Vty support different implementations of character width compuations (hopefully based on Unicode version) and let the Vty application choose, which consequently means exposing that choice to the end user.

I also want to say that while the issue is a thorny one, I very much would love it if we can improve the situation. The Matterhorn project I work on is particularly vulnerable to problems with character width because other Mattermost clients send wide emoji characters on a regular basis.

I don't have a solution yet, but I've been using this code to figure out the current terminal's behavior.
I think we could make an executable to generate an efficient wcswidth table for each Terminal that we're interested in and let people configure the vty package accordingly. The library already has a notion of a user-level configuration file.

I don't yet know if the choice of terminal emulator (and its attendant versions) is a good axis of configuration for Vty, but if it was, I'd be okay with that approach. In particular I don't know whether that makes for a good user experience, and we'd need to do some research on the correspondence between emulator versions and Unicode behavior. Lastly, I'd be concerned that the emulator version is not always a good proxy for Unicode support; in the case of VTE-based emulators like Gnome Terminal, the VTE version is likely to be the version of interest rather than the version of Gnome Terminal itself.
As far as configuration goes, even if Vty is configurable through its configuration file, I'd want to ensure that application developers can configure Vty programmatically without exposing Vty to their users.

Thank you both for your answers! I will think about it and come up with a more concrete proposal.
Some questions:
@jtdaugherty Is it possible for vty to know the terminal (emulator) it is running against, and maybe the version. Is this something that is provided by terminfo maybe?
@glguy What code are you using to determine the terminal behaivour?

@jtdaugherty Is it possible for vty to know the terminal (emulator) it is running against, and maybe the version. Is this something that is provided by terminfo maybe?

Not that I know of. I believe the only way we can get this information is for the user / application programmer to tell us. This is why my original thinking about this was about configuring based on Unicode version, not terminal emulator -- not because Unicode version is easier to detect (it isn't), but because then we don't have to track emulator versions and associate them implicitly with Unicode versions.

@u-quark @glguy meant to link to this: https://gist.github.com/glguy/a802eaaac5a00ab41e3c3ca33507634b

@glguy and I talked about this in person and concluded that another approach, if a tool based on the gist above is feasible, would be to have that tool interrogate the terminal to generate metadata that Vty-based applications could later load to influence their width computations. That would avoid needing to couple this to Unicode versions (which @glguy says won't be useful because they don't actually specify character widths anyway) or terminal emulators (which has the issues I mentioned before). This would also mean that application developers wouldn't need to worry about this issue at all because it could be done transparently by Vty based on the presence of, say, an environment variable or Vty configuration file.

Thanks for the info. I think the approach that you describe makes sense! I will look into this more and come back to you.

I did a small survey of the behaviour of terminal emulators based on the code from @glguy. Here are my results.
I tested the following terminal emulators on a fairly recent Debian unstable:

vte: GNOME Terminal 3.34.0 using VTE 0.58.0 +BIDI +GNUTLS
Konsole 19.08.0
Kitty 0.14.4
mlterm 3.8.8 (Features: otl ssh implugin imagelib(builtin) utmp)
urxvt (rxvt-unicode) v9.22 - released: 2016-01-23
XTerm 348
st 0.8.2

Also on a Mac:

iTerm2 3.2.9
Terminal.app (did not note the version)

I compared them to each other and to:

wcwidth from glibc 2.29-1 on Linux
wcwidth from libc on Mac Libsystem-1238.60.2
vty 5.25.1 implementation

I tested all the Unicode code points up to \x2FA1F. The following table shows in how many characters the different implementations disagree on.
             glibc  iterm2  kitty  konsole  maclibc  mlterm  st     terminalApp  urxvt  vte    vty    xterm
glibc        0      672     654    268      17814    497     2      1412         2      83     1412   2
iterm2       672    0       102    474      17785    181     670    2050         670    607    2050   672
kitty        654    102     0      386      17803    247     652    2042         652    571    2042   654
konsole      268    474     386    0        17690    617     266    1664         266    185    1664   268
maclibc      17814  17785   17803  17690    0        17920   17812  17103        17812  17812  17103  17814
mlterm       497    181     247    617      17920    0       495    1879         495    432    1879   497
st           2      670     652    266      17812    495     0      1410         0      81     1410   2
terminalApp  1412   2050    2042   1664     17103    1879    1410   0            1410   1479   0      1412
urxvt        2      670     652    266      17812    495     0      1410         0      81     1410   2
vte          83     607     571    185      17812    432     81     1479         81     0      1479   83
vty          1412   2050    2042   1664     17103    1879    1410   0            1410   1479   0      1412
xterm        2      672     654    268      17814    497     2      1412         2      83     1412   0

The code I used with the results can be found in https://gist.github.com/u-quark/fb8540813a3bdccda0d583c61a31f265. Also the character widths for each implementation are here and the lists of characters that have different width implementations here.

A few notes from these results:
Most terminal emulators differ between each other in several cases (vte_kitty.txt). xterm, urxvt and st seem to be in agreement with each other and with glibc (the 2 differences (urxvt_glibc.txt) are in control characters). The other differences can probably be attributed to subtle difference in implementation of various language scripts.
The good news is that these differences seem to be a few cases in various scripts but not disastrously too many and everywhere. Also glibc's wcwidth seem to be doing well across many terminals in Linux. Note also, that most of the differences occur in characters that seem to be supposed to be used in combinations with other characters.
It seems vty's wcwidth has the most incompatibility (vty_vte.txt) in Linux terminals, especially in latest Unicode versions (see emoji). Also it seems there are not many differences between glibc's wcwidth and vty's vendored implementation (vty_glibc.txt) - besides emoji.
Libc that comes with Mac seems to be completely wrong in most cases for both iTerm2 (iterm2_maclibc.txt) and Terminal.app (terminalApp_maclibc.txt).
vty's implementation seems to agree fully with Terminal.app. They both seem to assume wcwidth for a lot of wide characters to be one column.
@glguy Which terminal emulator/version did you use when you had issues using utf8proc in vty? What was the problem you had?

Based on this I think we could do the following (building on what you suggested before):

Have a file that configures how vty behaves with respect to terminal emulator behaviour. The path to this file can be read from an environment variable, say VTY_CONFIG and can default to $XDG_CONFIG_HOME/vty/config. This way this config can be overridden for different terminal emulators.

If the file is not present the first time a vty application starts we run some tests to determine the terminal behaviour and initialize the file. Any subsequent runs of vty applications are configured from the file without any tests. This way the end user will have a good experience without needing to run another tool before using the vty application and will still have good defaults in most cases.

We provide three ways of handling wcwidth in vty: a) the existing vendored implementation, b) an updated vendored implementation that returns 2 for newer wide characters according to the latest Unicode (especially emoji), c) use system's glibc. We make the config initialization process query the terminal behaviour and pick the implementation that fits best. On Mac we pick (a) or (b) based on the character width of an emoji and on Linux if glibc matches the character width we use option (c) or else (a) or (b) accordingly. This way we use the fast and future compatible implementation (c) when it is available.

For now only save in vty's config which one of the above tree implementations to use for wcwidth. This is enough to solve this issue easily/quickly and we have the infrastructure to improve the solution in the future to be even more compatible with terminal emulators.

vty already has an existing configuration file that we could extend.
I don't remember which characters broken last time we tried using utf8proc's implementation with vty, but I do remember noticing a lot more errors in Terminal.app in IRC channels at that time. I would guess it had to do with emojis but I don't remember. It's likely that this was on an earlier version of macOS than the current release and Terminal.app might have changed in the meantime.
Vty disagrees with Terminal.app on a bunch of characters. One example is U+2B50 (⭐) a wide character that vty considers to have width 1. By my count it differs on 1700 characters. That's to say I don't understand what you meant about vty agreeing with Terminal.app

We should use the existing configuration file, as you suggest.
I probably copy-pasted the wrong file for Terminal.app. Can you share the file you got from Terminal.app so we have all the information in one place?
Do you think overall this is a good solution? Especially if you had problem with emoji before this should address that. I think we can give it a try and test it on a few machines to see if you have any problems.

@u-quark thanks for all of your work on this, that's fantastic! A few comments:

If the file is not present the first time a vty application starts we run some tests to determine the terminal behaviour and initialize the file

Assuming I understand what you have in mind, I think this would be okay as an application-driven behavior, but not as an automatic Vty library behavior. The tests we'd need to run would take time and lead to a confusing startup delay of the application. So if Vty were to provide functionality to perform this testing and generate the config (or a function to mutate an existing config) that the application could call, I think that would work well.

On Mac we pick (a) or (b) based on the character width of an emoji and on Linux if glibc matches the character width we use option (c) or else (a) or (b) accordingly.

I'm not sure I understand what you mean here. When would this choice of implementation be made? And isn't there a bootstrapping problem? We don't know the width until we ask these implementations, so we can't use the width to choose between them.

Assuming I understand what you have in mind, I think this would be okay as an application-driven behavior, but not as an automatic Vty library behavior. The tests we'd need to run would take time and lead to a confusing startup delay of the application. So if Vty were to provide functionality to perform this testing and generate the config (or a function to mutate an existing config) that the application could call, I think that would work well.

That makes sense and we can figure out what the better default config is later. That said I am hoping that the checks are almost instant and imperceptible.

I'm not sure I understand what you mean here. When would this choice of implementation be made? And isn't there a bootstrapping problem? We don't know the width until we ask these implementations, so we can't use the width to choose between them.

The choice of the implementation would be done when you initialize the configuration after the tests mentioned above, and based on their results. When I say "based on the character width of an emoji" I mean the observed width from the terminal as returned by getCursorPosition0 from our test. After the decision is made of which implementation to use and saved to the vty config we always use that implementation.

That said I am hoping that the checks are almost instant and imperceptible.

I'm not sure we are talking about the same thing by "the checks" here. Above, I proposed that if we are going to use @glguy's tool functionality to determine what the local terminal thinks about Unicode characters, it would be good to make a tool that users can run that saves the results of the time-consuming analysis. Then, Vty would load that saved data and use it as an alternative to a width function like wcwidth. Do you mean something else?

The choice of the implementation would be done when you initialize the configuration after the tests mentioned above, and based on their results

Yeah, I think this is where the confusion is. Based on my suggestion above, there is no implementation selection at all, because the saved data is used instead. The implementation still matters, but only when there's no saved character width map to load. Does that make sense?

Above, I proposed that if we are going to use @glguy's tool functionality to determine what the local terminal thinks about Unicode characters, it would be good to make a tool that users can run that saves the results of the time-consuming analysis. Then, Vty would load that saved data and use it as an alternative to a width function like wcwidth.

What I don't like about this approach is that if an application uses emoji regularly it is going to be broken from the beginning. Requiring the user of this application to first run some other script that is going to take ~2h is not a very good experience. Same for having to write a configuration file. That's why I propose having a few build-in implementations and choosing the most compatible by doing a mini test on the first run of the application only with the intention to fix emoji. As I mentioned before this mini test can by done by the application and not by vty.

What I don't like about this approach is that if an application uses emoji regularly it is going to be broken from the beginning.

Why would this be the case? As I understand it, using the tool mentioned above would prevent this.

Requiring the user of this application to first run some other script that is going to take ~2h is not a very good experience.

I agree.

That's why I propose having a few build-in implementations and choosing the most compatible by doing a mini test on the first run of the application only with the intention to fix emoji.

What would the "mini test" be?

What I don't like about this approach is that if an application uses emoji regularly it is going to be broken from the beginning.

Why would this be the case? As I understand it, using the tool mentioned above would prevent this.

Because there is the caveat mentioned below. If the user does not use the tool mentioned above then they will have a broken experience.

Requiring the user of this application to first run some other script that is going to take ~2h is not a very good experience.

I agree.

That's why I propose having a few build-in implementations and choosing the most compatible by doing a mini test on the first run of the application only with the intention to fix emoji.

What would the "mini test" be?

I assume we can look at the width of a single emoji (or maybe two) using @glguy code and assume that the rest behave the same way.

I assume we can look at the width of a single emoji (or maybe two) using @glguy code and assume that the rest behave the same way.

Since there are many emoji added in each Unicode version, there is no single emoji that will predict how the terminal will behave for the rest. The terminal can also freely implement only part of the character ranges for each Unicode version, meaning that we can't rely on the terminal's behavior with one character in a particular version to represent how the terminal will handle others.

Since there are many emoji added in each Unicode version, there is no single emoji that will predict how the terminal will behave for the rest. The terminal can also freely implement only part of the character ranges for each Unicode version, meaning that we can't rely on the terminal's behavior with one character in a particular version to represent how the terminal will handle others.

Ok, what could we do instead then?

As far as I can tell, there are two extremes:

Extreme A: Do nothing, and things will continue to work (or not work) as they do right now. This approach means the level of brokenness will vary based on the factors you covered in your analysis above.
Extreme B: poll the terminal to get its width for as many wide Unicode code points as possible. This approach can be exhaustive and give good behavior, but it has the downside that it could be time-consuming and requires users to run an extra tool.

I've been advocating for Extreme B, partly because back when @glguy first made his tool I had the impression it didn't take that long to run. I don't remember exactly how long it took, though, or how many code points it queried. You mentioned two hours in your comment above - what is that number based on? If I recall, @glguy's script only took a few minutes.
The reason I'm glad you've spent so much time researching this is that I also don't like Extreme A. The emoji issue bites us routinely on the matterhorn project, and it sucks to not have a good story for this problem whenever it comes up. @glguy runs into this on his glirc project, too, so we are both interested in making progress on this.
As I see it, there are other options in between these two extremes like ones you have outlined here, where we make a better effort than we're currently making to get this right. I've been pushing for a more aggressive approach under the assumption that it's not that onerous for users, considering that right now things seem to work more or less okay, most of the time, for most people, as far as I know. As I see it, if running an extra tool that completely eliminates this problem is too much burden, not running the tool is a pretty acceptable solution if the fallback behavior is today's behavior.
Granted, making an extra tool also means we need to maintain it, and it means that tool needs to be available in binary form for as many platforms as we can support because we won't want to expect users to build Haskell programs from source.  So I'd also want to be sure it was worth it before going to all of that trouble.

I'll also add that a hybrid approach is also possible. Users who want to run the tool that will generate the 100% correct database of widths can always do so, and users who don't want that can rely on the library's smart choice of width implementation to get it right more of the time than Vty currently does.
In that case, though, I still want to figure out how that implementation will be selected. @u-quark gave a logic outline above, but I don't know what evidence we can use to choose from amongst them since as far as I can tell, sampling character widths won't tell us which one is the better one to use.

So, I finally got some time to come back to this and to write some code for it.
See #179 for my take on it. I've tested it on a few terminals and it seems to work pretty well. I haven't tested this on Mac. Please provide specific examples if/where this breaks for you, possibly using vty-wcwidth-tester utility.
This implementation, of course, is not perfect. But I think, if #179 is not controversial, we can merge it and continue discussing this issue here. #179 already fixes most of the issues I've seen.

@u-quark - thank you for your work on this! I have once again gotten time to come back around to this. After thinking on the results of your investigation and #179, and after talking about some concerns we had, @glguy and I are going to take new approach which I'll detail below. Consequently I'm going to close your pull request, although some of the work you did there will be migrated over (in particular, the tool to determine character widths). And the new approach is a hybrid based on ideas we have discussed on this ticket.
Here's what we have in mind now:

We'd like to avoid automatic character width table-generation for a number of reasons, so instead of having the library generate a table at startup by default, the library will use its built-in logic for computing widths instead. Put another way: unless told to do otherwise, it will do what it does now, and consult some width function that is compiled in.
The library will ship with a command-line tool to query the terminal for character widths. It will have the ability to write the results of that query to a binary file for later loading by the library. The format of the file will be a run-length encoding of the answers that the terminal gave; in particular, it will be a series of nine-byte sequences, the first four bytes indicating the starting code point of a range, the second four bytes indicating the end code point of a range, and the final byte indicating the column count for the characters in that range. The binary format will also have a header indicating the format version.
The table-generation logic used in (2) should also be exposed in the library API so that applications that want to invoke that logic can do so programmatically.
The library will provide an API for loading the tables described in (2). The library will also provide an API for installing the loaded tables into the address space of the Vty application for later consultation by the C width routine. The Vty library's API for installing tables should be thread-safe. Providing this API makes it possible for Vty's width functions to be used without using the rest of the Vty library to take control of the terminal, which may be important. (In particular, applications that need Vty's functions but that don't want to invoke mkVty will need this functionality; see below.)
The C width function will consult a global table pointer to determine whether a custom table has been installed. If the pointer is NULL, the built-in logic (i.e., today's logic) will be used to compute character widths. Otherwise the built-in table will be consulted. The in-memory table will be a contiguous C array whose indexes are code points and whose values are bytes indicating the column count for each code point. This is not the most memory-efficient, but it's easy to replace with a more efficient implementation later (e.g. 2 bits per code point, or 16 code points per array slot, say).
The Vty configuration file will be equipped with a new line syntax to allow users to indicate that Vty should automatically load and install particular tables based on the value of TERM. The mkVty function and its relatives that read the Vty configuration will be responsible for loading and installing the tables mentioned in the config. This is mostly to support applications like tmux and screen. But it also makes it easy to use the same Vty application on a system with different terminal emulators. The syntax will be something like width_map <TERM> <PATH>. If these lines are absent, or if they cannot be honored due to I/O errors, the default built-in width mapping will be used.

@glguy If I missed anything, let me know and I can edit.

My work on this will be happening on the feature/unicode-tables branch.

Steps (2) and (3) above are now complete.

As of 74a7c557fb895e8dc5bd739d2e27cb243f18d5b1, (4) and (5) are now complete, with some caveats:

Rather than provide a thread-safe, general API for swapping tables, @glguy and I decided to support at most one custom table installation. The reasoning is detailed in the commit log.
The wcwidth function is pure, but behaves impurely once a custom table has been installed. The reasoning for this is spelled out in the documentation for the newly-added functionality.

As of 9edfd029cb5275b7d2e43d83e2487e46c9d02c98, (6) is now complete.

@jtdaugherty All of this plan seems good! Except for one point. The default built-in behaviour. When someone tries one of the vty application they will not run some other command line tool to fix their broken layout. They will just assume that the application is broken. I see no reason for not switching the default implementation to the on from #179 if it makes things better.

If we have configurable tables I think it makes sense to update the default table if it actually is a better default

I see no reason for not switching the default implementation to the on from #179 if it makes things better.

I agree that switching the default implementation to a different one is worth doing if the new implementation results in more correct answers about widths, more of the time, with most terminal emulators. What I don't know is how to determine whether a new implementation actually is better. In the past, I've upgraded the wcwidth implementation in vty only to find that it actually made things worse, since it made the disagreement with the terminal emulator larger. That led me to conclude that the "best" thing to do for the time being is to keep the wcwidth implementation in vty at a similar level of staleness as the ones in most terminal emulators, since that generates a smaller disagreement about widths.

I would agree but:

A few years have passed since that decision and terminal emulators have been updated.
The short survey I did seems to suggest that the new implementation performs better.

Maybe you could try the implementation in #179  on a few terminals that you had problems with last time and see how it performs so that we get a better idea.

I haven't commented on the specific merits of the implementation in #179 because I've been focused on the bigger issues with the approach we've discussed in this ticket. With that said, I am not excited about merging the implementation in #179 because I'm not sure I trust the provenance of it (Musl is a great library for what it is but strikes me as a strange choice for where to borrow this from), I think its implementation is significantly more inscrutable than the current one, and I don't think the data you gathered above (as nice as it is!) makes the amount of improvement clear enough to make it the obviously-right choice.
If I was going to consider upgrading the built-in implementation again, I would much more prefer to use utf8proc because it is a well-maintained library specifically intended to be used for this kind of problem.

As of 16d5f27832ed04e77aefb9430473ee64c07c547b, the Config has a field allowCustomUnicodeWidthTables (defaulting to True) that allows a Vty user to suppress custom table loading that mkVty will now do when it finds a custom table declared in the Vty configuration file.
At this point I've also tested the custom table functionality with matterhorn with excellent results: once a custom table has been generated, Matterhorn transparently loads that and successfully renders wide characters that it previously had trouble with.
@glguy would it be possible for you to try feature/unicode-tables out with glirc?

I do agree the implementation based on musl code is more complex compared to the one currently in vty. The reason is performance and space of the Unicode tables. The implementation is using a two step lookup in C arrays so that it balances quick lookup with size of the produced binary (the array stored in code). I don't pretend I fully understand it. The existing code in vty is using a binary search, if I remember correctly. Given that wcwidth in vty is used during every operation is makes sense to keep the C function fast.
I am not too fuzzed about which implementation to use as the default built-in, as long as it gives the right answer according to Unicode for emojis.

I don't think the data you gathered above (as nice as it is!) makes the amount of improvement clear enough to make it the obviously-right choice.

What can we do to gauge the impact of changing the default implementation? How can we collect more data to make an informed decision?

If I was going to consider upgrading the built-in implementation again, I would much more prefer to use utf8proc because it is a well-maintained library specifically intended to be used for this kind of problem.

I could have a look at porting to vty the implementation of character width in utf8proc, if you think that would be valuable.

What can we do to gauge the impact of changing the default implementation? How can we collect more data to make an informed decision?

I'm not sure, really. Your findings above suggest that this is even more of a moving target than I originally thought, since the amount of computation/emulator disagreement is all over the place. The most exhaustive way to gauge the impact of a change in implementation would be to run the tests you ran above, but on all of the emulators we anticipate people will use. That's impractical. I think the only way to make this option realistic is to greatly constrain the set of emulators tested against, and even then, that requires access to testing systems to gather the data. Finally, this option assumes that compatibility is non-decreasing as terminal emulators evolve; I'm not sure that's safe to assume, since a newer release of a width computation may work better with many emulators and worse with others.
The other extreme is to upgrade the implementation purely based on its own merit. So, for example, if the latest version of utf8proc claims that it properly supports all of known Unicode at any given time, that makes it a good choice. However, the problem with this reasoning (as experienced directly by @glguy) is that a perfect width computation is not helpful if the terminal emulators lag behind and disagree, leading to behavior that is no better than what we have today.
Neither of those options seems like a great one to me. However, now that we have a pretty good story for how to deal with the compatibility issues, I think it's less of a problem. That does mean application developers (such as myself) will have to figure out how to make it easy for users to benefit from this new functionality, such as by distributing the Vty command-line tool that builds custom tables or by integrating it into their applications.

I could have a look at porting to vty the implementation of character width in utf8proc, if you think that would be valuable.

Since I have taken that approach before, I would be happy if we took it again. (I am willing to break things for the right reasons, and we have a good fallback plan if that happens.) I would like to know what @glguy thinks about it, first, though.

@jtdaugherty Your points are all valid. I do agree that testing the impact of the change is difficult, I also agree that the solution with loading external pre-computed tables is a good one and it solves all the problems for regular users. I am very happy we are seeing progress in this ticket. After @glguy's input and if you are still on board with the idea, I think it is a good ito give utf8proc another go. I had a brief look and it is not easy to separate the code for character widths from the rest of library. So we could either vendor it or depend on the system version of it.

The work in this ticket is now complete and has been released in 5.28. Thanks @u-quark and others for your persistence in helping us work on this ticket!

Thanks for all your work @jtdaugherty !

Hello,
I know this issue is closed, forgive me. But, about your statements @jtdaugherty,

this is even more of a moving target than I originally thought, since the amount of computation/emulator disagreement is all over the place. The most exhaustive way to gauge the impact of a change in implementation would be to run the tests you ran above, but on all of the emulators we anticipate people will use.

I have authored a specification for the python wcwidth library, and an automatic testing tool, ucs-detect, that asses each terminal emulator's compliance, and then I have gone ahead and performed testing on 20+ popular terminals and reported their results
I hope this is helpful, I have also written an article about this work https://www.jeffquast.com/post/ucs-detect-test-results/

@jquast thanks, I will take a look!


Option to serve downscaled large images if one is already cached

It would be good if, when a large version of an image has already been cached and a smaller version is requested, that the large one is supplied in a downscaled state.

Can you please provide an example, which demonstrates the use-case? Note that each derived (i.e., scaled) version is represented by its own URL and is cached separately. If you are referring to an updated original image, make sure you set invalidate to true in order to refresh the cache. Please see here for more information.

Closing this issue due to the time elapsed. Please feel free to either re-open the issue, contact our support at http://support.cloudinary.com or create a new ticket if you have any additional issues.

Can I pass invalidate to .upload method in the Javascript SDK?

Hi @prokilogrammer , unsigned uploads are restricted from deleting or overwriting resources. Therefore there is no use in invalidating. In order to update a resource you will have to use one of the server-side frameworks (Node.js, PHP, etc).

Oh interesting. I thought cloudinary_js code was also used in NodeJS where I use signed requests. Didn't realize there was a different package for it. Thanks!

The Node.js library indeed uses cloudinary_js, but while the core library is pure client-side - on node.js you can also generate signatures and perform signed requests.


Http LB tests integrated into Google smoke tests.

Wires two simple Http LB cases into the Google smoke tests. @ewiseblatt please review.

@ewiseblatt removed the inheritance. It's ugly but works.


Migrations as job

The job will likely be run earlier than the init container.

Closing in favor of https://github.com/galaxyproject/galaxy-helm/pull/271


Prevent iD uploading changes with broken multipolygons

This is getting out of hand. More and more often I have to fix iD uses breaking relations.
Force iD to check the entire relation and refuse upload where it is broken, or prevent uses from modifying them at all.

Can you provide some real examples please, including changeset id, and/or relation ids. Also describe is what ways relations were broken, etc. Otherwise, this is not be actionable in any way.

My latest repairs:
https://www.openstreetmap.org/changeset/106477831
https://www.openstreetmap.org/changeset/106359848
https://www.openstreetmap.org/changeset/101197011
Hope this can lead you to the problems.


Feature request: Drop isomorphic-fetch from implementation

Node 16 is no longer maintained
Node 18 onwards includes fetch by default.
Thus isomorphic-fetch should be replaced by plain fetch (either from browser or node)

Benefits

significant reduction in bundle size as isomorphic-fetch uses node-fetch which is not smallest (2.6.1 version)[https://www.npmjs.com/package/node-fetch/v/2.6.1]
isomorphic-fetch is no longer maintained

Thanks for considering this :+1:


Add MunifTanjim/nui.nvim

Repo URL:
https://github.com/MunifTanjim/nui.nvim
Pinging @MunifTanjim for awareness
Checklist:

[x] The plugin is specifically built for Neovim, or if it's a colorscheme, it supports treesitter syntax.
[x] The lines end with a .. This is to conform to awesome-list linting and requirements.
[x] The title of the pull request is Add/Update/Remove `username/repo`  (notice the backticks around `username/repo`) when adding a new plugin.
[x] The description doesn't mention that it's a Neovim plugin, it's obvious from the rest of the document. No mentions of the word plugin unless it's related to something else.
[x] The description doesn't contain emojis.
[x] Neovim is spelled as Neovim (not nvim, NeoVim or neovim), Vim is spelled as Vim (capitalized), Lua is spelled as Lua (capitalized), Tree-sitter is spelled as Tree-sitter.
[x] Acronyms should be fully capitalized, for example LSP, TS, YAML, etc.

Oops, this one's already there, I must have missed it. Closing.


Support for multiple providers during same session

Is there any support for providing tokens for multiple Omniauth providers? I'd like to be able to authenticate a current user using one strategy, but then authenticate the same user using a different strategy, and link the two accounts.
This would be fairly easy in a regular application because you could just store each user in the session separately, but I'm absolutely stumped on how to pass a token into DTA from the current session while hitting the auth endpoints for the other provider.

Just the saw the long discussion at https://github.com/lynndylanhurley/devise_token_auth/issues/23 ...excuse the faux pas, I've been up late trying to implement this myself -_-


[Tree][Closure] Build the tree of existing entity with parent property/column

Hi,
I've a entity User and this contain a property parent and this is quite similar to the structure that ClosureTree need:

1:admin (parent: null)
2:user1 (parent: 1)
3:user2 (parent: 1)
4:user3 (parent: 2)
5:user4 (parent: 4)

But now I need to make a real tree structure, and I'll use the ClosureTree type.
I make the config, add annotations, add extra entity, everything is ok. But if I update the entity, the closure table continue empty. But if add a new User, the closure table is populated.
How to I build the closure table of existing users?
Thk's

hi, you would need to backup the users table or rename it, and run through inserts for all users in the needed parent - child structure..

Hi,
I'll try this. But this is a bad solution :(
Many thank's. :D


Implement flist build for g8osgw

flist should contain nftables and dnsmasq bins

Staging code: https://github.com/maxux/g8os-template/blob/master/builder-g8osgw.sh


Out of Memory Crash Report Has sdk.integrations: Discarded invalid value （）

Environment
Flutter (Channel stable, 3.0.5, on macOS 12.1 21C52 darwin-x64, locale zh-Hans-CN)
• Flutter version 3.0.5 at /Users/zby0520/fvm/versions/stable
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision f1875d570e (6 weeks ago), 2022-07-13 11:24:16 -0700
• Engine revision e85ea0e79c
• Dart version 2.17.6
• DevTools version 2.12.2
• Pub download mirror https://pub.flutter-io.cn/
• Flutter download mirror https://storage.flutter-io.cn/
Checking Android licenses is taking an unexpectedly long time...[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
• Android SDK at /Users/zby0520/Library/Android/sdk
• Platform android-32, build-tools 32.0.0
• ANDROID_SDK_ROOT = /Users/zby0520/Library/Android/sdk
• Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
• All Android licenses accepted.
sentry_flutter: ^6.9.1
sentry_logging: ^6.9.1
sentry_dio: ^6.9.1
Steps to Reproduce

OutOfMemory
The OS most likely terminated your app because it overused RAM.
Actual Result
sentry cause iOS app crash
https://sentry.deepcoin.ski/share/issue/44d64605ca514a8189d151e712132ab4

any updates

Dupe https://github.com/getsentry/sentry-dart-plugin/issues/37


[UI] Transactions are displayed 3 times

windows 10 64 bit
transactions show up 3 times once confirmed
unconfirmed transactions only show up once

No the issue does not persist. after restart it correctly show each transaction once


Clean previously generated assets when regenerating

Hi,
Is it possible to configure jekyll-assets in such a way that it cleans previously generated assets when the assets gets regenerated by for instance jekyll serve? Currently the _site directory is filling up with no longer used assets. Restarting jekyll serve does clean the entire directory.
See https://github.com/jekyll/jekyll/issues/3742.

I found this issue too & made a quick workaround, but I need to do some additional work before having clean pull request. In the meantime you can have a look at my commit here https://github.com/didacte/jekyll-assets/commit/3d1ace7cadd380afda28bf5b0503197eda716299 which add a fix to avoid removing assets on empty rebuild.

Is this still the case on 2.0.0 alpha?

This issue has been open for more than 2 weeks without a response, it is being closed.  If there is a change in circumstance please ask for this ticket to be reopened answering the questions posed.


Multiselect value needed for drop down to close in IE11

Multiselect value needed for drop down to close in IE11. Also finding that 2 values are needed for drop down to close.
http://jsfiddle.net/wphqwvLf/15/

Duplicate of https://github.com/select2/select2/issues/3300.


ideas for more efficient arithmetic on 64 bit

@arigo was suggesting some improvements to get more efficient 32bit arithmetic on 64bit machines. The IRC logs of the discussion are here:
https://github.com/cornell-brg/pydgin/issues/new
The most immediately practical idea is to write overflow_from_add (and ..._sub) in a branchless way, and maybe special-case them on 64bit machines.

Thanks for the ideas @cfbolz and @arigo! The link doesn't seem to work, so here are the logs:
11:06 < berkin> an aside: supporting arithmetic ops on r_uint32 would actually be *very* useful for emulating 32-bit machines
11:06 < arigato> berkin: types like r_uint32 were probably a bad idea; we should have said that local variables must be full-sized integers, and have the smaller types only for storing into structures
11:06 < arigato> (or arrays)
11:07 < arigato> berkin: yes, and also for languages like Javascript whose efficient jitting requires 32-bit overflow checks
11:07 < arigato> but it's all unclear
11:07 < arigato> we could solve the latest problem by adding "ovfcheck32(a + b)" for example, which does the addition and overflow checking as 32-bit numbers
11:08 -!- amaury [amauryfa@nat/google/x-hjbtmjuzradhlheq] has quit [Ping timeout: 265 seconds]
11:08 < arigato> and keeping full 64-bit numbers as local variables
11:08 < arigato> simply ignoring the higher bits
11:08 -!- tifo<EMAIL_ADDRESS>has quit [Ping timeout: 246 seconds]
11:08 < arigato> that solution would be less work in the JIT's backends
11:08 < berkin> arigato: i'm wondering if it's possible to do such tricks in the backend, where 32-bit ops on a 64-bit machine usually involves some operation and then masking back to 32-bit etc
11:09 < berkin> so use more efficient asm for such patterns
11:09 < arigato> maybe, but I think we can simply ignore the high bits most of the time
11:09 < arigato> and use the existing "int_signext" operation in the rare cases were we can't (e.g. before a right shift)
11:10 < arigato> or even just add "int_rshift32"
11:10 < arigato> yes, probably a better idea to add int_rshift32 because we'd still need "int_eq32" etc.
11:11 < cfbolz> interesting idea to just ignore the higher bits
11:11 -!- Remi_M<EMAIL_ADDRESS>has quit [Quit: See you!]
11:12 < arigato> easier than sprinkling int_signext everywhere... and it works if we're careful in producing the jitcodes (e.g. we need int_signext when we cast a 32-bit int to 64-bit)
11:12 < arigato> maybe we could do the same with bools, which is a good idea at least on x86
11:12 -!- krono_ [~krono@unaffiliated/krono] has joined #pypy
11:13 < arigato> ...well, unclear if it's a good idea actually
11:13 < arigato> if we compare the simple operation "load a byte from memory" and the apparently more complex operation "load a byte and zero-extend it", then actually the latter might be more efficient
11:14 < arigato> because the former adds some dependency on the old content of the rest of the target register, or something
11:14 < cfbolz> arigato: I'm not sure in the concrete use case of the ARM emulator this aproach would work though
11:14 < cfbolz> e.g the memory should really be 32bit ints, for space reasons
11:15 -!- marky1991<EMAIL_ADDRESS>has joined #pypy
11:15 -!- marky1991<EMAIL_ADDRESS>has quit [Changing host]
11:15 -!- marky1991 [~lgfdev@unaffiliated/marky1991] has joined #pypy
11:15 < arigato> ah, all accesses by the emulator are done via an array and a zero-extended 32-bit offset
11:15 < cfbolz> yes
11:15 < arigato> but 32-bit overflow checks are signed
11:16 < arigato> bit of a mess
11:16 < cfbolz> also, implementing the ADD opcode is really not fun, because you need to set 32bit overflow flags etc
11:16 < cfbolz> and the carry, and whatnot
11:16 -!- krono [~krono@unaffiliated/krono] has quit [Ping timeout: 265 seconds]
11:17 -!- krono_ [~krono@unaffiliated/krono] has quit [Ping timeout: 265 seconds]
11:17 < arigato> right, but at least you hope that the next ADD opcode just afterwards will overwrite the flags and make the earlier computation not needed
11:17 < arigato> but you kind of have to write it all in a branch-less way
11:17 < cfbolz> true
11:18 -!- DragonSA<EMAIL_ADDRESS>has joined #pypy
11:18 < cfbolz> interesting point!
11:18 < cfbolz> I think atm there is a branch
11:18 -!- DragonSA<EMAIL_ADDRESS>has quit [Changing host]
11:18 -!- DragonSA [~DragonSA@freebsd/developer/dbn] has joined #pypy
11:18 -!- amaury [amauryfa@nat/google/x-jgtiaxqgkctebabk] has joined #pypy
11:19 < cfbolz> arigato: yes: https://github.com/cornell-brg/pydgin/blob/master/arm/utils.py#L184
11:19 < cfbolz> arigato: so the theory is that replacing the "and" with "|" might improve stuff
11:19 < arigato> "&"
11:19 < arigato> but there is "==" in RPython
11:20 < cfbolz> ?
11:20 < cfbolz> I thought we stopped making that use a guard?
11:20 < arigato> right, ok
11:20 < arigato> fwiw, a possibly more efficient way to write it now would be to shift all the values left 32 bits
11:21 < arigato> then the natural int_add_ovf applies
11:21 < arigato> (obscure)
11:21 < cfbolz> "all the values"?
11:22 < cfbolz> like right before you operate on them, but also store them like that in the register file?
11:22 < arigato> unclear
11:22 < cfbolz> :-)
11:23 -!- mstuchli<EMAIL_ADDRESS>has quit [Ping timeout: 244 seconds]
11:23 < arigato> actually you can check overflow_from_add() more easily if you know you are running on 64-bit
11:23 < arigato> with code like "x != (x << 32) >> 32"
11:24 < arigato> or even better:
11:24 < arigato> x != intmask(r_int32(x))
11:25 < arigato> the last line JITs to one "int_signext" operation followed by "int_eq"
11:26 < cfbolz> arigato: overflow_from_sub is the same, I guess?
11:26 < arigato> yes
11:26 < arigato> you can even store "x - intmask(r_int32(x))" in the overflow "flag", which is interpreted later as either zero or non-zero
11:27 -!- DragonSA [~DragonSA@freebsd/developer/dbn] has quit [Ping timeout: 258 seconds]
11:28 < cfbolz> why is that better?
11:28 < cfbolz> int_eq vs int_sub
11:28 < arigato> int_eq is more costly in terms of assembler
11:28 < cfbolz> eh, ok
11:28 < arigato> at least if it's not immediately followed by the guard on it
11:29 < cfbolz> which it isn't, we hope
11:29 < arigato> ah, or it is
11:29 < arigato> unclear if it is
11:29 < arigato> I mean in common cases we'd expect the whole int_signext/int_eq to be killed
11:29 < arigato> but not if there is an unrelated guard
11:30 < arigato> in that case indeed a int_sub is better :-)
11:30 < cfbolz> I think we should look at traces, really
11:30 < arigato> yes
11:30 < cfbolz> which I don't have right now
11:30 < cfbolz> dmlockhart: should I make an issue out of these ideas?
11:31 < arigato> obviously, a good enough backend should be able to optimize "int_sub / int_nonzero / guard_true" the same as "int_ne / guard_true"...
11:31 < cfbolz> yes, well, at some point in the future we may need a better backend :-P
11:32 < arigato> or else it can be the optimizer's job to replace "int_sub / int_nonzero" with "int_ne"
11:32 < arigato> :-)
11:41  * arigato -> dinner
11:41 -!- arigato<EMAIL_ADDRESS>has quit [Quit: Leaving]
11:47 -!- tinyblak_<EMAIL_ADDRESS>has quit [Remote host closed the connection]
11:52 -!- tinyblak<EMAIL_ADDRESS>has joined #pypy
11:53 < berkin> cfbolz, arigato: thanks for the ideas
11:54 < berkin> cfbolz: does "branchless" mean converting "and" to "&" so that it doesn't have a guard?
11:54 < cfbolz> berkin: yes
11:54 < cfbolz> Because and short circuits
11:55 -!- MrScout [~MrScout@unaffiliated/mrscout] has joined #pypy
11:56 < berkin> cfbolz: ah yeah, short circuiting is probably worse in jit because of the guards
11:56 < berkin> interesting
11:56 < cfbolz> Yes
11:56 < cfbolz> Bit non intuitive
11:56 < cfbolz> But we should have noticed when looking at the traces
11:57 < berkin> cfbolz: i'm sure it was there somewhere, this will probably improve the performance slightly
11:58 < cfbolz> berkin: maybe a bit more than slightly, because it means that the flags computation can be fully removed if the flag is never read
12:00 < berkin> cfbolz: ah i see, when there is a guard, you can't optimize the flag computation away
12:00 -!- tinyblak<EMAIL_ADDRESS>has quit [Remote host closed the connection]
12:00 < cfbolz> Yes

If I'm reading the code correctly, I think it assumes anyway that the host is 64-bit.  With a 32-bit host the existing overflow detection doesn't work.  In that case, the most efficient way is to use "x == intmask(r_int32(x))"...  well, or at least if r_int32 existed in rarithmetic.  Add it there with the line:
r_int32 = build_int('r_int32', True, 32)


vagrant ssh -c does not work

With version 0.8.0.dev and vagrant 1.8.5 I get this:
Y:\repo\tmp\managed-servers>vagrant ssh -c "ls -la /vagrant" managed_linux
==> managed_linux: The machine is backed by a cloud provider (managed), caching
==> managed_linux: will be disabled.
Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.5.9/lib/childprocess/windows/process_builder.rb:87:in `create_process': The system cannot find the file specified. (2) (ChildProcess::LaunchError)
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.5.9/lib/childprocess/windows/process_builder.rb:34:in `start'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.5.9/lib/childprocess/windows/process.rb:69:in `launch_process'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.5.9/lib/childprocess/abstract_process.rb:82:in `start'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/ssh.rb:187:in `exec'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/ssh_run.rb:53:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builder.rb:116:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `block in run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/busy.rb:19:in `busy'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/call.rb:53:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builder.rb:116:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `block in run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/busy.rb:19:in `busy'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/call.rb:53:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/home/.vagrant.d/gems/gems/vagrant-managed-servers-0.8.0.dev/lib/vagrant-managed-servers/action/warn_networks.rb:14:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builder.rb:116:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `block in run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/busy.rb:19:in `busy'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `run'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:225:in `action_raw'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:200:in `block in action'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:182:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:182:in `block in action'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:186:in `call'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:186:in `action'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/plugins/commands/ssh/command.rb:49:in `block in execute'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/plugin/v2/command.rb:235:in `block in with_target_vms'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/plugin/v2/command.rb:229:in `each'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/plugin/v2/command.rb:229:in `with_target_vms'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/plugins/commands/ssh/command.rb:41:in `execute'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/cli.rb:42:in `execute'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/environment.rb:302:in `cli'
        from Y:/tools/vagrant/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/bin/vagrant:174:in `<main>'

Not an issue with vagrant-managed-servers. Same thing happens with any other provider in my environment...


[WIP] Feature/add auditing monkeypatch

Description

This PR adds the preliminary implementation of extension based auditing. Looking forward to feedback as well as some input on how to test this.
to do:

[x] Add documentation
[ ] Test through implementation (locally)
[x] Test through unit tests

Checklist for success

[x] This change isn't already implemented by a pending pull request
[x] This change is a child commit of a reasonably recent commit in the master branch

Requests need not be a single commit, but should be a linear sequence of commits (i.e. no merge commits in your PR)

[x] All tests pass locally (see the testing section for more info)
[ ] All New behaviors or major functional changes have tests

My last commit should clean up some of Ben's code changes. I'm running into some issues developing a functional example, but still working at it. I'm going to hold off on doc changes until I figure out implementation.
Also what should I do instead of throwing an error? Could I use console.warn or something like that?

🎉woo! implementation works! I modified the code from the tutorial wiki with my auditer (used npm link to use local version of package). Notice in the top right there is a list of 'audits' being output from line 30 of the file on the left. Thomas' published chaos event is sorta broken, tomorrow I'll be able to focus on a pr updating it.
Only thing left for this PR is documentation and deciding what to do about the error line.

I have opened a PR on Thomas' extension that once merged will reference in the documentation for this feature addition https://github.com/trstringer/azure-chaos-fn-webapp-startstop/pull/1

Getting close on this one. I'd like to include an example extension (Thomas' thing) in the documentation. Now that I've settled the --resources issue I have to retest everything and make sure it is working as expected too.


Different namespace for TypeAlias converters

Hi.
I am working with C#
From reading the code of this class:
https://github.com/Microsoft/bond/blob/9b8aa2fd15d72383ea86d6017c4e887c1efb8807/cs/src/core/expressions/TypeAlias.cs
I understand that in order to create a type-alias converter, my converter namespace must be the same as the namespace of the .bond scheme.
I think that this limitation is too harsh.
I want to create my own converters library with the namespace Core/Bond/Converters
Now, my DateTime converter (for example) will be Core/Bond/Converters/DateTimeConverter.cs
The MyStruct.Bond namespace does not have any relation to the namespace "Core/Bond/Converters/DateTimeConverter" (actually, it's namespace is "MyProgram/Models", But I do want it to use the converters from my core library.
Is it possible to tell the gbc.exe to use a different namespace for the type converters (other than the MyStruct.Bond namespace)?

There's nothing in gbc or the C# libraries that will permit this today. Your current recourse--if you want to keep them in separate namespaces--is to have converter implementations in MyProgam.Models that delegate to the ones in Core.Bond.Converters, which can be tedious. Static classes cannot inherit from one another, so there's lots of boilerplate typing. And, you can't just move the BondTypeAliasConverter type into the namespace System, to try to get a match for System.DateTime as the converter must be found in the same assembly as the type.
To enable something like this, we'd need to come up with a way to annotate the C# field/property/type with the location of the converters that is used in addition/instead of the current rules. Then, we'd need to come up with a way to tell gbc to emit such an annotation. Nothing too hard, but this one should start from a design proposal, not just a code change.

Having heard nothing else, closing this issue.

To enable something like this, we'd need to come up with a way to annotate the C# field/property/type with the location of the converters that is used in addition/instead of the current rules. Then, we'd need to come up with a way to tell gbc to emit such an annotation. Nothing too hard, but this one should start from a design proposal, not just a code change.

I want to suggest a different approach to tackle the problem:
Allow specifying conversion method in a 'dependency injection' style - so that the writers/readers will look up for custom converters when it needs one.


chore(*): bump to rust toolchain 1.73

What this PR does / why we need it:
Special notes for your reviewer:
If applicable:

[ ] this PR has an associated PR with documentation in akri-docs
[ ] this PR contains unit tests
[ ] added code adheres to standard Rust formatting (cargo fmt)
[ ] code builds properly (cargo build)
[ ] code is free of common mistakes (cargo clippy)
[ ] all Akri tests succeed (cargo test)
[ ] inline documentation builds (cargo doc)
[ ] all commits pass the DCO bot check by being signed off -- see the failing DCO check for instructions on how to retroactively sign commits

/version patch

/version patch


[Feature]Please support darwin amd64 for older mac machines in release

Thanks for reporting. Fixed in 0.8.0


chore: import @bufbuild/connect instead of @bufbuild/connect-web

We are importing @bufbuild/connect-web a few places; this has been deprecated. We should instead import @bufbuild/connect.
See for example in index.d.ts:
/**
 * @deprecated please import from \@bufbuild/connect instead
 */
Code,

The new imports are:
@bufbuild/connect-web -> @connectrpc/connect-web
and
@bufbuild/connect -> @connectrpc/connect.
I noticed that we use quite old versions of these packages, along with @bufbuild/protobuf.
I started upgrading versions to v2 for protobuf and connect. (note that connect has a rc for v2)
However, v2 is a quite big breaking change for us.
The generated messages are no longer classes. They are now plain objects:
export class User extends Message<User> {
  /**
   * @generated from field: uint64 ID = 1;
   */
  ID = protoInt64.zero;

  /**
   * @generated from field: bool IsAdmin = 2;
   */
  IsAdmin = false;

  ...

  constructor(data?: PartialMessage<User>) {
    super();
    proto3.util.initPartial(data, this);
  }

  ...
}

Where previously we'd create new instances of a message as such:
import { User } from "./proto/qf/types_pb"
const user = new User()
const userWithID = new User({ ID: 10 })

Generated messages for v2 uses plain objects:
export type User = Message<"qf.User"> & {
  /**
   * @generated from field: uint64 ID = 1;
   */
  ID: bigint;

  /**
   * @generated from field: bool IsAdmin = 2;
   */
  IsAdmin: boolean;

  ...
};

To create a new instance of a user:
import { create } from "@bufbuild/protobuf"
import { UserSchema } from "./proto/qf/types_pb"
// the below returns object of type User
const user = create(UserSchema)
const userWithID = create(UserSchema, { ID: 10 })

There are also changes for how we clone objects:
// old
import { User } from "./proto/qf/types_pb"

const user = new User()
const clonedUser = user.clone()

// new
import { clone, create } from "@bufbuild/protobuf"
import { UserSchema } from "./proto/qf/types_pb"

const user = create(UserSchema)
const clonedUser = clone(user)

Timestamps also work differently now:
// old
import { Timestamp } from "@bufbuild/protobuf"

export const timeFormatter = (deadline: Timestamp): Deadline => {
    const timeToDeadline = deadline.toDate().getTime()
    ...
}

// new
import { Timestamp, timestampDate } from "@bufbuild/protobuf/wkt"

export const timeFormatter = (deadline: Timestamp): Deadline => {
    const timeToDeadline = timestampDate(deadline).getTime()

Our custom client also needs tweaking in order to support passing in an error handler and returning wrapped messages
current client.ts
// client.ts
export type Response<T extends AnyMessage> = {
    error: ConnectError | null
    message: T
}

// example of client usage
const response = client.getUser()

if (response.error) {
    // handle error
}

// do something with response.message

I'm sure I'll encounter more differences, but I figured I'd document some of the bigger breaking changes.


Fix usage of WOODPECKER_ROOT_PATH

I had experienced some issues running Woodpecker behind a reverse-proxy, resulting from not defining the WOODPECKER_ROOT_PATH environment variable in #2477.
As suggested by @qwerty287, specifying WOODPECKER_ROOT_PATH=/foo mostly solved the issue of running the woodpecker server at an url like https://example.org/foo.
However, the webhook urls and badge urls were generated excluding the configured WOODPECKER_ROOT_PATH.
This PR (mostly) fixes issues related to non-empty WOODPECKER_ROOT_PATH.

I was just wondering, because for example here the OAuthHost is extended by the RootPath:
https://github.com/woodpecker-ci/woodpecker/blob/4e1db732e9214d99f661fc8dabd593ebf447f3be/server/forge/gitea/gitea.go#L106
So I guess the correct way would be to never use the server.Config.Server.RootPath in conjunction with any server.Config.Server.*Host variable?
Then the RootPath also should be default-initialized with the path component of the url specified in WOODPECKER_HOST...

You might have a look now.
The RootPath is primarily used for

specifying the path at which files are served and requests are routed from in the web server.
redirecting after login.
specifying the absolute path of assets in the index.html.

I think for the index.html it should be sufficient to provide relative paths.
In the first two cases, I think it is safer to keep specifying absolute paths.

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@842e8a8). Click here to learn what that means.
Patch has no changes to coverable lines.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2485   +/-   ##
=======================================
  Coverage        ?   40.57%
=======================================
  Files           ?      185
  Lines           ?    11032
  Branches        ?        0
=======================================
  Hits            ?     4476
  Misses          ?     6204
  Partials        ?      352

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


flutter release isn't making Api call working on debug[android/emulator]

On development everything is working fine but on production version api not being called.
using http library[http: any]
i made apps before using same code. may be after updating it started to occur.
what i tried so far.
1.flutter clean
2.creating new project (still happens)
3.flutter build apk --release
everytime works on emulator or phone but app-release fails.
Steps to Reproduce

...
...
...

Logs

[  +37 ms] executing: [C:\flutter\] git rev-parse --abbrev-ref --symbolic @{u}
[ +130 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[   +3 ms] origin/stable
[        ] executing: [C:\flutter\] git rev-parse --abbrev-ref HEAD
[  +91 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] stable
[        ] executing: [C:\flutter\] git ls-remote --get-url origin
[  +96 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter.git
[        ] executing: [C:\flutter\] git log -n 1 --pretty=format:%H
[  +85 ms] Exit code 0 from: git log -n 1 --pretty=format:%H
[        ] 8661d8aecd626f7f57ccbcb735553edc05a2e713
[        ] executing: [C:\flutter\] git log -n 1 --pretty=format:%ar
[  +83 ms] Exit code 0 from: git log -n 1 --pretty=format:%ar
[        ] 8 weeks ago
[   +1 ms] executing: [C:\flutter\] git describe --match v*.*.* --first-parent --long --tags
[  +96 ms] Exit code 0 from: git describe --match v*.*.* --first-parent --long --tags
[        ] v1.2.1-0-g8661d8aec
[ +501 ms] executing: C:\android\platform-tools\adb devices -l
[  +48 ms] Exit code 0 from: C:\android\platform-tools\adb devices -l
[        ] List of devices attached
[  +18 ms] No connected devices.
[  +14 ms] "flutter run" took 433ms.
[        ] "flutter run" took 433ms.

#0      throwToolExit (package:flutter_tools/src/base/common.dart:24:3)
#1      RunCommand.validateCommand (package:flutter_tools/src/commands/run.dart:240:7)
<asynchronous suspension>
#2      FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:525:11)
<asynchronous suspension>
#3      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:482:33)
<asynchronous suspension>
#4      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:142:29)
<asynchronous suspension>
#5      _rootRun (dart:async/zone.dart:1124:13)
#6      _CustomZone.run (dart:async/zone.dart:1021:19)
#7      _runZoned (dart:async/zone.dart:1516:10)
#8      runZoned (dart:async/zone.dart:1463:12)
#9      AppContext.run (package:flutter_tools/src/base/context.dart:141:18)
<asynchronous suspension>
#10     FlutterCommand.run (package:flutter_tools/src/runner/flutter_command.dart:473:20)
#11     CommandRunner.runCommand (package:args/command_runner.dart:196:27)
<asynchronous suspension>
#12     FlutterCommandRunner.runCommand.<anonymous closure>
(package:flutter_tools/src/runner/flutter_command_runner.dart:396:21)
<asynchronous suspension>
#13     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:142:29)
<asynchronous suspension>
#14     _rootRun (dart:async/zone.dart:1124:13)
#15     _CustomZone.run (dart:async/zone.dart:1021:19)
#16     _runZoned (dart:async/zone.dart:1516:10)
#17     runZoned (dart:async/zone.dart:1463:12)
#18     AppContext.run (package:flutter_tools/src/base/context.dart:141:18)
<asynchronous suspension>
#19     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:356:19)
<asynchronous suspension>
#20     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:111:29)
#21     new Future.sync (dart:async/future.dart:224:31)
#22     CommandRunner.run (package:args/command_runner.dart:111:11)
#23     FlutterCommandRunner.run (package:flutter_tools/src/runner/flutter_command_runner.dart:242:18)
#24     run.<anonymous closure> (package:flutter_tools/runner.dart:60:20)
<asynchronous suspension>
#25     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:142:29)
<asynchronous suspension>
#26     _rootRun (dart:async/zone.dart:1124:13)
#27     _CustomZone.run (dart:async/zone.dart:1021:19)
#28     _runZoned (dart:async/zone.dart:1516:10)
#29     runZoned (dart:async/zone.dart:1463:12)
#30     AppContext.run (package:flutter_tools/src/base/context.dart:141:18)
<asynchronous suspension>
#31     runInContext (package:flutter_tools/src/context_runner.dart:48:24)
<asynchronous suspension>
#32     run (package:flutter_tools/runner.dart:51:10)
#33     main (package:flutter_tools/executable.dart:52:9)
<asynchronous suspension>
#34     main (file:///E:/b/rr/tmpdrhwmy/w/archive/flutter/packages/flutter_tools/bin/flutter_tools.dart:8:3)
#35     _startIsolate.<anonymous closure> (dart:isolate/runtime/libisolate_patch.dart:298:32)
#36     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

Analyzing jansamparklifedotcom...
No issues found! (ran in 29.4s)

[√] Flutter (Channel stable, v1.2.1, on Microsoft Windows [Version 10.0.17134.706], locale en-US)
    • Flutter version 1.2.1 at C:\flutter
    • Framework revision 8661d8aecd (8 weeks ago), 2019-02-14 19:19:53 -0800
    • Engine revision 3757390fa4
    • Dart version 2.1.2 (build 2.1.2-dev.0.0 0a7dcf17eb)

[!] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at C:\android
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = C:\android
    • Java binary at: C:\Program Files\Android\AndroidStudio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    ! Some Android licenses not accepted.  To resolve this, run: flutter doctor --android-licenses

[√] Android Studio (version 3.3)
    • Android Studio at C:\Program Files\Android\AndroidStudio
    • Flutter plugin version 33.4.1
    • Dart plugin version 182.5215
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[√] VS Code (version 1.33.1)
    • VS Code at C:\Users\danish\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 2.25.0

[!] Connected device
    ! No devices available

! Doctor found issues in 2 categories.

the problem was with the INTERNET permission...

I have this issue as well
You mean i need to put a permission for using internet?

I have this issue as well
You mean i need to put a permission for using internet?

yes. you need INTERNET permission.
for that you need to goto android/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.INTERNET"/>

Fixed on Android.
But on iOS I still can't do requests!

Thanks a lot my problem is gone!.

yes. you need INTERNET permission.

i already have this INTERNET permission but still not working on real device. any idea?

yes. you need INTERNET permission.

i already have this INTERNET permission but still not working on real device. any idea?

can you share your AndroidManifest.xml

`

<application
    android:name="io.flutter.app.FlutterApplication"
    android:label="my_app"
    android:icon="@mipmap/ic_launcher">
    <activity
        android:name=".MainActivity"
        android:launchMode="singleTop"
        android:theme="@style/LaunchTheme"
        android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
        android:hardwareAccelerated="true"
        android:windowSoftInputMode="adjustResize">
        <intent-filter>
            <action android:name="android.intent.action.MAIN"/>
            <category android:name="android.intent.category.LAUNCHER"/>
        </intent-filter>
    </activity>
    <!-- Don't delete the meta-data below.
         This is used by the Flutter tool to generate GeneratedPluginRegistrant.java -->
    <meta-data
        android:name="flutterEmbedding"
        android:value="2" />

    <meta-data
        android:name="com.google.android.gms.ads.APPLICATION_ID"
        android:value="ca-app-pub-3940256099942544~3347511713"/>
</application>

`

`

<application
    android:name="io.flutter.app.FlutterApplication"
    android:label="my_app"
    android:icon="@mipmap/ic_launcher">
    <activity
        android:name=".MainActivity"
        android:launchMode="singleTop"
        android:theme="@style/LaunchTheme"
        android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
        android:hardwareAccelerated="true"
        android:windowSoftInputMode="adjustResize">
        <intent-filter>
            <action android:name="android.intent.action.MAIN"/>
            <category android:name="android.intent.category.LAUNCHER"/>
        </intent-filter>
    </activity>
    <!-- Don't delete the meta-data below.
         This is used by the Flutter tool to generate GeneratedPluginRegistrant.java -->
    <meta-data
        android:name="flutterEmbedding"
        android:value="2" />

    <meta-data
        android:name="com.google.android.gms.ads.APPLICATION_ID"
        android:value="ca-app-pub-3940256099942544~3347511713"/>
</application>

`

here is my code: https://codeshare.io/5DRJqO

have you solved this issue ? I have the same as on Emulator I have Android version 8.1.1 and real device has 8.1.0 but on real device the network doesnt work, I added permission to main also to debug and also to profile to test it, none of that works

I have this issue as well
You mean i need to put a permission for using internet?

yes. you need INTERNET permission.
for that you need to goto android/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.INTERNET"/>

Thanks, this solved my issue

Ditto. Fixed the problem. Thanks.
It's odd that nothing throws a warning or error of any sort (anywhere, that I can see).

I have this issue as well
You mean i need to put a permission for using internet?

yes. you need INTERNET permission.
for that you need to goto android/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.INTERNET"/>

Thanks, this solved my issue

the problem was with the INTERNET permission...

Thanks ❤

I have same problem, i also added Internet permission but not working in signed apk or in release mode..

The same issue Any solutions?

Hello I have fixed it
U have to use https in your API URL instead of http.

i have added internet permission,but its not working yet.

In my case I was using two differente urls to fetch data. And one of them wasn't working despite I included internet permission. This solution works for me https://github.com/flutter/flutter/issues/40719#issuecomment-533202279

You need to add this line in the manifest file.
Go to manifest file android/app/src/main/AndroidManifest.xml
add below line in application with internet permission.

This too does not solve the issue,Here is my manifest file
https://codeshare.io/5M6nA3

It is now working by adding the internet permission.

Same problem, it works with adding internet permision in debug mode but it doesn't work in release mode. Any solition for http?
Not https!

i have same problem too for http library. I already added permission to  the manifest file but it's not working in release apk.

i have same problem too for http library. I already added permission to the manifest file but it's not working in release

i have same problem too for http library. I already added permission to the manifest file but it's not working in release apk.

Are you using http or https links??

i have same problem too for http library. I already added permission to the manifest file but it's not working in release

i have same problem too for http library. I already added permission to the manifest file but it's not working in release apk.

Are you using http or https links??

The Problem was in my release AndroidManifest.xml file but i solved, thanks.


[RELEASE]  2023-05-02T22-48-10Z

Release 2023-05-02T22-48-10Z has changes that impact the docs:

[ ] Allow readiness probe to skip auth in K8s environments (Commit d15990d)
[ ] Improve kes identity new command (Commit 8a9e169)

For readiness probe, see also https://github.com/minio/kes/blob/master/server-config.yaml#L54-L97