Remove MTCLIM from Classic Driver

This removes MTCLIM, supporting routines, and documentation from the Classic Driver.  This will require more commits but most of the MTCLIM stuff is gone with 70e4efd.
TODO:

[ ] Remove 2 forcing file option (no longer needed since forcings will need to be at the model timestep).
[ ] Stand-alone MTCLIM disaggregator.
[ ] Write vic_force: read forcing files and pack atmos_data_struct.
[ ] Update test datasets
[ ] Update docs - I started but can't complete until we decide on a few other things.

CC: @tbohn, @bartnijssen
Closes #240
Related and may close #17, #18, #33

@bartnijssen is OK with skipping a stand-alone C disaggregator and moving directly to a python disaggregator.  The key criterion is to evaluate the differences between the new tool and VIC 4.2 in output_force mode at 1-hour time step, and to demonstrate either (a) exact equivalence or (b) any differences that exist are due to bugs in VIC 4.2 rather than the python tool.

Thanks @tbohn. Per our previous discussion, I think we can proceed with this task breakdown:
@tbohn:

Remove 2 forcing file option (no longer needed since forcings will need to be at the model timestep).
Write vic_force: read forcing files and pack atmos_data_struct.

@jhamman:

Stand-alone MTCLIM disaggregator.
Update test datasets

All:

Update docs - I started but can't complete until we decide on a few other things.

Also, the failing test above is a Travis bug and not related to the first commit here.

As I work on vic_force(), I am discovering that it would be much easier to "subroutinize" it if I could loop over the variables in the atmos_data struct instead of calling them by explicit names.  E.g., right now, to assign precip from the forcing_data structure to atmos_data, we have:
(embedded within a loop over major and minor time steps)
atmos_data[rec].prec[i] = forcing_data[PREC][uidx];
If atmos_data looked more like:
atmos_data[PREC][rec][i]
I could loop over all forcing variables as follows (more or less):
define some array forcing_types to be the list of variables that we're going to use, then
for (tidx = 0; tidx < Ntypes; tidx++) {
populate_forcing(&atmos_data[forcing_types[tidx]],&forcing_data[forcing_types[tidx]]);
}
What do you think?

@tbohn - this sounds like a good idea.  However, this change is likely to propagate pretty far down into the code.  You'll have to think if you really want to chase the atmos struct around that much.

@jhamman - I'd prefer not to do something that invasive - I brought it up because changing to that type of data structure is what we'd need to make an effective subroutine for assignment.  So, I will just have to write a separate assignment loop for each variable.  I will avoid redundant code to the extent practical.

@jhamman - looking at vic_force() from image mode, I see that there are some functions I can take advantage of, e.g., average(), q_to_vp(), air_density(), etc..  So, first of all, is it safe to assume that your tool will produce a standard set of variables similar to the set assumed by the image mode version of vic_force()?  In other words, can I have vic_force() expect the following required variables:
prec
air_temp
shortwave
longwave
pressure
q  (specific humidity)
wind
from which I can have vic_force() compute vapor pressure, vpd, density, etc.?
And, should we move these functions average(), air_density(), q_to_vp(), and will_it_snow() to the "shared" folder?

I like it when I can say "yes" to all.  Let's make a forcing_utils.c module in driver/shared/.

So, will you handle unit conversion in your tool?  Presumably the same units that WRF outputs?

I was thinking I would provide a units option that looked like UNITS: {VIC, ALMA}.  I haven't gotten there yet.

Seems to me like we should require 1 consistent set of units - the same units that the image mode driver expects - and your tool should convert to those units...  vic_force() in image mode does not understand the "ALMA" option, and it seems silly to maintain that logic only in the classic driver.  If people are already running your tool to convert daily to sub-daily, etc., why not also do the conversion to preferred units there?  And remove all the ALMA logic from VIC?

I would like to keep the 2-forcing file option, since the veg_hist inputs are quite different from the meteorological inputs.  They have different dimensions (extra veg-tile dimension), very likely different time step (hard to imagine them at smaller than daily), and likely will not be supplied by an atmospheric model like WRF.  Allowing 2 forcing files makes it easier for someone to turn veg_hist on/off: the meteorological input files don't need to change; you just change a couple of global parameter file settings.
This also frees up your tool to not have to deal with them.  Same goes for channel_in and the lake option.
What about renaming the 2nd forcing file to be an optional "veg_hist" file?  And channel_in could be a 3rd optional input file - a "channel_in" file?  Or, you could just allow the optional 2nd forcing file setup to remain as it is...

I much prefer the idea of separate files for the channel_in and veg_hist data streams, why don't we specifically list them in the filenames_struct and we can treat them totally separate from the forcings.  That is much cleaner.

I much prefer the idea of separate files for the channel_in and veg_hist data streams, why don't we specifically list them in the filenames_struct and we can treat them totally separate from the forcings.  That is much cleaner.

OK, thanks.  Regarding units: so, it's still not clear to me: does vic_force() need to handle the ALMA option, or will your tool handle it?  Can I remove the ALMA input option from VIC entirely?

I'm fine with taking the ALMA units out.  @bartnijssen - any objections?

So, I'm going through the code to replace the various 2-element force* arrays with a single element and adding 2 new sets of similar (optional) elements: "veg_hist" and "channel_in".  Just FYI, it makes the code somewhat uglier in that we replace a loop with a bunch of extra, similar lines of code.  I'm now wondering whether it would have made more sense to just keep the old arrays (maybe increase to 3 elements to accommodate channel_in) and insert some logic to enforce that FORCING1 is meteorological forcings, with time step that must equal the snow step.  What do you think, should I proceed with my current path or go back to a 2- or 3- element array with loops?

I'm not exactly sure what you are saying here.  The best way to handle this is to have separate subroutines to handle each forcing stream (atmos, veg_hist, channel_in).

There are many subroutines that deal with the forcing files and their metadata (start date, n_types, dt, etc).  Sure, some, like read_forcing_data(), could be split out into separate functions.  But others can't.  For example, get_global_param() previously looped - incrementing file_num each time it encountered a new forcing section.  Display_current_settings() looped over file_num to display forcing info.  These will now have 3 forcing sections instead of 1.  The global_param and param_set data structures now have 3 sets of metadata variables (start date, dt, etc).
Considering that the handling of forcing, veg_hist, and channel_in files is very similar, doesn't it actually make more sense to keep them as just multiple instances of a single set of forcing file structures?  Splitting out the functions creates extra work whenever we decide to change how files are read - we lose the ability to make just 1 change and have it work for all files.
More and more, I'm thinking that these files are more similar than they are different.  Can you remind me of the benefits of splitting them out?  Why did you originally want to get rid of the 2nd forcing file - what was the cost of keeping it?

Also, I'm basically finished with the patching up of the removal of MTCLIM - that was the easy part, didn't touch many files.
The task of giving different names to the different forcing files (and creating separate routines for them) is much more involved, touches many files in the classic, image, and python drivers, and will likely take me a while longer to finish.  Plus, if I understand correctly, this isn't a show-stopper in terms of getting a testable version of VIC running...
Perhaps the task of separating the forcing files is a lower priority right now, and could be moved into a separate issue?

@tbohn - I'm fine with either a) not making the changes forcing files structure or b) doing it as a later step.  We don't have to rewrite everything at once :smirk:.
Why don't you commit what you have and issue a PR against this branch.  @bartnijssen and I will review and we'll go from there.

@bartnijssen - this is ready for your review.  As you may expect, this is a pretty big PR so get a cup of coffee before setting off.
I've started the work of comparing VIC 5 with 4.2 results but I'd like to apply any changes I make to a separate branch.  So, once you give it a sign off, I'd like to merge this.

@jhamman, @tbohn : This is not ready for merging. A large number of completely unrelated commits have crept into this PR and need to be split out as separate PRs. The whole point is that we should be able to review and roll-back related sets of changes. There are changes to files like read_soilparam.c that having nothing to do with MTCLIM. Please split this out first.

This is really weird.  I didn't make any changes to these files in my
branch and/or pull request.
On Sat, Oct 24, 2015 at 2:57 PM, Bart Nijssen<EMAIL_ADDRESS>wrote:

@jhamman https://github.com/jhamman, @tbohn https://github.com/tbohn
: This is not ready for merging. A large number of completely unrelated
commits have crept into this PR and need to be split out as separate PRs.
The whole point is that we should be able to review and roll-back related
sets of changes. There are changes to files like read_soilparam.c that
having nothing to do with MTCLIM. Please split this out first.
—
Reply to this email directly or view it on GitHub
https://github.com/UW-Hydro/VIC/pull/288#issuecomment-150854053.

If you continue to make changes to the branch on which the PR is based, then these changes will automatically become part of the PR once you push it to github. This greatly facilitates iterative code reviews and improvements, but also requires that you have separate branches for separate PRs (at least that is the easiest way to handle this). Joe for example has many branches related to different issues that he is working on. My guess is that you have kept adding commits to the same branch, which has led to the inclusion of a lot of extraneous commits. Fortunately that will make it very easy for you (or Joe) to split things out.

Except that I haven't made any changes to any vic code after submitting my
pull request to Joe.  And my pull request was for him to merge my branch
into his.  So his pull request contains my pull request plus his own stuff,
I assume.
On Sat, Oct 24, 2015 at 3:08 PM, Bart Nijssen<EMAIL_ADDRESS>wrote:

If you continue to make changes to the branch on which the PR is based,
then these changes will automatically become part of the PR once you push
it to github. This greatly facilitates iterative code reviews and
improvements, but also requires that you have separate branches for
separate PRs (at least that is the easiest way to handle this). Joe for
example has many branches related to different issues that he is working
on. My guess is that you have kept adding commits to the same branch, which
has led to the inclusion of a lot of extraneous commits. Fortunately that
will make it very easy for you (or Joe) to split things out.
—
Reply to this email directly or view it on GitHub
https://github.com/UW-Hydro/VIC/pull/288#issuecomment-150855594.

It may be that he merged in more code than he intended. We'll let him take a look first.

Looking at all of the nodes on the network graph starting from my commits
and following all of Joe's subsequent commits along the "remove_mtclim"
branch, none of them appear to have involved read_soilparam.c...
On Sat, Oct 24, 2015 at 3:10 PM, Bart Nijssen<EMAIL_ADDRESS>wrote:

It may be that he merged in more code than he intended. We'll let him take
a look first.
—
Reply to this email directly or view it on GitHub
https://github.com/UW-Hydro/VIC/pull/288#issuecomment-150856193.

https://github.com/UW-Hydro/VIC/pull/288/files
Not worth arguing about whether or not it is in there or not, because under "Files changed" it is clearly there.

@bartnijssen and @tbohn - Give me a few minutes and I'll be able to explain what the changes to read_soil_param are doing here or if they are here in error.

@bartnijssen - part of removing the MTCLIM code included removing the OUTPUT_FORCE option.  What you are seeing in the diff is the removal of the logic that supported that option.  The way Github is putting together the diff, it is hard to see exactly what I did, but it is basically just whitespace changes (except where I removed the if !options.OUTPUT_FORCE {...}.
Does that make sense?

There is more than just that: Please split this up a little more atomically. There are changes in for example read_soilparam that have to do with neither, so splitting out those commits into a separate PR should be easy. I can accept that one first, so there should be no coding involved, just packaging of the PRs

@bartnijssen - I think we talked through this in person but just to review:

The changes in read_soil_param and similar modules are from removing the OUTPUT_FORCE option, this was part of removing the MTCLIM code.
The whitespace changes in read_soil_param, for example are from removing the if !options.OUTPUT_FORCE {...} blocks, of which there were many.  The github diff does not do these changes justice and makes it appear that the changes were not as simple as they were.  I'd recommend viewing the diff in split mode or just by viewing the whole file on my feature/remove_mtclim branch.

@wietsefranssen has been starting to do his own comparisons of the image and classic drivers and this merge will help with both of our efforts in that regard.

@jhamman : OK - the diff looks complicated even in split view, but I'll take your word for it. I'll go ahead and merge. Ted's issue can be fixes separately.


Series details page (backoffice): identifiers missing

as in tittle, identifiers tab is not there (compare with the document)

It is there as in the document:


Improvement [Admin/Interesting facts block]  Admin can change the order of WOW facts after they are created

Environment: OS, Windows 11
Browser: Google Chrome Version 114.0.5735.135
Reproducible: always
Build found: 1b79c16
Preconditions

Go to http://<IP_ADDRESS>/admin-panel
Login as  Admin
A WOW facts are added

Steps to reproduce

Go to the Interesting facts block - WOW - факти
Verify if Admin can change the order of WOW facts

Actual result
Admin cannot change the order of WOW facts after they are created
Expected result
Admin can reorder WOW facts after they are created
User story and test case links
E.g.: "User story #123
[Test case] #361
https://github.com/ita-social-projects/StreetCode/assets/109734235/bca47e7b-322a-44b8-a069-213a7e3ad755
Labels to be added
"Bug", Priority (" hight"), Severity ("medium"), Type ("Functional")

Duplicate - Actual Bag report -> #865


[Q]Whether the searched parameters support parameter nesting

I recently searched for parameters according to the following conditions, it seems that this format cannot be used
wandb.errors.CommError: Invalid sweep config: invalid hyperparameter configuration: optimizer
  sweep_config:
    project: In_Housing
    entity: chem_mtl
    metric:
      name: val_loss
      goal: minimize
    method: bayes
    parameters:
      optimizer:
        name:
          values: [ 'Adam','SGD' ]
        lr:
          values: [ 1e-3,5e-3,1e-4,5e-4,1e-5,5e-5 ]
        weight_decay:
          values: [ 1e-3,5e-3,1e-4,5e-4,1e-5,5e-5 ]
      lr_scheduler:
        name:
          value: StepLR
        step_size:
          values: [ 50,100 ]
        gamma:
          min: 1e-1
          max: 1.0
      bs:
        values: [ 64,128 ]
      G1:
        name:
          value: G1_Decoder
        p:
          values: [ 0.1,0.2,0.3,0.4,0.5 ]
      G2:
        name:
          value: G2_Decoder
        p:
          values: [ 0.1,0.2,0.3,0.4,0.5 ]
      G3:
        name:
          value: G3_Decoder
        p:
          values: [ 0.1,0.2,0.3,0.4,0.5 ]
      G4:
        name:
          value: G4_Decoder
        p1:
          values: [ 0.1,0.2,0.3,0.4,0.5 ]
        p2:
          values: [ 0.1,0.2,0.3,0.4,0.5 ]

this is ok,Sometimes parameter nesting is more convenient.Maybe consider this feature in the next version

Hi @PussInCode just checking in here to see if you tried the above .yaml file and if that would resolve the issue for you? thanks!

thanks,I'II try it

this is ok, thanks, it's very good


chore(docs): Add LocalPV quickstart link to front page

Signed-off-by: Niladri Halder<EMAIL_ADDRESS>Added the link to the github quickstart doc for hostpath and device localpv. This replaces the links to the uglocalpv-device.md and uglocalpv-hostpath.md pages.

Closing this PR. LocalPV docs changes to uglocalpv-device.md and uglocalpv-hostpath.md will be handled separately in a different PR.


Class properties resulting in false positive

In the following JS I get a "no-undef" on webpack compilation because of inputPicks and contextTypes static class properties but it does not happen firing ESLint from command line.
export default class Input extends React.Component {
    static contextTypes = {
        formName: PropTypes.string.isRequired,
    }

    static inputPicks = ['minLength', 'maxLength', 'pattern']

    constructor(props) {
        super(props, context);
    }
...and so on

Package.json
    "eslintConfig": {
        "parser": "babel-eslint",
        "parserOptions": {
            "ecmaVersion": 2017,
            "sourceType": "module",
            "impliedStrict": true,
            "ecmaFeatures": {
                "jsx": true,
                "experimentalObjectRestSpread": true
            }
        },
        "env": {
            "browser": true,
            "node": true,
            "jest": true,
            "es6": true
        },
        "plugins": [
            "import",
            "react",
            "jest"
        ],
        "extends": [
            "eslint:recommended",
            "plugin:react/recommended",
            "plugin:jest/recommended"
        ],
        "rules": {
            "no-console": "off",
            "no-unused-vars": "off",
            "no-unsafe-finally": "off",
            "react/no-unescaped-entities": "off",
            "react/prop-types": "off"
        }
    },

Dependencies:
    "devDependencies": {
        "autobind-decorator": "^1.3.4",
        "autoprefixer": "^7.1.1",
        "babel-core": "^6.7.4",
        "babel-eslint": "^7.1.1",
        "babel-loader": "7.0.0",
        "babel-plugin-react-css-modules": "^2.6.0",
        "babel-plugin-syntax-class-properties": "^6.13.0",
        "babel-plugin-transform-decorators-legacy": "^1.3.4",
        "babel-polyfill": "^6.20.0",
        "babel-preset-env": "2.0.0-alpha.3",
        "babel-preset-react": "^6.5.0",
        "babel-preset-stage-0": "^6.16.0",
        "babili": "^0.1.2",
        "babili-webpack-plugin": "^0.1.1",
        "clean-webpack-plugin": "0.1.16",
        "compression-webpack-plugin": "^0.4.0",
        "css-loader": "^0.28.4",
        "cssnano": "^3.10.0",
        "eslint": "^4.0.0",
        "eslint-loader": "^1.6.1",
        "eslint-plugin-import": "^2.2.0",
        "eslint-plugin-jest": "^20.0.3",
        "eslint-plugin-react": "^7.0.1",
        "extract-text-webpack-plugin": "^2.1.0",
        "favicons-webpack-plugin": "^0.0.7",
        "file-loader": "^0.11.1",
        "history": "^4.6.1",
        "html-webpack-plugin": "^2.25.0",
        "img-loader": "^2.0.0",
        "json-loader": "^0.5.4",
        "material-ui": "^0.18.1",
        "material-ui-community-icons": "^0.15.0",
        "node-sass": "^4.1.1",
        "open-browser-webpack-plugin": "^0.0.5",
        "postcss-load-config": "^1.1.0",
        "postcss-loader": "^2.0.5",
        "postcss-scss": "^1.0.0",
        "query-string": "^4.3.4",
        "react": "^15.4.1",
        "react-addons-perf": "^15.4.2",
        "react-create-app": "^1.0.3",
        "react-create-store": "^1.0.4",
        "react-dimensions": "^1.3.0",
        "react-dom": "^15.4.1",
        "react-ga": "^2.2.0",
        "react-highcharts": "^12.0.0",
        "react-moment": "^0.2.4",
        "react-redux": "^5.0.1",
        "react-router": "^4.1.1",
        "react-router-dom": "^4.1.1",
        "react-router-redux": "next",
        "react-sizeme": "^2.3.2",
        "react-svg-inline": "^2.0.0",
        "react-swipeable-views": "^0.12.3",
        "react-tap-event-plugin": "^2.0.0",
        "react-transition-group": "^1.1.3",
        "recompose": "^0.23.4",
        "redux": "^3.6.0",
        "redux-action-buffer": "^1.0.1",
        "redux-actions": "^2.0.3",
        "redux-devtools-extension": "^2.13.2",
        "redux-logger": "^3.0.6",
        "redux-persist": "^4.1.0",
        "redux-persist-crosstab": "^3.5.2",
        "redux-promise": "^0.5.3",
        "redux-thunk": "^2.1.0",
        "sass-loader": "^6.0.5",
        "string-replace-loader": "^1.2.0",
        "style-loader": "^0.18.2",
        "stylelint-config-sass-guidelines": "^2.1.0",
        "stylelint-config-standard": "^16.0.0",
        "stylelint-scss": "^1.4.4",
        "stylelint-webpack-plugin": "^0.7.0",
        "url-loader": "^0.5.8",
        "webpack": "^2.5.1",
        "webpack-bundle-analyzer": "^2.8.2",
        "webpack-chunk-hash": "^0.4.0",
        "webpack-notifier": "^1.5.0"
    }

I don't know where this bug is... here is the eslint ref
https://github.com/eslint/eslint/issues/8720

wow I didn't see that: did you just open the same issue some hours before I did? xD
Maybe something due to latest updates?

I opened it last night, but they (eslint) suggested the bug may be here.

This is serious.
In order to compile I'm now with "no-undef": "off": I just lost the whole morning (~3hours) to track down a simple not defined variable which eslint would spot in milliseconds.

Sorry this is causing issues. I'm not familiar with the codebase much. Could someone dive in a bit and provide a suggested solution/pull request?

Is think broken in eslint 3.19 as well or just 4.0?

@soda0289 hell yeah, fixed in 4.0! Thanks so much ( @jsg2021 )

i don't see any changes in this or eslint... how is it fixed?

@soda0289 this wasn't a problem in eslint 3.x.x

If this problem started with eslint v4 it probably has something to do with the new scope analysis fork eslint uses. It was changed from escope to eslint-scope.
I think the problem is that babel-eslint is failing to find and monkey patch eslint-scope. It might be finding escope from some other installation of eslint. How are you running eslint?

Rules in package.json
    "eslintConfig": {
        "parser": "babel-eslint",
        "parserOptions": {
            "ecmaVersion": 2017,
            "sourceType": "module",
            "impliedStrict": true,
            "ecmaFeatures": {
                "jsx": true,
                "impliedStrict": true,
                "globalReturn": false,
                "experimentalObjectRestSpread": true
            }
        },
        "env": {
            "browser": true,
            "node": true,
            "jest": true,
            "es6": true
        },
        "plugins": [
            "import",
            "react",
            "jest"
        ],
        "extends": [
            "eslint:recommended",
            "plugin:react/recommended",
            "plugin:jest/recommended"
        ],
        "rules": {
            "no-console": "off",
            "no-unused-vars": "off",
            "no-unsafe-finally": "off",
            "no-unreachable": "off",
            "react/no-unescaped-entities": "off",
            "react/prop-types": "off"
        }
    },
    "stylelint": {
        "extends": [
            "stylelint-config-standard",
            "stylelint-config-sass-guidelines"
        ],
        "rules": {
            "indentation": 4,
            "no-missing-end-of-source-newline": null,
            "selector-list-comma-newline-after": null,
            "max-nesting-depth": null,
            "selector-pseudo-class-no-unknown": [
                true,
                {
                    "ignorePseudoClasses": [
                        "global",
                        "local"
                    ]
                }
            ]
        }
    },
    "babel": {
        "presets": [
            [
                "env",
                {
                    "targets": {
                        "browsers": [
                            "chrome 54",
                            "safari 9",
                            "firefox 50",
                            "edge 12",
                            "explorer 11"
                        ]
                    },
                    "modules": false,
                    "useBuiltIns": true,
                    "debug": true
                }
            ],
            "stage-0",
            "react"
        ],
        "plugins": [
            "transform-decorators-legacy",
            "transform-class-properties",
            [
                "react-css-modules",
                {
                    "filetypes": {
                        ".scss": "postcss-scss"
                    }
                }
            ]
        ]
    },
    "jest": {
        "bail": true
    },
    "dependencies": {
        "array-to-csv": "^1.0.1",
        "axios": "^0.16.2",
        "better-console": "^1.0.0",
        "child-process-promise": "^2.2.1",
        "compare-versions": "^3.0.1",
        "cssnano-preset-default": "^4.0.0-rc.0",
        "csv-parse": "^1.2.0",
        "csvjson": "^4.3.3",
        "docdash": "^0.4.0",
        "esdoc": "^0.5.2",
        "fs-promise": "^2.0.2",
        "jest": "^20.0.4",
        "jsdoc": "^3.4.3",
        "jsdoc-strip-async-await": "^0.1.0",
        "jws": "^3.1.4",
        "koa": "^2.2.0",
        "koa-basic-auth": "^2.0.0",
        "koa-better-ratelimit": "^2.1.2",
        "koa-body": "^2.1.0",
        "koa-compress": "^2.0.0",
        "koa-cors": "0.0.16",
        "koa-json": "^2.0.2",
        "koa-mount": "^3.0.0",
        "koa-passport": "^3.0.0",
        "koa-remove-trailing-slashes": "1.0.0",
        "koa-router": "^7.2.0",
        "koa-send": "^4.1.0",
        "koa-sslify": "^2.1.2",
        "koa-useragent": "^1.0.0",
        "lodash": "^4.17.4",
        "moment": "^2.18.1",
        "moment-timezone": "^0.5.13",
        "mz": "^2.6.0",
        "node-schedule": "^1.1.1",
        "node-xlsx": "^0.7.4",
        "nodemailer": "^4.0.1",
        "passport": "^0.3.2",
        "passport-facebook": "^2.1.1",
        "passport-google-oauth": "^1.0.0",
        "pm2": "^2.4.4",
        "sequelize": "^4.1.0",
        "slack-node": "^0.1.8",
        "sleep-promise": "^2.0.0",
        "sqlite3": "^3.1.8",
        "stripe": "^4.22.0",
        "tsv": "^0.2.0",
        "xlsx-populate": "^1.6.0",
        "yargs": "^8.0.1"
    },
    "devDependencies": {
        "autobind-decorator": "^1.3.4",
        "autoprefixer": "^7.1.1",
        "babel-core": "^6.7.4",
        "babel-eslint": "^7.1.1",
        "babel-loader": "7.0.0",
        "babel-plugin-react-css-modules": "^2.6.0",
        "babel-plugin-syntax-class-properties": "^6.13.0",
        "babel-plugin-transform-decorators-legacy": "^1.3.4",
        "babel-polyfill": "^6.20.0",
        "babel-preset-env": "2.0.0-alpha.3",
        "babel-preset-react": "^6.5.0",
        "babel-preset-stage-0": "^6.16.0",
        "babili": "^0.1.3",
        "babili-webpack-plugin": "^0.1.0",
        "case-sensitive-paths-webpack-plugin": "^2.1.1",
        "clean-webpack-plugin": "0.1.16",
        "compression-webpack-plugin": "^0.4.0",
        "css-loader": "^0.28.4",
        "cssnano": "^3.10.0",
        "deep-diff": "^0.3.8",
        "eslint": "^4.0.0",
        "eslint-loader": "^1.8.0",
        "eslint-plugin-import": "^2.2.0",
        "eslint-plugin-jest": "^20.0.3",
        "eslint-plugin-react": "^7.1.0",
        "extract-text-webpack-plugin": "^2.1.0",
        "favicons-webpack-plugin": "^0.0.7",
        "file-loader": "^0.11.1",
        "history": "^4.6.2",
        "html-webpack-plugin": "^2.25.0",
        "img-loader": "^2.0.0",
        "json-loader": "^0.5.4",
        "material-ui": "^0.18.1",
        "material-ui-community-icons": "^0.15.0",
        "node-sass": "^4.1.1",
        "open-browser-webpack-plugin": "^0.0.5",
        "postcss-load-config": "^1.1.0",
        "postcss-loader": "^2.0.5",
        "postcss-scss": "^1.0.0",
        "query-string": "^4.3.4",
        "react": "^15.6.1",
        "react-addons-perf": "^15.4.2",
        "react-create-app": "^1.0.3",
        "react-create-store": "^1.0.4",
        "react-dimensions": "^1.3.0",
        "react-dom": "^15.6.1",
        "react-ga": "^2.2.0",
        "react-highcharts": "^12.0.0",
        "react-moment": "^0.2.4",
        "react-redux": "^5.0.1",
        "react-router": "^4.1.1",
        "react-router-dom": "^4.1.1",
        "react-router-redux": "next",
        "react-sizeme": "^2.3.2",
        "react-svg-inline": "^2.0.0",
        "react-swipeable-views": "^0.12.3",
        "react-tap-event-plugin": "^2.0.0",
        "react-transition-group": "^1.1.3",
        "recompose": "^0.23.4",
        "redux": "^3.6.0",
        "redux-action-buffer": "^1.0.1",
        "redux-actions": "^2.0.3",
        "redux-devtools-extension": "^2.13.2",
        "redux-logger": "^3.0.6",
        "redux-persist": "^4.1.0",
        "redux-persist-crosstab": "^3.5.2",
        "redux-promise": "^0.5.3",
        "redux-thunk": "^2.1.0",
        "sass-loader": "^6.0.6",
        "string-replace-loader": "^1.2.0",
        "style-loader": "^0.18.2",
        "stylelint-config-sass-guidelines": "^2.1.0",
        "stylelint-config-standard": "^16.0.0",
        "stylelint-scss": "^1.4.4",
        "stylelint-webpack-plugin": "^0.7.0",
        "url-loader": "^0.5.8",
        "webpack": "^2.5.1",
        "webpack-bundle-analyzer": "^2.8.2",
        "webpack-chunk-hash": "^0.4.0",
        "webpack-notifier": "^1.5.0"
    }
}

I'm on npm 5, node 8.
I execute eslint by: eslint --ext .js,.jsx ./src
my eslint config is pretty simple...
{
	extends: 'eslint:recommended',
	parser: 'babel-eslint',
	parserOptions: {
		ecmaVersion: 2017,
		sourceType: 'module',
		ecmaFeatures: {
			experimentalObjectRestSpread: true,
			impliedStrict: true,
			globalReturn: false
		}
	},

	env: {
		es6: true,
		node: true,
		browser: true,
		jest: true
	}
}

@jsg2021
So you are using the globally installed eslint. Can you try the locally installed one:
node node_modules/.bin/eslint --ext .js,.jsx ./src

that was the local one. I have ./node_modules/.bin/ on my path. I don't use global packages.

I'm always local:
    "scripts": {
        "pm2": "./node_modules/pm2/bin/pm2",
        "lint": "./node_modules/eslint/bin/eslint.js --ignore-path .gitignore --ignore-pattern **/*.min.js .",

Why is this issue closed? There are multiple reports here and in eslint:
eslint/eslint#8720

@Lyncredible very weird: I don't exactly know what was updated in my last "yarn upgrade" but this is broken again...

This issue was resolved for me when upgrading to 7.2.2 (might have even worked before that, only know it did not work with 6.1.2).

Occurs for me on 7.2.2 and eslint 4.3.0

Same here on 7.2.3 and ESLint 4.3.0.

This was fixed for us by explicitly invoking the ./node_modules/eslint/bin/eslint.js script rather than using the binary in ./node_modules/.bin/eslint I would suggest that perhaps there's some issue with the binary being generated in 4.3.x?

It just went away for me... don't know when/how it changed.
I'm on node 8.2.1, npm 5.3.0, latest eslint 4.x, latest babel 6.x, babel-eslint 7.x

I'm having this problem with node v6.11.1, npm 5.3<EMAIL_ADDRESS><EMAIL_ADDRESS>

Same for me! ALARM!

I can confirm I'm having this same issue as well. no-undef false positive with babel-esint: 7.2.3 and eslint: 4.4.1

I was getting this just in the webpack build, and not when running eslint from the command line. I deleted the node_modules folder and did a yarn install and all the problems went away. babel-esint: 7.2.3 and eslint: 4.4.1

FWIW it appears to occur when packages are installed with Yarn instead of NPM. Running npm rebuild after a fresh yarn install fixes the problem for me.
These are the content of the package.json file:
{
  ...
  "scripts": {
    ...
    "lint": "eslint --cache --ext .jsx,.js .",
    ...
  },
  ...
  "devDependencies": {
    ...
    "eslint": "4.4.1",
    "eslint-plugin-import": "2.7.0",
    "eslint-plugin-jsx-a11y": "5.1.1",
    "eslint-plugin-react": "7.2.0",
    ...
  },
  ...
}

These are the versions of Node, NPM, and Yarn running on my machine:
node --version
v6.11.2

npm --version
3.10.10

yarn --version
0.27.5

We also have this exact problem after upgrading to latest eslint and babel-eslint.

I am having this issue after upgrade.
My devDependencies are:
  "devDependencies": {
    "autoprefixer": "^7.1.1",
    "babel-core": "^6.25.0",
    "babel-eslint": "^7.2.3",
    "babel-loader": "^7.1.0",
    "babel-plugin-angularjs-annotate": "^0.8.0",
    "babel-plugin-flow-runtime": "^0.11.1",
    "babel-plugin-syntax-flow": "^6.8.0",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "babel-plugin-transform-decorators-legacy": "^1.3.4",
    "babel-preset-env": "^1.6.0",
    "babel-preset-flow": "^6.23.0",
    "babel-preset-stage-3": "^6.24.1",
    "css-loader": "^0.28.4",
    "del-cli": "^1.1.0",
    "eslint": "^4.6.1",
    "eslint-import-resolver-webpack": "^0.8.3",
    "eslint-plugin-flowtype": "^2.34.0",
    "eslint-plugin-import": "^2.7.0",
    "extract-text-webpack-plugin": "^3.0.0",
    "file-loader": "^0.11.1",
    "firebase-tools": "^3.9.2",
    "flow-bin": "^0.53.1",
    "flow-runtime-mobx": "^0.14.0",
    "html-loader": "^0.5.0",
    "html-webpack-plugin": "^2.30.1",
    "mobx-logger": "^0.6.0",
    "ngtemplate-loader": "^2.0.1",
    "node-sass": "^4.5.3",
    "postcss-loader": "^2.0.6",
    "raw-loader": "^0.5.1",
    "sass-loader": "^6.0.6",
    "style-loader": "^0.18.1",
    "svgo": "^0.7.2",
    "svgo-loader": "^1.2.1",
    "webpack": "^3.3.0",
    "webpack-dev-server": "^2.4.5",
    "webpack-notifier": "^1.5.0",
    "xo": "^0.18.2",
    "yargs": "^8.0.1"
  }

Seems like I've encountered this problem after adding flow related packages:
    ...
+   "babel-cli": "^6.26.0",
    "babel-core": "^6.26.0",
    "babel-eslint": "^7.2.3",
    "babel-jest": "^21.0.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-lodash": "^3.2.11",
    "babel-plugin-module-resolver": "^2.7.1",
    "babel-plugin-ramda": "^1.4.3",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "babel-plugin-transform-object-rest-spread": "^6.26.0",
    "babel-plugin-transform-react-constant-elements": "^6.23.0",
    "babel-plugin-transform-react-inline-elements": "^6.22.0",
    "babel-plugin-transform-react-remove-prop-types": "^0.4.8",
    "babel-preset-env": "^1.6.0",
+   "babel-preset-flow": "^6.23.0",
    "babel-preset-jest": "^21.0.0",
    "babel-preset-react": "^6.24.1",
    ...
    "eslint": "3.19.0",
    "eslint-config-airbnb": "^15.1.0",
    "eslint-import-resolver-babel-module": "^3.0.0",
    "eslint-import-resolver-webpack": "^0.8.3",
    "eslint-plugin-babel": "^4.1.2",
    "eslint-plugin-compat": "^1.0.4",
    "eslint-plugin-fetch": "0.0.1",
    "eslint-plugin-import": "^2.7.0",
    "eslint-plugin-jsx-a11y": "5.1.1",
    "eslint-plugin-react": "^7.3.0",
    "eventsource-polyfill": "^0.9.6",
+   "flow-bin": "^0.54.1",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^21.0.1",
    ...,
+   "prettier-eslint": "^8.0.0",
    ...

Same issue.

I can also confirm that we have no-undef false positive with babel-esint: ^7.2.3 and eslint: ^4.4.1.

+1

Is anyone taking a look at this? It seems this should be done by someone who knows a bit about eslints and this plugins internals to make the correct adjustments needed for the newer eslint versions?

+1

also happening with eslint 3.19.0 and babel-eslint 7.2.3
Any workaround for this for the time being without just disabling no-undef ?

i solved this by updating the relative dependencies.
new version :
"eslint": "^4.8.0",
"babel-eslint": "^8.0.1"
hope it helps

It works by eslint . --ext js,jsx
But it is broken when use new CLIEngine().executeOnFiles(['path/to/files'])
node 8.4.0
babel-eslint 8.0.1
eslint: 4.9.0

Still occurring with
eslint ^4.11  babel-eslint ^8.0.2 node 8.9.1

There's been changes in the class-properties implementation in 7.0.0-beta.x after v6.x see.
http://www.mergely.com/4LNZVfuO/

Was seeing this error, then updated to "eslint": "^4.12.0" and "babel-eslint": "^8.0.2" and now it's fixed. Updating only babel-eslint did not fix it. I'm on node v8.8.1 and npm v5.5.1

This issue started for me when I added prettier-eslint. prettier-eslint is using eslint 4.5, if I add<EMAIL_ADDRESS>(when they're using eslint 3.1.9) the issue goes away. I'm using "eslint": "3.19.0" and "babel-eslint": "7.2.3"

I am experiencing the same issue. If I use<EMAIL_ADDRESS>the errors goes away. But if I upgrade to<EMAIL_ADDRESS>I start to get these errors.
I don't use Eslint, but Standard instead. First I thought it was something wrong with Standard, but after finding this issue I understand that it has something to do with babel-eslint.
How come these kind of issues starts to occur with<EMAIL_ADDRESS>$ node -v
v9.2.0

$ npm -v
5.5.1

  devDependencies: {
    ...
    "babel-eslint": "7.2.3",
    "husky": "^0.14.3",
    "snazzy": "^7.0.0",
    "standard": "^10.0.3",
    ...
  },
  "lint-staged": {
    "*.js": [
      "yarn run lint",
      "git add"
    ]
  },
  "config": {},
  "standard": {
    "parser": "babel-eslint",
    "globals": [
      "describe",
      "test",
      "jest",
      "expect",
      "fetch",
      "navigator",
      "__DEV__",
      "XMLHttpRequest",
      "FormData",
      "React$Element"
    ]
  }

Same here, tried above and it's the same.

From what I can tell, this is now fixed in babel-eslint ^8.1.0, which drops the unreliable monkey-patching in favor of the new eslint parser API. To make this work, you also need eslint ^4.14.0, which is the first release that includes the new parser API. Otherwise, babel-eslint will fall back on the old & buggy code path.

Still happens to me:
    "babel-core": "^6.26.0",
    "babel-eslint": "7.2.3",
    "babel-plugin-transform-decorators-legacy": "^1.3.4",
    "babel-preset-react-native": "^4.0.0",
    "eslint": "4.13.1",
    "eslint-config-standard-react": "^5.0.0",
    "eslint-plugin-flowtype": "^2.44.0",
    "standard": "^10.0.3"

  "standard": {
    "plugins": [
      "flowtype"
    ],
    "parser": "babel-eslint"
  }

{
  "parser": "babel-eslint",
  "extends": [
    "standard",
    "standard-react"
  ]
}

@swansontec I tried with<EMAIL_ADDRESS>and<EMAIL_ADDRESS>and also had the issues

With eslint at ^4.18.1 and babel-eslint at ^8.2.2 the issue is gone for me.
If, like me, you have a global installation of eslint, you might want to check where the command is being run from. In my case the eslint version inside the project dependencies, where it was being run from, was an older one. After updating the dependency, props are no longer producing lint errors for me. Hope this helps!

hey, it seems this still break eslint@3,  maybe should pick a patch?

I verified it's caused by the monkey patch.
I started with the following versions and it worked perfectly.
"eslint": "^3.19.0",
"babel-eslint": "^7.2.3",

Then, after I upgraded to webpack 4, everything broke without changing any eslint related versions.
I found that webpack 4 depends on eslint-scope now, and it seems like babel-eslint will try to use this eslint-scope instead of its own dependency escope.
I verified this by doing npm rm eslint-scope, then the eslint error goes away.
I don't know an easy way to get rid of eslint-scope, so I chose to upgrade to the eslint 4 and eslint-babel 8 in the end.

What @Bobgy said - just experienced the same thing with the same versions after an update to webpack 4.

Just ran into this issue when I bumped the version of an unrelated dependency (not eslint/webpack/babel-eslint). Our organization has numerous modules using eslint v3 and babel-eslint v7. One common dependency was updated that uses eslint v4. This caused eslint-scope to bubble up to the top node_modules directory which switched eslint v3 to now load it. This resulted in all our module eslints now falsely failing for no-undef errors for things like static class properties.
Seems like this issue should really be addressed as bringing in an unrelated dependency should not cause this sort of side affect.

upgrade eslint latest
npm i -D eslint@latest

This bug is back with babel-eslint@^9 and instance fields containing bound methods:
class asdf {
  m = () => { // <-- BORK rule: 'no-undef', add eslint-disable-next-line no-undef above to workaround

  }
}

Same issue:
    "eslint": "^4.13.1",
    "babel-eslint": "^10.0.1",
node: 10.10.0

class CustomDialog extends React.Component {
    state = {  // <----[eslint] 'state' is not defined. (no-undef)
        open: false,
    };

In a yarn workspace, you can prevent hoisting of the problematic eslint-scope package with the following config in your package.json:
"workspaces": {
  "packages": [
    "packages/*"
  ],
  "nohoist": [
    "**/eslint-scope"
  ]
},

I updated all my packages, see my package.json:
"devDependencies": {
    "babel-eslint": "^8.2.1",
    "eslint": "^4.16.0",
    "eslint-config-airbnb": "^16.1.0",
    "eslint-plugin-import": "^2.8.0",
    "eslint-plugin-jsx-a11y": "^6.0.3",
    "eslint-plugin-react": "^7.5.1"
  },

and my .eslintrc is so
"parser": "babel-eslint",
  "extends": "airbnb",
  "plugins": [
    "react",
    "jsx-a11y",
    "import"
 ],

I hope I can have helped

It's worked for me!

In a yarn workspace, you can prevent hoisting of the problematic eslint-scope package with the following config in your package.json:
"workspaces": {
  "packages": [
    "packages/*"
  ],
  "nohoist": [
    "**/eslint-scope"
  ]
},

@kryops  this method is work for me! 👏

Upgrading to:
{
"eslint": "^5.16.0",
"babel-eslint": "^10.0.1",
}
Solved the issue for me.

Having these issues with:
{
"eslint": "^3.1.0",
"babel-eslint: "^9.0.0"
}

Having the same issue with
{
"eslint": "^3.6.0",
"babel-eslint": "^9.0.0","
}

Any update on this issue? i have the same problem btw after upgrading babel from v6 to v7. using eslint 3.19.0 and babel eslint version 9.0.0

Since people (including me recently) are still running into this issue, and I just spent quite some time figuring it out, I would like to post a long answer with some reference.
TL;DR
This should be solved by
    "eslint": ">=4.14.0",
    "babel-eslint": ">=8.1.0"
}```
with fix from `eslint/eslint` [PR#8755](https://github.com/eslint/eslint/pull/8755),
and fix from `babel/babel-eslint` [PR#542](https://github.com/babel/babel-eslint/pull/542).
Thanks to @mysticatea for the hard work.
------

**Explanation:**
This is a issue caused by the 'scope analyze' in eslint. In the older version (<4.14.0), eslint analyze the scope by its own without passing in the correct `childVisitorKeys` attribute.

// eslint.js
// gather scope data that may be needed by the rules
scopeManager = escope.analyze(ast, {
ignoreEval: true,
nodejsScope: ecmaFeatures.globalReturn,
impliedStrict: ecmaFeatures.impliedStrict,
ecmaVersion,
sourceType: currentConfig.parserOptions.sourceType || "script",
fallback: Traverser.getKeys
});
https://github.com/eslint/eslint/blob/421aab44a9c167c82210bed52f68cf990b7edbea/lib/eslint.js#L892

A correct scope after `parser.parse()` requires corresponding `childVisitorKeys` defined by different parser, as [PR#8755](https://github.com/eslint/eslint/pull/8755#issue-126156112) explains.

After those two PRs, `babel-eslint` will generate the `scopeManager` with its own modification.
`eslint` will consume the `scopeManager` (`const scopeManager = parseResult.scopeManager;`), and apply rules over it.

**Take away:**
It costs time to locate the root cause for this kind of issue if you are not familiar with the massive project like eslint  (or ast concept), not to mention it normally comes with several more issues together. If you really want to do the research, start from the `rules` folder and analyze specific rule to get two cents about what goes wrong, in this case, it's the scope. It would be much easier then to narrow down your research in scope-related logic in `parse()`.

The easy way to 'just make things work' is always upgrade version.

Getting this issue (using React Native).
I previously only had "eslint-config-rallycoding": "^3.2.0" in my dev dependencies and eslint installed globally and everything was going smoothly. I had to install babel-eslint 7.2.1 to get operator chaining (a?.b) working without errors.
This then caused this issue where arrow functions on classes were "undefined". I upgraded to the latest babel-eslint, and this gave me the error that it couldn't find "eslint-scope".
Installing the latest eslint in dev dependencies then broke everything and files stopped linting at all.

Thank you for the issue. Now that @babel/eslint-parser has been released, we are making this repository read-only. If this is a change you would still like to advocate for, please reopen this in the babel/babel monorepo.


eth/node/syncing reports is_syncing: false during optimistic sync

Description
When teku has optimistically imported all blocks but the EL is still syncing, the /eth/v1/node/syncing api reports that the node is syncing while the logs say it is.
Logs:
2022-04-04 01:49:03.760 | beaconchain-async-3       | INFO  | teku-event-log                 | Syncing started
2022-04-04 01:49:12.426 | TimeTick-1                | INFO  | teku-event-log                 | Syncing     *** Target slot: 32621, Head slot: 32575, Remaining slots: 46, Connected peers: 3
2022-04-04 01:49:20.666 | beaconchain-async-4       | INFO  | teku-event-log                 | Beacon chain syncing complete, waiting for Execution Client
2022-04-04 01:49:24.811 | TimeTick-1                | INFO  | teku-event-log                 | Syncing     *** Target slot: 32622, Head slot: 32621, Remaining slots: 1, Connected peers: 3
2022-04-04 01:49:36.439 | TimeTick-1                | INFO  | teku-event-log                 | Syncing     *** Target slot: 32623, Head slot: 32622, Remaining slots: 1, Connected peers: 3
2022-04-04 01:49:48.434 | TimeTick-1                | INFO  | teku-event-log                 | Syncing     *** Target slot: 32624, Head slot: 32623, Remaining slots: 1, Connected peers: 6

Syncing response:
{
  "data": {
    "head_slot": "32625",
    "sync_distance": "0",
    "is_syncing": false
  }
}

We probably need to clarify how this API should work post-merge.


Create dataset loader for CC100-Sundanese

#self-assign

I'm currently running the cc-net tools to regenerate the corpus matching the paper's configuration.

For simplicity, statmt has the datasets from 2018 saved: https://data.statmt.org/cc-100/

The existing dataset hub already supports Sundanese (using su), it's just not well documented.
from datasets import load_dataset

dataset = load_dataset("cc100", lang="su")
dataset['train'][0]

generates
{'id': '0', 'text': '"Enya. ali ieu mah sok pipindahan"\n'}


Question about data

Hi!
I have another question about data. I download your dataset and find that the images is black background and edge part is white. I want to know if black background and white edge part would affect the encoder to encode the images. Looking forward to your reply!
Thanks!

Hi,
I think it's not difficult for the neural network to learn the patterns of rendered images.


Migrate to Nextclade v3

Description of proposed changes
Migrates the ingest workflow to Nextclade v3, which simplifies the workflow since we no longer need to use two Nextclade datasets. Includes additional clean ups to use augur curate rename and augur merge, see commits for details.
Related issue(s)
Resolves https://github.com/nextstrain/mpox/issues/280
Checklist

[x] Checks pass
[x] Trial run of ingest workflow

Trial run successfully completed and uploaded results to s3://nextstrain-data/files/workflows/mpox/branch/ingest-nextclade-v3/
There are many changes across the metadata.tsv compared to the production metadata.tsv because of changes in the Nextclade columns. A majority of these changes come from the divergence column, which is expected because the v3 MPXV Nextclade dataset uses NC_063383.1 as the reference instead of a reconstructed ancestor.
I took a quick look at clade changes:

A majority of clade changes are a result of the new Ia & Ib clade distinctions
42 sequences that used to fail now get assigned clades
12 sequences that were labelled as outgroup now fail

 Detailed clade change counts

prod_clade
v3_clade
count

I
6

II
4

IIa
5

IIb
10

Ia
2

outgroup
15

I
II
1

I
IIb
1

I
Ia
518

I
Ib
133

I
outgroup
22

II
IIa
1

II
IIb
14

II
outgroup
7

IIa
II
2

IIa
IIb
24

IIa
outgroup
1

IIb
outgroup
2

outgroup

12

outgroup
I
1

outgroup
II
4

outgroup
IIa
4

outgroup
IIb
28

These changes seem reasonable to me, but would like a quick review from @corneliusroemer


NavigationRegion2D Not Rendering Cutout Consistently

Godot version
v4.1.2.stable.official [399c9dc39]
System information
Windows 10
Issue description
When adding another outline with add_outline() to the NavigationRegion2D to create a hole via make_polygons_from_outlines(), sometimes the renderer does not create a hole and instead layers another polygon on top. This issue is somehow location-dependent, and is shown in the minimal reproduction project when the CollisionPolygon2D is placed at (-256,120) vs (360,-100) for example.
Unsure if this is necessary, but expanding the Polygon via offset_polygon() is used to create the outline in the project. I was unable to reproduce this bug without it. But I do not know if it's necessary.
Based on how the NavigationRegion2D is rendered with and without the hole, I believe this is not just a rendering issue but the hole is not actually created. As when the hole renders correctly it also is triangulated correctly, and when it doesn't the original NavigationRegion2D remains a single polygon.

Steps to reproduce

Create a NavigationRegion2D with a NavigationPolygon large enough to encompass another shape inside of it.
Create another Polygon inside of the NavigationRegion2D
Expand the Polygon  via offset_polygon()
Add the Polygon vertices as an outline to the NavigationPolygon and then make_polygons_from_outlines()
Move the Polygon around until it does not render a hole correctly. This part is a bit of a mystery to me, but the minimal production project starts the Polygon in a position (-256,120) where it doesn't render correctly. Moving the Polygon to (360,-100) will cause the NavigationPolygon2D to render the hole correctly.

Minimal reproduction project
navigation_polygon_rendering.zip

Can you please try this with a supported version, like 4.1.3 (only the latest patch version is supported, and it might already be fixed)

Same as https://github.com/godotengine/godot/issues/70823
The make_polygons_from_outlines() function has a lot of known problems that break the navigation mesh creation easily, hence why that function is deprecated in Godot 4.2 that adds 2D navigation mesh baking.

Looks like this issue is irrelevant! Thanks


WORK IN PROGRESS: Include current reader position in errors

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The current state of this PR is "gather feedback". It's nowhere near ready to merge.
I was trying to debug why the ion-hash-test.ion file couldn't load. Our error infrastructure doesn't include anything to make this easy. Backtraces don't work even. So I wanted to first fix the debug-ability issue before fixing the actual issue.
The core idea is that IonCError will carry file pos information. There is a new field called pos that is optionally hydrated by the reader (writers will never set it). The type is currently the C triple verbatim (bytes, line number, offset).
Q1: do we want to carry positional information in case of error? (my opinion: yes)
Q2: is a field in IonCError the right place to carry the information? (my opinion: yes)
Q3: should this be enabled by default? (my opinion: yes, and it cannot be disabled)
Q4: should we create a type that at least names the fields? (my opinion: yes - and this is not done)
To carry the information, I added a new method to the reader trait that returns the position (using the underlying ionc call).
Q5: should this method be part of the reader trait? (my opinion: yes)
With this in place, the question then became: how to use the new reader API to update the IonCError generated by the reader. Initially I tried to change all error-creators to do so with a macro. Here's an example of how it looked:
#[inline]
fn read_string(&mut self) -> IonCResult<StrSliceRef> {
    let mut value = ION_STRING::default();
    with_pos!(self, ionc!(ion_reader_read_string(self.reader, &mut value)))?;
    // make a str slice that is tied to our lifetime
    let str_ref = value.as_str(PhantomData::<&'a u8>::default())?;
    Ok(StrSliceRef::new(self, str_ref))
}

Note with_pos!(self, ..., which added the result of self.pos() to the IonCError. The problem is that I had to add this everywhere (no big deal, but the compiler didn't help me..) and I missed one! In particular, see the value.as_str(..)? above. That's the actual bug I was hunting. So, instead, I did manual delegation:
impl<'a> IonCReader for IonCReaderHandle<'a> {
    #[inline]
    fn next(&mut self) -> IonCResult<ION_TYPE> {
        let reader = self.reader;
        self._next()
            .map_err(|err| include_current_position(&reader, err))
    }

this is a real pain to maintain as it has to be done for the entire trait surface area, so I'm looking for suggestions on how to make it less painful. However, on the plus side, it completely covers all ways errors can be created.

And, here is it working:
     Running [...]/ion-rust/target/debug/deps/ion_hash_tests-43dcc1c1cca61933

running 1 test
test ion_hash_tests ... FAILED

failures:

---- ion_hash_tests stdout ----
Error: IonCError { source: IonCError { code: 8, message: "IERR_NULL_VALUE", additional: "iERR Result", pos: Some((26874, 1, 7)) } }
thread 'ion_hash_tests' panicked at 'assertion failed: `(left == right)`

I updated my ion_hash_tests.rs to test the entire suite and it failed. Note the pos above:
// symbols
{ ion:$0,            expect:{ identity:(update::(0x0b) update::(0x71) update::(0x0e)
                                        digest::(0x0b 0x71 0x0e)) } }

(That fails when loader calls read_string() for the empty symbol.)

Q1: do we want to carry positional information in case of error? (my opinion: yes)

I agree.

Q2: is a field in IonCError the right place to carry the information? (my opinion: yes)

Yes, and we probably want a field that helps capture other context too.  E.g. ionc! should be able to stringify! the expression itself plus file! and line! to make it a bit clearer where the error originated--this should actually be the easiest thing to add.

Q3: should this be enabled by default? (my opinion: yes, and it cannot be disabled)

I agree.

Q4: should we create a type that at least names the fields? (my opinion: yes - and this is not done)

Yes, this should be structured in the error somehow--whether nested or not, I don't have a strong opinion.

Q5: should this method be part of the reader trait? (my opinion: yes)

I agree, this seems reasonable.

this is a real pain to maintain as it has to be done for the entire trait surface area, so I'm looking for suggestions on how to make it less painful. However, on the plus side, it completely covers all ways errors can be created.

I really think the only place we can do this is with some kind of delegation/decoration macro over the entire reader surface which probably needs some kind of procedural macro (I'm sure someone has done something like this before).  We will invariably miss something, and this code is not too shy around the use of ? for various places that IonCError can pop up including translation from other more standard errors.
Even if we had to manually use a attribute macro on every method, that would be acceptable:
#[position_error]
fn read_string(...) -> IonCResult<...>

I think extending ionc! with stringify! and line! and file! is probably the easiest first step.  But something like the above is probably what we want for the code you're writing manually.
Alternatively, we manually decorate every method.  The above macro is just shorthand for something vaguely like:
fn actual_read_string(...) -> IonCResult<...> {...}

fn read_string(...) -> IonCResult<...> {
  match self.actual_read_string(...) {
    Ok(v) => Ok(v),
    Err(e) => Err(include_current_position(&self.reader, e))
  }
}

#[position_error]

That's a good suggestion. I'll look into it when I get a chance.
Agree with the ionc! suggestion, but while that's easier to implement it also has less utility for me right now.

Agree with the ionc! suggestion, but while that's easier to implement it also has less utility for me right now.

Added #242 for this.


Link with wrong locale

Someone sent me this link: https://www.boolder.com/de/p/7981
(Not sure if they used the Android app or the iOS app.)
It seems that they had their phone in German and that the link was generated with a non-existing locale (de) instead of the default one (en)

@nmondollot This is coming from the android app, I'll take care of fixing it


Changing ProductInstance interface when loading and finding

*** REVIEWERS ***
@hmunfru @flopezag
*** DESCRIPTION ***
Changing retunr type in loads and finds methods

No merge till the PaaS Manager change has been introduced


Automated release notifications and tweets

- Do you want to request a feature or report a bug?
Feature
- What is the current behavior?
When release gets cut, they have the possibility to be unnoticed or not tweeted.
- What is the expected behavior?
I recently started working with Netlify Functions and wrote one that auto-tweets and blogs when a release is created. I am curious if the CMS team is interested in this. I realize there is no blog on the netlifycms.org, but this would be a great addition to the site somewhere.
I am working on a tutorial/blog post soon, but here is the function.
https://github.com/bdougie/create-react-app-lambda/blob/master/src/lambda/gitHubEventListener.js
Do you see a need for this?

Looks like there is a bit of crossover with #1045

@bdougie Agreed, I intended that issue to be for notifying users from inside the CMS, whereas this issue is for external notifications (tweets, etc). Feel free to combine them (or we can edit #1045 to clarify).

Closing in favor of #1036


Adding prims algorithm Graph code in JavaScript

💥 Proposal
I saw a lot of code missing in the JavaScript section, I happen to have just learned JS would like to try and implement it
in this particular issue I would like to work on prims algorithm

/assign


Error executing your script

I there!
I followed the steps to export my last.fm playcount, cloned this repository and, when i try to use it, i receive this error:
Traceback (most recent call last):   File "/home/custard/navidrome_lastfm_import/main.py", line 184, in <module>     main()   File "/home/custard/navidrome_lastfm_import/main.py", line 160, in main     track_id = get_track_id(db_cursor,                ^^^^^^^^^^^^^^^^^^^^^^^   File "/home/custard/navidrome_lastfm_import/main.py", line 112, in get_track_id     {"artist": artist.lower(), "name": name.lower()}),                ^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'lower'
What i'm doing wrong?
Thanks in advance

Hi, @CustardFlan !
Sorry for a late reply. It appears that a track name has not been read properly. Can I have a look at your file from the last.fm export site? If not, then I would need to know how recent was the export done. There might have been changes in the export data format that I've missed.

Nice!
The output.log is empty after running the script
and, this is a sample of the export, these are the loved tracks, i have another with the entire scrobbling history, but it's a 168 MB file, the format is the same, the tool used is https://mainstream.ghan.nl/export.html
Sample:
https://paaster.io/jPHeZCfenNUfGFyuLenfz#_ph8r-28DcC36yts5fqP2wkFmB3XGsQvuBCJlrdv_bs
These files doesn't have newlines, it's that normal?

The "loved tracks" seems to be working on my end so far.
The "scrobbling history" is not supported though - I haven't worked on it yet. Let me know if I should look into that.
And no new lines in JSON is OK - they are frequently minified this way to shrink the size.
I've pushed a small commit that would indicate if the input format is invalid.

It would be neat if you work in play count syncing! :D
I have opened a discussion here, but nobody replied so far
https://github.com/navidrome/navidrome/discussions/3454
The idea is:
Make a GET request like this:
https://ws.audioscrobbler.com/2.0/?autocorrect=1&username=CustardSandS&format=json&api_key=THEAPIKEY&method=track.getInfo&artist=Daft+Punk&track=WDPK+83.7+FM
Using the data already available on the DB, and, updating the loved status or the play count, i'm not so good in python or SQL, but, i think you get the idea, and can do this!
Why updating using request to the last.fm API and not use the export instead? Well, because the scrobbles get autocorrected anyway in that service, and, i assume if one use the navidrome software, the tags existing in the music collection are well organized, and, i have tested a small batch of albums, and the data can be correllated easilly
It's only an idea, but, take that on mind

OK. I will try to accomodate for the "play count" feature in the existing code, but it will take time.

Thank you!


What happened to 'delete' function?

The examples show how to use delete function

###Usage:
•Deletes all object that touch the 10x10 rectangle starting at position 10x10:
• var myDelCount = myRTree.delete({x:10, y:10, w:10, h:10});
•Delete only specific object if it touches the 10x10 rectangle starting at position 10x10:
• var myDelCount = myRTree.delete({x:10, y:10, w:10, h:10}, specific_object);

But no such function appears to be defined.

Seems delete was renamed to remove, see actual docs just above your quoted example.
Also, the test/deletion.test.js test passes, using the remove function.


Provide data scientists and engineers with a live stream of sampled, shadowed production data into Kafka (24 hr retention) to enable more current and real-time model training and testing

helps sync dev/local and prod environments by allowing devs/data scientists to tap directly into the production stream during development and model training

Related to https://github.com/fluxcapacitor/pipeline/issues/77


ignore the case on http headers like sap-message

according to RFC 2616 HTTP Header "Field names are case-insensitive." (4.2 Message Headers, https://www.ietf.org/rfc/rfc2616.txt).
The current implementation in the ODataMessageParser only checks for "sap-message" headers, case -sensitive.
This PR changes that.

 You should sign our Contributor License Agreement in order to get your pull request merged.Mark Schmale seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account ([for further information, click here] (https://help.github.com/articles/why-are-my-commits-linked-to-the-wrong-user/#commits-are-not-linked-to-any-user)).

@themasch I will have a look into the failing build.
npm install does not succeed which seems to be related to the node version in use (still 0.10).


Use pytest-httpbin to test

http://httpbin.org has been unstable recently. Time to consider use pytest-httpbin.

I would like to try refactoring the tests to use pytest-httpbin.


Abnormal playback of the new fvp version m3u8

[flutter_meedu_videoplayer] Exception: No Stream for textureId: -1.
[flutter_meedu_videoplayer] #0      MdkVideoPlayer.videoEventsFor (package:fvp/src/video_player_mdk.dart:207:5)
#1      VideoPlayerController.initialize (package:video_player/video_player.dart:488:10)

#2      MeeduPlayerController.setDataSource (package:flutter_meedu_videoplayer/src/controller.dart:533:7)

#3      _M3u8ExamplePageState._setDataSource (package:flutter_meedu_videoplayer_example/pages/m3u8_page_example.dart:285:7)

There is another issue, I am not sure if it is a problem with fvp. If the network is disconnected and restored, the buffered video will not continue to play!

I think it's related to fvp, yes it's from it's logic
Also on what platform?

Playing on the web is normal, but this error occurred on Android!

Just disable fvp on Android from the initmeedu
It will work using official video_player

If there is a problem with even such a basic playback format, I am very concerned about the robustness of the fvp library!

Is fvp not used by default?

Yes it's not used by default, but on the example I think I left it enabled

If there is a problem with even such a basic playback format, I am very concerned about the robustness of the fvp library!

I tested it using macos, and it was working very good,
Also it's in early stages so bugs are logical to occur, right now the only problem if windows or Linux has bugs those will be a problem

I think it's related to fvp, yes it's from it's logic
Also on what platform?
I would love if you can open an issue for fvp explaining the steps to reproduce it
Just to help the package maintainer fix and improve the package

I am running M3u8ExamplePage in the example. You can try running it on an emulator or Android phone, and an error will be reported directly!

Yes, after disabling, the playback of the m3u8 sample is normal!

opened an issue
https://github.com/wang-bin/fvp/issues/15

Thank you very much, you are still dealing with the problem so late!

Thank you very much, you are still dealing with the problem so late!

i had to suffer today to avoid making people suffer
hopefully we get the package back to working state
thank you for keeping up with me and helping when you had stuff to do <3

I really like flutter_meedu_videoplayer and I look forward to more people using it! I hope it becomes more and more stable!

I see that the author of Fvp has released a new version that solves playback issues in Windows and M3U8. Please update our version dependency!

I saw that the author of Fvp has released a new version that addresses playback issues in Windows and M3U8. Please update our version dependencies after testing!

they didnt release yet

they published, flutter upgrade should be enough

Okay, waiting for your testing and updates!

Okay, waiting for your testing and updates!

i tested it on fvp it was working correctly, so everything should be good, let me know if there is another problem

I just tested the new version, but unfortunately, I still cannot compile it!
FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':fvp:configureCMakeDebug[arm64-v8a]'.

[CXX1429] error when building with cmake using C:\Users\Pub\Cache\hosted\pub.flutter-io.cn\fvp-0.2.1\android\CMakeLists.txt: -- android-16 is not supported for arm64-v8a. Using minimum supported LP64 version 21.
-- Android: Targeting API '21' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64'
-- Android: Selected unified Clang toolchain
-- The CXX compiler identification is Clang 12.0.8

CMake Error at CMakeLists.txt:67 (include):
include could not find requested file:
  C:/Users/Pub/Cache/hosted/pub.flutter-io.cn/fvp-0.2.1/android/mdk-sdk/lib/cmake/FindMDK.cmake

I have upgraded to 7.4.2 for Gradle ！

can you do flutter clean and try again, i think the code failed to download the file, since i just tested and it works

Flutter clean I have tried! Invalid, I checked the error message and found that the file FindMDK.cmake is not available in the directory fvp-0.2.1!
Cache/hosted/pub.flutter-io.cn/fvp-0.2.1/android/mdk-sdk/lib/cmake/FindMDK.cmake
android/mdk-sdk/    The directory 'mdk sdk/' does not exist

I found the reason, https://sourceforge.net/projects/mdk-sdk/files/nightly/mdk-sdk-android.7z Not a valid download address, it's just a download page!

it does download for me
let me do a flutter clean and try again

The actual download address is https://altushost-swe.dl.sourceforge.net/project/mdk-sdk/nightly/mdk-sdk-android.7z I have submitted an issue to the author for correction!


Added Mobile Support

modified:   src/mjxgui.js
modified:   src/mjxgui.min.js
modified:   src/modules/ui.js

Hey @AvaterClasher, it looks like you've run an autoformatter on the whole ui.js file, which has changed the indentation from 4 spaces to 2 spaces and much more. Please only include the new changes that you have added.
Also, you don't need to run Grunt and generate the other MJXGUI source files, just updating ui.js is enough 👍🏻


V2 Single Token Exit Reverting

User: 0xf2267EeA216037d29e4e7206183caB1640706117
Pool: https://balancer.fi/pools/gnosis/v2/0x4d7d7b769cc617a00ed98ff3426c350a50e727120001000000000000000000c1/remove-liquidity
Non-nested
5 asset pool
To a single token USDC.e: 0x2a22f9c3b484c3629090feed35f17ff8f88f76f0
humanBptIn: 6228.440443578208593374
The same BalancerQueries query is run when we call simulate and same BAL#306 error happens:
https://dashboard.tenderly.co/agualis/v3/simulator/b6cc9e01-89d2-4092-a5f0-c9dca9f50679
Error in WeightedMath:
// Calculate the factor by which the invariant will decrease after burning BPTAmountIn
uint256 invariantRatio = bptTotalSupply.sub(bptAmountIn).divUp(bptTotalSupply);
_require(invariantRatio >= _MIN_INVARIANT_RATIO, Errors.MIN_BPT_IN_FOR_TOKEN_OUT);

Ya
Esteban Angulo
On Mon, Sep 16, 2024, 1:58 AM John Grant @.***> wrote:

User: 0xf2267EeA216037d29e4e7206183caB1640706117
Pool:
https://balancer.fi/pools/gnosis/v2/0x4d7d7b769cc617a00ed98ff3426c350a50e727120001000000000000000000c1/remove-liquidity
Non-nested
5 asset pool
To a single token USDC.e: 0x2a22f9c3b484c3629090feed35f17ff8f88f76f0
humanBptIn: 6228.440443578208593374
The same BalancerQueries query is run when we call simulate and same
BAL#306 error happens:
https://dashboard.tenderly.co/agualis/v3/simulator/b6cc9e01-89d2-4092-a5f0-c9dca9f50679
Error in WeightedMath:
// Calculate the factor by which the invariant will decrease after burning BPTAmountIn
uint256 invariantRatio = bptTotalSupply.sub(bptAmountIn).divUp(bptTotalSupply);
_require(invariantRatio >= _MIN_INVARIANT_RATIO, Errors.MIN_BPT_IN_FOR_TOKEN_OUT);
—
Reply to this email directly, view it on GitHub
https://github.com/balancer/b-sdk/issues/414, or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5F2DFCMFQI4ZJT5JXLCJALZW2MR3AVCNFSM6AAAAABOI2NQZ2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGUZDOOBXHE3TCMQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

The issue is that Weighted pools never allow for trades that are larger than 30% of the pool's balance and a similar check is applied to the invariant on non-proportional add/removes. So the single asset exit is basically seen as a swap and in this case the user is trying to exit with > 30% so it reverts. Proportional exits don't have that condition so all is fine there. Its only going to be an edge case on low liquidity pools.
FE going to change: > 25% block single asset exits and show warning


leveraging cloudant query to get docs back in the correct order based on properties.timestamp.

@rajrsingh @bradley-holt @bradnoble. Please review and test to make sure docs are coming back in proper order. Thanks!

having trouble with this. I got an error, modified app.js line 386 to get back a more verbose error message, and see this: "err: {"error":"no_usable_index","reason":"No index exists for this sort, try indexing by the sort fields."}".
@seanbarclay is there some index you created manually that needs to be added to the couchapp in some way?

@rajrsingh I believe this was the index I created:
{
"index": {
"fields": ["properties.timestamp"]
}
}
here's what the doc looks like in my location tracker DB:
https://gist.github.com/seanbarclay/9ee118740df594e7d82d
At some point, I recommend updating the tutorial with the reference to CQ and creating the index.


[release-0.26] Certificate rotation based on shared secrets

This is a manual cherry-pick of #3128.
Release note:
Multiple important changes:
* operator-managed kubevirt installations will rotate outdated or broken certificates
* the operator is now mandatory. Installation manifests without the operator will not be generated  anymore
* The operator requires now that the KubeVirt-CR is created in the same namespace like the operator resides.

/check-dco

/skip

/cc @davidvossel

/override pull-kubevirt-e2e-k8s-1.14
/override pull-kubevirt-e2e-k8s-1.15-ceph
/override pull-kubevirt-e2e-k8s-1.16
/override pull-kubevirt-e2e-k8s-cnao-1.17
/override ci/prow/images
By accident opened the PR at first against master. Overriding all test lanes from master.

/close
too late for this disruptive change on this branch.


Add proposer_id to BeaconBlock

I didn't realize #1339 had been closed! Creating this issue for some fresh eyes. This is something I'd definitely like to revisit and maybe have auditors look at this month.
See #1339 for the original description of the issue.
The tldr is that without a proposer_id and a way for a quick signature verification on an incoming block, an attacker can create blocks with some amount of skip slots that require database look ups and state transitions to even validate the proposer signature. In the current case, that attacker doesn't even need to be in the validator set.
By adding the proposer_id we at least reduce the set that can craft such blocks to the validators and limit them to a 1 per slot allowance (due to gossip conditions and slashing conditions). This rate limit combined with peer scoring (and dropping!) at least begins to bound this problem.

You have my support for this change. And prefer proposer_index over proposer_id to be more consistent with the other places where we use validator indices.

+1 on this change. This will result in better client code

Closing this issue, the proposer id (index) has been introduced in #1626, to be featured in upcoming v0.11.0 release.


Bug: wrapper.find('button').trigger('mouseenter') can't trigger new element generation detection

Describe the bug

I have a content panel that can be displayed through hover (I created it under the body for the first time), and I triggered the hover function through 'mouseenter', but I didn't detect the element I expected to be displayed, even though I waited 1000ms.
To Reproduce

  it('hover', async (done) => {
      const wrapper = mount(Popup, {
        slots: {
          default: <button id="btn">btn</button>,
          content: <div id="content">`${content}`</div>,
        },
        attachTo: document.body,
      });
      expect(document.querySelector('#content')).toBeNull();
      wrapper.find('button').trigger('mouseenter');
      wrapper.vm.$nextTick();
      // await mockDelay(1000);

       expect(document.querySelector('#content')).not.toBeNull();
    });
  });

get
AssertionError: expected null not to be null

Expected behavior

Related information:

  "@vue/test-utils": "^2.4.1",
Additional context

I can provide more detailed test case repo if you want.

Hi @RSS1102
Can you provide us a small repro online using https://stackblitz.com/github/vuejs/create-vue-templates/tree/main/typescript-vitest?file=src%2Fcomponents%2F__tests__%2FHelloWorld.spec.ts ?
It only takes a few minutes, and we'll be able to take a look

This is an example I created temporarily. Although it is not exactly the same as mine, I think it can still represent my current situation.
https://stackblitz.com/edit/github-pkkfbw?file=src%2Fcomponents%2F__tests__%2FHelloWorld.spec.ts,src%2Fcomponents%2FHelloWorld.vue,src%2FApp.vue,package.json

Thanks for the repro. If you want to request to find element you're creating manually in the DOM, you need to use attachTo: document.body.
This makes your unit test succeed:
const wrapper = mount(HelloWorld, {
  props: {
    tooltipText: 'tooltip title',
    tooltipContent: 'tooltip content',
  },
  attachTo: document.body
});
expect(wrapper.text()).toContain('tooltip title');
wrapper.find('h1').trigger('mouseenter');
wrapper.vm.$nextTick();

expect(document.querySelector('h5')).not.toBeNull();

😟😟
Thank you, it seems you are right.
but I still got an error when I did this in the actual project even if i use attachTo: document.body. Maybe I did not do it correctly. I will continue to try.
thanks again.


CSUB-918: Integration tests for burn/burn_all

Description of proposed changes

Practical tips for PR review & merge:

[ ] All GitHub Actions report PASS
[ ] Newly added code/functions have unit tests

[ ] Coverage tools report all newly added lines as covered
[ ] The positive scenario is exercised
[ ] Negative scenarios are exercised, e.g. assert on all possible errors
[ ] Assert on events triggered if applicable
[ ] Assert on changes made to storage if applicable

[ ] Modified behavior/functions - try to make sure above test items are covered
[ ] Integration tests are added if applicable/needed

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (eebe47b) 26.97% compared to head (818e5ed) 13.68%.

:exclamation: Current head 818e5ed differs from pull request most recent head 75daa3e. Consider uploading reports for the commit 75daa3e to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##              dev    #1435       +/-   ##
===========================================
- Coverage   26.97%   13.68%   -13.29%
===========================================
  Files          31       31
  Lines        1001     1001
  Branches      128      128
===========================================
- Hits          270      137      -133
- Misses        721      864      +143
+ Partials       10        0       -10

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update dependency pre-commit to ^3.7.0

This PR contains the following updates:

Package
Type
Update
Change

pre-commit
dev
minor
^3.5.0 -> ^3.7.0

Release Notes

pre-commit/pre-commit (pre-commit)
v3.7.0
Compare Source
==================
Features

Use a tty for docker and docker_image hooks when --color is specified.

#3122 PR by @glehmann.

Fixes

Fix fail_fast for individual hooks stopping when previous hooks had failed.

#3167 issue by @tp832944.
#3168 PR by @asottile.

Updating

The per-hook behaviour of fail_fast was fixed.  If you want the pre-3.7.0
behaviour, add fail_fast: true to all hooks before the last fail_fast
hook.

v3.6.2
Compare Source
==================
Fixes

Fix building golang hooks during git commit --all.

#3130 PR by @asottile.
#2722 issue by @pestanko and @matthewhughes934.

v3.6.1
Compare Source
==================
Fixes

Remove PYTHONEXECUTABLE from environment when running.

#3110 PR by @untitaker.

Handle staged-files-only with only a crlf diff.

#3126 PR by @asottile.
issue by @tyyrok.

v3.6.0
Compare Source
==================
Features

Check minimum_pre_commit_version first when parsing configs.

#3092 PR by @asottile.

Fixes

Fix deprecation warnings for importlib.resources.

#3043 PR by @asottile.

Fix deprecation warnings for rmtree.

#3079 PR by @edgarrmondragon.

Updating

Drop support for python<3.9.

#3042 PR by @asottile.
#3093 PR by @asottile.

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Update in pedantic v2


Make travis more resilient to composer self-update

composer self-update has been failing regularly on travis recently.
As composer is already installed and it's not strictly essential to have the very latest version, this change allows the build to continue, even if composer can't self-update

proof it works:

So  || true a way to ignore the return code. Fair enough, thanks Dan!

I'll go and add this to some other repos (without PRs to save time) when I can


Logs dont correctly show set @id

Describe the bug
The @id plugin parameter is used to give each plugin its unique id. This id  should be included in logs generated by said plugin.
This seems to work inconsistent between different plugin types.
I tested the json filter plugin (for which the id is not shown) and the loki output plugin (which shows it)
working example:
<source>
  @type sample
  @id id_sample_too_long
  sample {"data":"this log is way to long to fit into one chunk"}
  tag sample.long
</source>

<match **>
  @type loki
  @id id_too_long_message
  url ...
  ...
<buffer>
  @type file
  chunk_limit_size 5
  ...
</buffer>
</match>

results in logs:
2024-07-26 13:28:37 +0000 [warn]: #0 [id_too_long_message] chunk bytes limit exceeds for an emitted event stream: 64bytes
2024-07-26 13:28:37.073654307 +0000 fluent.warn: {"message":"[id_too_long_message] chunk bytes limit exceeds for an emitted event stream: 64bytes"}

Both contain the "id_too_long_message" id of the plugin the error occured in.
But for the filter plugin:
<source>
  @type sample
  @id id_sample_not_json
  sample {"data":"this is not json}"}
  tag sample.json
</source>

<filter sample.json>
  @type parser
  @id id_not_json
  key_name data
  <parse>
    @type json
  </parse>
</filter>

The error:
2024-07-26 13:21:57 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'this is not json}'" location=nil tag="sample.json" time=2024-07-26 13:21:57.079027108 +0000 record={"data"=>"this is not json}"}
2024-07-26 13:21:57.079191057 +0000 fluent.warn: {"error":"#<Fluent::Plugin::Parser::ParserError: pattern not matched with data 'this is not json}'>","location":null,"tag":"sample.json","time":1722000117,"record":{"data":"this is not json}"},"message":"dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=\"pattern not matched with data 'this is not json}'\" location=nil tag=\"sample.json\" time=2024-07-26 13:21:57.079027108 +0000 record={\"data\"=>\"this is not json}\"}"}

does not contain the id "id_not_json" of the filter plugin that caused the error.
To Reproduce
The config:
<source>
  @type sample
  @id id_sample_not_json
  sample {"data":"this is not json}"}
  tag sample.json
</source>

<filter sample.json>
  @type parser
  @id id_not_json
  key_name data
  <parse>
    @type json
  </parse>
</filter>

results in:
2024-07-26 13:21:57 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'this is not json}'" location=nil tag="sample.json" time=2024-07-26 13:21:57.079027108 +0000 record={"data"=>"this is not json}"}
2024-07-26 13:21:57.079191057 +0000 fluent.warn: {"error":"#<Fluent::Plugin::Parser::ParserError: pattern not matched with data 'this is not json}'>","location":null,"tag":"sample.json","time":1722000117,"record":{"data":"this is not json}"},"message":"dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=\"pattern not matched with data 'this is not json}'\" location=nil tag=\"sample.json\" time=2024-07-26 13:21:57.079027108 +0000 record={\"data\"=>\"this is not json}\"}"}

Expected behavior
The @id fiel, if set, shows up in logs prodcued by the plugin. In this example I would expect:
2024-07-26 13:21:57 +0000 [warn]: #0 [id_not_json] dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'this is not json}'" location=nil tag="sample.json" time=2024-07-26 13:21:57.079027108 +0000 record={"data"=>"this is not json}"}
2024-07-26 13:21:57.079191057 +0000 fluent.warn: {"error":"# [id_not_json] <Fluent::Plugin::Parser::ParserError: pattern not matched with data 'this is not json}'>","location":null,"tag":"sample.json","time":1722000117,"record":{"data":"this is not json}"},"message":"dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=\"pattern not matched with data 'this is not json}'\" location=nil tag=\"sample.json\" time=2024-07-26 13:21:57.079027108 +0000 record={\"data\"=>\"this is not json}\"}"}

Your Environment
- Fluentd version: 1.17.0
- Package version:
- Operating system: Garden Linux 1443.9
- Kernel version: 6.6.35-cloud-arm64

Your Configuration
<source>
  @type sample
  @id id_sample_not_json
  sample {"data":"this is not json}"}
  tag sample.json
</source>

<filter sample.json>
  @type parser
  @id id_not_json
  key_name data
  <parse>
    @type json
  </parse>
</filter>

### Your Error Log

```shell
2024-07-26 13:21:57 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'this is not json}'" location=nil tag="sample.json" time=2024-07-26 13:21:57.079027108 +0000 record={"data"=>"this is not json}"}
2024-07-26 13:21:57.079191057 +0000 fluent.warn: {"error":"#<Fluent::Plugin::Parser::ParserError: pattern not matched with data 'this is not json}'>","location":null,"tag":"sample.json","time":1722000117,"record":{"data":"this is not json}"},"message":"dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=\"pattern not matched with data 'this is not json}'\" location=nil tag=\"sample.json\" time=2024-07-26 13:21:57.079027108 +0000 record={\"data\"=>\"this is not json}\"}"}

### Additional context

_No response_

@DavidKnacker is the id missing even for logs generated by input plugin?

@Athishpranav2003 I haven't checked it for error logs thrown by an input plugin yet.
But am I correct to assume that using @id to track errors should work for all kind of plugins? (or at least the ones that are shipped by default with fluentd?)

Ideally yes but since it's not behaving in that way I was checking what's the difference in underlying code
Will check it meanwhile
Just ping me later with the case of input plugin

another scope of Idea: track plugin which emit error event in router side, then emit more user friendly error message.

Thanks for your report.
This is Fluentd's specification.
This dump an error event log is generated by RootAgent (Fluentd's common logic).
The filter_parser plugin just emits the error event to the router.
So, this log does not have its id.
https://github.com/fluent/fluentd/blob/e763c0761c44d9734b6aa374371387a2e8406522/lib/fluent/root_agent.rb#L344-L354
Although this is the current specification, it looks like not so good.
Maybe it could be improved somehow.

Although this is the current specification, it looks like not so good.
Maybe it could be improved somehow.

It would be better to include the id of the plugin that emits the error event.
It may be easy to manually include the id of filter_parser in the error info that it emits.
https://github.com/fluent/fluentd/blob/e763c0761c44d9734b6aa374371387a2e8406522/lib/fluent/plugin/filter_parser.rb#L101-L103
However, it is not so good because every plugin that emits error events needs to consider this issue.
As @kenhys says in https://github.com/fluent/fluentd/issues/4567#issuecomment-2259468182, it would be great if we could create a common mechanism for this.

another scope of Idea: track plugin which emit error event in router side, then emit more user friendly error message.

@daipom so basically we need to avoid change in function signature but capture the caller plugin id?

so basically we need to avoid change in function signature but capture the caller plugin id?

Yes.
We can't change the signature of emit_error_event() because it is related to Event Emitter plugin helper API.
We need to keep compatibility for the plugins that use this function.

How don't we use log.error method to generate the error log than emitting it as event(It's indirect but works the same way right)?

@Athishpranav2003 to come back to your question, I do see the @id field for logs generated by the input plugin.

Why don't we use log.error method to generate the error log than emitting it as event(It's indirect but works the same way right)?

@daipom what about this?

Are there any updates on this?

Sorry for my late response.

Why don't we use log.error method to generate the error log than emitting it as event(It's indirect but works the same way right)?

Filter plugin should use emit_error_event().
It outputs events as @ERROR label.
It is intended to rescue collected data that could not be processed.
On the other hand, the events from Fluentd logger mean logs of Fluentd, not collected data by input plugins.
Fluentd logger outputs events as @FLUENT_LOG label.
Using both would be a possible solution.
However, it would be better if it could be resolved through a common mechanism.

I made a PR for this issue.

#4632

However, I'm concerned about some race conditions of the related existing logic...

I cannot say anything about potential race conditions, but I just tested the PR in my cluster and it seems to exactly achieve what I hoped for.
Thanks @daipom! :)
Would be great to have this in.

@DavidKnacker Thanks! I will check the race condition!

Hey @daipom, are there any updates on the ticket?

Sorry, we need to consider the race condition issue more.
I wish we could get this into 1.18.0, but I am not sure...

I confirmed the race condition issue.

https://github.com/fluent/fluentd/pull/4632#issuecomment-2503344686

We need to fix this race condition first.


feat: try to proxy body even after body-parser middleware

If a body parser middleware is added before the proxy is configured, proxying POST requests with bodies fails. This fixes that and provides a test, as requested by @chimurai in #460.
Code from #460 by @sarumont and from #320 by @repl-chris used -- thanks to both!
Fixes #90
Fixes #320
Fixes #417

Coverage decreased (-1.2%) to 87.077% when pulling 439832fa89fae4037e71db18ecc62927358f8145 on midgleyc:body-parser-test into f77121f0c10715db127e7436ea88b0e2435f9fcb on chimurai:master.

Hi, any updates on this PR? Thanks for doing this.

Thanks @midgleyc for this PR. (Sorry for late reply)
Could you separate this.fixBody function to external handler? Similar to responseInterceptor's implementation. https://github.com/chimurai/http-proxy-middleware/tree/master/src/handlers
This'll make fixBody optional, since not everyone is having this issue. And it'll prevent bloat in the core functionality.
It's usage will be like this:
const { createProxyMiddleware, fixBody } = require('http-proxy-middleware');

const proxy = createProxyMiddleware({
  /**
   * Fix bodyParser
   **/
  onProxyReq: fixBody,
});

Planning to work on a preset-like functionality where something like recommended contains a collection of handlers for the majority of use-cases.

@chimurai Done, how does it look now?

Changes made.
You should be able to make changes to my branch directly, if you'd like.

Thanks @midgleyc !

You're welcome, thanks for merging :)

This workaround doesn't seem to work anymore.
In later versions, the proxyReq etc. are set inside the on property:

on: {
                        /**
                        * Fix bodyParser
                        **/
                        proxyReq: fixRequestBody
                    }

In such versions, I get an error in the fixRequestBody method (where it sets the content type header), and is says (something like) 'cannot change headers after they are sent to the client'.

@dhurtrci Can you please open a new issue and provide a reproduction so it can be investigated?


add randomBool to generate random Prim.Boolean value

Suggested implementation:
randomBool = (< 0.5) <$> random

:+1: I'll put this in when I get to releasing an 0.7 RC of it.


Fix building on RasPi

Ubuntu on Raspberry Pi reports "arm-linux-gnueabihf". Since it still needs to link curl, I added the check for this triple ending.

Hi, it could make sense to simply check for if triple.endswith("linux") != -1: to catch every Linux variant, since looking at the history it looks like that that line was added to support all Linux variants. The only thing I can't be 100% sure about is if that works for the IBM LinuxOne too, but I guess it should.

I agree with @uraimo . We should probably check if we can reduce the checks to a single check for "linux". If that isn't feasible, the current solution looks good to me. Thanks.

Okay. I've changed the check to if triple.find("linux") != -1:. I've tested it with some common triples and it worked.

This looks good for now. Thanks.
@swift-ci test and merge

@swift-ci test and merge


feat: add configuration to open exported file by system default app

Hi! I'm not seeing this as an enumeration but as an editable field for which it's almost impossible to figure out the allowed values without looking at the source.