Feature: more logging variables available for the access log

At the moment, the log format of the access log in APICast is the following:
https://github.com/3scale/APIcast/blob/bfa3ac9be38c99da5c01cfba4becd0688513b0fe/gateway/http.d/apicast.conf.liquid#L1
There is a variable there called $post_action_impact, which is a variable created by the APICast Lua extensions. It would be great if more variables are set, so that I can set up a log format most suitable for me. Right now, the use case would be to see which service was called, and which method of that service. With that data, working with the logs in ELK or similar would become much easier (filtering on service  and method names instead of paths matching a particular regex etc.)
The change could be completely backwards compatible by just making the variables available but leaving the default logging format as it is.

Hi @magnusvage
Some work has been done over the last weeks to address this, I'll keep you posted in the following days.
Regards.

@eloycoto great news! Look forward to seeing the results :)

Hi @magnusvage
I sent the PR, need to be reviewed, but mostly address your behaviour, the example policy configuration will be something like this:
{
  "name": "apicast.policy.logging",
  "configuration": {
    "enable_access_logs": false,
    "custom_logging": "[{{time_local}}] {{host}}:{{server_port}} {{remote_addr}}:{{remote_port}} \"{{request}}\" {{status}} {{body_bytes_sent}} ({{request_time}}) {{post_action_impact}}",
  }
}

This is the PR:
https://github.com/3scale/APIcast/pull/1089
I will love your input here, and be able to provide to you the right solution that fixes your use case. :-)

Hi @eloycoto
Looks good!
So, this policy would completely override the access log format set at the NGINX level? And without this policy on a service, the access logs would be formatted according to the configured NGINX format?
My biggest concern here is that we would in our case consume the access logs with Logstash, and we would need a predictable format so that our parsing pipeline can parse every line of the access log, regardless of whether the new policy is there or not. Therefore I was hoping to be able to control the logging format at the gateway level. If (for example) the 3scale base policy set a bunch of variables (basically everything you could think of), then we could pick and choose from them in the nginx.conf and every log entry would look exactly the same.
Perhaps this is achievable with the policy approach as well? Can I bulk add/change policies on all services within a given 3scale domain?

Hi @magnusvage
I'll think about your use case
Thanks

@eloycoto thanks
The only important thing is that every access log row is parseable with the same logic, regardless of the configuration applied to any services. That could very well be achieved in a policy approach, but then I think that the custom logging info configured in the policy would have to be placed within some sort of delimiters in the access log row. Example:
log_format time '[$time_local] $host:$server_port $remote_addr:$remote_port "$request" $status $body_bytes_sent ($request_time) $post_action_impact <$custom_logging_policy_data>';
Then we could parse everything within the <> as a single entity.
But, then we wouldn't be able to parse the inside of the <>, since that format would be a service specific string. If we wanted to visualise the logs of a single service (or even a single method within a service), then we would have to know where to get the method name from the log row, every time.

Hi @magnusvage
We have some discussion about your proposal in the PR#1089. New changes will be addressed this week.
Thanks.

@eloycoto I'm still investigating the upstream_response_time.
One other thing: {{host}} used to become the APICast hostname with the old log format, but now with the custom logging we see the backend hostname here instead. Should we use some other variable names to get the APICast value again?

Hi @magnusvage
Good catch, I made some changes in the code and a PR is already open. I'll try to get this in the new stable release.
https://github.com/3scale/APIcast/pull/1116
Regards

@eloycoto do you have any suggestions on how we can get information into the access log identifying the consumer? Application identifier, identifier of the account owning the application or something along those lines?

Hi @magnusvage what information do you want? The only information that I can think that can be provided is the number of metrics that were increment. Oher information belongs to different components.
Regards.

@eloycoto we want the log entry to reflect which application made the call to the API. Application name (and preferably also application plan system name) would be perfect. I know that information is not stored in the spec.json configuration used by the gateway, so the only way I can think of is that, when APICast makes the auth/authrep call to the 3scale backend to authenticate the call, the response from the backend contains these pieces of information, which we then could put into the log entry using the custom logging policy.

Hi @eloycoto , is it possible to log the mapping rule that matched? We want to group log entries based on endpoints, and if the path contains path variables an access log entry containing /get/variable1 and /get/variable2 will be hard to match. But, if we could log "mappingrule: /get/{pathVariable}" or something similar, it would be a breeze to do analytics based on endpoints.

Hi @eloycoto we are still seeing the host name bug (backend host name instead of gateway host name) in the logs. Using image quay.io/3scale/apicast:v3.7.0. Did the fix make it into the release?

could you share your config?
Regards

local logging_policy_config = cjson.decode([[
    {
      "enable_access_logs": false,
      "custom_logging": "[{{time_iso8601}}] [{{req.headers.X-Forwarded-For}}] {{host}}:{{server_port}} {{remote_addr}}:{{remote_port}} \"{{request}}\" {{status}} {{body_bytes_sent}} ({{request_time}}) ({{upstream_response_time}}) {{post_action_impact}} {{service.id}} {{service.system_name}}"
    }
]])

produces the log entry
[2019-12-18T08:54:25+00:00] [<REDACTED>] echo-api.3scale.net:8080 <REDACTED>:10380 "GET /echo/3scale HTTP/1.1" 200 832 (0.292) (0.292) 0 <REDACTED> <REDACTED>
echo-api.3scale.net is the backend host.

@magnusvage instead of {{host}}:{{server_port}}, please use: {{original_request.host}}:{{server_port}}
Regards

@eloycoto thanks, that solved it! Is this documented somewhere? Can't find the information on https://github.com/3scale/APIcast/tree/master/gateway/src/apicast/policy/logging


Broken link/document: Support for Logical nullish assignment

The link/document Support for Logical nullish assignment shows no data. It is also listed in the ES2021 document  and the ES2021 feature support list.
Related to #6600 and #6601.

Thanks! Link has now been updated/corrected to https://caniuse.com/mdn-javascript_operators_nullish_coalescing_assignment


ElasticSearch Store

Create an ElasticSearch backed Gaffer store

Closing as the issue is now being tracked in the gaffer-experimental repository: https://github.com/gchq/gaffer-experimental/issues/7


Samplesheet for nanopore reads not working

Hi! I am attempting to use a samplesheet for some nanopore reads but Cecret seems to interpreting the reads as paired-end. I am able to run the pipeline just fine when the fastqs are in a directory. My sample sheet is formatted as follows:
sample,fastq_1,fastq_2
2024028971,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028971.fastq.gz,nanopore
2024028978,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028978.fastq.gz,nanopore
2024028980,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028980.fastq.gz,nanopore
The pipeline reports that paired-end reads were found:
Paired-end Fastq files found :<PHONE_NUMBER>
Paired-end Fastq files found :<PHONE_NUMBER>
Paired-end Fastq files found :<PHONE_NUMBER>
And then fails at the fastqc step.
Do you have any advice? Thanks so much in advance!!

!!!
It looks like the documentation doesn't match what the sample sheet is actually supposed to contain. (meaning, this is my issue, not yours)
What happens if you use ont for column 3 (the fastq_2 column) instead of nanopore?
sample,fastq_1,fastq_2
2024028971,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028971.fastq.gz,ont
2024028978,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028978.fastq.gz,ont
2024028980,/Projects/Research/Cecret/Cecret_v3.12.20240227_nano/nanopore/2024028980.fastq.gz,ont

Thanks! Strangely when I try that, it seems to be interpreting 'ont' as the name of one of the reads as artic begins to proceed (according to the command error):
usage: artic [-h] [-v]
{extract,basecaller,demultiplex,minion,gather,guppyplex,filter,rampart,export,run}
...
artic: error: unrecognized arguments: ont]
Also, I am getting this error (repeated for each of the samples):
[4f/64a662] NOTE: Missing output file(s) artic/2024028971_filtered.fastq.gz expected by process CECRET:cecret:artic_read_filtering<PHONE_NUMBER>) -- Execution is retried (1)
Thanks again for your help!!

I'm going to have to look into this. My kiddo is home because of a school break, but I'm confident I can fix this by tomorrow.

Thanks a bunch!!! I really appreciate it.
On Mon, Oct 21, 2024 at 12:19 PM Young @.***> wrote:

I'm going to have to look into this. My kiddo is home because of a school
break, but I'm confident I can fix this by tomorrow.
—
Reply to this email directly, view it on GitHub
https://github.com/UPHL-BioNGS/Cecret/issues/375#issuecomment-2427411120,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJBA2KGD7L3GKPKYVL43VGTZ4VARPAVCNFSM6AAAAABQGLPF2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXGQYTCMJSGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

I've created a PR that should fix the issue: https://github.com/UPHL-BioNGS/Cecret/pull/378
Basically, the nanopore file was read in with an extra string which was causing the error.
This PR is currently undergoing testing, but once testing is finished, version 3.15.24296 will be released and shouldn't have that extra string that is throwing the error.

It should be fixed now. Let me know if you run into issues!

Thanks!!!! Appreciate your assistance!!
On Tue, Oct 22, 2024 at 4:48 PM Young @.***> wrote:

I've created a PR that should fix the issue: #378
https://github.com/UPHL-BioNGS/Cecret/pull/378
Basically, the nanopore file was read in with an extra string which was
causing the error.
This PR is currently undergoing testing, but once testing is finished,
version 3.15.24296 will be released and shouldn't have that extra string
that is throwing the error.
—
Reply to this email directly, view it on GitHub
https://github.com/UPHL-BioNGS/Cecret/issues/375#issuecomment-2430456253,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJBA2KCDFUTLM36JVHDJQLDZ43I4BAVCNFSM6AAAAABQGLPF2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZQGQ2TMMRVGM
.
You are receiving this because you authored the thread.Message ID:
@.***>


Render connection matrix with jinja2

KNTK-294

display measurement data with coloring
detailed data in hover text
navigate to timeseries view when clicked on specific cell
support 60 x 60 matrix

This PR was replaced by #63.


Home brew version of netcat does not support -U

I had netcat installed through home brew and it does not support the -U parameter.
I fixed it by uninstalling netcat in home brew brew uninstall netcat
I do not know what you can do about it but if others runs into the issue it is at least documented here.

Thanks a lot, I will put this in the README file.


Implement Home page

Hey @Satyam1923,
I would like to work on the home page implementation based on the new design.

Can you assign this to me?

you can work on it just keep updates of new commits.


kubectl unknown flag (--short) for github runners

Hi Team,
I experience this when attempting to use cce-credential-action. This is not passed in my workflow but as a test from the action. The flag has been deprecated, can this be removed

please use the<EMAIL_ADDRESS>just released


metadata generation failed

I was trying to create extension with this template. I'm using Ubuntu 20.04 and Python 3.11.
I've used copier to copy the template, I selected frontend extension. Then I run pip install -e "." and got below error message:
pip install -e "."
Obtaining file:///home/piotr/sandbox/myextension
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [46 lines of output]
      INFO:hatch_jupyter_builder.utils:Running jupyter-builder
      INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
      INFO:hatch_jupyter_builder.utils:With kwargs: {'build_cmd': 'install:extension', 'npm': ['jlpm'], 'source_dir': 'src', 'build_dir': 'myextension/labextension'}
      INFO:hatch_jupyter_builder.utils:Installing build dependencies with npm.  This may take a while...
      INFO:hatch_jupyter_builder.utils:> /tmp/pip-build-env-ewv84fco/overlay/bin/jlpm install
      Usage Error: The nearest package directory (/home/piotr/sandbox/myextension) doesn't seem to be part of the project declared in /home/piotr.

      - If /home/piotr isn't intended to be a project, remove any yarn.lock and/or package.json file there.
      - If /home/piotr is intended to be a project, it might be that you forgot to list sandbox/myextension in its workspace configuration.
      - Finally, if /home/piotr is fine and you intend sandbox/myextension to be treated as a completely separate project (not even a workspace), create an empty yarn.lock file in it.

      $ yarn install [--json] [--immutable] [--immutable-cache] [--check-cache] [--inline-builds] [--mode #0]
      Traceback (most recent call last):
        File "/home/piotr/sandbox/myextension/extenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
          hook = backend.prepare_metadata_for_build_editable
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/piotr/sandbox/myextension/extenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/piotr/sandbox/myextension/extenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/piotr/sandbox/myextension/extenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 176, in prepare_metadata_for_build_editable
          whl_basename = build_hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ewv84fco/overlay/lib/python3.11/site-packages/hatchling/build.py", line 78, in build_editable
          return os.path.basename(next(builder.build(wheel_directory, ['editable'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ewv84fco/overlay/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 150, in build
          build_hook.initialize(version, build_data)
        File "/tmp/pip-build-env-ewv84fco/normal/lib/python3.11/site-packages/hatch_jupyter_builder/plugin.py", line 87, in initialize
          raise e
        File "/tmp/pip-build-env-ewv84fco/normal/lib/python3.11/site-packages/hatch_jupyter_builder/plugin.py", line 82, in initialize
          build_func(self.target_name, version, **build_kwargs)
        File "/tmp/pip-build-env-ewv84fco/normal/lib/python3.11/site-packages/hatch_jupyter_builder/utils.py", line 113, in npm_builder
          run([*npm_cmd, "install"], cwd=str(abs_path))
        File "/tmp/pip-build-env-ewv84fco/normal/lib/python3.11/site-packages/hatch_jupyter_builder/utils.py", line 229, in run
          return subprocess.check_call(cmd, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/piotr/.pyenv/versions/3.11.5/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/tmp/pip-build-env-ewv84fco/overlay/bin/jlpm', 'install']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

To solve the issue I needed to remove the yarn.lock from my home directory.


latex needs attention

I love this theme but the highlighting when editing latex documents is pretty rough for math

@musm can you post an example/screenshot? Thanks!

As you can see as I switch to min and to other themes a lot of the symbols are not highlighted in this latex document. thanks.


ASTC Texture error

PVRTexLib uses astcenc.exe to decode ASTC format Texture, need to put it in the libraries directory.

Ok thanks. I will add it in the next update

Done in aa86762cc07b3f306f76ad8cfffd6e7bbf8b87b7


Update CI toolchain to 1.63.0

Currently, the cargo clippy or the CI Check Rust code with rustfmt and clippy job failed.
It warns clippy::derive_partial_eq_without_eq as unknown lints but it exists in 1.63.0 release. You can see it through https://rust-lang.github.io/rust-clippy/master/.
At the #4046, it updated the toolchain's version to 1.62.1 from stable and the stable version was 1.63.0.
But I see the pull request description while writing this pull request's description:

I need time to check if the failure is our bug or rust regression.

I don't know its context so it may be intended and if then, this pull request should be closed.

hmm, why is this not fixing the error?

hmm, why is this not fixing the error?

Hmm.. I thought it is because the GitHub Actions follows the main branch but it seems not the situtation I thought...
I'll look more. 🙏🏻

When I saw the debug log, the clippy command's version was not 1.63.0 🤔 I think it's a bug of dtolnay/rust-toolchain or rustup 🤔 .

I'll push with the commit to use stable version.


[WIP] Dynamic theme migration

Details
This is a WIP for the dynamic theme migration – done via a codemod script. This is the diff between another branch and Rory-MigrateStylesToThemeHooks, where the codemod is defined, so it's just the result of the codemod.
Fixed Issues
$ https://github.com/Expensify/Expensify/issues/318357
Tests
We should run a full regression suite on this branch.

[x] Verify that no errors appear in the JS console

Offline tests
Full regression suite.
QA Steps
This can affect prettymuch the whole app, so just regression tests should be sufficient here.

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

I think 95% of the time, this will be right, but it currently has a few drawbacks:

When a static style is used outside of a component, we need to do something different to address that. Going to look at the result of eslint to find the places where that happens, and maybe address some common patterns via the codemod or otherwise just address them manually

this.props.theme and this.props.themeStyle can be a bit verbose in class components. It's a bit odd looking sometimes but I'd say in general it's not worth hoisting them with a statement like this:
const theme = this.props.theme;
const themeStyles = this.props.themeStyles;

because those are just redundant variables, and we're getting rid of class components soon anyways

:test_tube::test_tube: Use the links below to test this adhoc build on Android, iOS, Desktop, and Web. Happy testing! :test_tube::test_tube:

Android :robot:
iOS :apple:

https://ad-hoc-expensify-cash.s3.amazonaws.com/android/30544/index.html
https://ad-hoc-expensify-cash.s3.amazonaws.com/ios/30544/index.html

Desktop :computer:
Web :spider_web:

https://ad-hoc-expensify-cash.s3.amazonaws.com/desktop/30544/NewExpensify.dmg
https://30544.pr-testing.expensify.com

@roryabraham can you link the commits that you did manually versus via the script?

@roryabraham can you link the commits that you did manually versus via the script?

They're kind of mixed up, but some that I did with the script are:

https://github.com/Expensify/App/pull/30544/commits/980654db81ceb3ad2333b8dc9d4196dd729c6ac7
https://github.com/Expensify/App/pull/30544/commits/5179b1fea44318f1347073936dd98193b4ce6a8a
https://github.com/Expensify/App/pull/30544/commits/3177c88d6aeb8a67ab30b1313e8b9b63c4c58482

Most others are either commits from me building the script or from manual migrations. Most of the commits where I manually migrated something have messages like:

Manually migrate X
Fix X
Migrate X

There are a few other commits that touch lots of files where I just ran eslint and/or prettier.

Login - Error page appears after trying to log in with magic link in another tab
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Go to https://30544.pr-testing.expensify.com/
Enter an email you have access to.
Click on "Continue".
Go to email inbox and copy the magic link.
Change it to "30544.pr-testing.expensify.com" and go to it in another tab.
Wait for the Abracadabra page to appear.
Return to original tab.

Expected Result:
User should be logged in with no issues.
Actual Result:
An error page shows up on original tab.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/115492554/749d27b8-cb51-4316-a8f4-fb397b1a2849

View all open jobs on GitHub

Chat - Not found page opens when refreshing chat page
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Go to https://30544.pr-testing.expensify.com/
Open any chat
Refresh the page

Expected Result:
The same chat page opens
Actual Result:
User is redirected to https://30544.pr-testing.expensify.com/not-found
page
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] Windows: Chrome
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/78819774/c2ed89e3-868f-4ce8-9423-4899b31d8299

View all open jobs on GitHub

Bank account & Wallet - App crashes when adding bank account and enabling wallet
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:
Crash 1:

Go to https://30544.pr-testing.expensify.com/
Go to Settings > Workspaces > any workspaces > Bank account.
Click Connect manually.
Enter the bank account numbers and click Next.

Crash 2:
Precondition: User has added a bank account to the Wallet.

Click Enable wallet.

Expected Result:
App does not crash
Actual Result:
App crashes
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] Windows: Chrome
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/78819774/5020c412-2e81-4933-9ca7-62f95c19258a

View all open jobs on GitHub

Workspace - Crossed-out workspace is misaligned in the workspace list when deleted offline
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Go to https://30544.pr-testing.expensify.com/
Go offline.
Go to Settings > Workspaces > any workspace.
Delete the workspace.

Expected Result:
The crossed-out workspace in the list will not be misaligned
Actual Result:
The crossed-out workspace in the list is misaligned
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] Windows: Chrome
[ ] MacOS: Desktop

Screenshots/Videos

https://github.com/Expensify/App/assets/78819774/cbf49a04-3acb-4c82-a522-5ceac067afca

View all open jobs on GitHub

Hello @roryabraham @grgia! The QA team is facing difficulties while trying to install the Desktop app using the provided link: https://github.com/Expensify/App/pull/30544#issuecomment-1788173323.

Login - Language section on login page is not aligned and dropdown arrow overlaps Spanish
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Navigate to link https://30544.pr-testing.expensify.com/
Scroll down to language section

Expected Result:
Language section should be aligned and there should not be overlaping
Actual Result:
Language section is not aligned and dropdown arrow overlaps Spanish
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] Windows: Chrome
[ ] MacOS: Desktop

Screenshots/Videos

View all open jobs on GitHub

Expense report - App crashes when clicking on expense report title in the header
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Go to https://30544.pr-testing.expensify.com/
Go to workspace chat with expense
Click on the expense preview
Click on the expense report title in the header

Expected Result:
App does not crash
Actual Result:
App crashes
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] Windows: Chrome
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/78819774/c0ff078f-7fbe-4aba-9303-c4854a2b003e
logs.txt

View all open jobs on GitHub

Unable Sign In with Apple account. Error displayed: "Cannot get account details
Reproducible in staging?: n
Reproducible in production?: n
Action Performed:

Go to the sign-in page
Click on the Apple button
Enter or select a Apple Account

Expected Result:
Verify the user is logged correctly into NewDot
Actual Result:
Error displayed: "Cannot get account details, please try again or contact<EMAIL_ADDRESS>Unable to sign in with Apple account
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] Windows: Chrome
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/78819774/fdffe95b-685a-4b7e-977c-bd67281664dd

View all open jobs on GitHub


[#32562] Incorporate Prism into the Beam Website.

A set of initial changes to present a public face to Prism for users and developers.

Runner documentation and Roadmap pages.
Refer to said pages in Test Your Pipeline, and the 2.59.0 Beam blog.
Add links to Indexes and side bars.

In addition:

Related: Adding support for prism to the GitHub auto labeler and github README.
Minor fix to direct runner page to link to the Test Your Pipeline guidance page.

Adding Prism to the Runner Capabikity matrix, quickstarts, examples etc, is out of scope for this change. This is to provide a basic starting point on the site for further changes.
Staged: https://apache-beam-website-pull-requests.storage.googleapis.com/32563/documentation/runners/prism/index.html
Fixes #32562.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

assign set of reviewers


SLU14 - Exercises notebook

There is a cell that is set as "manual grading" in the nbgrader configs, which makes the evaluation of this notebook manual and this is why you don't see the notebook being graded.
Should be fixed very soon.

The problem was solved!
Students will need to pull the repository and use Sam's method (aka good old Ctrl+C Ctrl+V) to update the Exercises Notebook in this unit to the newest version.
When submitting the new Exercises Notebook, the grading should happen automatically.


Support Argument-Labels

Swift's functions have Argument Labels and Parameter Names. As far as I understand, swift-bridge doesn't seem to support Argument Labels.
Something like:
//Rust
#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        #[swift_bridge(argument_labels=(argument_label))]
        fn get(parameter_name: i32);
    }
}

//Generated Swift

func get(argument_label parameter_name: Int){

}

Resource
Functions

Nice, we should totally support this.
As far as the API, what do you think about something like this:
#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        fn greet(
            #[swift_bridge(label = "firstName")] name: &str,
            age: u32
        )
    }
}

func greet(firstName name: RustStr, _ age: u32) {
    // ...
}

The argument attribute approach feels a bit cleaner than the function attribute approach.
Now that I think of it, this is probably a better approach for things like our get, get_with and args_into attribute as well... But we can handle those separately.

The argument attribute approach makes it a bit easier tell which label applies to which argument.

Sounds good! I didn't know the argument attribute. Can I implement this?

Go for it!

Closed via #156


Update README with public API usage

Create README with kotlin idiomatic public API.
It will look similar to https://github.com/ably/ably-chat-js/?tab=readme-ov-file

┆Issue is synchronized with this Jira Task by Unito

Fixed via https://github.com/ably/ably-chat-kotlin/pull/65


Fix content cover remaining visible.

Reproduction steps:

Launch the WPF test application
Select the EXPLORE button on the home page
Make a section from items on the left

Observed: The content cover remains visible after the left drawer is dismissed.
The reason is because the command was not passing any Dock value as its command parameter. This was causing all of the Is*DrawerOpenProperty to be set to true. thus keeping the content cover visible after only the left drawer is dismissed.
The change here fixes the issue by specifying that only the left drawer should be opened.

Ooops!  Thank-you!


number of MNIST example test results varies

Good afternoon. May I ask you a question?
After I ran the MNIST example, I found that the number of test data are different depending on the running environment. For example, when I ran CaffeOnSpark on the local mode of Spark, there were exactly 10000 test results when I typed cat ${CAFFE_ON_SPARK}/lenet_features_result/*. But in the cluster mode, on HDFS there were 9900 test results when I typed hadoop fs -cat some/path/lenet_features_result/*. What would be the reason?
Thank you for your help!

This is due to the partition sizes indivisible by batch size, some of the samples get discarded as the remainder. In lenet_memory_train_test.prototxt, set the test batch size to 1 (the batch size in the test phase), then you should get the correct number of samples.

@jsongcse Please close this bug if it works for you

Thanks! I will close this issue.


snapshot+layers/GC: Slow snapshot v4 import with "--in-memory" flag

Using Tezos branch nomadic-labs/vicall@tomjridge@layered_store with 2022-04-22_layers_rebased_on_3.2.0 we can attempt to import a v4 snapshot into an empty data-dir:
$ ./tezos-node snapshot import ../mainnet-2335199.rolling.with_version_4_maybe --in-memory --data-dir ./dd3

This runs very slowly (too slow to be usable).
Some figures: on the tezos main branch, this operation takes about 5 to 10s per percent imported; overall time < 10min; as above, each percent takes multiple minutes.
This seems different to the other snapshot issues: whilst snapshot import usually needs an index (by default, snapshot import uses the tezos context index, which then causes problems with layers), by using "--in-memory" we should preserve that the tezos context index is minimal, and we should avoid the problem that layers wants all indexes to be minimal. But still, there seems to be some bad interaction with the layers code.

Fixed by https://github.com/tomjridge/irmin/commit/f92b3830567af074f59343fce7f600d02f3bee82
Keeping issue open as a reminder to clarify the semantics of flush.

@tomjridge This is fixed right?

Yes. As I understand it, the semantics of flush is "just flush, don't fsync", so the removal of the fsync is the right fix.


Create a kubevirt compatible image from an iso image

I am trying to create a Kubevirt compatible image using an iso file . I am using kubevirt/cirros-registry-disk-demo as the base image and adding my iso on top of it . Below are the contents of my Docker file :
FROM kubevirt/cirros-registry-disk-demo ADD VYOS.img /disk
I am able to create an image using this docker file but when I create a VM using this image , effects of the iso image are not seen in the created VM .

@mbhatnagar11 Can you please share virt-launcher and virt-handler logs?


No Data when building apk

Hi, when i use the plugin on "flutter run", it works properly and i get the mac address from the device as i want. When i build the apk and start the app, the mac address of the same device as before, is empty.
Is there any problem with the plugin or i' m doing something wrong?
The output of "flutter doctor" is:

[√] Flutter (Channel stable, v1.12.13+hotfix.8, on Microsoft Windows [Version 10.0.18362.657], locale el-GR)
[√] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
[√] Android Studio (version 3.5)
[√] VS Code, 64-bit edition (version 1.42.1)
[√] Connected device (1 available)

Add the method to the WifiInfoPlugin class
  static Future<String> get macAdress async {
    return await _channel.invokeMethod('getWifiDetails');
  }

Please this method on your project instead.
Future<String> getMacAddress() async {
  WifiInfoWrapper wifiObject;
  String macAddress;
  try {
    macAddress = (await WifiInfoPlugin.wifiDetails).macAddress;
  } catch (_) {
    macAddress = await WifiInfoPlugin.wifiDetailsByString;
  }
  return macAddress;
}

hope it's helpful

This issue will be updated in the next release


[Local test public] Automatically retarget PRs to the live branch

Your PR targeted the live branch, which is not allowed. The PR has been redirected to the default branch (main).

Docs Build status updates of commit f2c75d4:
:warning: Validation status: warnings

File
Status
Preview URL
Details

csidevdocs-public/1.md
:warning:Warning

Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

csidevdocs-public/docfx.json
:white_check_mark:Succeeded

csidevdocs-public/1.md

Line 1, Column 1: [Warning: author-missing - See documentation] Missing required attribute: 'author'. Add the current author's GitHub ID.
Line 1, Column 1: [Warning: title-missing - See documentation] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Warning: ms-author-missing - See documentation] Missing required attribute: 'ms.author'. Add the current author's Microsoft alias.
Line 1, Column 1: [Warning: ms-prod-or-service-missing - See documentation] One of the following attributes is required: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 1, Column 1: [Suggestion: description-missing - See documentation] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Your PR targeted the live branch, which is not allowed. The PR has been redirected to the default branch (main).

Docs Build status updates of commit f2c75d4:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: PullRequestMergeConflict] Cannot sync git repo to specified state. Please make sure the pull request is mergeable.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit f2c75d4:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

 It might be caused your branch is deleted or force pushed. If it is not this case, please open a ticket on https://aka.ms/SiteHelp

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


cmd/go: go mod -vendor prunes non-package directories

What version of Go are you using (go version)?
Does this issue reproduce with the latest release?
go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?
#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?
Successful build.
What did you see instead?
# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

We should probably preserve .s files too.

The cause here seems to be that we prune out directories that are not Go packages.
There is no voAAC.h in github.com/gen2brain/aac-go/aacenc/, but I do see one in github.com/gen2brain/aac-go/aacenc/external/aacenc/include/.

Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason.
Mostly cgo deps, but sometimes other required resources.
...
Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there.

I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to go mod -vendor) is snapshot the current module dependencies as opposed to the packages. Doing so would seem to address all of these points?
So then a go mod -modvendor (command/flags to be :bike:-shed-ed) would create a directory (say ./modvendor) that could then be used by GOPROXY=file:///path/to/modvendor.
That said, I don't use go mod -vendor and likely never will. But I do maintain an append-only cache of the modules that my CI then uses via GOPROXY=file://...

I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:

Non-Go code dependencies vendoring. This is what this issue is about.  And C dependencies are hard. Some approaches to solve this problem (excluding the painful choice of teaching the go tool about C dependencies... the horror):

Let module authors declare their non-Go code dependencies in their go.mod. Something like
external "github.com/gen2brain/aac-go/aacenc/external/aacenc"

Define and document a convention for such dependencies (e.g. external directory, similar how we have conventions for internal and testdata).

Explicitly give up and document that CGo is fundamentally incompatible with go mod -vendor. After all, even if you do have the C code in your vendor, there are no guarantees that that C code doesn't have some dependencies of its own, so hoping that /vendor will save you from build failures is rather naïve.

Binary tool dependencies. This may or may not be a problem Go modules want to solve. @rsc said it in #24051.

It is not a goal to expand into "linux distribution" territory.

But then this, too, should be documented explicitly. It would be a shame though, as AFAIK other systems, like Ruby's Bundler and JS's NPM support some version of it.

One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import.
I think this is only a problem because the c and h files live in a separate directory.

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.
If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :)

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files 😔

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace go mod vendor or fix it so it copies everything. What is the reason for only vendoring half of the repository?

@nomad-software this issue here was initially reported when the original reporter was using cgo.
Are also hitting this with cgo, vs. are you hitting this without using cgo?
If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:
https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience)

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files

I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here:
https://github.com/nomad-software/vend
Any suggestions to improve it are welcome, just open an issue there.

https://github.com/nomad-software/vend

Yo @nomad-software this should be for of Dep tool itself, awesome work! solved my issue brilliantly 👍
Cheers!

@nomad-software I wanted to briefly double-check on the approach for https://github.com/nomad-software/vend. It seems to be intended for use with CGO (if I am not mis-understanding the README there) include when C/C++ headers and files are outside of directories with .go files.
It seems that approach of having C/C++ headers and files outside of directories with .go files might be dangerous for build issues? (Forgetting about vendoring for a second; just wanted to ask about building).
Comments above from Russ seemed to indicate:

The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

And (with emphasis added):

modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries.

Also, the CGO documentation now includes (also with emphasis added):

Note that changes to files in other directories do not cause the package
to be recompiled, so all non-Go source code for the package should be
stored in the package directory, not in subdirectories.

Sorry if I am mis-understanding...

go mod vendor command can not copy c and other files to the vendor directory, this makes a lot of packages have problems, maybe can add go mod vendor -a.

I just wonder if all these vendoring problems will disappear one day...  Don't get me wrong, I like the vendoring concept way more than downloading dependencies over and over. Offline builds are great.   When it will all end?!

We also run into problems with this as we have non-Go files in a repository that contain e.g. SQL scripts (for database migrations) or config files (for testing). Before trying to switch to go modules we used vendoring via git submodules, which worked well and allowed us to produce deterministic builds. Now we would like to switch to go modules (mainly to be able to easily pin versions of external dependencies as well). We tried to use go mod vendor to install the selected versions of our dependencies into the vendor folder and then use the test config from there, but since it only copies Go files all the config files are missing.
It would be great if there was a flag that one could set to copy the entire repository to the vendor folder. This should not be difficult to implement as go mod already stores the entire repositories in $GOPATH/pkg/mod. The alternative would be to manually check out the repository and initialize the required config files, which is cumbersome as we'd have to find a way to fetch the commit corresponding to the one referenced in the go.mod file to obtain the files belonging to the specific commit.

@adewes, why is it important that the SQL scripts and config files be in the vendor directory in particular? I don't know of any reason you couldn't combine Git submodules for those non-Go dependencies and Go modules for everything else.

Well the SQL scripts belong to a specific version of the Go code (they define the database tables for the models defined in Go), so they should be versioned together. Of course we could use submodules for that but this would introduce another type of versioning (go module version and git submodule version), so I really don't like it. The files don't need to be in the vendor directory but the go mod vendor would provide a convenient and predictable way to check out the repositories belonging to the Go code that's used. Most other "modern" languages (e.g. Python, Javascript, C#) support bundling non-code files when building packages, so I really don't understand why Go doesn't.
How are package maintainers and package providers supposed to do this? I couldn't find anything in the documentation. If I want to publish my SQL scripts together with my Go code how would I do it such that people can use go get to fetch my package and my package knows where it should look for config files?

@bcmills how about this case, I have a lib with a few packages but no root go file. The mod won't bring anything from the root, that includes the README, LICENSE and so on. Maybe I want to read the README without having to open my browser to go to the lib page on git, or check the lib license. Or even worse, it doesn't bring the go.mod of that lib, and, from what I've read (maybe I read wrong), libs should have a go.mod file.

@adewes, the key thing to understand is that go mod vendor operates on packages, not on modules. Any files that are colocated with a package (transitively) imported by the main module will be included in go mod vendor.
If you want to unpack entire modules instead, try go mod download -json.

@bcmills thanks for that info, I understand that go mod vendor checks out packages, but (I think) that's what we need, as we want to check out the files belonging to a package with a specific version used by the module. So a -a flag that would allow us to check out all files (not only Go files) would be really great. Using go mod download -json seems quite cumbersome again, as we would need to parse out the exact file locations from the JSON, which I think is unnecessary: Go already downloads a specific version of a package, so why not provide the files of that package in a standardized path that matches the import path of the package? That would make working with non-code files so much easier.
Right now we solved the problem by merging our two dependent packages together (so that we don't need to go get the second one), which works but is not a viable strategy for third-party dependencies.

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.
If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

following @rsc's hint, I managed to make the following "hack" in my Makefile:
versionPath=$(shell go list -f {{.Dir}} github.com/gogo/protobuf/version)
gogoprotos=$(versionPath:version=protobuf)
gogoprotosTarget:=./vendor/github.com/gogo/protobuf/

.PHONY: vendor
vendor:
	@go mod tidy -v
	@go mod vendor -v
	@go mod verify
	# work around to save proto files in gogoproto
	@mkdir -p $(gogoprotosTarget)
	@cp -rf --no-preserve=mode $(gogoprotos) $(gogoprotosTarget)

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories.

Cute idea.  This would be a nice example to what devs out in the wild might actually need.

I found this issue after strugging with https://github.com/go-goracle/goracle/issues/155.
@bcmills the use case is github.com/go-goracle/goracle needs the odpi directory, which contains cgo header and implementation files, but it's not referenced as a package outside of the cgo comments requiring those header files. I'm not going to use git submodules + go modules, because then I have to keep multiple versions of things in sync, or use a custom build script, which I'm not interested in creating or maintaining.
The frustration on my end is that I feel like I'm being punished for using go mod -vendor. How am I supposed to keep my dependency's dependency in sync with my project when there's no clear, simple, concise way for that to happen, or even understand that it was missing in the first place?

The same problem exists if one tries to use Go modules for pulling it a dependency which specifies its interface as a set of *.proto files. There is no way to use those in your project as they will not be included in the vendor directory. Right now the solution seems to go back to writing bash scripts.

@boreq, the Go source files generated from .proto files are supposed to be committed to version control.
Per https://golang.org/s/go1.4-generate:

once things are settled, the author commits the generated files to the source repository, so that they are available to clients that use go get

Compiling .proto files can depend on .proto files from other sources though.

Comments above from Russ seemed to indicate:

The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

And (with emphasis added):

modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries.

Also, the CGO documentation now includes (also with emphasis added):

Note that changes to files in other directories do not cause the package
to be recompiled, so all non-Go source code for the package should be
stored in the package directory, not in subdirectories.

This is notably in conflict with how TensorFlow includes C headers from its Go bindings: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/attrs.go#L20
Since the C headers are shared, the only way to put them in the Go package directory would be to duplicate the header files, which seems like a poor solution. Is there a better way? I'm happy to fork TensorFlow to make it vendor properly if necessary.

@mxplusb  @fedepaol and anyone else coming here from the future:
I've solved this problem for goracle / godror by adding dummy golang files to the directories containing external C source code.
Then to ensure that go mod tidy doesn't remove them, you require each of those packages from the root package.
Lastly, you put build tags on the dummy golang files so that you don't encounter the error C source files not allowed when not using cgo
You can see it done here: https://github.com/godror/godror/pull/38/files

@veqryn thanks! I do believe it still doesn't address the root of the problem but it does help. :)

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace it or fix it so it copies everything. What is the reason for only vendoring half of the repository?

any news here ? this issue is blocking ...

This issue is now two and a half years old and it affects a lot of people. Has there been any movement whatsoever on this problem?

This issue is closed, and has been for almost 2.5 years. People don't generally follow closed issues, so please open a new issue if you feel there is more information available here. Thanks

The issue the way it is currently seems to be very well explained and valid. @rsc closed it back then asking for someone from the community to fix it ? but its quite bizarre, because go mod was supposed to fix dep, but this feature actually works well on dep, but there is no clear way to make it work with go mod. I dont understand what is the point to make everyone move to go mod, when it is actually not feature complete.
For now, I managed to solve it with the work around of using (https://github.com/nomad-software/vend) but in reality, this is something that should be fixed here.

Wow, two years have passed and still no proper vendoring with the go mod. At this point, I think it's a political decision from Google. People want full repo dependencies copy into the vendor directory, I think it's trivial, yet we still don't have it.

Wow, two years have passed and still no proper vendoring with the go mod. At this point, I think it's a political decision from Google. People want full repo dependencies copy into the vendor directory, I think it's trivial, yet we still don't have it.

@radu-munteanu Try this: https://github.com/nomad-software/vend

@radu-munteanu Try this: https://github.com/nomad-software/vend

lol this is a 2.5 year old issue. why does everything have to suck?

lol this is a 2.5 year old issue. why does everything have to suck?

this still an issue, its crazy how we still need to use vend to solve the package issue....

We use go mod vendor with a CGo module that references .C and .H files stored in its subdirectory, and they are not copied by the tool.
Perhaps, tool could support an option in go.mod file to list modules that should be fully copied, instead of stripped?

@hrissan Partially repeating an older comment:
If you are hitting this with cgo, and if we set aside vendoring for a moment, there is a decent chance you are also having a more fundamental problem even just during non-vendor-based building/recompiling due to the behavior described here:
https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

Does that seem like it applies to you?

We have a legal requirement to add the license file of some packages to their vendor directory. Unfortunately just copying go files and not having an option for other files can become a serious issue.

Any chance we can restart this discussion? A flag to declare assets for a package would make sense? I guess something similar happens when using the embed annotation.

Looking https://github.com/golang/go/issues/26366#issuecomment-405683150, it seems that using a tree of C files will not work correctly.
But I still have to use code from a library that is organized as a tree.
Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory?

Here is a workaround that may be useful for other projects:
For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

All,
I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.
I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.
So, what is the best way to deal with this issue?
Thanks,
William

I have packaged the vend tool from @nomad-software  on Gentoo, thanks for writing it.
IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.
We will use vend for now, but please count me in as someone requesting that this discussion be restarted.
Thanks much for your time,
William

@williamh
Is it acceptable to install athens  as a private proxy for Go modules?
In this way, only athens needs to access the internet.

https://github.com/gomods/athens
https://gomods.io/
https://docs.gomods.io/intro/protocol/
https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/go-repositories
https://jfrog.com/blog/why-goproxy-matters-and-which-to-pick/

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.

Why is this closed?

See https://github.com/golang/go/issues/26366#issuecomment-405683150.


SciHub

Reframe sci hub as just nuance  to avoid Tyranny make it about infinite threading with multi home and link to other sites with some cannonicalness

sentences are a brain cage. they are both too large in that they contain both claims and evidence, and also too small to defend themselves adequately. They also don't show linkages to other sentances well. Logichub will be a tool where the top level data strcutures will be a limited subset of sentences but smaller.... but how do we get a flow of prose this way... now that's the right question!

How To Win Arguments Through Collaborative Truth-Seeking https://intentionalinsights.org/win-arguments-collaborative-truth-seeking/

https://advanced-hindsight.com/
https://fs.blog/mental-models/


Test VERIFY_PEER / VERIFY_NONE work against real cluster

Same as #545 but merged into master branch.

As explained in #545 these are somewhat flaky but provide value, merging.


scale down INT AKS nodes

Can all our pods run on a single node?

Can all our pods run on a single node?

Asking cause otherwise it will scale out to 2 or three anways.

Can all our pods run on a single node?

Actually e2e is also set to one but I believe it always runs on 2 nodes :)

But yeah good point, maybe we should just set both to 2 to start with

I assume the reason is the current quota limitation, right? The new request form does still not work and the support ticket wasn't processed yet.

I assume the reason is the current quota limitation, right? The new request form does still not work and the support ticket wasn't processed yet.

yeah, I also tried that :(
what do you think, should we set the minimum for both e2e and int to 2nodes since that seems to be what we require anyway?


Updates to Angular 12

Updates to Angular 12 (avoids a sass bug when using newer versions of Carbon while also getting a head start on v10 leaving LTS in a few months: https://angular.io/guide/releases#support-policy-and-schedule)
Specifies Node 12
Fixes breadcrumb routes/hrefs so that links are defined
Small formatting fixes

@nrbarry


Add YAML and Sass to the Pygments lexer to Confluence language map.

Fixes #150
Signed-off-by: James Bungard<EMAIL_ADDRESS>

Note: planning to merge in once #148 is resolved.

@tonybaloney I may have misunderstood @jdknight, but I believe the test case failure is due to an upstream change as described in #148 and not related to my changes?

@verdesmarald, correct. I will most likely submit a change which skips the code unit test for now until a resolution for that issue is made.


After accepting credential offer, blank screen displayed

Occurs when there is a notification present on the Home Screen to accept/reject a credential offer. Upon accepting the offer, a modal is displayed indicating success/failure of acceptance. When the modal is dismissed, a blank screen is displayed. No errors are logged to the console.

Resolved by #107


Warning for first time usage of a path

Warn the user that all files not releated to the backup in this folder would be deleted.

[ ] Show Msg on path change
[ ] Don't show Msg for already setuped / installed plugin
[ ] Add a more clear warning in the documentation
[ ] Add a information in the plugin configuration

@JackGruber - Is this still the case, where all files in the backup directory are deleted? I have not been able to replicate this on Fedora Linux.
I created a testdir and created a test file, and then ran the backup, which did not seem to remove the test file:
[root@workstation testdir]# ls -ltrh
total 40K
-rw-r--r--. 1 workstation workstation   0 May 29 16:42 test.txt
drwxr-xr-x. 3 workstation workstation 36K May 29 16:46 joplin_backup_05M.29D.2022Y.16H.46M

Changes will be published with the next version


Add Portuguese language & Add Cookie info translation

While using this library, I found a need to use texts in Portuguese. To accomplish it, I've made some changes:

Create the Portuguese text file with texts translated to Portuguese pt-br.language.ts.
Create the type TranslatedText;
Update the NgxCookieConsentComponent by adding dynamic translation using the TranslatedText type;
Update interfaces and classes to include TranslatedText to the fields:

CookieItem interface:

name;
description;
privacyPolicyUrl.

CookieDetail interface:

description;
duration.

NgxCookieConsentConfigService class:

privacyPolicyUrl;
imprintUrl;

@CarlosNassif Thank you for PR - i will look at it. Its a huge PR - and there are some things that i want to look closer.

Thanks, @giacomo!
It's the first PR I write to a public lib. Please let me know if there are any adjusts I should do on code structure and logic.


Large wasm file adds significant build latency

Bug report

What is the current behavior?
I am trying to load a large wasm file into my web app. The build works, however it's incredibly slow:
npx webpack --mode development
asset c37ad502f0b49101ff66.module.wasm 22.6 MiB [emitted] [immutable] (auxiliary name: main)
asset main.js 797 KiB [emitted] (name: main)
runtime modules 4.99 KiB 10 modules
cacheable modules 610 KiB (javascript) 22.6 MiB (webassembly)
  modules by path ./wasm/pkg/ 78.7 KiB (javascript) 22.6 MiB (webassembly)
    ./wasm/pkg/wasm.js 139 bytes [built] [code generated]
    ./wasm/pkg/wasm_bg.js 74.5 KiB [built] [code generated]
    ./wasm/pkg/wasm_bg.wasm 4.04 KiB (javascript) 22.6 MiB (webassembly) [built] [code generated]
  ./src/index.js 344 bytes [built] [code generated]
  ./node_modules/lodash/lodash.js 531 KiB [built] [code generated]
webpack 5.80.0 compiled successfully in 25725 ms

If the current behavior is a bug, please provide the steps to reproduce.
I have created a small project that reproduces the problem based on the webpack quickstart here. To reproduce, do:
cd /tmp
git clone -b webpack-wasm-slow https://github.com/Dig-Doug/bug-reproductions.git
cd bug-reproductions/webpack-wasm-slow/
npm install
npx webpack --mode development

The wasm library is built from a rust library. I have included the generated JS/wasm files so you do not need to build them, but if you would like to regenerate them you can do the following:

Setup rust: https://www.rust-lang.org/learn/get-started
Setup wasm-pack: https://github.com/rustwasm/wasm-pack
Do:

cd wasm
wasm-pack build

What is the expected behavior?
I do not need the wasm file to optimized or changed in any way. As far as I can tell, webpack is not modifying the file so I'd expect the wasm file to not impact the build latency significantly.
Other relevant information:
webpack version:   5.80.0
Node.js version: v16.13.1
Operating System: Ubuntu
Additional tools: n/a

After profiling webpack, I believe this issue is caused by the wasm parser. I've opened a PR to fix the underlying issue.

Yeah, I start to looking into this and found webpack doesn't do something special here


updated step to include a new 4th item

I wanted to add a 4th step to clarify this procedure for customers.

Proposed changes

Merge timing

Related issues (optional)

just doing a demo video.


Add provider health status event

Feature or Problem
From the discussions of the following issues, providers should emit the health events on the wasmbus.evt topic.
Please let me know if I misunderstood what was being asked in those issues or if there is something missing.
Related Issues

#517
#519

Release Information
Next
Consumer Impact
N/A
Testing

Built on platform(s)

[x] x86_64-linux

Tested on platform(s)

[x] x86_64-linux

Unit Test(s)
I've added a test in test/host_core/provider_tests.exs to start a provider from file and wait on the health_check events. It first waits for the health_check_passed event when the provider is started and afterwards waits for the health_check_status event once, which is emitted every 30 seconds.
Manual Verification

I've launched nats and subscribed to wasmbus.evt.* and to ">".
I've used make run from the dashboard project and from the web page started a provider from file to checkout that the health_status event was being correctly sent.

I'm going to hold off merging for now since we're hoping to cut a release for v0.61 this week, but thanks again!

No problem!


Adding support for FluentValidations and swagger schema.

I wish to register FluentValidation rules, so that they show up in swagger schema.
Given the sample code below:
public class Validator : AbstractValidator<ClosureScheduleCommand>
{
    public Validator()
    {
        RuleFor(x => x.Warehouse).NotEmpty();
    }
}

We should be able to correctly mark the property as required.
Example of the wanted schema presentation in Swagger

The required field is marked by a red asterix. In this example i relied on the attribute [JsonRequired]
Similar implementations in MVC aspnetcore 3.1:

https://github.com/micro-elements/MicroElements.Swashbuckle.FluentValidation.WebApi
https://github.com/zymlabs/nswag-fluentvalidation

Comments regarding this issue on Discord:
Dĵ ΝιΓΞΗΛψΚ: yeah looks like we're gonna have to write a nswag schema processor that's aware of fluentvalidations. could you pls maybe create a feature request on github for this? someone may pick up the task as i don't have the capacity to look into it myself for the next couple of weeks.
Full dump

Related to https://github.com/dj-nitehawk/FastEndpoints/pull/116/commits
Schema is correctly created, if not using inheritance.
However the below does not work for my setup. (Might be related to startup config, still tracking it down)
CarRequest : BaseRequest{}
BaseRequest
{
Public Guid Identifier {...}
}

@NeskireDK are you using v4.2 released today?

@wjax will you be able to look in to this?

The primary issue seems to be, that all fluentvalidation rules gets ignored as soon as inheritance is used. With inheritance the 4 last properties have no schema rules: Required, Length etc. But without inheritance, simply copy paste, they do. Exactly as described by @wjac on discord: https://discord.com/channels/933662816458645504/955771546654359553/974938790546255882s
In conclusion the PR 116, that added the below hasn't solved the issue with base class inheritance.
 // Add properties from base class
ApplyRulesToSchema(context, schema.InheritedSchema, validator);

There's a secondary issue. If we look at the example with Inheritance, the property 'Identifier' becomes required, even though it is clearly not, as it defaults to new guid. Furthermore, there are no fluentvalidation rules or attributes assigned to 'Identifier'
Example here with inheritance.

public class SaveRule : EndpointWithRepository<HomeStationRuleRequest, HomeStationRuleResponse, IHomeStationRuleRepository>
{
    public override void Configure()
    {
        Put("Rules/HomeStation");
        base.Configure();
        Summary(summary => summary.Description = $"Saves a new or updates already existing 'Home Station Rule'");
    }

    public override async Task HandleAsync(HomeStationRuleRequest req, CancellationToken ct)
    {
        var result = await Repository.SaveAsync(Mapper.Map<HomeStationRule>(req));
        await SendAsync(Mapper.Map<HomeStationRuleResponse>(result), cancellation: ct);
    }
}

public class HomeStationRuleRequest : BaseIdentifier
{
    public string HomeStation { get; set; }
    public Guid ClosingDayIdentifier { get; set; }
    public Guid FlightOperatorId { get; set; }
    public DateTime ValidFrom { get; set; }
}

public class BaseIdentifier
{
    public Guid? Identifier { get; set; } = Guid.NewGuid();
}

public class RequestValidator : Validator<HomeStationRuleRequest>
{
    public RequestValidator()
    {
        RuleFor(x => x.HomeStation).HomeStationValidation();
        RuleFor(x => x.ClosingDayIdentifier).NotEmpty().NotNull();
        RuleFor(x => x.FlightOperatorId).NotEmpty();
        RuleFor(x => x.ValidFrom).NotEmpty().NotNull();
    }
}

Example without inheritance.

public class SaveRule : EndpointWithRepository<HomeStationRuleRequest, HomeStationRuleResponse, IHomeStationRuleRepository>
{
    public override void Configure()
    {
        Put("Rules/HomeStationNoInheritance");
        base.Configure();
        Summary(summary => summary.Description = $"Saves a new or updates already existing 'Home Station Rule'");
    }

    public override async Task HandleAsync(HomeStationRuleRequest req, CancellationToken ct)
    {
        var result = await Repository.SaveAsync(Mapper.Map<HomeStationRule>(req));
        await SendAsync(Mapper.Map<HomeStationRuleResponse>(result), cancellation: ct);
    }
}

public class HomeStationRuleRequest
{
    public Guid? Identifier { get; set; } = Guid.NewGuid();

    public string HomeStation { get; set; }
    public Guid ClosingDayIdentifier { get; set; }
    public Guid FlightOperatorId { get; set; }
    public DateTime ValidFrom { get; set; }
}

public class RequestValidator : Validator<HomeStationRuleRequest>
{
    public RequestValidator()
    {
        RuleFor(x => x.HomeStation).HomeStationValidation();
        RuleFor(x => x.ClosingDayIdentifier).NotEmpty().NotNull();
        RuleFor(x => x.FlightOperatorId).NotEmpty().NotNull();
        RuleFor(x => x.ValidFrom).NotEmpty().NotNull();
    }
}

I will look at it.
It worked in my examples, but you know... ;)
Can you provide a repo with an "non-working" example?

I will look at it. It worked in my examples, but you know... ;)
Can you provide a repo with an "non-working" example?

Yes, its the least I can do. https://github.com/NeskireDK/FastEndpointsPlayground

Ok, After some investigation, I have some thought to throw to you all.
Inheritance is now working in my branch, however there are some things to discuss if we want to allow first level swagger validation support.
Inheritance. Validation at Request type level
public class Request : BaseIdentifier
{
    public Guid? NullableIdentifier { get; set; }
    public string? Message { get; set; }
}

public class BaseIdentifier
{
    public Guid Identifier { get; set; } = Guid.NewGuid();
}

public class RequestValidator : Validator<Request>
{
    public RequestValidator()
    {
        RuleFor(x => x.Identifier).NotEmpty();
        RuleFor(x => x.NullableIdentifier).NotEmpty();
        RuleFor(x => x.Message).Length(3, 100);
    }
}

This is working.
Inheritance. Validation at any level
public class Request : BaseIdentifier
{
    public Guid? NullableIdentifier { get; set; }
    public string? Message { get; set; }
}

public class BaseIdentifier
{
    public Guid Identifier { get; set; } = Guid.NewGuid();
    public string baseMessage { get; set; }
}

public class RequestValidator : Validator<Request>
{
    public RequestValidator()
    {
        RuleFor(x => x.Identifier).NotEmpty();
        RuleFor(x => x.NullableIdentifier).NotEmpty();
        RuleFor(x => x.Message).Length(3, 100);
    }
}

public class BaseIdentifierValidator : Validator<BaseIdentifier>
{
    public BaseIdentifierValidator()
    {
        RuleFor(x => x.baseMessage).Length(10, 100);
    }
}

In this case we take into account the validators of the parents so the child object (the request) has also inherited the validators from the parents.
It seems right to me to support this case.
###Composition
What about composition. Currently it is not supported. Imagine the following:
public class Request
{
    public Guid? NullableIdentifier { get; set; }
    public string? Message { get; set; }
    public BaseIdentifier baseid { get; set; }
}

public class BaseIdentifier
{
    public Guid Identifier { get; set; } = Guid.NewGuid();
    public string messageBase { get; set; }
}

public class RequestValidator : Validator<Request>
{
    public RequestValidator()
    {
        RuleFor(x => x.NullableIdentifier).NotEmpty();
        RuleFor(x => x.Message).Length(3, 100);
        RuleFor(x => x.baseid.Identifier).NotEmpty(); // This does not work currently
    }
}

public class BaseIdentifierValidator : Validator<BaseIdentifier>
{
    public BaseIdentifierValidator()
    {
        RuleFor(x => x.messageBase).Length(10, 100);
    }
}

Swagger should provide validation for composition.
BaseIdentifierValidator is ingnored and not incorporated into swagger data.
But even worse, a Rule with a field from an "internal" object neither.
So what cases should be supported.
All of them seems good to me

first two cases I'd say yes.
third case/ composition I'd say nah.
I would certainly not write validators like that personally.
let's see what others have to say.

Talking about the composition thing,
How would I validate this:
public class Request
{
    public Guid? NullableIdentifier { get; set; }
    public string? Message { get; set; }
    public SomeEntity[] entities { get; set; }
}

public class SomeEntity
{
    public Guid Identifier { get; set; } = Guid.NewGuid();
    public string messageBase { get; set; }
}

public class RequestValidator : Validator<Request>
{
    public RequestValidator()
    {
        RuleFor(x => x.NullableIdentifier).NotEmpty();
        RuleFor(x => x.Message).Length(3, 100);
    }
}

public class BaseIdentifierValidator : Validator<SomeEntity>
{
    public BaseIdentifierValidator()
    {
        RuleFor(x => x.messageBase).Length(10, 100);
    }
}

Is this case a non-sense?

"set validator for each entity" i believe: https://docs.fluentvalidation.net/en/latest/collections.html?#collections-of-complex-types

Ok,
I'll see what I can do. Will pull request soon!

@NeskireDK can you try v4.3.0-beta2 and give us some feedback on this?

Sure. The original issue has been resolved beautifully. Thanks a lot, this is 95% of all cases now correctly working in my private repository.
I've updated my repository to reflect as such. In addition i've added 2 more cases, that unfortunately is not working / supported either.
That being

Validation specifying a validator for child: RuleFor(x => x.HeadQuarters).SetValidator(new AddressValidator());
Inheritance Validators: RuleFor(x => x.Contact).SetInheritanceValidator(v => v.Add<Type>(new ..);
As described here

The validations are correctly run and submission gets stopped with status 400, including validation message.
However the swagger schema once again doesn't show up for child elements this time.
Source code: https://github.com/NeskireDK/FastEndpointsPlayground
Remaining failing schema endpoints:

SaveWithChildValidator
SaveWithInheritanceValidator

After some discussion with @dj-nitehawk we will keep the current support for a single validator against the request type of the endpoint. We will not make complex things injecting base validators automatically that could break the surface api or make things too complex.
Of course the requestDto can use inheritance and it will play nice with both the validator and the swagger support.
So, in case anyone needs to reuse a validator from a base type, he/she can use inheritance in the validator like this:
public class Request : BaseIdentifier
{
    public string? NullableIdentifier { get; set; }
    public string? Message { get; set; }
}

public class BaseIdentifier
{
    public string Identifier { get; set; }
    public string baseMessage { get; set; }
}

public class RequestValidator : BaseIdentifierValidator<Request>
{
    public RequestValidator()
    {
        RuleFor(x => x.Identifier).NotEmpty();
        RuleFor(x => x.NullableIdentifier).NotEmpty();
        RuleFor(x => x.Message).Length(3, 100);
    }
}

public class BaseIdentifierValidator<TDto> : Validator<TDto> where TDto : BaseIdentifier
{
    public BaseIdentifierValidator()
    {
        RuleFor(x => x.baseMessage).Length(10, 100);
    }
}

I quite agree with this approach. However, see post above which attempts to do explicit child inheritance, and schema not showing the validation.

Opened a new issue to track these missing pieces
https://github.com/dj-nitehawk/FastEndpoints/issues/127


[vcpkg] How do I force remove a package ignore dependencies

How do I force remove a package ignore dependencies?
I tried vcpkg remove -f with no luck.
I had to remove all downstream packages at the same time.
Suppose I change a little bit code of a port, then I need rebuild all downstream packages even that I'm pretty sure it won't break downstream packages.

Not possible.
If you build vcpkg yourself you might want to grab this https://github.com/microsoft/vcpkg-tool/pull/816


Cleanup all instances on node delete, not just the healthy ones.

This is clearly not a good idea.


Update to 25APR2022 build

Description
Update build, localization, and covariance files.  There should be some speedup with recent changes in GetValues.  Follow-on PRs will include more GetValues and IODA changes.
Issue closed
Closes #102
Tests completed

[x] 3dvar_OIE120km_WarmStart
[x] 3denvar_OIE120km_WarmStart
[x] 3dvar_OIE120km_ColdStart
[x] 3denvar_O30kmIE60km_WarmStart
[x] eda_OIE120km_WarmStart

Similar to my comment in #100, the observation-space verification wall-time increased for AMSUA in the 30km-60km test.  Will push increased wall-time change to this feature branch after it succeeds.


No symbol version section for<EMAIL_ADDRESS>on linux ARM64

I am running linux (fedora) on arm64 architecture (apple silicon). When I run make inside glfw/lib, after compilation I get the following error during linking step:

no symbol version section for versioned symbol<EMAIL_ADDRESS>
I do want to fix this and create a PR, but I am not really sure where to begin.
One thing I noticed that in the directory glfw/lib/glibcver there are platform specific files like glibc_2.9_x64.h, but no corresponding glibc file for arm64. Can this be one of the causes?
Thanks.

Okey, looks like wheybags/glibc_version_header#35 is outstanding for now. I've just disabled static glibc version linking for now on ARM64, which means the produced binary will be tied to the host's GLIBC version number (and thus not be backwards compatible), but at least the compilation should succeed now.

@Shinmera Thank you! I am able to build it now, and raised a PR. But I am not sure how would the new file be located by cffi? Does it need to have a particular naming pattern?


OpenCL support for Raspberry pi 4 GPU (VideoCore VI)

First of all thank you for building OpenCL compiler for VideoCoreIV GPUs. Just wanted to know is there is possibility to support for VC6 GPU devices.

Short answer: no, see also #79.
The VideoCore VI GPU is a very different architecture (e.g. with MMU, without 2 separate register files AFAIK), so supporting it require rewriting most of the compiler.
Also, I don't know of any publicly available hardware and instruction set documentation.

@doe300 As I see Eben Upton co-founder of the Raspberry Pi Foundation, has announced open source Vulkan graphics driver for the Raspberry Pi 4 family. So I guess He can share VideoCore VI specs with you as well.

I tried to get this working:
https://github.com/kpet/clvk
But first off it needs a whole lot of ram to build (4gb ram + 6 gb swap for me). And unfortunately failed to build. Installing Vulkan on RPi isn't straightforward either.
I'm not in urgent need to run OpenCL on RPi, but wanted to post this to point out that this ticket probably can't be closed.
(Unless I missed something and recent developments have led to a better working approach?)

I tried to get this working:
https://github.com/kpet/clvk
But first off it needs a whole lot of ram to build (4gb ram + 6 gb swap for me). And unfortunately failed to build. Installing Vulkan on RPi isn't straightforward either.
I'm not in urgent need to run OpenCL on RPi, but wanted to post this to point out that this ticket probably can't be closed.
(Unless I missed something and recent developments have led to a better working approach?)

I can confirm that CLVK in combination with the latest version of Mesa's v3dv Vulkan driver, you can get OpenCL on a rPi4.
Unfortunately, important extensions are missing. The v3dv Vulkan driver can't do FP16 (16 bit floats.) Nor can it do 8 bits integers. This limits the usefulness for me personally, as my kernels are written to operate on FP16 values.
If your kernels don't use FP16 / INT8, you may have better luck than me, though.

I suspect the best way to get a decent supported OpenCL will be using the mesa gallium driver in conjunction with the new mesa rusticl opencl stack. rusticl is being actively developed and a there has been support added for a bunch of other arm GPUs that support opencl (panfrost for MALI, etnaviv for Vivante).

the new mesa rusticl opencl stack.

It can find the rusticl platform.
$ clinfo --list
Platform #0: Clover
Platform #1: rusticl

But it cannot find any devices:
$ clinfo
Number of platforms                               2
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 23.0.2
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   rusticl
  Platform Vendor                                 Mesa/X.org
  Platform Version                                OpenCL 3.0
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions with Version                cl_khr_icd                                                       0x400000 (1.0.0)
  Platform Numeric Version                        0xc00000 (3.0.0)
  Platform Extensions function suffix             MESA
  Platform Host timer resolution                  0ns

  Platform Name                                   Clover
Number of devices                                 0

  Platform Name                                   rusticl
Number of devices                                 0
...

This is with Ubuntu 23.04 on rPi-400.

the new mesa rusticl opencl stack.

It can find the rusticl platform.
$ clinfo --list
Platform #0: Clover
Platform #1: rusticl

But it cannot find any devices:

There's probably some driver work required, but ultimately that will be the best/quickest way to get GPU HW OpenCL on the rpi4

I can confirm that CLVK in combination with the latest version of Mesa's v3dv Vulkan driver, you can get OpenCL on a rPi4.
Unfortunately, important extensions are missing. The v3dv Vulkan driver can't do FP16 (16 bit floats.) Nor can it do 8 bits integers. This limits the usefulness for me personally, as my kernels are written to operate on FP16 values.
If your kernels don't use FP16 / INT8, you may have better luck than me, though.

Can I make a build in docker on the amd64 platform with arm64 emulation ?  It might to decrease building time