./get_data.py missing

In the README, it asks to first run python ./get_data.py but the file is missing from the repo?

You can find the data you want in the kaggle link below


Drop internal trusty REST code in favor of client from trusty-sdk

Summary
This commit modifies the trusty evaluator to use the API client from
stacklok/trusty-sdk-go isntead of the in tree pacakge which is now
removed.
DO NOT MERGE: For review purposes, this is pulling the trusty-sdk @ HEAD, once https://github.com/stacklok/trusty-sdk-go/pull/14 merges, I'll cut a new release and repush to pull on the tag.
Signed-off-by: Adolfo García Veytia (Puerco<EMAIL_ADDRESS>Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[ ] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[x] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[ ] Added comments to complex or tricky code sections.
[x] Updated any affected documentation.
[x] Included tests that validate the fix or feature.
[x] Checked that related changes are merged.

coverage: 52.8% (+0.1%) from 52.676%
when pulling 1ea0d359dadce63b0bb8bc13d15b6629e50c7357 on puerco:minder-sdk
into 54d663da4e994de120f1b2e6a509071be5fe7414 on stacklok:main.

coverage: 52.795% (+0.1%) from 52.676%
when pulling b5432e788640b6f3435ed06e82b73b19e0eaed7d on puerco:minder-sdk
into 54d663da4e994de120f1b2e6a509071be5fe7414 on stacklok:main.

@dmjb - I've noticed you approved it, but @puerco mentions he wants to release the sdk before merging so I've converted this to draft until it's ready 👍

coverage: 53.251% (+0.1%) from 53.148%
when pulling 250ef40e18a3a839f66b1473ce09a69768afc448 on puerco:minder-sdk
into a21f1dd4ab31f40a789353d74f04b5544c6b5cc4 on stacklok:main.

OK, the trusty sdk has a new release. This should be ready.

@puerco the code looks good, but because we took our sweet time reviewing, there are now some conflicts in go.mod and sum. If you rebase, I'll approve right away. Sorry for the delays.

coverage: 53.367% (+0.1%) from 53.269%
when pulling 4d1a11994fb3105b1067df5d5ff7146bd89e1942 on puerco:minder-sdk
into 958e5463fbf40b9eefcfead1be87271fb60e11af on stacklok:main.


SSL story, actually TLS

It would be good to have everything related to setting up secure servers in one place.
Also a prerequisite for playing with HTTP/2

[x] SSL vs TLS summary TLS is replacement to SSL with some incompatibilities
[x] Upgrade vs explicit port. What is best practise and why
[x] Security headers relating to HTTPs
[x] Creating a self signing certificate
[ ] Automatically creating a certificate using the Acme (Automatic Certificate Management Environment) protocol and lets encryp
[ ] lets encrypt staging endpoint
[ ] What is a cacerts file http://www.phoenixframework.org/docs/configuration-for-ssl

Concepts

keyfile, private key stored 'path/to/keyfile.key'
certificate signing request (CSR), normally given to known certificate authority (CA)
certificate, verification of a CSR by some CA

How to generate a certificate
for development https://serversforhackers.com/video/self-signed-ssl-certificates-for-development
self signed http://how2ssl.com/articles/openssl_commands_and_tips/

Finished at a basic level. I do not currently need the advanced topics to will wait till they are required


Add a SARIF output formatter

This commit adds a formatter that outputs JSON in a specific SARIF format according to spec at [1].
This code is largely leveraged from an existing implementation found here [2].
SARIF format is very useful for integration into ecosystems such as GitHub's Actions.
[1] https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/sarif-v2.1.0-cs01.html
[2] https://github.com/microsoft/bandit-sarif-formatter
Closes #646

Here is a rendered example of using Bandit's SARIF format hooked into a GitHub Code Scanning Action:
https://github.com/securesauce/examples/security/code-scanning/8

Here is a rendered example of using Bandit's SARIF format hooked into a GitHub Code Scanning Action:
https://github.com/securesauce/examples/security/code-scanning/8

That's a 404 for me. I'm guessing that's a (maybe private) repo's private security scan results

Here is a rendered example of using Bandit's SARIF format hooked into a GitHub Code Scanning Action:
Code scanning alert

That's a 404 for me. I'm guessing that's a (maybe private) repo's private security scan results

Ah yes, code scanning alerts are probably only accessible to owners and maintainers of the repo. Here's a snapshot.

It looks really nice. Do you know if those alerts clear upon subsequent commits that fix the issue?

I'm curious how this interacts with baseline but this looks fine to me

It looks really nice. Do you know if those alerts clear upon subsequent commits that fix the issue?

They do. I believe GitHub keeps record of a fingerprint of the code to determine whether something gets fixed.


Create scripts for the calculation of the indicator and add sample data:

[ ] SPSS
[ ] R
[ ] Stata
[ ] Sample data file

wfp_cabs_syntaxe_1108.pdf


A scan should NOT crash if the notification doesn't work as expected

terrascan version:
Operating System:

Description
Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.
I'm tryout the helm chart for admission controller capabilities. When I create an insecure pod, a scan is triggered, but due to the failure in the notification flow, the scan fails and terrascan allows that insecure pod to get created inside a cluster.

A failure in the notification feature flow should not block/sabotage the whole violation scan flow. The notification should be dispatched as a seperate goroutine.
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Fixed here : https://github.com/accurics/terrascan/pull/1051


fix: typo in Apollo version

This PR fixes a bug that caused Swift to fail resolving dependencies because no version 1.0.00-alpha-1 of apollo-ios exists.

@lovetodream: Thank you for submitting a pull request! Before we can merge it, you'll need to sign the Apollo Contributor License Agreement here: https://contribute.apollographql.com/


Introduce error-defer handler / sagas / compensations

In order to undo partial results in case of failures, we need a way to let the user specify an error-defer handler/compensations (a saga library). These error-defer handlers should only run if a terminal error is thrown (e.g. explicitly by the user or in the form of a CancellationException as part of a cancellation).
The way we register or run the error-defer handlers needs to be able to handle the situation where a sideEffect call is repeatedly failing and a user then cancels the invocation. In this case, we need to make sure that an associated error-defer handler is still executed because the sideEffect could be visible in the external system.

Closing this issue since we don't need it with the initial milestone of the cancellation feature.


Auto Loot does not work on a brand new game.

Description/How to reproduce:
Although the configuration-holotape is given immediately after face-cofiguration is finished, it does not persist through the pre-war era and freeze :p . (latest version of mod used)
Expected result:
The character should be given the configuration holotape at the moment it persist leaving v111.
Further Info:
Probably have to wait until the character is thawed as the loot obtained from v111 persists. Please add the necessary checks in dubhAutoLootQuest.pex

Thank you for the report. I'll look into this.

The holotape is now added to the player when the Pip-Boy is activated for the first time, or when the mod is first loaded any time after that point. This issue is now resolved. Please update to the latest version.


Get error on build gogs via glide from source

Gogs version (or commit ref): 0.9.97 and current git master (7e15ff9)
Git version: 7e15ff9
Operating system: linux (Arch Linux)
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[X] SQLite

Can you reproduce the bug at https://try.gogs.io:

[ ] Yes (provide example URL)
[ ] No
[X] Not relevant

Log gist:

...
# github.com/gogits/gogs/vendor/golang.org/x/net/html/charset
vendor/golang.org/x/net/html/charset/charset.go:42: undefined: encoding.Encoder
vendor/golang.org/x/net/html/charset/charset.go:45: undefined: encoding.HTMLEscapeUnsupported
...

Description
The build process is breaking via glide on the current release (0.9.97) and the current master commit (7e15ff9). Looks like gogs using some objects the not exist in this package (golang.org/x/net) with the defined commit level (57bfaa875b96fb91b4766077f34470528d4b03e9).

Was an caching problem

Ok, I found the solution. The problem is by glide. I need to run glide cc before I build the package.


What's wrong?  Can't find the file DeepFace.

hi, I am a fresher of python and Github. I can't see the 'database' file like what's showed in the tutorial video. Is there anything wrong with my system?

ImportError                               Traceback (most recent call last)
 in
----> 1 from deepface import DeepFace
/srv/conda/envs/notebook/lib/python3.6/site-packages/deepface/DeepFace.py in
13 import pickle
14
---> 15 from deepface.basemodels import VGGFace, OpenFace, Facenet, FbDeepFace, DeepID, DlibWrapper, ArcFace, Boosting
16 from deepface.extendedmodels import Age, Gender, Race, Emotion
17 from deepface.commons import functions, realtime, distance as dst
/srv/conda/envs/notebook/lib/python3.6/site-packages/deepface/basemodels/Boosting.py in
6 import numpy as np
7 import gdown
----> 8 from deepface.commons import functions, distance as dst
9
10 def loadModel():
/srv/conda/envs/notebook/lib/python3.6/site-packages/deepface/commons/functions.py in
2 import numpy as np
3 import pandas as pd
----> 4 import cv2
5 from pathlib import Path
6 import gdown
/srv/conda/envs/notebook/lib/python3.6/site-packages/cv2/init.py in
3 import sys
4
----> 5 from .cv2 import *
6 from .data import *
7
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

it seems that your opencv installation is not oke. you should fix it because opencv is a dependency of the deepface library.

it seems that your opencv installation is not oke. you should fix it because opencv is a dependency of the deepface library.

Sincerely thanks for your help, serengil I have download opencv with anaconda. But now I met another trouble: Image data of dtype object cannot be converted to float. I have no idea about it.  Below is the screenshot.

Hi,
I'm facing same issue with opencv.
I have tried with different versions of pip 'opencv' package.
i.e.
opencv-python
versions=<IP_ADDRESS>, <IP_ADDRESS>
Log:
https://gist.github.com/AnkS4/c7a0397972d12f4c17a1d49fc781b397


Docs: Readme: To Develop Locally

Setting up Robyn locally to prepare for contributions. Testing the documentation in the README.md details below
Preliminary conclusions

I didn't have clang installed (sudo apt install clang)
It looks like Python 3.13.0 fails (uvloop?)
The instructions do not mention git cloning to get started.
I've not got things working yet.

I started with this toolchain on Debian 12:

Rust 1.82.0
Python 3.13.0
uv 0.4.27

I used this variation of the readme instructions:

Setup a virtual environment

uv init Robyn

Install required packages

uv add pre-commit poetry maturin

Install development dependencies

uv run poetry install --with dev --with test

First, failure was:
uv run poetry install --with dev --with test
Uninstalled 8 packages in 3ms
Installed 8 packages in 8ms

[tool.poetry] section not found in /home/dave/Code/Robyn/pyproject.toml

Therefore I copied the entire pyproject.toml from gitub.
uv run poetry install --with dev --with test
warning: No `requires-python` value found in the workspace. Defaulting to `>=3.13`.
error: Failed to prepare distributions
  Caused by: Failed to download and build `uvloop==0.19.0`
  Caused by: Build backend failed to build wheel through `build_wheel` (exit status: 1)
...

I figure this is due to Python 3.13.0 so
uv python pin 3.12.7

Also edited .python-version to contain 3.12.7
Try again:
uv run poetry install --with dev --with test
Using CPython 3.12.7
Removed virtual environment at: .venv
Creating virtual environment at: .venv
warning: No `requires-python` value found in the workspace. Defaulting to `>=3.12`.
error: Failed to prepare distributions
  Caused by: Failed to build `robyn @ file:///home/dave/Code/Robyn`
  Caused by: Build backend failed to build editable through `build_editable` (exit status: 1)

[stdout]
Running `maturin pep517 build-wheel -i /home/dave/.cache/uv/builds-v0/.tmpEU1nmd/bin/python --compatibility off --editable`

[stderr]
💥 maturin failed
  Caused by: Can't find Cargo.toml (in /home/dave/Code/Robyn)
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/dave/.cache/uv/builds-v0/.tmpEU1nmd/bin/python', '--compatibility', 'off', '--editable'] returned non-zero exit status 1

So I created a Caro.toml copied from github then
uv run poetry install --with dev --with test
warning: No `requires-python` value found in the workspace. Defaulting to `>=3.12`.
error: Failed to prepare distributions
  Caused by: Failed to build `robyn @ file:///home/dave/Code/Robyn`
  Caused by: Build backend failed to build editable through `build_editable` (exit status: 1)

[stdout]
Running `maturin pep517 build-wheel -i /home/dave/.cache/uv/builds-v0/.tmpLOg0kR/bin/python --compatibility off --editable`

[stderr]
💥 maturin failed
  Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
  Caused by: `cargo metadata` exited with an error: error: failed to parse manifest at `/home/dave/Code/Robyn/Cargo.toml`

Caused by:
  can't find library `robyn`, rename file to `src/lib.rs` or specify lib.path
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/dave/.cache/uv/builds-v0/.tmpLOg0kR/bin/python', '--compatibility', 'off', '--editable'] returned non-zero exit status 1
dave@geeky:~/Code/Robyn$ cargo build
error: failed to parse manifest at `/home/dave/Code/Robyn/Cargo.toml`

Caused by:
  can't find library `robyn`, rename file to `src/lib.rs` or specify lib.path

Hey @dave42w 👋
Thanks , will do now

Hey @dave42w 👋
Thanks , will do now

Thanks!
The upgrade to poetry.lock has helped me.
uv is absolutely awesome.
Upgrades to more recent packages would help as well eg pytest

No worries. I'll probably not update to uv in this PR. (I love uv but will be a lot of work for a small PR(happy to accept PRs 😄 )) I will make another issue for it.

No worries. I'll probably not update to uv in this PR. (I love uv but will be a lot of work for a small PR(happy to accept PRs 😄 )) I will make another issue for it.

Sorry, I agree. uv for everyone would be a big step. Just the steps to make it easier to use newer python with newer packages and remove blockages to uv would be great.
I'd also really appreciate adding a section to cover using a customer build of robyn in your own project. Once you build a Robyn with some changes (maybe testing someone else's PR) how do you move beyond Robyns own unit and integration tests to use that version of Robyn in your own project.
Thanks

I'd also really appreciate adding a section to cover using a customer build of robyn in your own project. Once you build a Robyn with some changes (maybe testing someone else's PR) how do you move beyond Robyns own unit and integration tests to use that version of Robyn in your own project.

@dave42w , I don't really get it. Can you explain a bit more?

I'd also really appreciate adding a section to cover using a customer build of robyn in your own project. Once you build a Robyn with some changes (maybe testing someone else's PR) how do you move beyond Robyns own unit and integration tests to use that version of Robyn in your own project.

@dave42w , I don't really get it. Can you explain a bit more?

Sorry I was unclear.
I have a directory with a clone of github Robyn in it. I make changes and complete the steps to the point where it passes all the tests.
Now I want to use that personal, updated version of Robyn in my own project which is in a different directory/github repository.
What steps do I need to do to replace the released version of Robyn in my project (installed with uv add robyn) with the version that is in the target directories where I have just run the tests?
Does that make more sense?

No , lol. I thought this was well known. You can use a regular pip install as well
I will add it in the docs


THe host is not enable

We have written the needed data into your clipboard because it was too large to send. Please paste

/duplicate


Relax CudaUVM requirement

Fixes #16.

Opted for a different solution: BVH::bounds() is now a host function


Added accordion functionality to building display page

I think I resolved the merge conflicts correctly. Still can't wrap my head around the notations very well, but  the accordions do  appear to working as intended.

@wassona fyi your data file for the timeline page looks like it's missing from the codebase: './project_sample_subsidy.csv'


daemon: add copyright header to metrics.go

Signed-off by: Ian Vernon<EMAIL_ADDRESS>

This change is

test-me-please


Make ToggleGroup a Form Component

Feature request type
Enhance component
Component name
No response
Is your feature request related to a problem?
MudToggleGroup doesn't take advantage of a form.
Describe the solution you'd like
Derive from IFormComponent.
Have you seen this feature anywhere else?
No response
Describe alternatives you've considered
No response
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@ScarletKuro Is this a good idea?


Lightweight event framework for microservices

The FFTM codebase has built-in eventing support with:

WebSockets support for inbound connections
Webhooks support for outbound connections
Workload managed mode: at-least-once delivery
Broadcast mode: at-most-once delivery
Batching for performance
Checkpointing for the at-least-once delivery assurance
CRUD on event streams, using DB backed storage

This PR adapts that framework to be generic, such that it can be embedded into any Golang microservice.
The source of the events can be anything with the following characteristics:

Events have a sequence identifier, that can be expressed as an alpha-numerically sortable string
Events can be replayed from a specific previous sequence ID

Some improvements in code structure have been made along the way, in making it generic.
Also E2E tests have been added that demonstrate how it could be embedded, and using a real DB + WebSockets/WebHooks delivery mechanism.

Codecov Report

Merging #105 (c5acf45) into main (6f614c3) will decrease coverage by 0.02%.
The diff coverage is 99.90%.

@@             Coverage Diff             @@
##              main     #105      +/-   ##
===========================================
- Coverage   100.00%   99.98%   -0.02%
===========================================
  Files           63       77      +14
  Lines         5048     6100    +1052
===========================================
+ Hits          5048     6099    +1051
- Misses           0        1       +1

Files
Coverage Δ

pkg/dbsql/database.go
100.00% <100.00%> (ø)

pkg/dbsql/provider.go
100.00% <ø> (ø)

pkg/dbsql/provider_sqlitego.go
100.00% <100.00%> (ø)

pkg/eventstreams/activestream.go
100.00% <100.00%> (ø)

pkg/eventstreams/config.go
100.00% <100.00%> (ø)

pkg/eventstreams/event.go
100.00% <100.00%> (ø)

pkg/eventstreams/eventstreams.go
100.00% <100.00%> (ø)

pkg/eventstreams/manager.go
100.00% <100.00%> (ø)

pkg/eventstreams/persistence.go
100.00% <100.00%> (ø)

pkg/eventstreams/webhooks.go
100.00% <100.00%> (ø)

... and 14 more


Roll Flutter Engine from 37a7a144b864 to 597d726c3b67 (1 revision)

https://github.com/flutter/engine/compare/37a7a144b864...597d726c3b67
2024-12-09<EMAIL_ADDRESS>Roll Skia to 14f8f6d984ff (flutter/engine#57068)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC bdero<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Optimize CASE expression to produce dictionary-encoded arrays in some cases

Is your feature request related to a problem or challenge?
Here is an example CASE expression that produces an array containing exactly two distinct values:
CASE WHEN expr THEN 1 ELSE 0 END

We currently produce an array containing ones and zeros (and nulls). It seems like it would be beneficial to produce a dictionary-encoded array in this case to reduce memory pressure. This could also make subsequent operations more efficient in some cases.

I think implementing operations like this will be much easier if we are able to add LogicalTypes
https://github.com/apache/datafusion/issues/11513
(I am thinking specifically that we could have expressions dynamically decide what encoding to use for their output rather than having it hard coded)


[dask] add tests on warnings, fix incorrect variable in log

While working on #3756 , I've been running mypy python-package/. This caught a bug in my refactoring from recent PRs (not sure which one).

python-package/lightgbm/dask.py:251: error: Name 'tree_learner' is not defined

This PR fixes that, and adds unit tests to cover the behavior under the two warnings in lightgbm.dask._train().

I've been running mypy python-package/.

Could you please setup mypy in lint task but allow it to fail (|| exit 0) so that everyone will be able to see logs and possibly prevent new issues to appear. We had the same strategy with cpplint. We had been checking logs manually but at some moment we simply removed || exit 0.
https://github.com/microsoft/LightGBM/blob/066720eff680606abca17a9b34d95905337d8afd/.ci/test.sh#L42

I've been running mypy python-package/.

Could you please setup mypy in lint task but allow it to fail (|| exit 0) so that everyone will be able to see logs and possibly prevent new issues to appear. We had the same strategy with cpplint. We had been checking logs manually but at some moment we simply removed || exit 0.
https://github.com/microsoft/LightGBM/blob/066720eff680606abca17a9b34d95905337d8afd/.ci/test.sh#L42

I've been running mypy python-package/.

Could you please setup mypy in lint task but allow it to fail (|| exit 0) so that everyone will be able to see logs and possibly prevent new issues to appear. We had the same strategy with cpplint. We had been checking logs manually but at some moment we simply removed || exit 0.
https://github.com/microsoft/LightGBM/blob/066720eff680606abca17a9b34d95905337d8afd/.ci/test.sh#L42

sure! Sounds like an excellent good first issue idea for people to fix warnings.

I've been running mypy python-package/.

Could you please setup mypy in lint task but allow it to fail (|| exit 0) so that everyone will be able to see logs and possibly prevent new issues to appear. We had the same strategy with cpplint. We had been checking logs manually but at some moment we simply removed || exit 0.
https://github.com/microsoft/LightGBM/blob/066720eff680606abca17a9b34d95905337d8afd/.ci/test.sh#L42

sure! Sounds like an excellent good first issue idea for people to fix warnings.


Added release 5.4.2400147

Updated release notes with release description. Fixed typo in date of the previous release.
Updated index to show reflect new release, date.

@amiramm Thanks!


Wallet locked

Hello
This morning I want to transfer funds from my Bitpay Wallet to my Bitpay card. I received the error message:
" Could not create transaction"
"WALLET_LOCKED: Wallet is locked"
What that means? I lost some money now because of the exchange rate. That is not accaptable!
What is the issue?

WALLET_LOCKED means that the wallet is been updated or certain operation is done on the wallet (like address scanning ) and you need to wait a bit to use it.


Nodes: Can not resolve #include <lights_pars>

This error is thrown by the nodes examples:
Uncaught Error: Can not resolve #include <lights_pars>
    at replace (VM253 three.js:10629)
    at String.replace (<anonymous>)
    at parseIncludes (VM253 three.js:10633)
    at new WebGLProgram (VM253 three.js:10848)
    at WebGLPrograms.acquireProgram (VM253 three.js:11131)
    at WebGLRenderer.initMaterial (VM253 three.js:14168)
    at WebGLRenderer.setProgram (VM253 three.js:14250)
    at WebGLRenderer.renderBufferDirect (VM253 three.js:13646)
    at WebGLRenderer.renderObject (VM253 three.js:14116)
    at WebGLRenderer.renderObjects (VM253 three.js:14099)

https://htmlpreview.github.io/?https://github.com/semleti/three.ts/blob/master/examples/webgl_materials_nodes.html
https://htmlpreview.github.io/?https://github.com/semleti/three.ts/blob/master/examples/webgl_mirror_nodes.html

Solved.
Missing ShaderChunk shaders.
"lights_pars" and "lights_template"

Hello! Could tell more details how this problem was resolve, please?

@Fgeniy
Globally set your shader chunck
THREE.ShaderChunk['lights_pars'] = your_shader_here;


Enable and document global components

As a user, I want to be able to:

Customize basic HTML tags like headings, em etc.
Make frequently used custom components available in all MDX pages, so that I don't need to import them each time I want to use them.

All this can be done by tweaking this object...
const components = {
  Head,
  pre: Pre,
  GlobalTest: ({children}) =>
{children}
,
  wrapper: ({ layout, ...rest }) => {
    const Layout = require(`../layouts/${layout}`).default
    return <Layout {...rest} />
  }
}

... but it would be nice to configure it in such a way that the user doesn't ever need to touch this file.

Low priority IMO. Waiting for requests for this.


🐛 [firebase_crashlytics] IOS can't be activated

Bug report
Describe the bug
Firebase won't enable IOS Crashlytics.

2021-07-11 09:10:51.594553+0200 Runner[276:4235] [connection] nw_endpoint_handler_set_adaptive_write_handler [C4.1.1 2a00:1450:4007:818::200a.443 ready channel-flow (satisfied (Path is satisfied), viable, interface: en0, ipv4, ipv6, dns)] unregister notification for write_timeout failed
2021-07-11 09:10:51.670451+0200 Runner[276:4220] 8.0.0 - [Firebase/Analytics][I-ACS025026] Deferring to Firebase for event data collection
[Crashlytics] The signal SIGABRT has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGBUS has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGFPE has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGILL has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGSEGV has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGSYS has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
[Crashlytics] The signal SIGTRAP has a non-Crashlytics handler (GADRegisterSignalHandlers).  This will interfere with reporting.
2021-07-11 09:10:52.874300+0200 Runner[276:4215] 8.0.0 - [Firebase/Crashlytics][I-CLS000000] Warning: NSUncaughtExceptionHandler is 'GADRegisterExceptionHandler' in '/private/var/containers/Bundle/Application/B6B00CCF-D671-43E1-BB53-7A1ACB79B1F6/Runner.app/Runner'
2021-07-11 09:10:55.150964+0200 Runner[276:4232] 8.0.0 - [Firebase/Analytics][I-ACS800023] No pending snapshot to activate. SDK name: app_measurement
2021-07-11 09:10:55.263445+0200 Runner[276:4214] 8.0.0 - [Firebase/Analytics][I-ACS023012] Analytics collection enabled

Steps to reproduce
Add Firebase core & Firebase Analytics & Firebase Crashlytics and follow instruction for IOS (Android work well).
Steps to reproduce the behavior:

Do all steps for each one of these package from flutter fire website
Run Xcode Release build
Look the console on xcode

Expected behavior
I would see no error and skip loading step on firebase crashlytics
Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
[✓] Flutter (Channel stable, 2.2.2, on macOS 11.3.1 20E241 darwin-arm, locale fr-FR)
    • Flutter version 2.2.2 at /Users/githubissue/Documents/Softwares/flutter
    • Framework revision d79295af24 (4 weeks ago), 2021-06-11 08:56:01 -0700
    • Engine revision 91c9fc8fe0
    • Dart version 2.13.3

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/githubissue/Library/Android/sdk
    • Platform android-30, build-tools 30.0.3
    • Java binary at: /Users/githubissue/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7351085/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5.1, Build version 12E507
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] Android Studio (version 4.2)
    • Android Studio at /Users/githubissue/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7351085/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Ultimate Edition (version 2021.1.1)
    • IntelliJ at /Users/githubissue/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] Connected device (1 available)
    • Chrome (web) • chrome • web-javascript • Google Chrome 91.0.4472.114
    ! Error: iPhone de githubissue is busy: Waiting for Device. Xcode will continue when iPhone de githubissue is finished. (code -10) (Iphone SE (1 gen) IOS 14.6)

• No issues found!

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 2.13.3
Flutter SDK 2.2.2
generations 8.2.4+86

dependencies:
- firebase_analytics 8.1.2 [firebase_analytics_platform_interface firebase_analytics_web firebase_core flutter meta]
- firebase_core 1.3.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_crashlytics 2.0.7 [firebase_core firebase_core_platform_interface firebase_crashlytics_platform_interface flutter stack_trace]
- flutter 0.0.0 [characters collection meta typed_data vector_math sky_engine]
- google_mobile_ads 0.13.2 [meta flutter]
- custom_template 0.0.0 [flutter cached_network_image http intl shared_preferences cupertino_icons just_audio audio_session audio_service url_launcher html app_settings flutter_local_notifications volume_controller flutter_inappwebview flutter_to_airplay device_info google_mobile_ads firebase_crashlytics collection provider share flutter_cache_manager]
- wonderpush_fcm_flutter 1.0.3 [flutter]
- wonderpush_flutter 2.1.3 [flutter]

dev dependencies:
- flutter_launcher_icons 0.9.0 [args image path yaml]
- flutter_native_splash 1.1.8+4 [image meta path xml yaml universal_io]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters charcode collection matcher meta source_span stream_channel string_scanner term_glyph typed_data]

transitive dependencies:
- app_settings 4.1.1 [flutter]
- archive 3.1.2 [crypto path]
- args 2.0.0
- async 2.6.1 [meta collection]
- audio_service 0.17.1 [audio_session rxdart flutter_isolate flutter_cache_manager js flutter flutter_web_plugins]
- audio_session 0.1.5 [flutter flutter_web_plugins rxdart meta]
- boolean_selector 2.1.0 [source_span string_scanner]
- cached_network_image 3.0.0 [flutter flutter_cache_manager octo_image]
- characters 1.1.0
- charcode 1.2.0
- clock 1.1.0
- collection 1.15.0
- crypto 3.0.1 [collection typed_data]
- csslib 0.17.0 [source_span]
- cupertino_icons 1.0.3
- device_info 2.0.2 [flutter device_info_platform_interface]
- device_info_platform_interface 2.0.1 [flutter meta plugin_platform_interface]
- fake_async 1.2.0 [clock collection]
- ffi 1.1.2
- file 6.1.2 [meta path]
- firebase 9.0.1 [http http_parser js]
- firebase_analytics_platform_interface 2.0.1 [flutter meta]
- firebase_analytics_web 0.3.0+1 [firebase firebase_analytics_platform_interface flutter flutter_web_plugins meta]
- firebase_core_platform_interface 4.0.1 [collection flutter meta plugin_platform_interface]
- firebase_core_web 1.1.0 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- firebase_crashlytics_platform_interface 3.0.6 [collection firebase_core flutter meta plugin_platform_interface]
- flutter_blurhash 0.6.0 [flutter meta pedantic]
- flutter_cache_manager 3.1.2 [clock collection file flutter http path path_provider pedantic rxdart sqflite uuid]
- flutter_inappwebview 5.3.2 [flutter]
- flutter_isolate 2.0.0 [flutter uuid]
- flutter_local_notifications 6.0.0 [flutter flutter_local_notifications_platform_interface platform timezone]
- flutter_local_notifications_platform_interface 4.0.0 [flutter plugin_platform_interface]
- flutter_to_airplay 2.0.0 [flutter]
- flutter_web_plugins 0.0.0 [flutter js characters collection meta typed_data vector_math]
- html 0.15.0 [csslib source_span]
- http 0.13.3 [async http_parser meta path pedantic]
- http_parser 4.0.0 [charcode collection source_span string_scanner typed_data]
- image 3.0.2 [archive meta xml]
- intl 0.17.0 [clock path]
- js 0.6.3
- just_audio 0.8.0 [just_audio_platform_interface just_audio_web audio_session rxdart path path_provider async uuid crypto meta flutter]
- just_audio_platform_interface 3.1.0 [flutter plugin_platform_interface]
- just_audio_web 0.3.2 [just_audio_platform_interface flutter flutter_web_plugins]
- matcher 0.12.10 [stack_trace]
- meta 1.3.0
- mime 1.0.0
- nested 1.0.0 [flutter]
- octo_image 1.0.0+1 [flutter flutter_blurhash]
- path 1.8.0
- path_provider 2.0.2 [flutter path_provider_platform_interface path_provider_macos path_provider_linux path_provider_windows]
- path_provider_linux 2.0.0 [path xdg_directories path_provider_platform_interface flutter]
- path_provider_macos 2.0.0 [flutter]
- path_provider_platform_interface 2.0.1 [flutter meta platform plugin_platform_interface]
- path_provider_windows 2.0.1 [path_provider_platform_interface meta path flutter ffi win32]
- pedantic 1.11.1
- petitparser 4.1.0 [meta]
- platform 3.0.0
- plugin_platform_interface 2.0.1 [meta]
- process 4.2.1 [file path platform]
- provider 5.0.0 [collection flutter nested]
- rxdart 0.27.1
- share 2.0.4 [meta mime flutter]
- shared_preferences 2.0.6 [meta flutter shared_preferences_platform_interface shared_preferences_linux shared_preferences_macos shared_preferences_web shared_preferences_windows]
- shared_preferences_linux 2.0.0 [flutter file meta path path_provider_linux shared_preferences_platform_interface]
- shared_preferences_macos 2.0.0 [shared_preferences_platform_interface flutter]
- shared_preferences_platform_interface 2.0.0 [flutter]
- shared_preferences_web 2.0.0 [shared_preferences_platform_interface flutter flutter_web_plugins meta]
- shared_preferences_windows 2.0.0 [shared_preferences_platform_interface flutter file meta path path_provider_platform_interface path_provider_windows]
- sky_engine 0.0.99
- source_span 1.8.1 [collection path term_glyph]
- sqflite 2.0.0+3 [flutter sqflite_common path]
- sqflite_common 2.0.0+2 [synchronized path meta]
- stack_trace 1.10.0 [path]
- stream_channel 2.1.0 [async]
- string_scanner 1.1.0 [charcode source_span]
- synchronized 3.0.0
- term_glyph 1.2.0
- test_api 0.3.0 [async boolean_selector collection meta path source_span stack_trace stream_channel string_scanner term_glyph matcher]
- timezone 0.7.0 [path]
- typed_data 1.3.0 [collection]
- universal_io 2.0.4 [collection crypto meta typed_data]
- url_launcher 6.0.9 [flutter meta url_launcher_linux url_launcher_macos url_launcher_platform_interface url_launcher_web url_launcher_windows]
- url_launcher_linux 2.0.0 [flutter]
- url_launcher_macos 2.0.0 [flutter]
- url_launcher_platform_interface 2.0.4 [flutter plugin_platform_interface]
- url_launcher_web 2.0.1 [flutter flutter_web_plugins meta url_launcher_platform_interface]
- url_launcher_windows 2.0.0 [flutter]
- uuid 3.0.4 [crypto]
- vector_math 2.1.0
- volume_controller 2.0.2 [flutter]
- win32 2.2.5 [ffi]
- xdg_directories 0.2.0 [meta path process]
- xml 5.1.2 [collection meta petitparser]
- yaml 3.1.0 [collection source_span string_scanner]

I finally achieve to enable it after  24h. I stop use this method to test :
FirebaseCrashlytics.instance.crash();

I use this to activate crashlytics on ios app :
try {
        throw Exception("test crash");
      } catch(error, stack) {
        FirebaseCrashlytics.instance.recordError(
            error,
            stack,
            reason: 'a test crash',
        );
      }


Structured events - TemplateEnumerator

Have merged https://github.com/NLog/NLog.StructuredEvents together with https://github.com/NLog/NLog
And then introduced a TemplateEnumerator, that only allocates when a proper message-template (non-positional)
Stolen the unit tests from #2126

@304NotModified I know that i have stolen your code, and this is mostly a proof-of-concept. But maybe you can take some of the ideas in this PR to hammer the last nails in place for NLog 4.5.
Not sure I like exposing the Hole-structs directly. Would prefer they were just "Index" + "Name" + "Format". But I like reusing the Hole-array-allocation, instead of creating a new container.

Not sure I like exposing the Hole-structs directly.

My idea was to use the event-properties for that. The holes renders fills the properties and if desired, users could overwrite them. Less complicated for the user and can use all the targets/layout renders which are built on the event-properties.

My idea was to use the event-properties for that.

Have also implemented that by providing the new property LogEventInfo.ParameterProperties.
But I recall this PR #1962 where someone wanted access to the holes and their format-options, and not just the parameter-names and the parameter-objects

But I recall this PR #1962 where someone wanted access to the holes and their format-options, and not just the parameter-names and the parameter-objects

I was under the impression that your proposal for that was: reparse the message.

prefer to keep the structured event logic in a separate package because:

Since the NLog-engine is going to be fully integrated with the structured-event-logic, then I find it hard to split them without it becoming very artificial.
I understand the need to rip out all the random targets and layout-renderers, but the core engine should stay in one package.

I was under the impression that your proposal for that was: reparse the message.

My initial thought was to provide a way to request a single parameter to be formatted as string, like specified in the message template. But after seeing that #1962 wants to use his own target-specific-formatter, then my initial thought would be too restrictive.

We lost the parse and render tests from https://github.com/NLog/NLog.StructuredEvents?

We lost the parse and render tests from https://github.com/NLog/NLog.StructuredEvents?

I will inject them

I understand the need to rip out all the random targets and layout-renderers

The idea is now, new code is in new packages and current stuff will be separated after NLog 5 RTM

But when looking at ASP.NET-CORE loggers that seems prepared for Structured-logging, then I find it less optimal that the NLog-core is not able to handle structured-logging out of the box.

It's still out of the box when those are multiple packages. I will think about it, but currently i'm not a fan to merge them.

Else we need to improve the LogEventInfo-Properties even further so one can have internal meta-properties

I don't see why we need that. Just write to the properties, drop everything else.
I agree that having the Tempate inside the EventLogInfo was not the best solution, but IMO it's now even less optimal. It too complicated. I really don't like to expose a Hole list in the LogEventInfo.

I really don't like to expose a Hole list in the LogEventInfo.

Glad you feel the same way :). Was not sure how far you wanted to go to cater for nlog-targets-seq.
But I don't like pushing the parameter-names directly into LogEventInfo.Properties, because of the name-clashing (that you have also tried to handle). It will give unexpected behavior, no matter what you do.
Instead of having the ParameterHoles. Then we could have something called ParameterNames, that is just a IList ?

But I don't like pushing the parameter-names directly into LogEventInfo.Properties, because of the potential name-clashing

IMO the structured syntax is another way to define properties. I don't see the benefit of having properties and parameters. Those are on the same level (logevent level) and so separation is only over complicating things IMO.

I don't see the benefit of having properties and parameters.

Then one will just be surprised the day one have a custom-logger than always inject some custom-properties, and that logger is suddenly presented with a message-template that clashes.
Also I like how one can get direct mapping from the parameter-names to the parameter-values without allocating a dictionary.
What if we introduce a new MetaProperties-dictionary (that extends the object-key with a meta-type-enum). This will give one properties-collection, that will just allow "duplicate" object-keys. The existing Properties-dictionary will then just return the values included from meta-type-property (and not meta-type-parametername)

Then one will just be surprised the day one have a custom-logger that always inject some custom-properties, and that logger is suddenly presented with a message-template that clashes.

That would be also the case now, as I could set the logevent properties with C#
We have now the following contexts:

GDC
MDC & MDLC
NDC & NDLC
event properties
event context (deprecated)

Adding a new on event level is makes it even more complex. That the event context is already deprecated is a burden. Only if it's absolute necessary I would consider it.

PS:

Merging #2208 into master will decrease coverage by <1%.
The diff coverage is 80%.

Another issue with merging here, I cannot see easily if the new code is tracked by the unit tests. It doesn't look like it (it was in the other github repos 95% coverage)
But maybe that's an issue that we have to tackle differently, maybe sending the code coverage to sonar would help (but I did not get that working)

That would be also the case now, as I could set the logevent properties with C#

Usually the logger will just receive the logevent-message and parameters, and not any properties. But now it is possible for the logger to unwittingly overwrite the properties introduced by message-template.

Another issue with merging here, I cannot see easily if the new code is tracked by the unit tests

Well that is how proof-of-concept works for me, first you shuffle things around until it looks good, and then you lock things down with unit-tests (and remove unneeded code).

@304NotModified Have now removed ParameterHoles, and now injecting message-template properties when asking for the normal Properties-dictionary.

@304NotModified Made a very simple performance test of this PR:

Logger Name
Messages
Size
Args
Threads
Loggers

SimpleLogger
5.000.000
128
2
1
1

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-Standard
21.361
234.061
0
18
37
23.015
113,258

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-Positional
24.747
202.036
0
18
36
26.156
118,078

The extra CPU time in"NLog-Positional" is caused by the scanning if the template is positional or not.

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

Serilog-Positional
34.779
143.762
0
32
64
69.156
102,852

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-Template
34.724
143.989
0
87
175
37.687
130,496

The extra allocations in"NLog-Template" is caused by allocation of mesage-template and stringbuilder for formatting.

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

Serilog-Template
34.039
146.888
0
32
65
67.750
100,773

I think serilog has an optimization where it caches the MesageTemplates, which helps Serilog in this test. I also think Serilog doesn't allocate the formatted string, but writes it directly to its TextWriter.
NLog could probably improve its performance by not allocating a StringBuilder for every template-format-operation, and maybe attempt to cache the recent message-templates (together with the hole-strings).

Cool! Can I see the test cases code some where?
We like to use cache, it was always in the plan. Are you using the mru cache in the test?
Serilog uses a dictionary with a 1000 limit and full clear when over limit.

Cool! Can I see the test cases code some where?

It is just the performance test-project where I'm injecting a log-message-arg (with a hole-name of 25 chars) for every 64 char (Thus one in the middle and one at the end of the 128 char log-message).
The serilog-config is this:
var serilogConfig = new Serilog.LoggerConfiguration().WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog.txt"));
var serilogger = serilogConfig.CreateLogger();

We like to use cache, it was always in the plan. Are you using the mru cache in the test?

No the two bonus perf-tests is just me hardcoding things to see how caching affect allocations. The MruCache is not super-concurrent and doesn't expect a constant high insert-rate. All Loggers will hit this code-path and NLog has made an effort in not grapping a global-lock. I have not made any concurrency-tests to see how things scales as the thread-count rises.

Serilog uses a dictionary with a 1000 limit and full clear when over limit.

They have changed to a HashTable as it allows them to do the lookup without holding a monitor-lock,

They have changed to a HashTable as it allows them to do the lookup without holding a monitor-lock.

Ow yes indeed: https://github.com/serilog/serilog/blob/dev/src/Serilog/Core/Pipeline/MessageTemplateCache.cs

An alternative approach would be to just cache the parameter-hole-names (for the property-dictionary). Since the MessageTemplate is just discarded after the rendering. Then modify the TemplateEnumerator to render without allocating any MessageTemplate (Would not be an easy task). The number of unique hole-names are probably much less than the number of unique-messages.

Sounds complicated and not really needed IMO

No the two bonus perf-tests is just me hardcoding things to see how caching affect allocations. The MruCache is not super-concurrent when there is a constant high insert-rate.

Maybe it's a good idea to start with this one? What do you think?

The MruCache is not super-concurrent when there is a constant high insert-rate.
Maybe it's a good idea to start with this one? What do you think?

Sure go ahead, you just have to find a lock-free-container with fast lookup, fast insert, fast cleanup and minimum allocation :)
The above test was merely to see how far from the reference-point (Serilog) the current implementation is. Right now it looks pretty close, so maybe it is just a matter of improve the template-parser, and NLog is ready.
Before you start on implementing this caching business, then we might want to create another test, that pushes the concurrency a little:

Have 6 threads that each are writing to their own log-level
Each log-level is redirected to their individual target/sink
Each of the 6 threads will shuffle between 300 unique log-messages.

For me the low hanging fruit is implementing a StringBuilder-cache, so that will probably be next two tasks (builder-cache + new test).

Yes it cannot be that complicated to find a lock-free-container with fast lookup, fast insert, fast cleanup and minimum allocation :)

Before you start on implementing this caching business, then we might want to create another test, that pushes the concurrency a little (Closer to what a web-application might do):

It's a bit hard to add commits on another fork ;)

Made a concurrency test of Serilog and NLog, where I have 6 threads doing random logging between 2000 log-messages at 5 log-levels.

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-6T
30.309
79.181
2
103
205
107.250
172,246

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

Serilog-6T
71.380
33.622
7
78
479
279.859
160,824

My machine only has 4 real cores. So also tried the test with only 4 cores:

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-4T
18.213
131.772
3
67
132
63.718
172,730

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

Serilog-4T
47.075
50.981
19
73
255
182.359
148,590

This is the Serilog-config used:
            var serilogConfig = new Serilog.LoggerConfiguration().MinimumLevel.Verbose()
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Verbose)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Verbose.txt")))
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Debug)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Debug.txt")))
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Information)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Info.txt")))
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Warning)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Warn.txt")))
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Error)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Error.txt")))
                .WriteTo.Logger(config => config
                    .MinimumLevel.Verbose()
                    .Filter.ByIncludingOnly(e => e.Level == Serilog.Events.LogEventLevel.Fatal)
                    .WriteTo.Async(a => a.File(@"C:\Temp\Log\Serilog-Fatal.txt")))
                    ;

Made some profiling of Serilog and NLog just see how the Serilog-Template-Cache worked out. It spent as much CPU-time on cache-lookups, as NLog spent on parsing. So the caching implemented in Serilog only helps when always logging the same 1000 messages over and over (with different args).
This is the result when doing random logging of 100 different messages using 4 threads (Allow the caching to work):

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

Serilog-4T
30.061
79.837
5
68
141
116.921
156,582

Unless we can find a magical cache-container, then I don't think NLog should go down that road. Prefer the cpu-overhead of re-parsing, instead of the increased allocations with blocking garbage-collections to follow.

Have now implemented a very custom dictionary, that is just a combination of two linked lists (Holes + Meta). The linked-list works best for small collections below 100 items. This allows one to clear all the Holes while leaving the Meta-data untouched.

Test Name
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-4T
12.585
190.693
1
29
58
46.359
170,984

Great work! I'm a bit busy this week but will try to check it in depth soon. Would be great if I could also run the tests.

Would be great if I could also run the tests.

I have made a lot of hacks to the PerformanceTest-project, and added dependency to Serilog. Not sure I want to add those changes to a PR.
Maybe also a good idea, that a different set of eyes conjure up a performance test. Maybe you will find something that I didn't

I have made a lot of hacks to the PerformanceTest-project, and added dependency to Serilog. Not sure I want to add those changes to a PR.

In another branch sounds good to me.

In another branch sounds good to me.

Well now you can have some fun with this random code (Modifications to NLogPerformance-project):
NLogPerformance.zip

@304NotModified The changes made to LogEventInfo (new dictionary, new formatter) will probably be binary breaking, so it is not possible have NLog 4.4 running side-by-side NLog 4.5.
Maybe the NLog 4.5 and NLog 5.0 should be merged together, and just become NLog 5.0 ?

In another branch sounds good to me.

Well now you can have some fun with this random code (Modifications to NLogPerformance-project):
NLogPerformance.zip

@304NotModified We might also need to add some restriction in the Json-Serializer (And Value-Renderer), so if they get hold of a ienumerable/idictionary that contains more than 100.000 items, then it will not try and serialize them all.
Can also see that the Value-Renderer does not have a cyclic-loop-level-restriction implemented (Like the Json-Serializer has).

@snakefoot
I'm sorry but I haven't found time to test this. Currently very busy with moving to a new house. (and got some delays)

@304NotModified Don't sweat. Houses are never easy, and always gives delays :) And having a roof is usually more important than having a git-repository ;)

@304NotModified Considering to change the implementation of the parameter-holes-dictionary into a linked list. The Properties-dictionary remains a union of both. Then create a new property on LogEventInfo allowing one to get/set the ParameterNames/ParameterKeys/ParameterHoles-ienumerable.
This will allow one to get the parameter-names directly related to the log-message, and in the correct order so it matches the LogEventInfo-Parameters-array.

Made a concurrency test of Serilog and NLog, where I have 6 threads doing random logging between 2000 log-messages at 5 log-levels.

Just a quick note, WriteTo.Logger has a performance penalty in Serilog, so it's probably not the best basis for the comparison. Not sure it's really important (looks like your perf is really healthy!) but the config below is closer to how Serilog would work for this particular scenario:
new LoggerConfiguration()
    .MinimumLevel.Verbose()
    .WriteTo.Async(writeTo =>
        writeTo.Map(le => le.Level, (level, wt) =>
            wt.File(@"C:\Temp\Log\Serilog-" + level + ".txt")))
    .CreateLogger();

Cheers!

@nblumhardt Just a quick note, WriteTo.Logger has a performance penalty in Serilog, so it's probably not the best basis for the comparison

Thank you for the suggestion. When having merged with main-branch again, then I will run the performance test one more time.

Repeated the performance tests with the updated code:

Structured Logging
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-4T
6.403
780.815
1
16
32
22.437
187,012

Serilog-4T
30.013
166.591
0
33
67
115.984
170,367

Serilog-4T-100
28.320
176.549
0
21
43
111.828
111,457

Positional Logging
Time (ms)
Msgs/sec
GC2
GC1
GC0
CPU (ms)
Mem (MB)

NLog-4T
6.491
770.200
1
12
23
17.843
165,109

Serilog-4T
29.541
169.251
1
17
42
116.656
121,934

Serilog-4T-100
28.441
175.796
0
22
44
111.453
110,777

Using the updated Serilog-Configuration have certainly helped on the allocations, so it now matches NLog. But NLog is still doing good after all the merge.

Serilog.Sinks.Async moves all work to a single background worker, so this'd explain the core utilization. Looks like NLog is doing really well! 👍

great results @snakefoot !

@snakefoot do you prefer comments or git pushes from me?
Any way, please stop force push for now, i'm losing some comments after a force push. We can squash at the end.

@304NotModified Whatever works for you. You can also just merge it, and make the changes you want afterwards.
I have been thinking about renaming the folder StructuredEvents into "MessageTemplate"

Renaming is fine to me!

Perform folder move with force-push to avoid too much noise. I have updated the code according to your comments, except the MessageTemplate.ISerializer-logic.

@304NotModified I think NLog is about to have a lot of parameter/property-renderers. I'm thinking that we only need three:

The IJsonConverter that should always render in JSON-format.
The stringify object.ToString() with IFormatProvider support.
The new IPropertyRenderer that just converts an object into custom string-format

It should be possible to specify in the MessageTemplate which of the 3 types should be used.

have renamed the namespaces to MessageTemplates (plural)

Use plural namespace names if it is semantically appropriate.

https://msdn.microsoft.com/en-us/library/893ke618(v=vs.71).aspx
;)

oops broke it. Don't fully understand the code then ;) Fixing it

don't get it. LogEventTemplateShouldHaveProperties_even_when_changed fails locallly, even on dedd1e2!

ow it only works when I run all tests from LoggerTests.cs at once.

@snakefoot
I don't see it where exactly is the side-effect, but LogEvent.HasMessageTemplateParameters has a side effect. Could you try to make this more explicit?

@304NotModified I don't see it where exactly is the side-effect, but LogEvent.HasMessageTemplateParameters has a side effect.

Cannot see it either, but this call has a sideeffect:
!ReferenceEquals(this.Message, this.FormattedMessage)
It renders the formatted message, and captures properties (if any)

@304NotModified Interesting. Your test LogAfterAwait_CleanNamesOfAsyncContinuationsIsFalse_ShouldNotCleanNames seems to be unstable also on AppVeyor.

@304NotModified Interesting. Your test LogAfterAwait_CleanNamesOfAsyncContinuationsIsFalse_ShouldNotCleanNames seems to be unstable also on AppVeyor.

Don't get it. But that's the risk of stack walking. Feel free to skip the test.

@304NotModified Do we need more changes. Or can I perform a squash of the commits? (Btw. updated the Serilog performance numbers, after I disabled its lossy default behavior).

I prefer to check it again, but don't think I will find big issues. Currently working on NLog.web which should be asp.net core 2 ready asap.
https://github.com/NLog/NLog/pull/2208#issuecomment-334583579 -> this will be another PR?

#2208 (comment) -> this will be another PR?

Maybe start as an issue, and when having decided on something, then create a PR. I have not seen any confirmation or rejection of any of my statements :)

I have not seen any confirmation or rejection of any of my statements :)

Unfortunately hadn't find time for that.

@304NotModified Think I have found an okay solution for making it possible to provide an external override of the object-serialization.

@304NotModified Squashed the commits and performed forced push.

bam!
Sorry for the delay.
thx!

@304NotModified No issue. Glad that it could be approved.
I have found some extra modification, that I would like to include. See #2319

This is cool.  Just found out about Serilog and got led here, I already use NLog all the time.
Does the target only support going to a Seq server?  Will it be possible to just dump into a file?

@ericnewton76 NLog extracts the named parameters from the structured-message, and injects them into existing collection of properties, these can be used by any type of Target/Layout. Please post your comments and questions here instead: #2332

PS for seq there is https://www.nuget.org/packages/NLog.Targets.Seq/1.0.0-dev-00032


Subclassed layers not automatically saved to checkpoint in TF2.2

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.1 and 2.2
Python version: 3.6 and 3.8
CUDA/cuDNN version: 10.2 / 7.6.5
GPU model and memory: TITAN X (Pascal) 12 GB

Describe the current behavior
With TF 2.1, I was able to subclass the tf.keras.Sequential class as part of a model and checkpoints would save fine as far as I knew.
With TF 2.2, I saw that tf.keras.Sequential layers are not tracked in the checkpoint unless the layers are also set as attribute of the subclass.
Describe the expected behavior
I'm curious to know if the behavior in TF 2.2 is the intended behavior.
There are workarounds like manually setting the attributes or instantiating a tf.keras.Sequential.
Standalone code to reproduce the issue
from pprint import pprint

import tensorflow as tf

class CustomSequential(tf.keras.Sequential):
    def __init__(self, set_attributes=False, name='CustomSequential', **kwargs):
        super().__init__([
            tf.keras.layers.Conv2D(2, 3, 2, name='conv2'),
            tf.keras.layers.Conv2D(4, 3, 2, name='conv3'),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(10)
        ], name=name, **kwargs)

        if set_attributes:
            self.conv1 = self.layers[0]
            self.conv2 = self.layers[1]
            self.dense = self.layers[3]

class CustomModel(tf.keras.Model):
    def __init__(self, set_attributes, name='CustomModel', **kwargs):
        super().__init__(name=name, **kwargs)
        self.convolution = tf.keras.layers.Conv2D(8, 3, name='conv1')
        self.sequential = CustomSequential(set_attributes, name='MySequential')

    def call(self, inputs):
        net = self.convolution(inputs)

        return self.sequential(net)

print('TensorFlow version')
print(tf.version.GIT_VERSION, tf.version.VERSION)
for set_attributes in [True, False]:
    print(f'Set sequential layer attributes: {set_attributes}')
    inputs = tf.random.uniform([5, 16, 16, 3])
    model = CustomModel(set_attributes)
    outputs = model(inputs)

    checkpoint = tf.train.Checkpoint(step=tf.Variable(1), model=model)
    manager = tf.train.CheckpointManager(checkpoint, f'./custom_model_{set_attributes}', max_to_keep=3)
    manager.save()

    pprint(tf.train.list_variables(manager.latest_checkpoint))
    print('------------------')

TF 2.1 Result
TensorFlow version
v2.1.0-rc2-17-ge5bf8de 2.1.0
Set sequential layer attributes: True
[('_CHECKPOINTABLE_OBJECT_GRAPH', []),
 ('model/convolution/bias/.ATTRIBUTES/VARIABLE_VALUE', [8]),
 ('model/convolution/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 3, 8]),
 ('model/sequential/layer-0/bias/.ATTRIBUTES/VARIABLE_VALUE', [2]),
 ('model/sequential/layer-0/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 8, 2]),
 ('model/sequential/layer-1/bias/.ATTRIBUTES/VARIABLE_VALUE', [4]),
 ('model/sequential/layer-1/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 2, 4]),
 ('model/sequential/layer-3/bias/.ATTRIBUTES/VARIABLE_VALUE', [10]),
 ('model/sequential/layer-3/kernel/.ATTRIBUTES/VARIABLE_VALUE', [16, 10]),
 ('save_counter/.ATTRIBUTES/VARIABLE_VALUE', []),
 ('step/.ATTRIBUTES/VARIABLE_VALUE', [])]
------------------
Set sequential layer attributes: False
[('_CHECKPOINTABLE_OBJECT_GRAPH', []),
 ('model/convolution/bias/.ATTRIBUTES/VARIABLE_VALUE', [8]),
 ('model/convolution/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 3, 8]),
 ('model/sequential/layer-0/bias/.ATTRIBUTES/VARIABLE_VALUE', [2]),
 ('model/sequential/layer-0/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 8, 2]),
 ('model/sequential/layer-1/bias/.ATTRIBUTES/VARIABLE_VALUE', [4]),
 ('model/sequential/layer-1/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 2, 4]),
 ('model/sequential/layer-3/bias/.ATTRIBUTES/VARIABLE_VALUE', [10]),
 ('model/sequential/layer-3/kernel/.ATTRIBUTES/VARIABLE_VALUE', [16, 10]),
 ('save_counter/.ATTRIBUTES/VARIABLE_VALUE', []),
 ('step/.ATTRIBUTES/VARIABLE_VALUE', [])]
------------------

TF 2.2 Result
TensorFlow version
v2.2.0-rc4-8-g2b96f3662b 2.2.0
Set sequential layer attributes: True
[('_CHECKPOINTABLE_OBJECT_GRAPH', []),
 ('model/convolution/bias/.ATTRIBUTES/VARIABLE_VALUE', [8]),
 ('model/convolution/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 3, 8]),
 ('model/sequential/conv1/bias/.ATTRIBUTES/VARIABLE_VALUE', [2]),
 ('model/sequential/conv1/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 8, 2]),
 ('model/sequential/conv2/bias/.ATTRIBUTES/VARIABLE_VALUE', [4]),
 ('model/sequential/conv2/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 2, 4]),
 ('model/sequential/dense/bias/.ATTRIBUTES/VARIABLE_VALUE', [10]),
 ('model/sequential/dense/kernel/.ATTRIBUTES/VARIABLE_VALUE', [16, 10]),
 ('save_counter/.ATTRIBUTES/VARIABLE_VALUE', []),
 ('step/.ATTRIBUTES/VARIABLE_VALUE', [])]
------------------
Set sequential layer attributes: False
[('_CHECKPOINTABLE_OBJECT_GRAPH', []),
 ('model/convolution/bias/.ATTRIBUTES/VARIABLE_VALUE', [8]),
 ('model/convolution/kernel/.ATTRIBUTES/VARIABLE_VALUE', [3, 3, 3, 8]),
 ('save_counter/.ATTRIBUTES/VARIABLE_VALUE', []),
 ('step/.ATTRIBUTES/VARIABLE_VALUE', [])]
------------------

This may be related to #37839.

I have tried in colab with TF-GPU version 2.1.0 , 2.2.0 and was able to reproduce the issue.Please, find the gist here.Thanks!

I am not seeing any issue with TF nightly version(2.3.0-dev20200520).Please, find the gist here..You could use tf-nightly for now and in the next couple of months new stable version will be released .Please, close this thread if this solves your question. Thanks!

Good to know it should no longer be an issue. Thanks for the quick response.