193 modularize error handling into a separate module

PR Description
Hey team,
This PR introduces a middleware for error handling in our Node.js application. Currently, our application lacks a centralized mechanism for handling errors, making it difficult to manage and respond to errors consistently. To address this, we have implemented a middleware that will handle errors in a centralized manner.
Changes Made

Created a new middleware for error handling in our Node.js application.
The error handling middleware captures any errors that occur during the request/response lifecycle and provides a centralized mechanism for handling them.
The middleware ensures that errors are properly logged and appropriate responses are sent back to the client, with consistent error messages and status codes.
The error handling middleware is added to the application's middleware stack to ensure it is executed for every incoming request.

Notes for Reviewers

Kindly review the changes made to ensure that the error handling middleware effectively captures and handles errors in a centralized manner.
Verify that the middleware is correctly integrated into the application's middleware stack.
Pay attention to the logging of errors and the consistency of error messages and status codes.
If there are alternative approaches or suggestions to improve the implementation, please feel free to provide feedback.

Thank you for your time and consideration. Your feedback and suggestions are highly appreciated.
Best regards,
Louay HICHRI

Your error handling functions are well-structured and cover the most common scenarios you'd need to handle in an Express.js application. However, there are a couple of improvements that could make the code even better:
Specific error handling: The errorHandler function treats all errors as equal. However, in practice, you might want to handle specific errors differently. For example, if a validation error occurs, you might want to send a 400 status code instead of 500. One way to do this is to check the error type or name and add conditional logic for different error types.
Error logging: It's often useful to log errors for debugging or auditing purposes. You could add code to log the errors to the console, a file, or a logging service.
Here's an updated version of the errorHandler function incorporating these improvements:
`const errorHandler = (err, req, res, next) => {
// Log the error (could also be to a file or external service)
console.error(err);
if (err) {
    let status = err.status || 500;
    let message = err.message || 'An error occurred during the request.';
    let name = err.name || 'Error';

    // Handle specific error types
    if (err.name === 'ValidationError') {
        status = 400;
        message = err.message;
    }

    const body = {
        name,
        message,
        status
    }

    // render the error page
    res.status(status).json(body);
} else {
    next();
}

}
`

Thank you for reviewing the error handling functions in our Express.js
application. I appreciate your feedback and suggestions for improvement. I
wanted to provide an update on the changes made based on your
recommendations.
I have made the following modifications to the errorHandler function:
- Added console.log statements to log errors specifically for the
testnet environment.
- Removed the check for error type, allowing us to view all errors in
the logs.
These changes will allow us to log errors in the testnet environment for
better debugging and troubleshooting purposes. By removing the check for
error type, we can capture and review all errors that occur.
PR : https://github.com/SaTT-Wallet/Backend/pull/226
Le ven. 26 mai 2023 à 12:49, Bros G @.***> a écrit :

Your error handling functions are well-structured and cover the most
common scenarios you'd need to handle in an Express.js application.
However, there are a couple of improvements that could make the code even
better:
Specific error handling: The errorHandler function treats all errors as
equal. However, in practice, you might want to handle specific errors
differently. For example, if a validation error occurs, you might want to
send a 400 status code instead of 500. One way to do this is to check the
error type or name and add conditional logic for different error types.
Error logging: It's often useful to log errors for debugging or auditing
purposes. You could add code to log the errors to the console, a file, or a
logging service.
Here's an updated version of the errorHandler function incorporating these
improvements:
`const errorHandler = (err, req, res, next) => {
// Log the error (could also be to a file or external service)
console.error(err);
if (err) {
let status = err.status || 500;
let message = err.message || 'An error occurred during the request.';
let name = err.name || 'Error';
// Handle specific error types
if (err.name === 'ValidationError') {
    status = 400;
    message = err.message;
}

const body = {
    name,
    message,
    status
}

// render the error page
res.status(status).json(body);

} else {
next();
}
}
`
—
Reply to this email directly, view it on GitHub
https://github.com/SaTT-Wallet/Backend/pull/221#issuecomment-1564272374,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5UYTAHG6J5TSLHBODMNF63XICKF7ANCNFSM6AAAAAAYQALZCM
.
You are receiving this because you authored the thread.Message ID:
@.***>


NTR request "massively parallel reporter assay"

OBI:0002675	massively parallel reporter assay
New term is last one in sheet.
One issue is that my edit using MS Excel seemed to have a very inneficient diff - all rows of previous file removed, and all new ones added + new row.   The README has information: https://github.com/obi-ontology/obi#line-endings
Can you please try that and see if it fixes the diff?


bump electron version to v1.0.1

Electron has been updated to v1.0.1 :smile:
Relevant:
http://electron.atom.io/blog/2016/05/11/electron-1-0

Thanks for the PR!
For electron versions, we use an explicit version (i.e. no semver ranges like ^ since there's high potential for breakage and we want to upgrade explicitly.)
I'll update the version in a separate PR.

https://github.com/feross/webtorrent-desktop/pull/488


ES6 Arrow functions sample (fixes #148)

r: @jeffposnick

Thanks, reviewed. Sorry that the starter template leaves some confusion about what's required vs. what's just there to be illustrative.

No worries at all. We launch and iterate :) Feedback addressed. PTAL when you have a chance.

:+1:


Support for MOD150

Please explain your feature request in a one or two sentences.
MOD150 is recognzized by the plugin and can be switched off. Lights also work but the purifer can't be turned on. Any chance to add this to supported devices?

Is your feature request related to a problem? Please describe.

Any particular Meross devices that this relates to?
MOD150

Anything else?

Same here :-( Hope you can add support for this device soon
+1


net.eval for dropout

Using net.eval() for fixing the BN layer parameters make the dropout layer to act as in test time. Shouldn't only be used for BN layers only and not the whole network?

Our Resnet model does not have dropout, so it is fine here.

Oh, my mistake I saw it in the code code and I thought it was used.


Add DEFAULT_RENEW variable

Allow to override day to renew the cert.
Certificate from some CA like Buypass has 180 days lifetime, so you might want to renew it at day 170 instead of day 60
See https://github.com/acmesh-official/acme.sh/wiki/BuyPass.com-CA

Hi. Thank for the PR, unfortunately I'm currently blocked by #887

@buchdag
I just add test unit, could you please review ?

This would be really helpful, especially for in-house CAs with a even shorter forced refresh cycle than the default 60 days. acme-companion does currently not work with shorter CA policies, that issue certificates only for 30 days. @buchdag

Something I could do to help move this along?


Stream with small gap at start keeps buffering on very first play

We have come across an issue with the fix that resolved https://github.com/google/shaka-player/issues/600.
We noticed that this issue still occurs during the very first playback attempt of a stream with a small gap at the start. This was reproducible in the demo page of the latest release, 2.0.8.
Reproduction Steps:

Create a new user profile and clear cache (may also happen in incognito mode)
Go to demo page
Load https://akadash0-test.akamaized.net/test/clear/test-multi-audio/master.mpd. This should now be whitelisted. Or use the stream you used to test this change.

Expected: Stream plays
Actual: Stream infinitely buffers

@joeyparrish Looks like it still not available outside of our network. Sorry for not getting back to you sooner. We were able to create a workaround that resolved the issue based on our findings. After testing it in v2.1.0, it looks like we ended up coming up with the same solution. We noticed that this would occur when the gap was before the start time and so we would seek to the start time.
We should be able to remove the workaround once we migrate to v2.1.0. Thanks!


Convert dev_docs.rst to markdown

The link on dev_install.md was broken. This fixes it and makes its format consistent with the other docs that are written in markdown.

@jasongrout Care to merge?

@gnestor - if you have time to look at the other PRs I've been submitting, I'd love another set of eyes on them.

Just one small comment, and I think this is good to merge.

Updated! 👌

👍


Calendar mistakes clicks for drags on touchpad

When I click on an event in month view, the calendar mistakes my click for a drag, and does not open the edit dialog. However, tapping the event on my touchscreen works as expected.
This is on a Dell XPS 15.

What browser?

Chrome

Thanks. Devices that have mouse and touch events are pretty tricky to handle, I'm just mulling it over at this point.

Is it possible to use the @click.native along with what you already have for handling drag events?

In this instance, adding a native modifier won't fix it.


Fatal error: Uncaught Error: Call to undefined function split()

Con PHP 7.x la funzione split() deve essere sostiuita con explode(), preg_split() o str_split()
https://github.com/levith1/sofaskin-CW9009-italiano/blob/master/text.php.tmpl#L17
Fonte: http://php.net/manual/en/function.split.php

I have fixed this issue in my latest release of the skin. Split() is no longer supported so I used preg_split(). https://github.com/weatherstorm/Sofaskin-CW9009

It looks like the latest version of sofaskin-CW9009-italiano has the change as well.


Added android XML images support

So, now you can use PDF images for iOS and XML for android
I apologize for my code in advance =)
I think it can be useful for somebody.

@daniel-luberda
Sorry for disturb.
Have you looked through this PR yet?

@AndreiMisiukevich Sorry, not yet - I'll need to refactor some things here, so I'll need some more time for that (after fixing other issues)

I would prefer the same approach as SVG files. IVectorDataResolver should be used:
    public interface IVectorDataResolver : IDataResolver
    {
        int VectorWidth { get; }

        int VectorHeight { get; }

        bool UseDipUnits { get; }

        Dictionary<string, string> ReplaceStringMap { get; set; }
    }

As you see, it already contains VectorWidth / VectorHeight properties. We don't need to modify TaskParameter, ImageInformation and generating custom key, as it does it automatically for us when this interface is used.
We could create AndroidXmlDataResolver and iOSPdfDataResolver, similarly to SvgDataResolver.cs https://github.com/luberda-molinet/FFImageLoading/blob/master/source/FFImageLoading.Svg.Shared/SvgDataResolver.cs
Then you would load images as:
var task = ImageService.LoadCompiledResource("someFile.xml").WithCustomDataResolver(new AndroidXmlDataResolver(width: 200, height: 200)), etc
The problem is that IVectorDataResource.Resolve must return a Stream but we can return just a null and add another property to it like string ResourceId { get; }, I could refactor the code to support that scenario (pass that id to decoders).

Or even we could create platform specific extensions like:
var task = ImageService.LoadXmlResource("someFile.xml", width: 200, height: 200))
var task = ImageService.LoadPDFResource("someFile.pdf", width: 200, height: 200)), etc

hmm, I thought about it.. but it seemed as over design in XML files case.
because we've got separate Image View for Svg - MvxSvgCachedImageView / SvgCachedImage
Maybe, it will be strange to create new type for this (e.g. XmlCachedImage)

The only reason why there're a separate SVG controls is that SVG support is provided in a different nuget package. If we'll have xml/pdf support in base packages, we can handle loading of those files without any problem.


[RadzenDataGrid] InsertRow() does not work in certain situations

Describe the Bug
We have been migrating our grids to RadzenDataGrid from RadzenGrid. After getting everything else working, it appears that RadzenDataGrid.InsertRow() is not functioning. We have implemented everything from this example and cannot find any differences.
Steps to Reproduce

Clone this repo locally: https://github.com/CloudNimble/RadzenDataGridProblem
Set a breakpoint here.
Hit F5 and let the app run & load.
Click the "Add New Record" button.
Step through it adding a new item.
See that it does not put the row into edit mode.

Expected Behavior
I expect it to behave like the previous RadzenGrid and put a new row in the table and put that row into Edit Mode, just as in this example:

Screenshots

This is what the screen looks like after you click the button in our reproduction project. Confirmed that the breakpoint has been hit at the method has been called. No rows are added, and no additional errors are listed in the Console.
Environment

OS: Windows 11
Browser: Edge
Version: 109.0.1518.70

Additional context
This is currently blocking us from moving to Production.

I've replaced RadzenDataGrid and RadzenDataGridColumn with RadzenGrid and RadzenGridColumn however the insert didn't worked again.

Thanks for looking into it! Odd. I wonder if it's a .NET 7 thing? I'll roll it back to .NET 6 and see if it works.
The last version of Radzen.Blazor that I can guarantee it worked in for RadzenGrid was 3.6.8.
If you want to set those changes as a Pull Request on my repro project, I'll pull it down and see if I can roll back to the last verison / runtime it worked in.

didn't worked with 3.6.8 as well - pull request submitted.

I believe I have identified the problem. The code previously worked on .NET 5.0 with version 3.8.6. I was able to reproduce it locally in an older version of BurnRate.
In previous iterations, calling await gridInstance.InsertRow(item); was sufficient to get the Grid into Edit mode. The record was only committed to the underlying List if it was actually edited.
Instead, I'm now having to call:
ViewModel.Entities.Insert(0, item);
await gridInstance.EditRow(entity);

...and simply remove the blank item from the collection if it is deleted before it is persisted.
I will push an update that shows the codebase working, as well as a repo of it working the way it is supposed to on .NET 5.0. I will update this thread when those two branches have been pushed.

Here's the working version on .NET 7 that bypasses gridinstance.InsertRow(): https://github.com/CloudNimble/RadzenDataGridProblem/commit/187555826e84044f22a6bad8703c321ff409b161
Here's a repro of a working version on .NET 5.0: https://github.com/CloudNimble/RadzenDataGridProblem/commit/bfd06d45a291dfb416a514fa6445c007f86fb057

Try latest Radzen.Blazor with .NET 5 - if it works then the problem is caused by a change in the EF DbSet. We've not changed anything related to insert.

So I figured out what the problem is. Because the datasource we're using is remote, when the data is loaded well after the Grid is rendered, it's not refreshing the Grid with the new data. We had to set our custom grid wrapper to watch the ViewModel for when the LoadingStatus changed, and call gridInstance.Reload().
@enchev I would suggest that you folks add a RadzenDataGrid.Refresh() method that makes it clear you are just going to update the rendering from the existing data, not reload the underlying data source.
I'll go ahead and close this now. If any future readers want to see the fix, let me know.

I have the very same problem and it's very frustrating. I thought the problem was with me and my code, but thankfully, I did a series of tests to figure out the insert method.
Strangely, the insert works the first time but doesn't work the second time I call the insert method.
My project is using .NET 8, so I assume it shares the same underlying issue as with .NET 7.
I need to get this fixed for a project I am working on for a client.
Any attempts at fixing this issue would be very much appreciated :)

So I figured out what the problem is. Because the datasource we're using is remote, when the data is loaded well after the Grid is rendered, it's not refreshing the Grid with the new data. We had to set our custom grid wrapper to watch the ViewModel for when the LoadingStatus changed, and call gridInstance.Reload().
@enchev I would suggest that you folks add a RadzenDataGrid.Refresh() method that makes it clear you are just going to update the rendering from the existing data, not reload the underlying data source.
I'll go ahead and close this now. If any future readers want to see the fix, let me know.

I can confirm that this problem only happens when the data source is remote and has changed.
What is the best way to work around this problem?

Hello @robertmclaws , I'm interested in having this fixed. Did you manage to find a workaround for .NET 7/8?

Will this ever be fixed?

Will this ever be fixed?

Try this:
AllowVirtualization="true" in your RadzenDataGrid


disable cudf_udf tests for #2521

Signed-off-by: Peixin Li<EMAIL_ADDRESS>temply disable cudf_udf test to unblock nightly tests. We should enable that back when resolve #2521

build

The tests passed on the recent 4 Databricks runs. Not sure whether it is good to disable them for Databricks.
It is likely to be an env issue, I am debugging this.
Approving this as required.


Add GitHub link in footer

I want to add a GitHub link in the footer section.

I want to add GitHub link in footer.
Please assign this issue to me @codervivek5

no need to add GitHub link ....we should not share our source code publically guys


Doesn't Parse HTML Block as Being Part of the Definition List

The second HTML block is not considered to be part of the definition list. pandoc includes the second HTML block in the second definition list:
HTML In a Definition List
:

    One space

More HTML In a Definition List
:

    Two Spaces

Events from running `cargo run -- -D -e`
0..108: Start(DefinitionList)
0..26: Start(DefinitionListTitle)
0..25: Text(Borrowed("HTML In a Definition List"))
0..26: End(DefinitionListTitle)
26..74: Start(DefinitionListDefinition)
30..72: Start(HtmlBlock)
30..30: Text(Borrowed(" "))
30..36: Html(Borrowed("
\n"))
37..63: Html(Borrowed("   One space\n"))
64..72: Html(Borrowed("
\n"))
30..72: End(HtmlBlock)
26..74: End(DefinitionListDefinition)
74..105: Start(DefinitionListTitle)
74..104: Text(Borrowed("More HTML In a Definition List"))
74..105: End(DefinitionListTitle)
105..107: Start(DefinitionListDefinition)
105..107: End(DefinitionListDefinition)
0..108: End(DefinitionList)
110..153: Start(HtmlBlock)
110..110: Text(Borrowed("  "))
110..116: Html(Borrowed("
\n"))
116..144: Html(Borrowed("    Two Spaces\n"))
144..153: Html(Borrowed("
\n"))
110..153: End(HtmlBlock)
EOF

Looking into this a little more it seems this ins't just related to HTML blocks. It's also paragraphs, and presumably anything else that's separated from the : by two newlines. Again, pandoc includes the paragraph in the definition list.
Definition
:

  Some text

Events from running `cargo run -- -D -e`
0..16: Start(DefinitionList)
0..11: Start(DefinitionListTitle)
0..10: Text(Borrowed("Definition"))
0..11: End(DefinitionListTitle)
11..13: Start(DefinitionListDefinition)
11..13: End(DefinitionListDefinition)
0..16: End(DefinitionList)
16..26: Start(Paragraph)
16..25: Text(Borrowed("Some text"))
16..26: End(Paragraph)
EOF

One more test case. Here's how pandoc handles it.
Definition

:

  Some text

Events from running `cargo run -- -D -e`
0..17: Start(DefinitionList)
0..11: Start(DefinitionListTitle)
0..10: Text(Borrowed("Definition"))
0..11: End(DefinitionListTitle)
12..14: Start(DefinitionListDefinition)
12..14: End(DefinitionListDefinition)
0..17: End(DefinitionList)
17..27: Start(Paragraph)
17..26: Text(Borrowed("Some text"))
17..27: End(Paragraph)
EOF


New protocol # 446

🥼 Protocol Request
Great, we just need the ID number and its title; fill in below:
Protocol ID: 446
Protocol Title: Liver Reference Set-Rapid-Application: Nilsson Glycobond (2018)

Added 446


Test: script for integration test from scenario manger to deploy compute

We need a script for integration test. The script can be a bash or python script including functions like:

Create request body for topology, service-config, network-config, and compute-config.
deploy/check/delete topology, network, and compute
get the return message.
identify the test pass or failed by check the return message or code.

Will do


lowdown: 0.8.0 -> 0.8.1

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 112516 at 44f1a10a run on x86_64-linux 1

5 packages built:

hydra-unstable
lowdown
nix-update
nixFlakes (nixUnstable)
nixpkgs-review

1 suggestion:

warning: missing-phase-hooks
configurePhase should probably contain runHook preConfigure and runHook postConfigure.
Near pkgs/tools/typesetting/lowdown/default.nix:17:3:
   |
17 |   configurePhase = ''
   |   ^

Result of nixpkgs-review pr 112516 at 44f1a10a run on aarch64-linux 1

5 packages built:

hydra-unstable
lowdown
nix-update
nixFlakes (nixUnstable)
nixpkgs-review

1 suggestion:

warning: missing-phase-hooks
configurePhase should probably contain runHook preConfigure and runHook postConfigure.
Near pkgs/tools/typesetting/lowdown/default.nix:17:3:
   |
17 |   configurePhase = ''
   |   ^


Implement dttm column configuration through db extra config

CATEGORY
Choose one

[ ] Bug Fix
[x] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
Provides an ability for the superset admin to configure the date time columns per name so they will be automatically created on the table creation.
It helps a lot for the companies that have conventions & automatic day columns.
Reimplements https://github.com/apache/incubator-superset/pull/9353 to be more in line with the https://github.com/apache/incubator-superset/issues/6360
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

unit tests
local testing

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS
@john-bodley @villebro

Codecov Report

Merging #9444 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9444      +/-   ##
==========================================
+ Coverage   59.00%   59.02%   +0.01%
==========================================
  Files         378      379       +1
  Lines       12166    12189      +23
  Branches     3007     3014       +7
==========================================
+ Hits         7178     7194      +16
- Misses       4807     4814       +7
  Partials      181      181

Impacted Files
Coverage Δ

superset-frontend/src/explore/controls.jsx
38.80% <0.00%> (-0.89%)
:arrow_down:

superset-frontend/src/explore/controlUtils.js
89.36% <0.00%> (-0.86%)
:arrow_down:

superset-frontend/src/chart/chartReducer.js
19.04% <0.00%> (ø)

...d/src/dashboard/components/SliceHeaderControls.jsx
11.62% <0.00%> (ø)

.../src/explore/components/ControlPanelsContainer.jsx
66.66% <0.00%> (ø)

...et-frontend/src/components/ListView/Pagination.tsx
72.72% <0.00%> (ø)

...dashboard/components/FilterIndicatorsContainer.jsx
69.35% <0.00%> (+0.50%)
:arrow_up:

...rset-frontend/src/components/ListView/ListView.tsx
82.50% <0.00%> (+0.68%)
:arrow_up:

...ntend/src/dashboard/util/activeDashboardFilters.js
92.30% <0.00%> (+1.92%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d8a2b8...c0f973d. Read the comment docs.

@bkyryliuk one other question I had, and maybe the community should weigh in here, is whether i) defaults should be persisted on tables etc. or ii) whether undefined values should fallback to the defaults.

@willbarrett and @john-bodley addressed the comments, please let me know if you have other suggestions

@willbarrett addressed the comments, thanks

@willbarrett , @john-bodley - is there anything else I can do to get this merged?

No further comments from me. I recommend pinging a couple more contributors to review.

Generally LGTM
I think personally I prefer dynamic configuration hooks, where you'd define a function that receives a live table object upon insert and/or modify, and mutate it with whatever business/semantical logic make sense to your org, keeping all the logic on your side.
Intricate static configuration like the one defined here is hard to discover, often requires to read code to understand what it does / can do, and often has shortcomings / requires code change to the core package to get things done.

Thanks @mistercrunch , I think as an alternative, we could benefit from making them high level db configuration in the ui. It would make things more transparent. However this would be probably out of scope of this PR.
@willbarrett is PR in a good state to merge?

@bkyryliuk I have no further blockers, but I:

Don't understand this part of the system well enough to give you a merge approval
Am not a commiter, so my approval carries little weight.

Again, please reach out to additional committers to see if this is viable. It sounds like @mistercrunch would prefer an alternate solution.

@bkyryliuk I have no further blockers, but I:

Don't understand this part of the system well enough to give you a merge approval
Am not a commiter, so my approval carries little weight.

Again, please reach out to additional committers to see if this is viable. It sounds like @mistercrunch would prefer an alternate solution.

Thanks for the context. Will do.

Here's an example of what I meant:
https://github.com/apache/incubator-superset/compare/master...mistercrunch:example_config?expand=1
Using this, you can implement whatever logic to change table configuration based on naming conventions and such.
Would that work for you?

https://github.com/apache/incubator-superset/compare/master...mistercrunch:example_config?expand=1

I like it, will redo the pr

@mistercrunch  - updated the PR to follow the mutator pattern

test failure doesn't seem to be releated.

@bkyryliuk I restarted CI, I had a similar problem yesterday.

@bkyryliuk do you think there is merit in addressing this comment in a follow up PR? I'm a little concerned that there's two different mechanism for defining the Python date-format within the extra blob.


Leafmap (default backend) and Streamlit for vector tile layers

hi 👋
Environment Information

leafmap version: 0.7.7
steamlit version: 1.5.1
Python version: 3.9.0
Operating System: MacOS

Description
I want to run leafmap in stream lit with custom vector tile layers.
The page is displayed but the browser console overflows with errors. Eventually it goes unresponsive with this error looping infinitely:
Uncaught (in promise) Error: Invalid LatLng object: (53.24878214338736, NaN)
    at new B (leaflet-src.js:1365:11)
    at Object.unproject (leaflet-src.js:1667:12)
    at Object.pointToLatLng (leaflet-src.js:1506:28)
    at n.unproject (leaflet-src.js:3963:29)
    at n.layerPointToLatLng (leaflet-src.js:3971:17)
    at n.containerPointToLatLng (leaflet-src.js:4028:17)
    at n._update (leaflet-src.js:5493:10)
    at n.whenReady (leaflet-src.js:4428:15)
    at n.onAdd (leaflet-src.js:5469:9)
    at n.addTo (leaflet-src.js:4757:44)

The error does not occur when falling back to folium instead, so my guess is that I'm not supposed to use the default plotting backend. However folium does not allow me to use m.add_vector_tile_layer().
What I Did
Streamlit app
from leafmap import leafmap

m = leafmap.Map(
    center=[53.25, 13.37],
    zoom=8,
    draw_control=False,
    measure_control=False,
    fullscreen_control=True,
    attribution_control=False,
)

m.to_streamlit()

Unfortunately, folium does not yet support vector tile layers. Although ipyleaflet supports vector tile layers, it doesn't work well when converting it to streamlit. Essentially, the ipyleafmet Map object is converted to an HTML file. All bidirectional functionality will be lost. There is not much leafmap can do about this unless folium offiically supports vector tile layers.

https://github.com/python-visualization/folium/issues/937
https://github.com/python-visualization/folium/pull/943

As an FYI @giswqs, I addressed this upstream and added the needed plugin to support it to folium in #1576. I don't know when and if it will be accepted. In case it is not accepted, what is your take on 3rd party plugins for folium?

@iwpnd If the PR is not accepted upstream, you are welcome to submit it to leafmap, just like what I did in https://github.com/giswqs/leafmap/pull/218
I have been trying to get the SideBySide plugin. However, folium still has not accepted the PR since it was submitted two years ago. Therefore, I adapted the code and added it to leafmap directly. You can do the same if you want to.
https://github.com/python-visualization/folium/pull/1292


@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I rerendered the recipe in conda-forge/openconnect-feedstock#13.


fix: make favicon transparent

Description
Made the favicon transparent so that it doesn't appear weird with a white background while viewing the website in dark mode.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] My code follows the style and structure of this project
[x] My code follows the principles of clean code
[x] My changes generate no new warnings, bugs or errors

@0l1v3rr I have made the required changes, please have a look :)

@burhanharoon could you resolve the conflict? I can't merge this.

Sure I'll do that.

@burhanharoon, I still can't merge because of the conflicts. I think the easiest solution would be to sync the main branch of your fork (which contains the favicon with the white bg) and work from there. Then you could make another PR without conflicts.


Shadow css variable names are incorrect on this page

What happened?
Please refer to shadow documentation page https://ix.siemens.io/docs/theming/shadow
In this page there are shadow variable names are not correct and are not getting applied
Note: n stands for 1,2,3.....
--theme-shadow-n
I found these working
--theme-box-shadow-lvl-n

@negideepak We can not detect any issues regarding css shadow variables. We use them in components as well as in the documentation itself.


New version of PASTA's .eln file

include signature using minisign (#56)
include all metadata as PropertyValue into main json file (see #58)

A few remarks while manually checking the JSON-LD:

Nested entries like sdPublisher, author, etc. should ideally be flattened and only referenced via their @id.
Some entries like the author or the flexible metadata are missing a @type.
Side note: Have we discussed yet whether variableMeasured is the best place for the flexible metadata beyond the initial suggestion?

SampleDB currently fails to import the .eln file as the file ./temporary_pastaTest/StandardOperatingProcedures/SEM.md is referenced, however it does not exist. The temporary_pastaTest directory only contains a file data_hierarchy.json, nothing else.
Aside from that, there seem to be a lot of additional properties there, which I'm not sure whether they are useful to display to a user, e.g. _attachments.v0.json.digest to _attachments.v51.json.digest (and the respetive lengths, revpos, etc). These are also listed for File objects,  not just for Dataset objects, which isn't valid.
The example directory README should be updated to reflect the new ro-crate-metadata.json, right?
@nicobrandt I don't think so, but going by schema.org, a Dataset can contain a PropertyValue either as variableMeasured or variablesMeasured, aside from it being used as identifier. variablesMeasured is retired and recommends using variableMeasured, so unless we want to deviate from the existing properties, variableMeasured is the most fitting.

A few remarks from my side:

As per specification, the folder name should match the name of the archive.
According to the discussion in #9, the hasPart property in the second object in the @graph, should have only the relative paths of the folders. It should be updated in the specification too.
Additionally, some folders, for eg: {         "@id": "./temporary_pastaTest/Steel/"       },       {         "@id": "./temporary_pastaTest/SurfaceEvolutionInTribology/"       },   are listed in the hasPart but are not present in the archive.
Furthermore, the files referenced in the JSON-LD are missing in the archive.

@SteffenBrinckmann, a few remarks:

you should run the eln2md script so the PR includes changes to the ro-crate-metadata.json in the diff, and it becomes easier to point things
instead of using ORCID in the @author node, you should use identifier:

"identifier": "https://orcid.org/0000-0003-3432-2297"

what is the shasum property? The hashing function name must be mentioned.
regarding the signature, it's great, I could verify it :+1:

@nicobrandt @jmanideep @FlorianRhiem @NicolasCARPi Thank you all for the feedback: helpful and I will incorporate it in the next version.
We have not agreed an all of the "methods" used here and the idea here is not to fix anything. It is meant as a test area for ideas and see if/how they could work. Trying something in code is - in my view - better than theoretical discussions. Once we finally agree on things, then we can write tests for them and ... . Nevertheless, this is just the output of code and that code is easily changed, no hard feelings if we agree on alternative methods.

OK, this is an improved version that addresses all mentioned issues but one:

@jmanideep "According to the discussion in #9, the hasPart property in the second object in the @graph, should have only the relative paths of the folders. It should be updated in the specification too."
I do not understand: does that mean that './' cannot contain data?

Obviously from the failed github action, the tooling is also broken: I have some fixes but I did not want to mingle them into this.

@NicolasCARPi Sorry my bad

@NicolasCARPi I will look into that

@SteffenBrinckmann In the previous version of your example (here), the hasPart property in the second object(where "@id": "./") in the @graph has the following structure:
"hasPart": [         {           "@id": "./temporary_pastaTest/Steel/"         },         {           "@id": "./temporary_pastaTest/SurfaceEvolutionInTribology/"         },         {           "@id": "./temporary_pastaTest/IntermetalsAtInterfaces/"         },         {           "@id": "./temporary_pastaTest/data_hierarchy.json"         }       ],.
However, according to specification, the hasPart property in this node should contain only relative paths of the folders. This means that entries like  {           "@id": "./temporary_pastaTest/data_hierarchy.json"         }, should not be included here. Instead, such files should be listed in the hasPart of the respective Dateset node.

Should now fit


Fix Error Message Displayed on the Manage Organizations Page (Global View role)

🗣 Description

Fixed organizations not displayed on "Manage Organizations" as Global View.

💭 Motivation and context
Currently, there are no organizations displayed on the Manage Organizations page under the global view role. Under global view, you are able to view organizations but not modify. With this fix, it restores that functionality to a user under global view.
This resolves CRASM - 326

📷 Screenshots (if appropriate)

✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All future TODOs are captured in issues, which are referenced
in code comments.
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[ ] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[ ] Tests have been added and/or modified to cover the changes in this PR.
[x] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Revert dependencies to default branches.
[ ] Finalize version.

✅ Post-merge checklist

[ ] Create a release.

Looks and works great. Line 862 (const id =...) seems unnecessary though.

Good catch! The line in question has been removed.


studio: Display profile pic when available on the top right of the Menu

Similar to the one in Kavach Web: https://github.com/factly/kavach-web/issues/68

Requires server changes. https://github.com/factly/dega/issues/70


(PUP-10408) Remove deprecated '-noidme' flag from pkgdmg package provider

This pull request supersedes #8072.
The pkgdmg package provider uses the '-noidme' flag with hdiutil, which was deprecated in macOS 10.15. See man hdiutil for more information. In macOS 10.15 the '-noidme' flag causes hdiutil to output a deprecation warning, which then causes pkgdmg to fail to parse the plist xml that hdiutil generates, which then causes the package installation to fail in puppet.
This change removes the '-noidme' flag from pkgdmg in order to prevent the deprecation warning from hdiutil, and restore functionality to pkgdmg in macOS 10.15 and above. Additonally, after reviewing the man page for hdiutil in macOS 10.14, it seems the '-noidme' flag is unnecessary for older versions of macOS as hdiutil appears to not perform idme actions by default. This change also removes the '-noidme' flag from the pkgdmg test in order to bring it up-to-date with the change in pkgdmg.

Can one of the admins verify this patch?

CLA signed by all contributors.

Jenkins please test this on osx1014-64,osx1015-64


Unable to find coinbase order with store id

I'm not able to view order details from Admin panel,
1 exception(s):
Exception #0 (Magento\Framework\Exception\NoSuchEntityException): Unable to find coinbase order with store id "000000128"
Exception #0 (Magento\Framework\Exception\NoSuchEntityException): Unable to find coinbase order with store id "000000128"
#0 /var/www/html/qa_v2/aroma2gom2/app/code/CoinbaseCommerce/PaymentGateway/Block/Adminhtml/Order/View/Details.php(36): CoinbaseCommerce\PaymentGateway\Model\CoinbaseRepository->getByIncrementId('000000128')
#1 /var/www/html/qa_v2/aroma2gom2/app/code/CoinbaseCommerce/PaymentGateway/Block/Adminhtml/Order/View/Details.php(46): CoinbaseCommerce\PaymentGateway\Block\Adminhtml\Order\View\Details->getCoinbaseRecord()
#2 /var/www/html/qa_v2/aroma2gom2/app/code/CoinbaseCommerce/PaymentGateway/view/adminhtml/templates/details.phtml(4): CoinbaseCommerce\PaymentGateway\Block\Adminhtml\Order\View\Details->getCoinsDetail()

Still I'm having this same issue. magento2.3.2 with php7.2

We solved it. Try:
/vendor/coinbase/coinbase-commerce-magento/Block/Adminhtml/Order/View/Details.php
lines: 44 ~ 63
public function getCoinsDetail()
{
    try {
        $record = $this->getCoinbaseRecord();
        $data['expectedCoins'] = $record->getCoinsExpected() . ' ' . $record->getReceivedCurrency();
        $data['amount'] = $record->getCoinsReceived() . ' ' . $record->getReceivedCurrency();
        $data['status'] = $record->getCoinbaseStatus();
        $data['code'] = $record->getCoinbaseChargeCode();
        $data['transactionId'] = $record->getTransactionId();
        $data['totalPaid'] = $record->getTotalPaid();
        $data['orderPlacedCurrency'] = $this->getOrder()->getOrderCurrencyCode();
    } catch (\Exception $e) {
        $data['expectedCoins'] = '';
        $data['amount'] = '';
        $data['status'] = '';
        $data['code'] = '';
        $data['transactionId'] = '';
        $data['totalPaid'] = '';
        $data['orderPlacedCurrency'] = '';
    }

Hi folks, unfortunately we are moving towards deprecating support for the Magento plugin as it hasn't captured enough audience for us to maintain it at this time. However, we may come back to it in the future. The php SDK however will be kept alive. Feel free to clone this repository to make modifications for your use, which it seems by now you may have, or post any PR's you would like to fix for the php SDK to share with the community. Sorry we couldn't be more helpful at this time.


Fixed missing backtick in docs

Thanks so much for the fix. Can you also fix this in the TidierData.jl GitHub repo at: https://github.com/TidierOrg/TidierData.jl?
We are planning to convert Tidier.jl into a meta-package. I'm not 100% sure what we will do with the Tidier.jl documentation -- we will probably link out to the individual documentation pages for the growing collection of Tidier packages.
(If you can't get to it, no worries -- I'll wait to merge this until it's fixed on TidierData as I don't want to lose track of this)

Actually, nevermind - I'll just fix it on TidierData.jl right now. Thank you once again!


Add CHN address inspector

Description

Motivation and Context

How has this been tested?

Types of changes

[ ] Maintenance (no change in code, maintain the project's CI, docs, etc.)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

My git management seems to be in chaos, I will resubmit the PR


[Linux] Revise functional tests for cross-device mounts

On POSIX systems, where a rename(2) returns EXDEV when trying to move a file or directory between mounted filesystems, the functional tests which do this currently fail if the
repository is mounted on a separate device (as is the case on Linux).
Therefore we create a RepositoryMountsSameFileSystem test category for use on VFSForGit implementations which allow renaming across repository boundaries (i.e., Windows PrjFlt
and Mac kext), and mark a number of tests with this category designation as they move files or folders in or out of the repository, or create hard links across the repository boundary.
In the ModifiedPathsTests suite, two specific tests which check the contents of the modified-paths database after remounting and after hard-linking, specifically, are subdivided into pairs of tests.  Each pair consists of one test which only checks changes that do not cross the repository boundary, suitable for all implementations, and one test which check changes spanning the repository boundary and which is marked with the RepositoryMountsSameFileSystem
category.

@wilbaker -- I think I've managed to simplify this so that we just ignore the tests which don't apply if cross-device renaming (and hard-linking) are not supported; I can't, on reflection, see much value in emulating them by copying files.  The exception, sort of, are the two main tests in ModifiedPathsTests, which I split up as described into one each which makes inside-repo-only changes, and one which makes repo-boundary-crossing changes (and can thus be excluded on Linux).

Just as an added thought, @wilbaker, I can also submit these changes as a series of smaller PRs -- e.g., one each for Mac -> POSIX renaming, adding FileSystemSupportsFileMode, and adding RepositoryMountsSameFileSystem.  I'd be happy to do so if that would make your life easier!  :-)

I can also submit these changes as a series of smaller PRs -- e.g., one each for Mac -> POSIX renaming, adding FileSystemSupportsFileMode, and adding RepositoryMountsSameFileSystem. I'd be happy to do so if that would make your life easier! :-)

@chrisd8088 if you don't mind making that change I think it would be easier/quicker to review these changes in chunks.  Thanks!

/azp run GitHub VFSForGit Mac Functional Tests

/azp run GitHub VFSForGit Mac Functional Tests

/azp run GitHub VFSForGit Mac Functional Tests

/azp run GitHub VFSForGit Mac Functional Tests

OK @wilbaker -- after much duelling with some hanging CI tests, this is ready after being trimmed down to just the cross-device changes.  The POSIX-related ones I moved over to #1529.


Network error when using string value matching the id of the node in links

Hi @davelandry,
I've been using d3plus-react extensively now; I like everything about it. It takes a couple of days to know all the names of the props you need. But, once you know them it's amazing.
So I tried to use the network graph, but for some reason, it only accepts this format for links (index of the node in the nodes array).
    const methods = {
      links:  [
        {source: 0, target: 1},
        {source: 0, target: 2},
        {source: 3, target: 4},
        {source: 3, target: 5},
        {source: 5, target: 0}
      ] ,
      nodes:  [
        {id:'alpha',   x: 1,   y: 1},
        {id:'beta',    x: 2,   y: 1},
        {id:'gamma',   x: 1,   y: 2},
        {id:'epsilon', x: 3,   y: 2},
        {id:'zeta',    x: 2.5, y: 1.5},
        {id:'theta',   x: 2,   y: 2}
      ],
    }

    let network = <Network config={methods} />

when I try to create the links based on the string value matching the id of the node, it just doesn't work.

    const methods = {
      links:  [
        {source: 'alpha', target: 'beta'},
        {source: 'alpha', target: 'gamma'},
        {source: 'epsilon', target: 'zeta'},
        {source: 'epsilon', target: 'theta'},
        {source: 'theta', target: 'alpha'}
      ] ,
      nodes:  [
        {id: 'alpha',   x: 1,   y: 1},
        {id: 'beta',    x: 2,   y: 1},
        {id: 'gamma',   x: 1,   y: 2},
        {id: 'epsilon', x: 3,   y: 2},
        {id: 'zeta',    x: 2.5, y: 1.5},
        {id: 'theta',   x: 2,   y: 2}
      ],
    }

    let network = <Network config={methods} />

@AlmahaAlmalki thank you for the kind words! glad you're enjoying using d3plus 😄
This is actually intended behavior, because internally there's no way to know if the integer passed as the source/target is an index or an actual ids (because ids can be integers). The value for source and target need to either be the index of the node in the nodes array, or the actual node itself! For example, you could transform your links array like this:
const methods = {
  links:  [
    {source: 'alpha', target: 'beta'},
    {source: 'alpha', target: 'gamma'},
    {source: 'epsilon', target: 'zeta'},
    {source: 'epsilon', target: 'theta'},
    {source: 'theta', target: 'alpha'}
  ] ,
  nodes:  [
    {id: 'alpha',   x: 1,   y: 1},
    {id: 'beta',    x: 2,   y: 1},
    {id: 'gamma',   x: 1,   y: 2},
    {id: 'epsilon', x: 3,   y: 2},
    {id: 'zeta',    x: 2.5, y: 1.5},
    {id: 'theta',   x: 2,   y: 2}
  ],
}

methods.links.forEach(function(link) {
  link.source = methods.nodes.filter(function(node) {
    return node.id === link.source;
  })[0];
  link.target = methods.nodes.filter(function(node) {
    return node.id === link.target;
  })[0];
});

Let me know if that doesn't work for you!

Hi @davelandry
Thanks that makes a lot of sense, there's another issue with dynamic nodes position. However,  I used d3 force layout to generate the positions dynamically. But, I was wondering if there's another way to set the x and y using d3plus only?

@AlmahaAlmalki force-directed layouts have not been implemented in 2.0 yet: https://github.com/d3plus/d3plus-network/issues/28


Update adios2_config.xml for grad3d test

Remove unused io handles and match io handle name with code

@pbartholomew08 , thanks I think you sorted the error I had with ADIOS2

Ok the issue with ADIOS2 is still there. Do you want me to have a look?

the error i got is the following
[Tue Nov 07 09:33:26 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_at_io: [Tue Nov 07 09:33:26 2023] [ADIOS2 EXCEPTION] <Helper> <adiosType> <CheckForNullPtr> : found null pointer for adios2_adios, in call to adios2_at_io

 2DECOMP&FFT ERROR
   errorcode:                4
   error in file  /home/rfj82982/GIT_Work/2decomp-fft/src/io.f90
            line          1878
   error message: adios2_at_io grad-io
 2DECOMP&FFT ERROR
   errorcode:                4
   error in file  /home/rfj82982/GIT_Work/2decomp-fft/src/io.f90
            line          1878
   error message: adios2_at_io grad-io
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 4.

You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

If the tests pass I think this should now be good to go

Yes I am looking at the runner


Removed errors from README.md

@DhruvamSharma, thanks for the edits! Can we send you some LBC in appreciation?


Evolution Changes - Next, All Future, All Past

OK, little bit of a code dump, sorry. Fixes #6 .
Please do read through this; I am completely new to TypeScript.
Major Changes:

New methods on pokemon.parser to fill future/past evolution links
Tree class to handle future evolutions
Test re-organization to make pokemon-specific tests MUCH easier

Let me know if you need anything at all changed.

@proegssilb All in all the PR is fine. The comments I made are just for consistency or a better readability. So thanks a lot!
Maybe we should consider in the future to split up PokemonParser, but that can be done in another PR.

Splitting up PokemonParser is definitely something I'd want to dive into a bit further (maybe in an issue or over Discord) before I worked on it. Just to make sure I understand the reasoning and goal.

@proegssilb Some review change-requests are still pending. Are you working on them or have you missed them?

@BrunnerLivio I can't figure out what changes you're referring to, so I guess I missed them :) .
The only comments I had were "I like that!" in reference to the pokemon-specific tests, and a request to not use shorthand, which I thought was in reference to the Pokemon Parser and the tree datatype in the Pokemon Model file.

which I thought was in reference to the Pokemon Parser and the tree datatype in the Pokemon Model file.
@proegssilb Oh. No that's no what I meant. The review with EvolutionTree is about the recursive implementation, which is inconvenient as a user. I suggest to flatten the recursive EvolutionTree. More details in the review itself

Yeah, that comment doesn't exist on my view in github. Do you have an incomplete review?
I'll see what I can do about EvolutionTree, but it was extremely difficult to balance both 3-stage evolutions and eevee's/tyrogue's branching structure in the same tree. This is compounded by my lack of familiarity with Gen3+, which might have introduced more branching or other quirks.

@proegssilb Oh I'm so sorry, I did not submit. I'm not used to Github reviews haha.

I'll see what I can do about EvolutionTree, but it was extremely difficult to balance both 3-stage evolutions and eevee's/tyrogue's branching structure in the same tree. This is compounded by my lack of familiarity with Gen3+, which might have introduced more branching or other quirks.

My approach for this would be the "evolutionStage" attribute


[BUG/ISSUE] Cannot install nano from scoop using WingetUI (but 'scoop install nano' does work in Powershell)

Before submitting the issue

[X] I have checked that the bug is not a duplicate or, if it is, it needs to be reopened
[X] I have checked that I am running the latest version of WingetUI
[X] I have tried resetting and reinstalling WingetUI and the issue persisted or happened again
[X] I have checked the FAQ section for solutions
[X] I have read the contributing guidelines and I agree with the Code of Conduct

Describe the bug
When I try to install nano from scoop in WingetUI, I get an error message.
To Reproduce
Steps to reproduce the behaviour:

Open WingetUI and make sure scoop is enabled
Search for 'nano'
Two results will appear. Right-click on the scoop result and click Install
This always produces the following error:

Starting installation...
Installing 'nano' (7.2-19.3) [64bit] from main bucket
Downloading https://github.com/okibcn/nano-for-windows/releases/download/v7.2-19.3/nano-for-windows_win64_v7.2-19.3.zip (279.0 KB)...
Checking hash of nano-for-windows_win64_v7.2-19.3.zip ... ok.
Extracting nano-for-windows_win64_v7.2-19.3.zip ... Exception calling "ExtractToFile" with "3" argument(s): "The archive entry was compressed using an unsupported
compression method."
At
C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\Microsoft.PowerShell.Archive\Microsoft.PowerShell.Archive.psm1:1056
char:25
+ ...             [System.IO.Compression.ZipFileExtensions]::ExtractToFile( ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidDataException
done.
Running pre_install script...
Linking ~\scoop\apps\nano\current => ~\scoop\apps\nano\7.2-19.3
Creating shim for 'nano'.
Can't shim 'nano.exe': File doesn't exist.

Installing scoop manually from the command line (Powershell in Windows Terminal) works fine.
Expected behaviour
I expect the scoop nano package to install without error when using WingetUI.
Screenshots
How to trigger the bug:

The error message after installing:

Installing nano manually from the command line (Powershell in Windows Terminal):

The Log
C:\Users\MarkFarragher\AppData\Local\Programs\WingetUI\sudo\gsudo.exe
C:\Users\MarkFarragher\AppData\Local\Programs\WingetUI\sudo
🟢 checkQueue Thread started!
🟢 Using cached lang file
🟡 Using bundled lang file (forced=True)
It took 0.22644734382629395 to load all language files
🔵 Language file up-to-date
🔵 Starting main application...
Scanning for instances...
Found lock file(s), reactivating...
🔵 Starting update check
🟢 Downloaded latest metadata to local file
🔵 Loaded metadata from local file
🔵 Version URL: https://versions.marticliment.com/versions/wingetui.ver
🟢 Updates not found
Reactivation signal ignored: RaiseWindow_[1679916032.3743653]
b'v1.4.10173\r\n'
None
updating winget
b'0.10.15\r\n'
None
b"Current Scoop version:\nv0.3.1 - Released at 2022-11-15\n\n'main' bucket:\n2d7aa7410 task: Update to version 3.23.0\n\n\n"
None
{'wingetFound': True, 'scoopFound': True, 'chocoFound': True, 'sudoFound': True, 'wingetVersion': 'v1.4.10173\r', 'scoopVersion': 'v0.3.1 - Released at 2022-11-15', 'chocoVersion': '0.10.15\r', 'sudoVersion': 'gsudo v2.0.4 (Branch.tags-v2.0.4.Sha.506efa024af0cef6e4b0cfec42e0c8c5d0b1472c)\r'}
Reached main ui load milestone
Can't disconnect
🟢 Starting winget search, winget on C:\Users\MarkFarragher\AppData\Local\Programs\WingetUI\winget-cli\winget.exe...
🟢 Starting scoop search...
🔵 Starting choco search
🟢 Discover tab loaded
🟢 Starting choco search, choco on choco.exe...
Can't disconnect
🟢 Starting winget search, winget on C:\Users\MarkFarragher\AppData\Local\Programs\WingetUI\winget-cli\winget.exe...
🟢 Starting scoop search...
🟢 Starting choco search, choco on choco.exe...
🟢 Upgrades tab loaded
Can't disconnect
🟢 Starting winget search, winget on C:\Users\MarkFarragher\AppData\Local\Programs\WingetUI\winget-cli\winget.exe...
🟢 Starting scoop search...
🟢 Starting choco search, choco on choco.exe...
🟢 Discover tab loaded
{'System language': 'default', 'Arabic - عربي\u200e (95%)': 'ar', 'Bulgarian - български (87%)': 'bg', 'Bangla - বাংলা (25%)': 'bn', 'Catalan - Català': 'ca', 'Czech - Čeština': 'cs', 'Danish - Dansk (47%)': 'da', 'German - Deutsch': 'de', 'English - English': 'en', 'Spanish - Castellano (99%)': 'es', 'French - Français': 'fr', 'Hindi - हिंदी (75%)': 'hi', 'Hungarian - Magyar (99%)': 'hu', 'Italian - Italiano (99%)': 'it', 'Indonesian - Bahasa Indonesia (22%)': 'id', 'Japanese - 日本語 (81%)': 'ja', 'Korean - 한국어 (94%)': 'ko', 'Dutch - Nederlands': 'nl', 'Polish - Polski': 'pl', 'Portuguese (Brazil)': 'pt_BR', 'Portuguese (Portugal)': 'pt_PT', 'Russian - Русский (99%)': 'ru', 'Serbian - Srpski (20%)': 'sr', 'Slovene - Slovenščina': 'sl', 'Turkish - Türkçe (78%)': 'tr', 'Ukranian - Yкраї́нська (78%)': 'ua', 'Simplified Chinese (China)': 'zh_CN', 'Traditional Chinese (Taiwan) (99%)': 'zh_TW'}
['System language', 'Arabic - عربي\u200e (95%)', 'Bulgarian - български (87%)', 'Bangla - বাংলা (25%)', 'Catalan - Català', 'Czech - Čeština', 'Danish - Dansk (47%)', 'German - Deutsch', 'English - English', 'Spanish - Castellano (99%)', 'French - Français', 'Hindi - हिंदी (75%)', 'Hungarian - Magyar (99%)', 'Italian - Italiano (99%)', 'Indonesian - Bahasa Indonesia (22%)', 'Japanese - 日本語 (81%)', 'Korean - 한국어 (94%)', 'Dutch - Nederlands', 'Polish - Polski', 'Portuguese (Brazil)', 'Portuguese (Portugal)', 'Russian - Русский (99%)', 'Serbian - Srpski (20%)', 'Slovene - Slovenščina', 'Turkish - Türkçe (78%)', 'Ukranian - Yкраї́нська (78%)', 'Simplified Chinese (China)', 'Traditional Chinese (Taiwan) (99%)']
en
{'default': 'System language', 'ar': 'Arabic - عربي\u200e (95%)', 'bg': 'Bulgarian - български (87%)', 'bn': 'Bangla - বাংলা (25%)', 'ca': 'Catalan - Català', 'cs': 'Czech - Čeština', 'da': 'Danish - Dansk (47%)', 'de': 'German - Deutsch', 'en': 'English - English', 'es': 'Spanish - Castellano (99%)', 'fr': 'French - Français', 'hi': 'Hindi - हिंदी (75%)', 'hu': 'Hungarian - Magyar (99%)', 'it': 'Italian - Italiano (99%)', 'id': 'Indonesian - Bahasa Indonesia (22%)', 'ja': 'Japanese - 日本語 (81%)', 'ko': 'Korean - 한국어 (94%)', 'nl': 'Dutch - Nederlands', 'pl': 'Polish - Polski', 'pt_BR': 'Portuguese (Brazil)', 'pt_PT': 'Portuguese (Portugal)', 'ru': 'Russian - Русский (99%)', 'sr': 'Serbian - Srpski (20%)', 'sl': 'Slovene - Slovenščina', 'tr': 'Turkish - Türkçe (78%)', 'ua': 'Ukranian - Yкраї́нська (78%)', 'zh_CN': 'Simplified Chinese (China)', 'zh_TW': 'Traditional Chinese (Taiwan) (99%)'}
Can't disconnect
🟢 Starting scoop search...
🟢 Settings tab loaded!
🟢 About tab loaded!
🟢 Main application loaded...
🟢 Chocolatey uninstallable packages search finished
🟢 Searching for string ""
🟢 Scoop search finished
🟢 Searching for string ""
Failed in attempting to update the source: winget
main https://github.com/ScoopInstaller/Main.git 3/27/2023 6:28:42 AM      1180
🟢 Scoop bucket search finished
🟢 Scoop search finished
🟢 Searching for string ""
🟢 Chocolatey search finished
🟢 Searching for string ""
Name                                            Id                                      Version       Available
🟢 Winget search finished
🟢 Searching for string ""
🟢 Searching for string ""
🟡 Can't get custom interval time! (got value was '')
🟢 Total packages: 40
🟢 Winget uninstallable packages search finished
🟢 Searching for string ""
🟢 Searching for string ""
🟢 Total packages: 243
None
🟢 Winget search finished
🟢 Scoop search finished
🟢 Total packages: 5535
🟢 Found reactivation lock file...
0
0
🟢 Searching for string "n"
🟢 Searching for string "na"
🟢 Searching for string "nan"
🟢 Searching for string "nano"
🟢 Searching for string "nano"
args
🟢 Waiting for install permission... title=Nano, id=nano, installId=1679917731.2013974
🔵 Current program set to<PHONE_NUMBER>.2013974
🟢 Have permission to install, starting installation threads...
scoop: main
powershell -Command scoop install main/nano
🟢 scoop installer assistant thread started for process <Popen: returncode: None args: 'powershell -Command scoop install main/nano '>

It's because that particular zip file was compressed using Deflate64

Which the ExtractToFile method off the ZipFileExtensions class doesn't support, so the Powershell cmd Expand-Archive fails to extract it. Not sure where the cmd is getting called though as I'd assume scoop would use the bundled 7zip instead of the Powershell cmd.

So, this isn't an issue with WingetUI after all?

At its heart no. When the scoop install via Powershell is being launched as a secondary process from python it isn't detecting\using 7zip to extract the zip file and is defaulting back to the Expand-Archive Powershell method which doesn't support the gzip compression algorithms like deflate. Just not sure why. @mdfarragher if you try to install as admin or if you run:
scoop config 7ZIPEXTRACT_USE_EXTERNAL $true

in a normal Powershell window first does it make any difference?

To confirm I just tried it on my own box and the scoop nano install worked fine via WingetUI

yes, for me nano also installs fine

scoop install nano also works fine for me


non blocking match method for gazetteer

For getting a gazetteer session started, it can be useful to to try to match a target records against all of the records. Maybe a match_all method.

This is turning out to not be a problem, closing.


Add default_js_lib config

This PR adds default_js_lib/0 to address the inconvenience of having to set js_lib for the component every time it is used.

Awesome. Thanks @nallwhy


Document not initialized properly?

https://github.com/tmpvar/jsdom/blob/2390f917749608deda0b5dabff959659eaf7e1c9/lib/jsdom/level1/core.js#L1498
Looks fishy. Node expects an ownerDocument and gets passed a string?
(Just noting that here, have not dug into it yet.)

Agreed, surprised this doesn't break everything horribly. I guess people don't look at document.ownerDocument very often (or anything related to it).


Talabi/login page UI

Resolves #001
Description

This was an issue - Login Page UI
Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not
work as expected)
This change requires a documentation update
Documentation update

New Feature - Login
New feature - Login Page
I duplicated the image folder from the src folder to the components folder for ease of access. It threw an error when I tried to import it from the src folder to the login page
How Has This Been Tested?

There are no tests for this component yet.
Screenshots

You'll need to fix your merge conflicts as well

It looks like there are many unrelated changes in this PR, like your changes to Dashboard and Deposits. Your PR should only include changes to the Sign In feature. Is there a reason you changed those files?
If not, please revert those files and just include your changes for the sign in component.
Your PR is failing to build. It looks like there are some compile errors.


iOS 14 Privacy - SDK pops up permission request to "find and connect to devices on local network"

Confirm by changing [ ] to [x] below:

[x] I've gone though Developer Guide and API reference
[x] I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device
What are you running the sdk on?
iOS 14.2
Describe the question
On iOS 14, initializing the cognito identity client results in a pop-up asking for permission to allow the app to find and connect to devices on local network. Disallowing permission still works, but this is a privacy concern.
Why does it need this permission?
auto cognito_id_client = std::make_shared<Aws::CognitoIdentity::CognitoIdentityClient>(config);

Logs/output

Hi @triton3 ,
Just wanted to write a bit of an update, this is the first time I see something like that so I'll try looking into it but it might take some time to figure out. I'll update as soon as I find something useful.

@KaibaLopez Thank you for the update. The sdk version is 1.4 on our side. Please let me know if you need any other information.

@triton3 ,
So, that is a very old version of the sdk , and it makes things harder to test, can you use a newer (newest would be better) version?

@KaibaLopez The reason we havent updated to the latest version is because there were many breaking changes in the sdk, that required a full overhaul. Is there anything we can do to check with the version we have? I can put up the aws sdk source zip file and share through a private message using dropbox, if that helps.

@KaibaLopez Just wanted to check if there is any update? Any advice is appreciated.

Sorry for the long wait, the only updates I've got is that this seems to be caused by a new iOS14 privacy feature, here's a helpful thread on the iOS dev forums.
Been trying to test things out but with no real luck finding the root cause, from what I can tell it looks like a false positive on iOS 14 local network privacy warning, which btw I can find a bunch of similar cases, I just haven't been able to prove that's the case.
I can say that the SDK version doesn't really matter, so no problem if you don't update.
Sorry I can't be of more help here, I'll keep trying but it does seem like it will be a while until I find it, I am not really an expert on iOS development and other issues will take priority over this at least for the time being.
Having said that, I will keep this open in the hopes that other community members might see this and contribute ideas.

@KaibaLopez Thank you for the update.

We're having the problem.  For us it happens when constructing the Aws::Client:ClientConfiguration.
_clientConfig = Aws::MakeSharedAws::Client::ClientConfiguration("mem_AwsClientManager");

We're having the problem.  For us it happens when constructing the Aws::Client:ClientConfiguration.
_clientConfig = Aws::MakeSharedAws::Client::ClientConfiguration("mem_AwsClientManager");

Update: we were able to get around the prompt by setting the AWS_EC2_METADATA_DISABLED environment variable to TRUE before initializing our ClientConfiguration.

    setenv("AWS_EC2_METADATA_DISABLED", "TRUE", true);
    _clientConfig = Aws::MakeShared<Aws::Client::ClientConfiguration>("mem_AwsClientManager");

Update: we were able to get around the prompt by setting the AWS_EC2_METADATA_DISABLED environment variable to TRUE before initializing our ClientConfiguration.

    setenv("AWS_EC2_METADATA_DISABLED", "TRUE", true);
    _clientConfig = Aws::MakeShared<Aws::Client::ClientConfiguration>("mem_AwsClientManager");

@jstock78 Thanks! We will try this as well...

@jstock78 Thanks! We will try this as well...


Leveraging Lambda Powertools Logger for querying in Athena or other AWS services

Runtime:
Python
Is your feature request related to a problem? Please describe
IHAC who was looking into the benefits of the logger formatting but was having a hard time getting Athena to ingest the logs from Cloudwatch and create a schema out of them and was wondering if there was any consideration on the logger format to be  more friendly to data ingestion by other services outside of Cloudwatch. For example as it is now this is how its outputting to cloudwatch with the powertools logger on:
2021-05-10T15:26:57.772Z {"level":"INFO","location":"checkOrderInput:50","message":"Checking required fields ['contact', 'address', 'id', 'expedite', 'installDate', 'addressValidationOverride', 'product', 'asset']","timestamp":"2021-05-10 15:26:57,771+0000","service":"new-order","xray_trace_id":"1-609950be-18655ee7e321f53ab8b4f629"}
with this format you could extract two columns

timestamp, 2. the entire message as a json struc column

or use Grok serDer in athena with regex to try to grab a pattern, in either case you would still need to run this through some type of data processing no different than if you used the built in python logger with custom configurations but slightly more challenging due to the JSON structure.
Describe the solution you'd like

Either for Powertools Logger to just output eachline as a JSON object (removing the timestamp at the beginning of the log so Athena/Glue can just use the built in JsonSerDer to parse it and create columns or some documentation or examples on how to leverage this logger formatting in queries and creating Metric Filters of them.
Describe alternatives you've considered

Using a messy DDL statement in Athena using Grok SerDer
CREATE EXTERNAL TABLE ugi(
loglevel string COMMENT 'from deserializer',
timestamp string COMMENT 'from deserializer',
service string COMMENT 'from deserializer',
traceid string COMMENT 'from deserializer')
ROW FORMAT SERDE
'com.amazonaws.glue.serde.GrokSerDe'
WITH SERDEPROPERTIES (
'input.format'='(?"level":"(.{4,10})),([^.]+)(?"timestamp":"(.{10,28}))"([^.]+)(?"service":"(.{3,10}))",([^.]+)(?xray_trace_id":"(.{30,40}))"')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://ugi/'
TBLPROPERTIES (
'CrawlerSchemaDeserializerVersion'='1.0',
'CrawlerSchemaSerializerVersion'='1.0',
'UPDATED_BY_CRAWLER'='powertoollogs',
'averageRecordSize'='191',
'classification'='powertoollogs',
'compressionType'='none',
'grokPattern'='(?"level":"(.{4,10})),([^.]+)(?"timestamp":"(.{10,28}))"([^.]+)(?"service":"(.{3,10}))",([^.]+)(?xray_trace_id":"(.{30,40}))"',
'objectCount'='1',
'recordCount'='1',
'sizeKey'='191',
'typeOfData'='file')
Just using the built-in python logger and use the traditional CW-Firehose-S3 architecture for streaming the logs into an S3 bucket in Athena to avoid parsing the JSON structure.
If you provide guidance, is this something you'd like to contribute?
I am not the best developer but sure! I could help
Additional context

Providing some examples of how others have leverage the powertools logger or any use cases where this logger has made operational tasks easier would be very valuable and easier to sell to customers. Right now outside of the nice structured uniform formatting it creates in Cloudwatch logs I do not see another benefit of then using this data efficiently

@heitorlessa Yeah I think your proposal makes sense, its more on documentation than feature per se.
I essentially was thinking that since Powertools already was structuring the logs as JSON that it would reduce the Pre-Processing steps that someone would have to do in order to create a centralized logging architecture with the least amount of cost (our customer is cost conscious) through the steps you mention below. So in my mind I thought it would be able to use the JSON SerDer that Athena/Glue have built in to quickly make the schema based on the JSON structure but that timestamp CW adds essentially messes it up. In short I was trying to leverage some type of documentation or use cases to show the customer how they can leverage this format to make their log aggregations easier and operational day to day analysis easier than just using the Python Built In logger and defining all the structure themselves similar to how the Metrics module significantly reduces the overhead of creating Custom Metrics and then being able to use those custom metrics for alarms and notifications.

"In other projects I exported CloudWatch Logs -> Lambda for pre-processing
-> Kinesis Firehose -> S3 -> Glue -> Athena. Pre-processing Lambda would do
two things: 1/ only use the payload nothing else, 2/ append a based64
newline to the payload because Kinesis Firehose doesn’t and it breaks Glue
and Athena"

@mwarkentin -> I was trying to figure out how to query in cloudwatch Insights but was not having great success I will take your example and re-try the only limitation for our use case with insight is that you would have to go into each individual accounts to look at the logs as opposed to managing them all in a central Security account.

@rickyaws querying with Cloudwatch Logs Insights works great [1] and you can automate queries via APIs, create dashboards, etc. The way Logs Insights work with the CWL string format directly makes querying this very easy. Also check using Contributor Insights as it is a very useful feature (watch out for the costs though)
Now, The idea of using Athena directly to parse these logs is going to become very complex (if at all possible) due to two facts:

the "schema" might change with each logged line. This will be both from a code perspective (as new structure may be logged, for example, if the user outputs a dictionary) and also because the user doesn't have complete control of what will be logged from the environment: i.e. the log lines that are created by the worker itself which aren't handled by the powertools logger (i.e. START, REPORT and END lines, exception errors, print copmmands, etc)
Athena doesn't handle variable schemas very well as the table definition is pretty static - Using ORC or Parquet may help a bit, but again, the nature of flexible logging makes it very hard for a more static tool to parse this at will.

How the transport from CWL to Firehose to S3 will wrap the messages to pass along the different layers.

Trying to use Grok or regex to parse the logs, will make it very hard to maintain and process for the same reasons.
As an alternative, you could use the JSON format on the subscription filter which will properly wrap all the output from the lambda logs (including powertools logger) into a single field named "message" as an encoded string, but Athena isn't very powerful to handle this back to json AFAIK.
i.e.
[ ...
{
      "id": "36168440376909863750225378866398863959406407651950067771",
      "timestamp":<PHONE_NUMBER>630,
      "message": "REPORT RequestId: 4fae6371-c5ce-432d-812e-67cd621e2819\tDuration: 298.08 ms\tBilled Duration: 299 ms\tMemory Size: 128 MB\tMax Memory Used: 90 MB\t\nXRAY TraceId: 1-60ab737e-47c4b2fd43ec6df90c90c3c4\tSegmentId: 59a6cccc6f62a3d8\tSampled: true\t\n"
},
{
      "id": "36168440376040134687482684563878970946773121553216831544",
      "timestamp":<PHONE_NUMBER>591,
      "message": "{\"level\":\"INFO\",\"location\":\"lambda_handler:76\",\"message\":\"Completed\",\"timestamp\":\"2021-05-24 09:35:58,591+0000\",\"service\":\"crawler_frontapi_save\",\"cold_start\":false,\"function_name\":\"ServerlessCrawlerStack-CrawlerFrontAPISaveFunction-1C5FWAYTV559J\",\"function_memory_size\":\"128\",\"function_arn\":\"arn:aws:lambda:us-east-2:123456789012:function:ServerlessCrawlerStack-CrawlerFrontAPISaveFunction-1C5FWAYTV559J\",\"function_request_id\":\"4fae6371-c5ce-432d-812e-67cd621e2819\",\"parse_id\":\"#2021-05-24T09:35:58.491645\",\"url_site\":\"\",\"xray_trace_id\":\"1-60ab737e-47c4b2fd43ec6df90c90c3c4\"}\n"
},
...]

[1] As an example about the power of CloudWatch Logs Insights, this is a very useful query:
stats count() by level, service as cnt
| filter not isempty(service ) and not isblank(level)
| sort cnt desc
| limit 20

After doing some additional research, I don't think Athena will be a good alternative to query structured logs due to the variety of fields that can be produced on each output and there is nothing in Lambda Powertools itself that could improve on this situation. If any, additional work should be done on Kinesis Firehose to export only specific records and of these, only specific fields.
I suggest closing this request as it is my understanding this is not something that Lambda Powertools itself could or should handle.

Agree, thanks a lot @pcolazurdo for taking this for a spin!
@rickyaws suggestion moving forward is to centralise these logs into ElasticSearch, and to add +1 to CloudWatch Log Insights cross-account search.
If ElasticSearch/ELK isn't an option here due to management, you can use Kinesis to centralize CloudWatch Logs into another account - helps with management but increases costs too.
Before we close it, I'll transfer it to Python repo, and update Logger FAQ to have this info more broadly available.
Thank you all!


Checksum label is missing when prune is false

When prune is true, child objects of Kustomizations have these labels:
labels:
  kustomize.toolkit.fluxcd.io/checksum: 21ff4959f5c2052e702115c585ea8cf694c67613
  kustomize.toolkit.fluxcd.io/name: flux-system
  kustomize.toolkit.fluxcd.io/namespace: flux-system

When you set prune to false, the checksum label is dropped:
labels:
  kustomize.toolkit.fluxcd.io/name: flux-system
  kustomize.toolkit.fluxcd.io/namespace: flux-system

Even if the user does not have garbage collection enabled, it is still very useful to have these checksum labels because you can use them to identify stale resources.
This would for example be useful in a flux tree view or a web interface like the Flux UI that shows which objects are managed and which are not.
I don't see any reason not to include these checksums.
It should be a relatively safe change for existing Kustomizations when kustomize-controller is updated, but it will cause every object to be patched with new labels.

We used to have the checksum labels no matter if GC was enabled and that caused many issues for our users, controllers like Kafka, MySQL and others would delete the stateful sets every time the label changes, we also spam with alerts since all objects are changed.

Fair enough, although that sounds like the behavior with prune is suboptimal in those cases.
Would a matching annotation do any harm?
Could we modify the GC to filter annotations and take a small performance hit?

An issue with the above suggestion is that the GC gets slower linearly as garbage grows, but maybe it's worth the tradeoff.
Is the issue with the Kafka/MySQL controller that the labels are being copied down to the child STS objects from a parent CR?  (and the labels aren't stable)
I assume using annotations would fix that since they are designed to be a RW extension point for API's.

Could we modify the GC to filter annotations and take a small performance hit?

I don’t think you can query based on annotations and fetching all objects of a kind could be expensive and due to rate limiting, on some distributions it will result in timeouts.

If the label selector is only the kustomization Name + Namespace, we can still query more efficiently, but it would include all of the garbage.

Noted in conversation with Stefan, a concern with merging #348 with the current state of kustomize-controller is that it will cause Events to be generated whenever the kustomize build output changes.
The exact desired Object Tracking behavior and constraints are not super clear to me yet.

I know the kustomize-controller's interaction with the Kubernetes API needs to be performant.
Users of the Kustomization Status fields and related Events should also have a good experience.

a.
Currently, when Prune is enabled and the checksum changes, every child object of the Kustomization will be updated with new labels.
This can sometimes cause undesireable reconcile behavior in other 3rd-party controllers.
If the checksum changes, I also expect an Event recording the Kustomization was updated.
Annotations are intended to be arbitrary extensions, so adding them when the labels are already changing in the Prune Enabled case seems inconsequential to me.
b.
Currently, when Prune is disabled, the checksum is not included in any objects, only the static Flux Kustoimzation labels are, so no updates to other objects can occur unless they actually change.
Objects that don't change will never be reconciled by downstream controllers.
Adding annotations is less likely to change this but there may be some additional overhead to consider now that it's necessary to update every child object of the Kustomization, even when Prune is disabled.
c.
I'm not fully understanding the relation to kubectl diff and Drift Correction.
Drift Correction seems possible to bolt-on before the objects are reconciled into the Cluster, regardless of what we do with the labels.
If we are trying to filter Events on Kustomizations only when the checksum changes, that already seems possible and adding the checksum annotation to downstream objects shouldn't affect that.
The only consequence I can think of:
if we introduce this behavioral change to kustomize-controller, teams who have Prune disabled will have a one-time change calculated, when they upgrade kustomize-controller and the downstream annotations are added.
This behavior is already true when you enable Prune on an existing Kustomization.

If we could clarify together what sort of behavioral constraints we want from kustomize-controller's Object Tracking, we'll come up with the right solution.
I'm almost inclined to suggest that we just use Prune-labels (not annotations) all the time and include a boolean annotation or spec field so that people who really need to opt-out of Object Tracking checksums can do so.

@stealthybox Are we able to control these Kubernetes events in some way or are these just a construct of updating the object in general and can't be avoided?

For users that have GC disabled this is what currently happens:

user: creates a Kubernetes Deployment, Kubernetes Service and a Flux Kustomization in Git
kustomize-controller: labels the Deployment and Service and applies them on the cluster
kustomize-controller: issues event with deployment created, service created
user: updates the Kubernetes Deployment in Git
kustomize-controller: issues event with deployment changed

If we merge Jonathan's PR:

user: updates the Kubernetes Deployment in Git
kustomize-controller: annotates the Deployment and Service and applies them on the cluster
kustomize-controller: issues event with deployment changed, service changed

As you can see with that PR we'll be sending "bogus" events, the service didn't changed in Git nor in cluster, but kubectl apply reports a change since the annotation that we generate did changed.
To resolve this issue, instead of relying on kubectl apply output, we should run kubectl diff first, parse the output, remove all objects where only the checksum annotation changed, and use this list for the events. Once we do that, we can drop the checksum label, and use the annotation for GC as well.

Hopefully cache-hits in the apply for all of the objects from the diff will make adding the diff unnoticeable.

So one thing I found when testing out kubectl diff in one of my clusters is that it failed. The reason it failed is because I have OPA Gatekeeper mutating webhook n the cluster. It returns an error message.
Error from server (BadRequest): admission webhook "mutation.gatekeeper.sh" does not support dry run

The reason this occurs is because the mutating webhook has set sideEffects: Unknown which will auto-reject dry run commands.
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#mutatingwebhook-v1-admissionregistration-k8s-io
We need to document that flux depends on dry run working and not  being limited by a webhook. I will try to get this solved in the gatekeeper project, but I am unsure if there are other projects which may cause flux to stop working.

kustomize-controller uses dry run when spec.validation is set, the diff comes after the validation step, so Gatekeeper would block Flux no matter if we add the diff step or not.

Yes but validation is an optional setting that is disabled by default? This feature would run dry run for every apply.

Validation is enforced at bootstrap, so anyone running bootstrap via the CLI or Terraform will run into this issue. I can see how doesn't apply to Azure... Guess we'll have to make diff optional too but then we are back were we're started, without diff you get events spam and so on.

@phillebaba I'm for dropping validation: none in v1beta2, so then it will no longer matter if we use diff, it will always fail.

I think responsibility should fall on the end user to make this work. I dont normally use the diff or dry run commands which is why I have totally missed this. I think we should move forward with having dry run as a required feature and document this requirement as I bet it will become more obvious when v1beta2 is released.
I created an issue in the gatekeeper repository, and my guess is that kyverno does not have this problem, but I will have a look just to be sure.

Every kyverno release is being e2e tested with every Flux release :) See https://github.com/fluxcd/flux2-multi-tenancy/pull/30

@phillebaba you may want to review https://github.com/fluxcd/kustomize-controller/pull/362 and express your concerns there, maybe in that PR we should run the diff only if validation is enable so that Flux doesn't crash when used with OPA Gatekeeper?

@phillebaba I looked it up and Kyverno uses sideEffects: NoneOnDryRun, a server side dry run or diff would trigger it:
$ k apply --dry-run=client -f fails-policy.yaml
kustomization.kustomize.toolkit.fluxcd.io/podinfo2 created (dry run)

$ k apply --dry-run=server -f fails-policy.yaml
$ k diff -f fails-policy.yaml
Error from server: admission webhook "validate.kyverno.svc" denied the request:

resource Kustomization/apps/podinfo was blocked due to the following policies

flux-multi-tenancy:
  serviceAccountName: 'validation error: .spec.serviceAccountName is required. Rule
    serviceAccountName failed at path /spec/serviceAccountName/'

We have in #379 an implementation that uses diff, but this falls short on numerous cases (see https://github.com/fluxcd/kustomize-controller/pull/379#issuecomment-874705131) given this situation I'm inclined on closing this issue as wontfix.

I'm going to go ahead and close the issue. Thanks for the discussion.


Add blog post for OME NGFF 2021 updates

As mentioned in the minutes of the last meetings, includes:

the NGFF paper
the 0.2 and 0.3 specification updates
the new samples

Staged for review at https://snoopycrimecop.github.io/www.openmicroscopy.org/2021/12/15/ngff.html

Now staged at https://snoopycrimecop.github.io/www.openmicroscopy.org/2021/12/16/ngff.html

Latest changes are deployed at https://snoopycrimecop.github.io/www.openmicroscopy.org/2021/12/16/ngff.html.

No more embedded browser since there is the confusion about how it loads?

Unlike the previous blog posts which were focused about a new spec and a set sample data, this blog post includes more information including the paper. I found that including the embedded browser was diluting the message and also further pushing the list of samples to the bottom of the page.
Probably a take home is to publish shorter blog posts more often ;)

Probably a take home is to publish shorter blog posts more often ;)

;) :+1:

LGTM. (Happy to postpone the cosmetic issues to a follow-up)


validate the keys of params file

Sometimes, the correct key may not be used because of a typo. The params object should check and not allow any key.

This would be useful. Also mentioned in https://github.com/jonescompneurolab/hnn/issues/22

It could be a good first issue for @rythorpe

@jasmainak Since I haven't encountered this problem myself, is the idea here to add a method referenced by Param.__init__(), Param.__getitem__(), and Param.__setitem__() that validates all new input keys according to a preconfigured set of possible param keys?

yep, you would need to the validation only in setitem I think. You should raise ValueError etc. when the right parameters are not present or they are not in the right ranges. Since the param files are all converted to dictionary ultimately, you could probably use JSON schemas to make your life easier.

This is no longer relevant as we are moving to create a new file format with @gtdang's work


Remove the tabindex?

Thanks for the library - looks great! Any chance of an option to remove the tabindex=0 - it causes an ugly rectangular outline of the control when it is focused.

I could make a style for :focus state that isn't as flashy as the default

should be all fixed up now!

Perfect, thanks!


How to set authorization config?

Hi,
I wasn't able to find out how to set the authorization settings. It seems the configmap generated always leaves them empty. As the configis not dynamically created the same way it happens in the temporalio/server docker, setting env vars on the worker service using overrides was also not successful.
Am I missing something obvious?

Hi @smndtrl !
Thanks for filling the issue. You're right, the authorization setting is missing in the TemporalCluster CR.
I'll add this for the next release (v0.16.0).


Optional flag to prevent evmtool from shutting down

In https://github.com/hyperledger/besu/issues/1470 an optional mode was introduced that allowed us to pipe in multiple tests over the stdin pipe.
Currently a lot of time during fuzzing is spent on setting up the JRE.
Having an optional flag (--keep or --pipemode or something) would enable us to spin up a besu instance and send state tests to it. This would greatly reduce the overhead of fuzzing.

Where can i find details on how this fuzzing happens, and how it interacts with besu via the command line?

This already works :see_no_evil: There was a tiny bug that was fixed some time ago

Re fuzzing: This is how we interact with besu: https://github.com/holiman/goevmlab/blob/master/evms/besubatcher.go