RFC: Moving to git-flow type development

Hey everyone,
bioconda has a nice bot which automatically updates the bioconda packages when there is a new release, so there is more reason than ever to switch to a more git-flow type development, as we can push patches to master, cut patch-releases and they will automatically get built into the bioconda package.
This will require a few changes, right now our development flow looks like this:

master should be bug free. new features and patches both get merged directly into master once they are shown to work
new features get forked from master, worked on their own branch and pushed back to master when they are complete
bug fixes get pushed directly into master
new releases happen once we've stabilized new features or there are major bug fixes that need to be in every version of bcbio
we increment version numbers as 1.xx.

This flow has very little overhead, but has some disadvantages. Namely, if we push new features or changes to master, users cannot get bug fixes without also pulling in the new features. It also makes 'master' a little less stable than it could be, since new features appear in master which is destabilizing.
The git-flow workflow looks like this:

A develop branch is created from master
A release branch is created from develop
Feature branches are created from develop
When a feature is complete it is merged into the develop branch
When the release branch is done it is merged into develop and master
If an issue in master is detected a hotfix branch is created from master
Once the hotfix is complete it is merged to both develop and master

so there is not that much extra overhead, mostly just to remember to implement new features from develop instead of on master. We can still do bugfixes to master like we have been doing. This will let us keep master more stable than it is currently and let users stick with it for longer while we work on other, non bugfix improvements. It also lets us use the new bioconda bot and keep our packages on bioconda up to date, bug-free and stable.
I think this is a good switch, it doesn't add very much process and has some useful benefits to it. There is a git-flow tool that makes these processes a little simpler to do: https://danielkummer.github.io/git-flow-cheatsheet/
A brief description of the git-flow workflow is here:
https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow
and here is the original git-flow post: https://nvie.com/posts/a-successful-git-branching-model/
@chapmanb @naumenko-sa @hackdna @lpantano what do you all think about switching over to this method?

I think is a good idea!

Yes, I think it's a great way to manage a larger project like bcbio. This should also make upgrading more intuitive since -u development would actually pull from the develop branch instead of the master.
Btw, any more info on that bioconda bot?

+1. I agree this way it is is more organized.
@Rory could you please clarify the interaction between bug fixing process and new development? Now when everything is happening in the master, people find and report bugs really fast, so they got fixed fast, because otherwise, the master is not working. Also, these bugfixes are one of the sources of new features. When we split bug fixing and dev, devs have to make sure they got the last bugfixes before merging, right? And they will need to spend more time testing, which was done by users before? S.

Hi Sergey,
With this method the -u development option will pull from the develop branch and -u stable will pull from master. The new master branch just has bugfixes. The new develop branch will be exactly what the master branch is now. It should still work, and should be bugfixed and also has new features in it.
I don't think it should add anything to our private testing, folks can still use -u development if they want to be on the cutting edge and folks running -u stable will still find bugs. This method should fix a bunch of issues that we get where the answer is 'just upgrade to development and it will fix it', since it will be already fixed on the stable branch.

@hackdna the new bioconda bot looks for releases on packages it knows about and automatically builds the package. So if we push regular hotfix type releases the bot will build the package for us, saving us a step. See https://github.com/bioconda/bioconda-recipes/pull/16410 as an example

As you mentioned, the nvie model is good, but I would prune all branches except master and feature branches. The (cognitive, newcomers) overhead of managing development, hotfix, etc .. is not worth it, in my experience.
master + feature branches is much faster (and you'll perhaps find that it does not break things that often), but as always YMMV.
/cc @vladsaveliev @ohofmann @pdiakumis

Great, I've been trying to develop using the git-flow model for the past few weeks, and I think I agree with Roman regarding keeping it the way it is with master + feature branches without the overhead of a separate develop branch for the time being. If we were doing a lot of feature development, or we had lots of folks developing, it might make more sense to move to that model. But right now it's just a few of us for the most part so the additional overhead isn't really useful, just aggravating. Thanks all for the comments!

Just to illustrate the benefit of git-flow with a concrete example, this could have been applied as a hotfix to the stable version and thus not require the user to switch to the development version (which could contain other bugs): https://github.com/bcbio/bcbio-nextgen/issues/2795#issuecomment-517561579


Ctrl + A not working!!!

I've tried every trouble shouting mentioned and the select all function (Ctrl + A) still DOES NOT WORK!!! Ctrl C, Ctrl V, Ctrl X, etc all work.  Ctrl A is the only one that does not work.  Just stopped worked suddenly.  Has anyone had this issue that can help me?  It doesn't work in anything.  Not in Word, Excel, or anything else.  I use Windows 10 Pro.  Have a Logi Ergo K860 keyboard & a HP laptop. I've downloaded every possible driver and nothing either. There are no Windows updates, BIOS updates, or anything like that needed.  HEEELLLPPP MEEEE!

Hey @gabycmc,
thank you for your submission.
I'm not sure, but this plugin doesn't bind the hotkey CTRL+A. By default, the only hotkeys are CTRL+T, while selecting an entry and CTRL+ALT+T for global auto-type. So this plugin shouldn't be responsible for your issue. Did you install other software, like AutoHotKey?
I'm busy right now, but I will try giving you further advise this evening. Please stay patient.
Kind regards

@tiuub
Couldn't figure it out either huh?

As I said, this plugin does not bind the hotkey CTRL-A by default. So this issue can't happen due to this plugin.
The only thing which is possible, is that you have bind CTRL-A as global hotkey for entering the otp. Therefore please navigate to KeePass->Tools->KeeOtp2->Settings and set the hotkey sequence to anything else than CTRL-A.
If this doesn't work for you, you have to give me more details about this issue. Like, does it also happen if you completely close KeePass. At the moment, I can't reproduce your issue.
Kind regards


Add Diamond exercise

Add Diamond exercise

@masters3d @nywilken I update my version of configlet and generated a new UUID


Add release notes for 2.2.1

Signed-off-by: Mohammad Qureshi<EMAIL_ADDRESS>Description
Adds release notes for 2.2.1
Issues Resolved
[List any issues this PR will resolve]
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #517 (527b2f3) into 2.2 (fc0b0a5) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                2.2     #517   +/-   ##
=========================================
  Coverage     61.45%   61.45%
  Complexity      112      112
=========================================
  Files            73       73
  Lines          2467     2467
  Branches        266      266
=========================================
  Hits           1516     1516
  Misses          773      773
  Partials        178      178

Flag
Coverage Δ

opensearch-notifications
61.45% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Standard 'This extension requires the Microsoft ODBC Driver 11 for SQL Server', but unsolvable

I had the 11 Driver installed for the past year, but recently installed the 13 Driver.
Now, php 5.6 keeps returning the standard driver 11 error.
I've confirmed that both 11 and 13 are installed, in the ODBC Administrator, and also re-installed driver 11.
And, I've got
[PHP_SQLSRV_56_NTS]
extension=php_sqlsrv_56_nts.dll
extension=php_pdo_sqlsrv_56_nts.dll
in the php.ini file, along with the actual files in the ext folder.
But I keep receiving the error.  Will I need to uninstall 13?  I'm using it for non-Php purposes.

Hi @GHExplorer,
ODBC 13 should not really affect your PHP5 applications, because connection string is hard-coded to use ODBC 11. You can check the source code here.
Note that, we do not maintain PHP5 drivers anymore. Is there any reason why you are not using PHP7+?

I just tried PDO_SQLSRV with PHP5 on a VM which has both ODBC11 and ODBC13 installed and was able to connect.
php -v

PHP 5.6.28 (cli) (built: Nov  9 2016 06:40:27)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies

ODBC 11 version: 12.2
ODBC 13 version: 13.1

I use PHP 7 for personal projects.  But our agency, is using 5.5 and 5.6, and they're not ready to upgrade, yet.  I've been able to test and run most of their code in PHP 7, but some of the code doesn't work there, and it's in some of our legacy libraries.

Hi @GHExplorer ,
Were you able to solve the issue?

Not yet.  I was off yesterday, and will play with it, during today.

Same error.  It was originally non-thread safe.  But for the heck of it, I moved to thread safe, replacing the nts files with nt in the extensions folder and confirmed with phpinfo.
No go.

After reinstalling VS 2012 runtime, and a reboot, I checked both the odbc administrator and the registry.  ODBC Driver 11 for SQL Server is no longer in the list.  Driver 13 is still there.
I tried re-installing #11, and it ran successfully.  But it still doesn't show up, in the odbc administrator, nor the registry.
This is the problem, obviously.  At some point, driver 11 was removed, and can't be reinstalled.  I'm going to try uninstalling driver 13, even though some of my User DSNs depend on it, and then re-re-install driver 11.

Ok, got it.  I don't know at what point the 11 drivers were uninstalled - I've been using them for a while.
But when trying to reinstall them, the components were unchecked, and therefore the reinstall wasn't even attempting to put them back.
Everything's good now, thanks.

@GHExplorer glad to hear that! I'll close the issue then


Pepr's UDS Core Soak Test Should have App Deployed

Is your feature request related to a problem? Please describe.
We test new Pepr features and releases by soaking them in an UDS Core cluster for a couple of days. We want to take this practice further and deploy an application like BookInfo or PodInfo to ensure the correct artifacts are generated at deployment time and continue to work over time.
Describe the solution you'd like

Given Pepr/KFC has a new change that and we want to ensure that it will not affect UDS Core/Delivery efforts
When we soak test to ensure everything is working properly
Then we also have an application deployed on top of UDS Core.

Describe alternatives you've considered
(optional) A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

we are focusing on the informer in our soak tests to focus our testing. Due to that, we are creating a cronjob that creates pods that pepr should delete. We do not need a another app.  We are using the hello-pepr-soak in pepr-excellent-examples


Support DML in IF/THEN/ELSE expressions

As we've discussed (and, tragically, recommended to users), we can
rewrite the conditionals into for loops:
    for b in COND union (
      {
	(for _ in (select () filter b) union (IF_BRANCH)),
	(for _ in (select () filter not b) union (ELSE_BRANCH)),
      }
    )

The main fiddly part is preserving the correct
cardinality/multiplicity inference. I did this by adding a
card_inference_override field to Set that specifies another set that
should determine the cardinality.
I don't love this, but I haven't thought of a cleaner approach that
doesn't give up the benefits of the desugaring approach.
We need more testing but I wanted to get something up for people to
look at / we can catch up on testing after the feature freeze if
needed.
Fixes #4437.

Follow-up tasks:

[ ] Support DML in coalescing ??
[ ] Don't generate conflict CTEs for potential conflicts between arms if the IF?


Can not open subtitles on hls

Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
v4.3.2
Can you reproduce the issue with our latest release version?
Yes
Can you reproduce the issue with the latest code from main?
Yes
Are you using the demo app or your own custom app?
No
If custom app, can you reproduce the issue using our demo app?
What browser and OS are you using?
Chrome 109.0.5414.87 macOS 13.1 (22C65)
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
What are the manifest and license server URIs?

What configuration are you using?  What is the output of player.getConfiguration()?

What did you do?

Open https://shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=en-US;uilang=en-US;asset=https://storage.googleapis.com/shaka-demo-assets/apple-advanced-stream-ts/master.m3u8;panel=ALL_CONTENT;panelData=HLS,MP2TS;build=uncompiled
Select a subtitles

What actually happened?

the following error throws, and subtitles not working.

Cannot read properties of undefined (reading 'push')

What did you expect to happen?
not to throw

v4.3.2  ❌
v4.3.1  ✅
v4.2.6  ❌
v4.2.5  ✅

the bug is introduce by https://github.com/shaka-project/shaka-player/pull/4824

@joeyparrish can you review it?

It seems that it broken for all HLS content.
https://shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=en-US;uilang=en-US;asset=https://storage.googleapis.com/shaka-demo-assets/angel-one-hls/hls.m3u8;panel=ALL_CONTENT;panelData=HLS,MP4;build=uncompiled

Acknowledged.  I'll work on it today, and delay release plans until this is fixed.

The fix will be in v4.3.3 and v4.2.7.


Add Spotless Script to hack/run.sh

Problem
Currently, spotless is run when someone runs ./hack/update-codegen.sh, or if they know how to manually use spotless itself. We should abstract that away from contributors and allow them to run ./hack/run.sh format-java and have it format all the java code for them.
Persona:
Which persona is this feature for?
Contributors
Exit Criteria
Running ./hack/run.sh format-java formats all the java code.
Time Estimate (optional):
How many developer-days do you think this may take to resolve? 1
Additional context (optional)
Add any other context about the feature request here.

Hello @Cali0707 I would like to work on this issue can you please guide me on how to get started

@MeenuyD thanks for picking this up! In general, the pattern for adding a command to ./hack/run.sh is to add the command into https://github.com/knative-extensions/eventing-kafka-broker/blob/main/hack/run.sh, and then in the big set of if statements there, run whatever needs to be run (either calling a new script or running a few bash commands). Here, you would want to run ${ROOT_DIR}/mvnw spotless:apply.
/assign @MeenuyD

/assign @MeenuyD

Completed by #3333


Generate Audit Report is too slow

Can we optimize the processing of this report?

#497 is an optimization that helps this.
If we get to a point where audit report generation is complicated enough that optimizations like that don't get us below the request timeout (do we know what that limit is?) we could consider decoupling report generation from request lifetime -- e.g. the client sends one request to start generation, which kicks off the process and returns immediately; then the client polls in separate requests to find out when the job is done.

Closing for now, will ask partners to resubmit issue if it's still too slow.


Apologies - More of a Question

First-Off, allow me to apologize. I wasn't sure how else to reach someone directly with experience in coding Gw2 Addons. Reddit felt like a little too much of a random chance of actually getting someone.
I'm actually considering attempting an addon myself, but I'm not sure how feasible it is. I was considering studying this addon, but I'm relatively new to C++ without much experience.
Essentially, the addon would do a very simple, non-combat thing: It would store dye sets for Outfits, Gliders, and Mounts, so if you change to a specific outfit, glider, or mount, it would automatically load the dyes you'd last used with that item.
Is this something that would be possible in the game, or would it be a fool's errand to attempt?

That's well outside the bounds of this addon or what you can do with simple addons. You'd have to get into reverse engineering, which is a pretty complex topic.
You should ask this kind of thing on the GW2 Addon Devs Discord server.


Recursive types?

I'm submitting a ...
[ ] bug report
[x] feature request
[ ] question about the decisions made in the repository
[ ] question about how to use this project

Summary

I'm wondering if it's possible to support recursive ADTs, for example a classic linked list. My own attempts at a generalised ADT library have failed in this regard since the TS compiler can't handle the recursive type aliases

Hey,
Thanks for the request!
it is technically possible to use an interesting trick with const declarations
const Obj = {
  prop: Obj // Error: Block-scoped variable 'Obj' used before its declaration.
}

const Obj2 = {
  prop: ()=> Obj2
}

console.log('hey it works', Obj2.prop())

Example: https://stackblitz.com/edit/recursive-type?file=index.ts
I was thinking about it, but in my own practice I haven't found a need to do it.
A potential API:
import {Union, of, Self} from 'ts-union';
const List = Union({
  Empty: of(null),
  Elem: of<number, Self>() // 'number' is a type of values to store
})

Then type Self can be substitute (I'm not sure if it will actually work in ts though).
Or something like that
import {Union, of} from 'ts-union';

const List = Union(a => ({
  Empty: of(null),
  Elem: of(a, ()=>List) // or a wrapper defer(()=>List)
}))

Note that the version above is a generic variant, so it is akin to just classic List.
I'm not sure if this complexity is worth it, but feel free to maybe sketch out a syntax + types to brainstorm :)

here's a use case for example, to model effects like PromiseAction - i implemented a quick equivalent in this case but would have used ts-union if there were an established way to reference self in types.
the Self idea is very interesting, i like it! i will see if i can come up with a scratchpad...

I kept plugging away - here's a proof-of-concept with the substitution approach

@m0nkfish that's awesome!
I wonder how matching will work with Self, essentially matching is also recursive. Do you have an API sketch in mind?

@twop match should be unaffected since Self is substituted in the case creation function. if you want to define a recursive function you can do so:
const List = Union({
  nil: Type<null>(),
  cons: Type<{ head: number, tail: Self }>(),
})
type List = typeof List.T

function sum(list: List) {
  return List.match(list, {
    nil: () => 0,
    cons: ({ head, tail }) => head + sum(tail)
  })
}

Upd: I didn't forget about this feature request. I'm working on a new feature: matching tuple of unions, and I don't want to focus on a new feature until I ship an MVP of matching tuples.
@twop  thanks again for the patience and working on PoV for recursive types ! <3


Feat/income tax

Changes made

Timeseries component - added tooltipItemSort to re-order tooltips, generateLabels prop used for mixed type (line and rect) legend items. The legend onClick function is coded out to ensure interactivity (can hide and show datasets) when generateLabels is used.
Income taxation page is created - singular stacked timeseries chart for now.

Considerations

dashboard index page should be updated with the correct route (income-taxation -> income-taxation/overview) from BE's end, after this PR is approved.
The tab layout is modelled after perkeso page, since election dashboard also uses it, maybe it should be refactored into a separate layout component

The tab layout is modelled after perkeso page, since election dashboard also uses it, maybe it should be refactored into a separate layout component

Good idea, if there are more similar dashboards in the future. Can create a ticket on trello for this. I'll merge this for now


Returns column headers.

Codecov Report

Merging #10 (2b3ecde) into main (a3f34e4) will increase coverage by 0.50%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #10      +/-   ##
==========================================
+ Coverage   81.74%   82.24%   +0.50%
==========================================
  Files          37       37
  Lines        2114     2174      +60
==========================================
+ Hits         1728     1788      +60
  Misses        347      347
  Partials       39       39

Impacted Files
Coverage Δ

internal/sql/type_mock.go
0.00% <ø> (ø)

internal/sql/engine/engine.go
100.00% <100.00%> (ø)

internal/sql/expr/binary.go
100.00% <100.00%> (ø)

internal/sql/expr/column.go
100.00% <100.00%> (ø)

internal/sql/expr/expression.go
100.00% <100.00%> (ø)

internal/sql/expr/expression_mock.go
100.00% <100.00%> (ø)

internal/sql/expr/scalar.go
100.00% <100.00%> (ø)

internal/sql/expr/unary.go
100.00% <100.00%> (ø)

internal/sql/parsing/parser/parser.go
100.00% <100.00%> (ø)

internal/sql/planning/plan/create.go
100.00% <100.00%> (ø)

... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3f34e4...2b3ecde. Read the comment docs.


Document all available API endpoints

Checklist

Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?

Yes

Are you sure you're not using someone else's docker image?

Yes

Have you searched for similar issues (both open and closed)?

Yes

Describe the bug
Several key API endpoints are not included in the OpenAPI schema; namely, deleting a proxy host, details around the required fields for programmatically creating a proxy host, etc. I've had to use browser developer tools to reverse-engineer the path to discover this endpoint (which helped tremendously in quickly deleting proxy hosts)
Nginx Proxy Manager Version
2.11.2
To Reproduce
Steps to reproduce the behavior:
There is nothing to reproduce - the behavior is that the feature is missing altogether.
Expected behavior
Expectation is for all available API endpoints to be included in the API schema document (or a Postman, or equivalent, collection provided for easy import)
Screenshots
Screenshot shows the only available schema details (as displayed in Firefox from /api/schema)

Missing from this screenshot is the /DELETE endpoint (curl --location --request DELETE 'https://npm.mydomaincom/api//nginx/proxy-hosts/HOST_ID_NUMBER' \ --header 'Authorization: Bearer BEARER_TOKEN_VALUE_HERE)
Operating System
Docker container running on Ubuntu Server 22.04 LTS - underlying OS shouldn't play a factor here.
Additional context
Happy to expand further if above information isn't sufficient, thanks!

https://github.com/NginxProxyManager/nginx-proxy-manager/discussions/3195
i talk about the API and wheres a good place to look for info here..

Bad bot - the issue isn't stale!

@jc21  would appreciate any guidance you can offer on this issue please and thank you!


OLS-583: Bump-up langchain-ibm to 0.1.7

Description
Bump-up langchain-ibm to 0.1.7
Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[x] Bump-up dependent library

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.54%. Comparing base (ed679d4) to head (25ddd19).
Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #970      +/-   ##
==========================================
- Coverage   95.18%   92.54%   -2.65%
==========================================
  Files          54       54
  Lines        2120     2118       -2
==========================================
- Hits         2018     1960      -58
- Misses        102      158      +56

see 5 files with indirect coverage changes

/retest

/approve


🐞 [BUG] - Card under text is not proper responsive it move out of the card , I fix this issue

Issue Verification

[X] I have verified that this is a new bug report.
[X] I'm interested in working on this issue
[X] I'm part of GSSOC organization

Describe the bug
🐞 [BUG] - Card under text is not proper responsive it move out of the card , I fix this issue, plz assign me this issue

Expected behavior
No response
Steps to reproduce
No response
Additional Context
No response

i found this issue , plz assign me for fix it.

Duplicate issue, someone is already working on it


Fix Iris-Clustering Tutorial: In v0.6 the LearningPipeline is in the …

…"Legacy" namespaces
In v0.6 the LearningPipeline API is in the "Legacy" namespaces, such as:
using Microsoft.ML.Legacy;
using Microsoft.ML.Legacy.Models;
Summary
Describe your changes here.
Fixes dotnet/docs#Issue_Number (if available)

@CESARDELATORRE should the .csproj file be updated to reference the 0.6 version of ML.NET? The same remark to other two tutorials updates.

We can say so, but I think it'd be better just to say that you need to install/use the latest ML.NET NuGet package (which happens to be v0.6), so we won't need to update this point when releasing 0.7, which will also work the same way, etc.

@CESARDELATORRE the tutorial text doesn't specify the version to install, so it implicitly suggest the latest.
I'm concerned about the project that is in this repo:
https://github.com/dotnet/samples/tree/master/machine-learning/tutorials/IrisClustering
Usually the projects are kept buildable.

Right, that .csproj in the tutorial folder should be changed to use v0.6. But that code is used just for showing code in the Tutorial?
However, the working sample is here:
https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/getting-started/Clustering_Iris

But that code is used just for showing code in the Tutorial?

Correct. And it's also working, in case readers of tutorial would like to examine the whole project.

I was thinking of switching those to use the * notation instead of having to update the csproj for every release.


v4 SDK - missing "node_modules/openai/dist/index.mjs"?

Describe the bug
Trying to work out the new v4 beta SDK (love the new features!) - but constantly get an error during initialization for "missing node_modules/openai/dist/index.mjs"
I can't even see this file anywhere in the v4 SDK
Am I not doing things correctly? Any pointers very welcome.
To Reproduce
install new v4 SDK - from a zip file download
Code snippets
No response
OS
macOS
Node version
Node 18.17.0
Library version
openai SDK v4 beta

Ah, it's not intended for use via zip download – can you npm install --save<EMAIL_ADDRESS>Is there a reason you want to download via zip?
If that's a requirement, you should be able to install it locally with npm install; npm build; cd dist; npm link or an equivalent.

🙏 Thank you. This worked.


Feature: maintenance mode for admin

As an admin I can activate a maintenance mode.
End users are redirected to a static page.
We can use a Django middleware for that.
This would allow to perform spec migration when needed with new request blocked.

Done and merged in dev branch


correcting overfitting with log-likelihood

Should correct the overfitting problem when optimizing hyperparameters with cobyla in smt 7.0

:x: Build SMT 0.6.812 failed (commit https://github.com/SMTorg/SMT/commit/39b024d14e by @Paul-Saves)

:x: Build SMT 0.6.813 failed (commit https://github.com/SMTorg/SMT/commit/021cc156d7 by @Paul-Saves)

:x: Build SMT 0.6.814 failed (commit https://github.com/SMTorg/SMT/commit/9fc0c47606 by @Paul-Saves)


apacheHttpd: 2.4.34 -> 2.4.35

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/apache-httpd/versions.
meta.description for apacheHttpd is: '"Apache HTTPD, the world's most popular web server"'.

Checks done (click to expand)

built on NixOS
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/dbmmanage had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/envvars-std had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/envvars had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/htpasswd had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/htdigest had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/htdbm had a zero exit code or showed the expected version
/nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/ab passed the binary check.
/nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/logresolve passed the binary check.
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/httxt2dbm had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/htcacheclean had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/rotatelogs had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/checkgid had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/fcgistarter had a zero exit code or showed the expected version
/nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/httpd passed the binary check.
/nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35/bin/apachectl passed the binary check.
4 of 15 passed binary check by having a zero exit code.
0 of 15 passed binary check by having the new version present in output.
found 2.4.35 with grep in /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35
directory tree listing: https://gist.github.com/2117024b99120c15da0c3ed8cdf1b912
du listing: https://gist.github.com/661389ea6912268993c34204d4765412

Instructions to test this update (click to expand)

One-time optional setup to skip building using Cachix:

Install cachix from https://cachix.org/
Use r-ryantm's cache:cachix use r-ryantm

Test this update by entering a nix shell, seeing what is inside the
result, and if applicable, running some binaries:
nix-shell --pure -I nixpkgs=https://github.com/r-ryantm/nixpkgs/archive/524c2120f92581791fea8bc98e1ea6e745b4db52.tar.gz -p apacheHttpd
ls -la /nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35

cc @lovek323 @peti for testing.

Unexpected error: command failed with exit code 1 on x86_64-darwin (full log)
Attempted: apacheHttpd
Partial log (click to expand)
building '/nix/store/lpxhxb3579dw6d2magqkq6z7h8gpa999-httpd-2.4.35.tar.bz2.drv'...
/nix/store/qjk2h301faawvbnhv5azhyiaw6vrqalg-builder.sh: fork: retry: Resource temporarily unavailable
copying path '/nix/store/b96db2pgsjq264jc7hc6yrqnlnlv6lv0-apr-util-1.6.1-dev' from 'https://cache.nixos.org'...
/nix/store/qjk2h301faawvbnhv5azhyiaw6vrqalg-builder.sh: fork: retry: Resource temporarily unavailable
/nix/store/qjk2h301faawvbnhv5azhyiaw6vrqalg-builder.sh: fork: retry: Resource temporarily unavailable
/nix/store/qjk2h301faawvbnhv5azhyiaw6vrqalg-builder.sh: fork: retry: Resource temporarily unavailable
/nix/store/qjk2h301faawvbnhv5azhyiaw6vrqalg-builder.sh: fork: Resource temporarily unavailable
builder for '/nix/store/lpxhxb3579dw6d2magqkq6z7h8gpa999-httpd-2.4.35.tar.bz2.drv' failed with exit code 254
cannot build derivation '/nix/store/grcspag7rqn61b87sl4g33f4xxfkh9cj-apache-httpd-2.4.35.drv': 1 dependencies couldn't be built
error: build of '/nix/store/grcspag7rqn61b87sl4g33f4xxfkh9cj-apache-httpd-2.4.35.drv' failed

Success on x86_64-linux (full log)
Attempted: apacheHttpd
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/5k7fhnica7ylrvsl46jlyp8jcsqf0abg-apache-httpd-2.4.35-man
gzipping man pages under /nix/store/5k7fhnica7ylrvsl46jlyp8jcsqf0abg-apache-httpd-2.4.35-man/share/man/
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/5k7fhnica7ylrvsl46jlyp8jcsqf0abg-apache-httpd-2.4.35-man
checking for references to /build in /nix/store/5k7fhnica7ylrvsl46jlyp8jcsqf0abg-apache-httpd-2.4.35-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/327g2ib8igvmj6qvhjc9250xdc8z54mm-apache-httpd-2.4.35-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/327g2ib8igvmj6qvhjc9250xdc8z54mm-apache-httpd-2.4.35-doc
checking for references to /build in /nix/store/327g2ib8igvmj6qvhjc9250xdc8z54mm-apache-httpd-2.4.35-doc...
/nix/store/a9cbndza1f3k9hg812qz0dr7f205j5wz-apache-httpd-2.4.35

Success on aarch64-linux (full log)
Attempted: apacheHttpd
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/glx141kn3mg4kghfqz038syfb1i55a0f-apache-httpd-2.4.35-man
gzipping man pages under /nix/store/glx141kn3mg4kghfqz038syfb1i55a0f-apache-httpd-2.4.35-man/share/man/
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/glx141kn3mg4kghfqz038syfb1i55a0f-apache-httpd-2.4.35-man
checking for references to /build in /nix/store/glx141kn3mg4kghfqz038syfb1i55a0f-apache-httpd-2.4.35-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/k7lz1hg19v11hgrpcdw0r8yfw83fv7zs-apache-httpd-2.4.35-doc
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/k7lz1hg19v11hgrpcdw0r8yfw83fv7zs-apache-httpd-2.4.35-doc
checking for references to /build in /nix/store/k7lz1hg19v11hgrpcdw0r8yfw83fv7zs-apache-httpd-2.4.35-doc...
/nix/store/1kzjmjvjm4jk0icdywqp5zmvgz88f24s-apache-httpd-2.4.35


Update Source Installation For Jetson Users

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I've agreed with the maintainers that I can plan this task.

Description
Update the current source installation documentation to include how to install and compile differently Autoware with arm64 hardware.
Purpose
For arm64 users, specifying differences in installation of dependencies and changes in build phase.
Possible approaches
Source installation section of the document will be updated for Jetpack users.
Definition of done
Autoware was compiled with a few different steps for Jetson that were not specified in the documentation.

Linked PR is closed, other PR is merged. @xmfcx can this issue be closed?

This issue is fixed by:

https://github.com/autowarefoundation/autoware.universe/pull/1472


failed to catch imdecode exception

Hi, I just found that I can not catch some IO exceptions if the program failed to load an image when using mxnet's image decoding API. The problem can be produced like below:
import mxnet as mx

print('MXNet version: %s' % mx.__version__)

with open('test.jpg', 'wb') as fd:
    fd.close()

try:
    with open('test.jpg', 'rb') as fd:
        buf = fd.read()
    im = mx.image.imdecode(buf, flag=1, to_rgb=True)
except:
    print('mxnet imdecode failed to load image.')

➜  ~ python3 test_imdecode_error.py
MXNet version: 1.3.0
OpenCV Error: Assertion failed (!buf.empty() && buf.isContinuous()) in imdecode_, file /home/lhy/Documents/Lib/opencv-3.4.0/modules/imgcodecs/src/loadsave.cpp, line 711
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/lhy/Documents/Lib/opencv-3.4.0/modules/imgcodecs/src/loadsave.cpp:711: error: (-215) !buf.empty() && buf.isContinuous() in function imdecode_

[1]    28041 abort (core dumped)  python3 test_imdecode_error.py

However, I would like the program to skip the corrupted images and just continue processing others. Is it a bug?

Hi @nicklhy - This is not a bug, underlying OpenCV is crashing. However, core dumped in any case is not the user experience expected from any library. Adding feature request label so we can visit this usecase and handle it more gracefully.
Thanks for reporting.

@sandeep-krishnamurthy , I just did a few more tests about the opencv APIs. cv2.imdecode behaves just like mx.image.imdecode. But what quite interest is, cv2.imread doesn't crash or raise exception at all, it just return None ( which is exactly what I want). Can I implement something similar via mxnet's APIs?


JWT Signature validation fails in .NET Core app targetting .NET Framework 4.7.1

From @fabiodaniele on February 13, 2018 15:55
Hi,
I was having an issue trying to authenticate users to a .NET Core WebAPI using a JWT bearer token generated by a WSO2 Identity Server.
The project targets .NET Framework 4.7.1 and references Microsoft.AspNetCore.Authentication.JwtBearerToken.
At first, I thought it was an issue related to my WSO2 IS configuration.
Then, I found this article:
https://www.jerriepelser.com/blog/manually-validating-rs256-jwt-dotnet/
and tried to execute the same code found there in a new .NET Core Console app: it worked!
So, I thought: "it maybe an issue of my WebAPI project".
I then made a few tries with two different brand new .NET Core WebAPI projects, one targetting .NET Core 2.0 and the other one targetting .NET Framework 4.7.1, using the same startup code in both.
Here is the code:
      // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            IConfigurationManager<OpenIdConnectConfiguration> configurationManager = new ConfigurationManager<OpenIdConnectConfiguration>($"{Configuration["OpenId:Authority"]}/.well-known/openid-configuration", new OpenIdConnectConfigurationRetriever());
            OpenIdConnectConfiguration openIdConfig = configurationManager.GetConfigurationAsync(CancellationToken.None).Result;

            services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                .AddJwtBearer(options =>
                {
                    options.IncludeErrorDetails = true;
                    options.TokenValidationParameters.ValidateIssuer = true;
                    options.TokenValidationParameters.ValidateAudience = true;
                    options.TokenValidationParameters.ValidateIssuerSigningKey = true;
                    options.TokenValidationParameters.ValidIssuer = Configuration["OpenId:Issuer"];
                    options.TokenValidationParameters.ValidAudiences = new[] { Configuration["OpenId:Audience"] };
                    options.TokenValidationParameters.IssuerSigningKeys = openIdConfig.SigningKeys;
                });

            services.AddMvc();
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseAuthentication();

            app.UseMvc();
        }

I decorated the default ValuesController with the Authorize attribute and tried to invoke it via Postman, with a new JWT token obtained from the WSO2 IS.
The results are different:

the app targetting .NET Core 2.0 simply works, giving me the expected JSON result from the Action invoked
the one targetting .NET Framework 4.7.1 replies with Bearer error="invalid_token", error_description="The signature is invalid"

So the question is:
is this the expected behavior or is it a bug?
Thanks in advance.
Copied from original issue: aspnet/Security#1649

I can confirm the same issue.
I have created a minimal repro for this as a console app targeting either .net 4.7 or .net core 2.0
I have removed all aspnet dependencies and narrowed the issue down to the System.IdentityModel.Tokens.Jwt 5.2.1 package.
The following works using .net core and fails when targeting 4.7:
        var jwtHandler = new JwtSecurityTokenHandler();

        var byteCert = Convert.FromBase64String(@"CERTPLACEHOLDER");

        var cert = new X509Certificate2(byteCert);

        var rsaSecurityKey = new RsaSecurityKey(cert.GetRSAPublicKey())
        {
            KeyId = "KEYID"
        };

        SecurityToken token;
        jwtHandler.ValidateToken(@"TOKENPLACEHOLDER",
        new TokenValidationParameters
        {
            RequireSignedTokens = true,
            ValidateIssuer = false,
            ValidateAudience = false,
            ValidateLifetime = true,
            IssuerSigningKey = rsaSecurityKey
        },
        out token);
        Console.WriteLine(token);
    }

Exception in 4.7:
Microsoft.IdentityModel.Tokens.SecurityTokenInvalidSignatureException occurred
HResult=0x80131500
Message=IDX10503: Signature validation failed. Keys tried: 'Microsoft.IdentityModel.Tokens.RsaSecurityKey , KeyId: KEYID
'.
Exceptions caught:
'System.NotSupportedException: IDX10641: Key is not supported: 'Microsoft.IdentityModel.Tokens.RsaSecurityKey'.
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.ResolveAsymmetricAlgorithm(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForVerifying(SecurityKey key, String algorithm)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(Byte[] encodedBytes, Byte[] signature, SecurityKey key, String algorithm, TokenValidationParameters validationParameters)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(String token, TokenValidationParameters validationParameters)
'.
token:
StackTrace:
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateSignature(String token, TokenValidationParameters validationParameters)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.ValidateToken(String token, TokenValidationParameters validationParameters, SecurityToken& validatedToken)
at jwt2.Program.Main(String[] args) in C:\Users\allevy\Source\Repos\jwt2\jwt2\Program.cs:line 36
Alon

@allevyMS I'm on vacation and can't look at this for a week or so, but should be able to work around this by plugging in a custom Cryptofactory using TokenValidationParameters

Is this patched in the 5.2.2 release? Seeing the same issue running v5.2.1 in an ASP.NET project targeting 4.6.2.

Hi,
I have a Security Token Service running on 4.5.2 and I have a requirement to support CNG certificates based tokens. What should be recommended approach for this?
For now I am experimenting with migrating from framework 4.5.2 to 4.6.2, but I am running into similar error as posted above in the thread while executing the following code.
Would this be resolved by the solution that you are working on. By when I can expect the resolution and which .net framework version? Is there any work around that I can adopt?
`               The exception is thrown at this line -> handler.WriteToken(secToken);
            var expires = DateTime.UtcNow.AddMinutes(10);

            var certName = ConfigurationManager.AppSettings["CertName"];

            X509Certificate2 certificate = GetCertificate(StoreName.My, StoreLocation.LocalMachine, certName);

            IdentityModelEventSource.ShowPII = true;

            var alg = certificate.GetRSAPrivateKey();

            Microsoft.IdentityModel.Tokens.RsaSecurityKey  securityKey = new Microsoft.IdentityModel.Tokens.RsaSecurityKey(alg);

            var value = certificate.SignatureAlgorithm.FriendlyName;

            var SigningCredentials1 =
                new Microsoft.IdentityModel.Tokens.SigningCredentials(securityKey,
                    SecurityAlgorithms.RsaSha256Signature);

            var header = new JwtHeader(SigningCredentials1);

            var now = DateTime.UtcNow;

            var payload = new JwtPayload
            {
                {"iss", "a5fgde64-e84d-485a-be51-56e293d09a69"},
                {"scope", "https://example.com/ws"},
                {"aud", "https://example.com/oauth2/v1"},
                {"iat", now},
            };

            var secToken = new JwtSecurityToken(header, payload);

            var handler = new JwtSecurityTokenHandler();

            var tokenString = handler.WriteToken(secToken);`

Exception:
IDX10641: Key is not supported: 'Microsoft.IdentityModel.Tokens.RsaSecurityKey'.
Stack trace:
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.ResolveAsymmetricAlgorithm(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
at CNGCertificateSupport.Program.Main(String[] args) in c:\OSI\SAF\Spikes\CNG\CNGCertificateSupport\CNGCertificateSupport\Program.cs:line 131
Thanks in advance,
Manish

@manish13oct @cvocvo this dropped off our radar.
What happened was a change to .Net between 4.5.2 and 4.6 where X509Certificate2.PublicKey returns a different type than RSACryptoServiceProvider, it returns RsaCng.
IdentityModel return as RSACryptoServiceProvider, which is null.
Then the runtime throws.
The NetStandard1_4 will work.
We are looking at a fix.

@Tratcher @manish13oct @cvocvo with this commit, this should now work.
5.2.3 will have this fix.
In 5.2.4 we plan on introducing a 4.6.0 or 4.6.1 target, that will solve the issue for applications that can use that target. As we plan on supporting 4.5.1 and 4.5.2 for a while, we need this fix as it has nice SHA2 support.
We plan on extending this adapter to use CNG for certs and ecdsa.

@brentschmaltz Thanks! -- What's the ETA for 5.2.3's release?

@cvocvo currently we only release on nuget.org.
Yea, 5.2.5 should have a 4.6.1 or 4.6.0 target.

Perfect; hopefully that 4.6.x target will resolve the issue for us. I rolled out 5.2.4 earlier today into production and our logging caught a couple of the error reoccurring. Do you know (approximately) what the timeline on 5.2.5 is? (I couldn't find a way to subscribe to nuget for alerts on releases)

@cvocvo been on vacation, back now.
5.2.5 targets Aug 10th.

@brentschmaltz No worries; your great support on this is much appreciated!

@brentschmaltz Just circling back; any updates on the 5.2.5 release targeting .NET 4.6.0+? Any plans to target 4.7.0 down the road too (not sure if even needed if there's a 4.6.0 target) ?

@cvocvo our next release is 5.3.0 (was 5.2.5) and we are just waiting for approval for our package id's.
Can you try our nightly packages (5.3.10) on myget here: https://www.myget.org/gallery/azureadwebstacknightly

@brentschmaltz Apologies -- missed this message. I just checked and see 5.3.0 is out now. I'm rolling that out to our staging environment for further testing. Thank you! (I'll edit post when I have test results)

@cvocvo I'll close this out, if you hit issues please reopen.
We added a 4.6.1 target, there is not much more in 4.7, from a crypto point of view that will add a lot of value, so we didn't add that target.

@brentschmaltz & @Tratcher  : I ended up here because I got the same problem as initially reported by @fabiodaniele and @allevyMS : ASP.NET Core web application targetting .NET 4.7.2 with the Microsoft.AspNetCore.Authentication.JwtBearer package that references System.IdentityModel.Tokens.Jwt v5.2.0.
I am implementing the OpenID Connect Authorization Code flow, but I'm stuck with the following error when validating the JWT signature:
IDX10503: Signature validation failed. Keys tried: 'Microsoft.IdentityModel.Tokens.JsonWebKey , KeyId: <my kid>
'.
Exceptions caught:
 ''.
token: <decoded JWT>

I'm 100% sure the JWT is valid and I'm using the correct JsonWebKey. If I manually validate the signature using the PEM in jwt.io it shows "signature verified".
Consoleapp targetting .NET Core 2.1 and using System.IdentityModel.Tokens.Jwt v5.3.0 to validate my JWT (JwtSecurityTokenHandler) using the same RSA key succeeds without any exception.
Consoleapp targetting .NET 4.7.2 (or lower) and using System.IdentityModel.Tokens.Jwt v5.3.0 (or lower) to validate my JWT (JwtSecurityTokenHandler) using the same RSA key gives me the same IDX10503 exception.
Is there any way to get around this issue? It's becoming a showstopper for the project I'm working on.
Thanks in advance!

@brentschmaltz any updates regarding this issue?

@RubenDelange if you look at the modulus of this key, you will see that the first byte is '0'. This is contrary to JWA here; https://tools.ietf.org/html/rfc7518#section-<IP_ADDRESS>
The frustrating thing here is we tried to 'fix' this by removing the first byte if ZERO and byte length == 257, but we started failing in other libraries.
Perhaps we should revisit this issue and put in some type of compatibility flag that one can turn on.

@brentschmaltz knowning the cause is a good starting point :)
I don't know if this explains why there are no problems with the same code in a Consoleapp targetting .NET Core 2.1 as opposed to a Consoleapp targetting .NET 4.7.2 (or lower), but I'm hoping this get's fixed somehow.
In my ASP.NET Core web application targetting .NET 4.7.2 (and System.IdentityModel.Tokens.Jwt v5.2.0), I have to us a dirty workaround to get passed signature validation :
SignatureValidator = delegate (string token, TokenValidationParameters parameters)
{
	return new JwtSecurityToken(token);
}

but I'd like to get rid of it asap.
Can you give some advice for a better and more secure workaround?

@brentschmaltz : jwk_uri is https://idp.iamfas.int.belgium.be/fas/oauth2/connect/jwk_uri ; currently the same JWK applies as used in the console app code sample here
I know that not validating the signature leaves a big security hole, but I had to make some progress in development. I'm looking forward to your suggestion to enable signature validation for the given JWK.

Thank you @brentschmaltz , I can use JWT signature validation again!

Hi guys (@brentschmaltz) sorry for the long delay on circling back to this (it got back burnered for a while and buried on the to-do list). I would love to see anything/something changed to help this issue in 5.3.1 (we're still seeing issues in our project targeting 4.6.2). After reading the above posts a few times, I'm thinking that we're likely seeing the same problem as @RubenDelange ; could you help us place your  modulus checking snippet of code as well?
Our validation code is heavily based on the sample here: https://github.com/Azure-Samples/active-directory-dotnet-webapi-manual-jwt-validation/blob/4b80657c5506c8cb30af67b9f61bb6aa68dfca58/TodoListService-ManualJwt/Global.asax.cs#L54
Here's the full snippet:
`public class JwtTokenValidationHandler : DelegatingHandler
{
private readonly IConfigManager _configManager;
	public JwtTokenValidationHandler()
	{
		_configManager = new ConfigManager();
	}

	private static bool TryRetrieveToken(HttpRequestMessage request, out string token)
	{
		token = null;
		IEnumerable<string> authheaders;
		if (!request.Headers.TryGetValues("Authorization", out authheaders))
		{
			if (!request.Headers.TryGetValues("X-Authorization-Token", out authheaders))
			{
				return false;
			}
		}

		var bearerToken = authheaders.FirstOrDefault();
		token = bearerToken.StartsWith("Bearer ", StringComparison.Ordinal) ? bearerToken.Substring(7) : bearerToken;
		return true;
	}

	/// <summary>
	/// Based on: https://github.com/Azure-Samples/active-directory-dotnet-webapi-manual-jwt-validation/blob/4b80657c5506c8cb30af67b9f61bb6aa68dfca58/TodoListService-ManualJwt/Global.asax.cs#L54
	/// </summary>
	/// <param name="request"></param>
	/// <param name="cancellationToken"></param>
	/// <returns></returns>
	protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
	{
		var authHeader = request.Headers.Authorization;
		if (authHeader == null && !request.Headers.Contains("X-Authorization-Token"))
		{
			// missing authorization header
			return await base.SendAsync(request, cancellationToken);
		}

		string userJwtToken;
		if (!TryRetrieveToken(request, out userJwtToken))
		{
			return new HttpResponseMessage(HttpStatusCode.Unauthorized);
		}

		SecurityToken parsedToken;
		try
		{
			var appliesToAddress = _configManager.GetConfigurationKey("JwtAllowedAudience");
			var tokenIssuerName = _configManager.GetConfigurationKey("JwtValidIssuer");
			SecurityKey securityKey = new SymmetricSecurityKey(KeyHolder.SymmetricKey);
			TokenValidationParameters validationParameters = new TokenValidationParameters
			{
				ValidAudience = appliesToAddress,
				ValidIssuer = tokenIssuerName,
				IssuerSigningKey = securityKey
			};

			JwtSecurityTokenHandler tokenHandler = new JwtSecurityTokenHandler();
			// Validate token -- this is where exceptions will originate
			ClaimsPrincipal claimsPrincipal = tokenHandler.ValidateToken(userJwtToken, validationParameters, out parsedToken);

			// Set the ClaimsPrincipal on the current thread.
			Thread.CurrentPrincipal = claimsPrincipal;
			// Set the ClaimsPrincipal on HttpContext.Current if the app is running in web hosted environment.
			if (HttpContext.Current != null)
			{
				HttpContext.Current.User = claimsPrincipal;
			}
		}
		catch (SecurityTokenInvalidSignatureException ex)
		{
			// the token was probably manipulated before it got to us
			// this could imply an old auth token (expired), bad code or a malicious actor
			Elmah.ErrorSignal.FromCurrentContext().Raise(ex);
			return new HttpResponseMessage(HttpStatusCode.Unauthorized);
		}
		catch (SecurityTokenValidationException ex)
		{
			// the error is more general, not sure. Someone could be sending us bad data
			Elmah.ErrorSignal.FromCurrentContext().Raise(ex);
			return new HttpResponseMessage(HttpStatusCode.Unauthorized);
		}
		// don't catch raw exceptions -- let them bubble up naturally

		if(parsedToken == null)
		{
			return new HttpResponseMessage(HttpStatusCode.Unauthorized);
		}

		return await base.SendAsync(request, cancellationToken);
	}
}`

I have a case where in .net core 2.2 when i containerize the app i get a Bearer error="invalid_token", error_description="The signature is invalid"
It is working fine when i host it on windows using IIS/IIS express.
My code -- The token generator is IBM API Connect it uses RSA 256 Algorithm to generate the key
var rsa = new RSACryptoServiceProvider();
string exponentvalue = "AQAB";
var e = Base64UrlEncoder.DecodeBytes(exponentvalue);
var N = "public key  put your value here"
var modulus = Base64UrlEncoder.DecodeBytes(N);
rsa.ImportParameters(
new RSAParameters()
{
Modulus = modulus,
Exponent = e
  });

var signingKey = new RsaSecurityKey(rsa);
var tokenValidationParameters = new TokenValidationParameters
{
// The signing key must match!
ValidateIssuerSigningKey = true,
IssuerSigningKey = signingKey,
            // Validate the JWT Issuer (iss) claim
            ValidateIssuer = false,
            ValidIssuer = issuer,

            // Validate the JWT Audience (aud) claim
            ValidateAudience = false,
            ValidAudience = audience,

            // Validate the token expiry
            //ValidateLifetime = true,

            // If you want to allow a certain amount of clock drift, set that here:
            //ClockSkew = TimeSpan.FromMinutes(1)
        };

Any idea why it wouldn't be working on a container hosted either locally on docker or AKS?

@loga0060 can you check if the modulus first byte is zero?

@brentschmalz I did try the code that was suggested above first byte was 190. I read  a post on docker forum saying ms provided nuget packages for jwt uses these fingerprint of public key where as docker expects a different format (lib trust fingerprint ) could this be the case?

I figured out my issue full solution is here  https://stackoverflow.com/questions/54585148/net-core-2-2-validation-of-jwt-failing-with-401-on-a-container

nice.

For anyone else landing here trying to figure out the "IDX10511: Signature validation failed." error, specifically with a "System.Security.Cryptography.CryptographicException: An internal error occurred" exception. This was the solution for us:
Our 2008R2 hosts needed to be configured to auto-start the "CNG Key Isolation" service.
Ref: https://blogs.msdn.microsoft.com/dsnotes/2012/08/17/cngkey-import-throws-an-exception-saying-an-internal-error-occurred-when-importing-a-key-blob-under-a-non-admin-user-from-a-wcf-web-service/

@jflord78 thanks for this post, very helpful.

@brentschmaltz
I'm being faced with with similar issue that's been reported in here. I have tried all the suggestions but I'm getting no where.
Am I missing something?
I have added a snapshot of my code
IDX10511: Signature validation failed. Keys tried: 'Microsoft.IdentityModel.Tokens.X509SecurityKey
I'm using MSAL interceptor in Angular 8 code to send access token to the service "C# .NET core 3.1"
//*****************************************************************************************************
public static JwtSecurityToken Validate(string token)
{
        string stsDiscoveryEndpoint = "https://login.microsoftonline.com/eddf9e42-9a7f-4639-8042-***********/.well-known/openid-configuration?appid=7391aebb-1a35-4ad7-8166-xxxxxxxxxx"; // https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration";

        ConfigurationManager<OpenIdConnectConfiguration> configManager = new ConfigurationManager<OpenIdConnectConfiguration>(stsDiscoveryEndpoint, new OpenIdConnectConfigurationRetriever());
        OpenIdConnectConfiguration config = configManager.GetConfigurationAsync().Result;

        TokenValidationParameters validationParameters = new TokenValidationParameters
        {
            ValidateAudience = false,
            ValidIssuer = config.Issuer,
            IssuerSigningKeys = config.SigningKeys,
            ValidateIssuerSigningKey = false,
            ValidateIssuer = true,
            ValidateActor = false,
            ValidateLifetime = true
        };

        IdentityModelEventSource.ShowPII = true;
        JwtSecurityTokenHandler tokendHandler = new JwtSecurityTokenHandler();

        SecurityToken jwt;
        var result = tokendHandler.ValidateToken(token, validationParameters, out jwt);
        return jwt as JwtSecurityToken;
    }

/*
IDX10511: Signature validation failed. Keys tried: 'Microsoft.IdentityModel.Tokens.X509SecurityKey, KeyId: 'HlC0R12skxNZ1WQwmjOF_6t_tDE', InternalId: '91ea3222-c9ea-4aa4-a515-83f2b195f989'. , KeyId: HlC0R12skxNZ1WQwmjOF_6t_tDE
'.
kid: 'HlC0R12skxNZ1WQwmjOF_6t_tDE'.
Exceptions caught:
''.
token: '{"alg":"RS256","typ":"JWT","nonce":"19yszC4xPiqhnbI587HT_08QFjftE7J3qE8F9xw_sqY","x5t":"HlC0R12skxNZ1WQwmjOF_6t_tDE","kid":"HlC0R12skxNZ1WQwmjOF_6t_tDE"}.{"aud":"00000003-0000-0000-c000-000000000000","iss":"https://sts.windows.net/eddf9e42-9a7f-4639-8042-110281cf41a2/","iat":1581413726,"nbf":1581413726,"exp":1581417626,"acct":0,"acr":"1","aio":"42NgYLgWut8sY6dad5urN6/nxuP8P7Z6ZXr1Wwl355xibZu8nxMA","amr":["pwd"],"app_displayname":"SoraxWeb","appid":"7391aebb-1a35-4ad7-8166-xxxxxxxx","appidacr":"0","family_name":"kwofie","given_name":"Ernest","ipaddr":"<IP_ADDRESS>","name":"Exxxx kxxxx","oid":"9f9efce3-37e5-4018-8035-7ffc6323e827","onprem_sid":"S-1-5-21-3789744388-407605227-1228871550-7770","platf":"3","puid":"100320003319DDA9","scp":"Mail.Send openid profile User.Read User.ReadWrite email","sub":"LhkC5zL_zJm-MZBjJDYCz5Asjm7Nwg9tHUehKVSH40A","tid":"eddf9e42-9a7f-4639-8042-110281cf41a2","unique_name":"Exxxx.kxxxx@xxxx.com","upn":"Exxxx.kxxxx@xxxxx.com","uti":"3RsEnJpEokmZb-Y3MwJzAA","ver":"1.0","xms_st":{"sub":"zSJz0JM6iB6SRwxi1ZZm_1F__aYqHhEDtjb5qkege_o"},"xms_tcdt":1443089411}'.
*/
Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddCors(options => options.AddPolicy(MyAllowSpecificOrigins, builder =>
{
builder.AllowAnyOrigin().AllowAnyMethod().AllowAnyHeader();
}));
services.AddControllers();
}
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseRouting();
        app.UseCors(MyAllowSpecificOrigins);
        app.UseHttpsRedirection();
        app.UseAuthorization();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }

@EkwofiePrax did you ever find a solution?

Any updates?

@EkwofiePrax were you able to find a solution for your problem?


Vickel: Watch menupro.org

Vickel requests the watch of the watch_keyword menupro\.org. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

menupro\.org has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


psycopg2 fails to install on mac arm64 m1 and misc installation issues

Hello,
Thanks for putting the project together, I tried a poetry install on mac and psycopg2 failed to install, and as a result (?) none of the vulndb admin commands work properly. Moreover the sql command described in the program assumes some program installed for users which might not be there so any additional context would be helpful

By the 'the sql command described in the program' are you referring to db/create.sql?
The primary means of database management is intended to be through the 'vulndb admin' command set. That does require psycopg2 to be working, but then so do most features in the code base, so it's something that you'll need to resolve to get much use from it. What is the issue that you're having?
Having said that, about 90% of the work to convert to sqlite3 and eliminate postgresql entirely is done, but it's uncertain right now when the final 10% will be completed.


Connection issue

Version of the custom_component
V1.0.0
Describe the bug
Will not connect
Debug log

NoneType: None
2022-04-24 06:59:36 ERROR (SyncWorker_4) [givenergy_modbus] Did not receive expected response type: ReadInputRegistersResponse != ModbusIOException
2022-04-24 06:59:38 ERROR (SyncWorker_4) [givenergy_modbus] Returned base register (0) does not match that from request (60).
2022-04-24 06:59:39 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up givenergy_local platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/givenergy_local/sensor.py", line 192, in async_setup_entry
    PVEnergyTodaySensor(
  File "/config/custom_components/givenergy_local/sensor.py", line 219, in __init__
    self._attr_unique_id = f"{self.coordinator.data.inverter.inverter_serial_number}_{entity_description.key}"
  File "/usr/local/lib/python3.9/site-packages/givenergy_modbus/model/plant.py", line 32, in inverter
    return Inverter.from_orm(self.inverter_rc)
  File "/usr/local/lib/python3.9/site-packages/pydantic/main.py", line 562, in from_orm
    values, fields_set, validation_error = validate_model(cls, obj)
  File "/usr/local/lib/python3.9/site-packages/pydantic/main.py", line 1001, in validate_model
    value = input_data.get(field.alias, _missing)
  File "/usr/local/lib/python3.9/site-packages/givenergy_modbus/model/register_getter.py", line 69, in get
    return getattr(self._obj, key, default)
  File "/usr/local/lib/python3.9/site-packages/givenergy_modbus/model/register_cache.py", line 26, in __getattr__
    val = self[register]
KeyError: IR:183

There's an issue in the underlying library that causes occasional glitches when fetching data (that's the Did not receive expected response type you're seeing in the logs. Sometimes this can happen on the very first try, in which case HA will report this error. This is expected to be sorted in the near future, but in the mean time, could you try to connect a few more times and see if you can get past this stage of the process?

Tried a few more times.  Logs maybe more legible:
Error while setting up givenergy_local platform for sensor 20:24:20 – (ERROR) Sensor (custom integration) Did not receive expected response type: ReadHoldingRegistersResponse != ReadInputRegistersResponse 20:24:19 – (ERROR) /usr/local/lib/python3.9/site-packages/givenergy_modbus/modbus.py - message first occurred at 20:24:16 and shows up 2 times Modbus Error: [Input/Output] Modbus Error: [Invalid Message] No response received, expected at least 164 bytes (0 received) 20:24:19 – (ERROR) /usr/local/lib/python3.9/site-packages/givenergy_modbus/modbus.py Transaction failed 20:24:19 – (ERROR) /usr/local/lib/python3.9/site-packages/givenergy_modbus/transaction.py Returned base register (60) does not match that from request (120). 20:24:17 – (ERROR) /usr/local/lib/python3.9/site-packages/givenergy_modbus/modbus.py

I'm getting the same issue, I cannot connect. I've tried dozens of times.
I also have GivTCP running on the network, and that is working (mostly) fine, so the inverter is online and available.
Logs:
Logger: givenergy_modbus
Source: /usr/local/lib/python3.9/site-packages/givenergy_modbus/modbus.py:82
First occurred: 10:40:33 (14 occurrences)
Last logged: 10:45:39

Returned base register (60) does not match that from request (0).
Returned base register (0) does not match that from request (180).
Returned base register (0) does not match that from request (60).
Returned base register (60) does not match that from request (120).
Returned base register (60) does not match that from request (180).

Logger: givenergy_modbus
Source: /usr/local/lib/python3.9/site-packages/givenergy_modbus/modbus.py:94
First occurred: 10:40:14 (18 occurrences)
Last logged: 10:45:40

Did not receive expected response type: ReadInputRegistersResponse != ModbusIOException
Did not receive expected response type: ReadHoldingRegistersResponse != ModbusIOException
Did not receive expected response type: ReadHoldingRegistersResponse != ReadInputRegistersResponse
Did not receive expected response type: ReadInputRegistersResponse != ReadHoldingRegistersResponse

I'm running 2022.7, it's still not working:
Logger: givenergy_modbus
Source: /usr/local/lib/python3.10/site-packages/givenergy_modbus/transaction.py:100
First occurred: 00:08:20 (4 occurrences)
Last logged: 00:09:13
Transaction failed

Logger: givenergy_modbus
Source: /usr/local/lib/python3.10/site-packages/givenergy_modbus/modbus.py:52
First occurred: 00:08:20 (8 occurrences)
Last logged: 00:09:13

    Modbus Error: [Input/Output] Modbus Error: [Invalid Message] No response received, expected at least 164 bytes (0 received)
    Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response

NoneType: None

Just to add here, I find a connection will fail if there is something else polling the device (givtcp for example)

Sorry my repo is private so you wont see the error. I saw this in my logs yesterday, I have not seen the error since and my HA is showing up to date data  today.  I will monitor if this occurs again and under what circumstances.

Closing for now as I don't think there's anything that could be changed in this project to reduce connectivity issues.


stuck at boot screen

after motheboard logo show up, it switch to windows logo and get stuck there. its manage to get back to stock normally, but i ran it 2 more time and it still have the same problem (even with J opttion)
here is the log  file:
setup.log
note: i do dual boot 2 different version of windows, maybe this cause something?

I don't fully understand your report.
I have encountered a similar situation where Windows boots very slowly (appears to be stuck but continues finally). It happened to me a few times on the same computers which usually work just fine, and it got better on the next boot. It's hard to say what connection it has to HackBGRT and if there's any possible fix. (Note that HackBGRT is not running anymore at that point, so the slowness is something that Windows does.) I would advise you to reinstall (with I) and reboot at least a few times very patiently to see if it gets better.
Your first try (with I) looks ok, there's proper boot log which says HackBGRT was used and working.
Your later tries (with J) look like HackBGRT was not run during boot. This is a common problem with that install method, Windows seems to restore the old boot sequence and skip HackBGRT. That's why it's the second alternative.
I don't multiboot Windows, so I can't say what effect that might have.

@Metabolix i have cap an image of boot screen with I option here

it got too this and freeze (i have wait 5 min but nothing happen on screen), even after reboot two time (like i said above)
weird

Yes, it took ages to boot for me too (probably 10–20 minutes) but finally it started working normally again. I'm not sure anymore if I changed the image during the process. Now it doesn't happen anymore so I can't even reasonably investigate it.
When the custom logo shows, your computer is already running the Windows boot loader. Whatever makes it so slow is a Windows thing. Maybe it's some obscure compability problem with the image file (a specific size or something). You could try another image with different dimensions or try either image=remove or image=black in config.
It's an interesting problem. If you get it working with some settings, please retry with that exact image and share exact details about what works and what doesn't.

@Metabolix thank for ur response, i will try it again and share it back here when i have time <3


Commenter returning "Ignoring - change not part of the current PR" for all comments

I have a workflow (below) that correctly parses my .tf files in a PR, but never actually comments on it. Already looked into permissions issues - no problem here. I have also cycle through various arguments to see if they would yield the result I wanted, but again I've had no luck.
name: tfsecPOC
on:
  pull_request:
jobs:
  tfsec-compliance:
    name: tfsec compliance check
    runs-on: [self-hosted, vault]
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3
      - name: tfsec
        uses: aquasecurity/tfsec-pr-commenter-action@main
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          tfsec_args: --debug

I just get this output at the end of my run
1 file(s) written: results.json
+ commenter
Starting the github commenter
Working in repository terraform
Working in PR 1333
TFSec found 5 issues
Working in GITHUB_WORKSPACE /github/workspace/
Preparing comment for violation of rule aws-ec2-add-description-to-security-group in ./terratest-poc/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-ec2-add-description-to-security-group-rule in ./terratest-poc/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-ec2-enable-at-rest-encryption in ./terratest-poc/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-ec2-enforce-http-token-imds in ./terratest-poc/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-ec2-no-public-ingress-sgr in ./terratest-poc/main.tf
Ignoring - change not part of the current PR

I can see where this error is coming from, but I am not able to see what the exact issue with the results.json file is.
for _, result := range results {
		result.Range.Filename = workingDir + strings.ReplaceAll(result.Range.Filename, workspacePath, "")
		comment := generateErrorMessage(result)
		fmt.Printf("Preparing comment for violation of rule %v in %v\n", result.RuleID, result.Range.Filename)
                err := c.WriteMultiLineComment(result.Range.Filename, comment, result.Range.StartLine, result.Range.EndLine)
		if err != nil {
			// don't error if its simply that the comments aren't valid for the PR
			switch err.(type) {
			case commenter.CommentAlreadyWrittenError:
				fmt.Println("Ignoring - comment already written")
				validCommentWritten = true
			case commenter.CommentNotValidError:
				fmt.Println("Ignoring - change not part of the current PR")
				continue
			default:
				errMessages = append(errMessages, err.Error())
			}
		} else {
			validCommentWritten = true
			fmt.Printf("Commenting for %s to %s:%d:%d\n", result.Description, result.Range.Filename, result.Range.StartLine, result.Range.EndLine)

Action command output
/usr/bin/docker run --name cbb2812a0638619a79478ab2aa7296e3716695_8d2681 --label cbb281 --workdir /github/workspace --rm -e "INPUT_GITHUB_TOKEN" -e "INPUT_TFSEC_ARGS" -e "INPUT_WORKING_DIRECTORY" -e "INPUT_TFSEC_VERSION" -e "INPUT_TFSEC_FORMATS" -e "INPUT_COMMENTER_VERSION" -e "INPUT_SOFT_FAIL_COMMENTER" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_EVENT_PATH" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "GITHUB_ACTION" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/github/_work/_temp/_github_home":"/github/home" -v "/home/github/_work/_temp/_github_workflow":"/github/workflow" -v "/home/github/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/github/_work/terraform/terraform":"/github/workspace" cbb281:2a0638619a79478ab2aa7296e3716695

tfsec --out=results.json --format=json --soft-fail --debug .

Based on this, it looks like there is an issue with my PR and the results.json file syncing up
// CommentNotValidError returned when the comment is for a file or line not in the pr
type CommentNotValidError struct {
	filepath string
	lineNo   int
}

Looked into this a little more today and was able to pull out the error that is triggering the ignore statement:
https://github.com/aquasecurity/tfsec-pr-commenter-action/blob/7a44c5dcde5dfab737363e391800629e27b6376b/vendor/github.com/owenrumney/go-github-pr-commenter/commenter/errors.go#L50-L51
Here is the results.json file and the errors:
{
	"results": [
		{
			"rule_id": "AVD-AWS-0099",
			"long_id": "aws-ec2-add-description-to-security-group",
			"rule_description": "Missing description for security group.",
			"rule_provider": "aws",
			"rule_service": "ec2",
			"impact": "Descriptions provide context for the firewall rule reasons",
			"resolution": "Add descriptions for all security groups",
			"links": [
				"https://aquasecurity.github.io/tfsec/v1.28.1/checks/aws/ec2/add-description-to-security-group/",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule"
			],
			"description": "Security group explicitly uses the default description.",
			"severity": "LOW",
			"warning": false,
			"status": 0,
			"resource": "aws_security_group.terratest-sg",
			"location": {
				"filename": "/github/workspace/terratest-poc/main.tf",
				"start_line": 9,
				"end_line": 18
			}
		},
		{
			"rule_id": "AVD-AWS-0124",
			"long_id": "aws-ec2-add-description-to-security-group-rule",
			"rule_description": "Missing description for security group rule.",
			"rule_provider": "aws",
			"rule_service": "ec2",
			"impact": "Descriptions provide context for the firewall rule reasons",
			"resolution": "Add descriptions for all security groups rules",
			"links": [
				"https://aquasecurity.github.io/tfsec/v1.28.1/checks/aws/ec2/add-description-to-security-group-rule/",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule"
			],
			"description": "Security group rule does not have a description.",
			"severity": "LOW",
			"warning": false,
			"status": 0,
			"resource": "aws_security_group.terratest-sg",
			"location": {
				"filename": "/github/workspace/terratest-poc/main.tf",
				"start_line": 12,
				"end_line": 17
			}
		},
		{
			"rule_id": "AVD-AWS-0131",
			"long_id": "aws-ec2-enable-at-rest-encryption",
			"rule_description": "Instance with unencrypted block device.",
			"rule_provider": "aws",
			"rule_service": "ec2",
			"impact": "The block device could be compromised and read from",
			"resolution": "Turn on encryption for all block devices",
			"links": [
				"https://aquasecurity.github.io/tfsec/v1.28.1/checks/aws/ec2/enable-at-rest-encryption/",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#ebs-ephemeral-and-root-block-devices"
			],
			"description": "Root block device is not encrypted.",
			"severity": "HIGH",
			"warning": false,
			"status": 0,
			"resource": "aws_instance.terratest-ec2-instance",
			"location": {
				"filename": "/github/workspace/terratest-poc/main.tf",
				"start_line": 1,
				"end_line": 7
			}
		},
		{
			"rule_id": "AVD-AWS-0028",
			"long_id": "aws-ec2-enforce-http-token-imds",
			"rule_description": "aws_instance should activate session tokens for Instance Metadata Service.",
			"rule_provider": "aws",
			"rule_service": "ec2",
			"impact": "Instance metadata service can be interacted with freely",
			"resolution": "Enable HTTP token requirement for IMDS",
			"links": [
				"https://aquasecurity.github.io/tfsec/v1.28.1/checks/aws/ec2/enforce-http-token-imds/",
				"https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#metadata-options"
			],
			"description": "Instance does not require IMDS access to require a token",
			"severity": "HIGH",
			"warning": false,
			"status": 0,
			"resource": "aws_instance.terratest-ec2-instance",
			"location": {
				"filename": "/github/workspace/terratest-poc/main.tf",
				"start_line": 1,
				"end_line": 7
			}
		}
	]
}

starting the github commenter
Working in repository terraform
Working in PR 1337
TFSec found 4 issues
Working in GITHUB_WORKSPACE /github/workspace/
Preparing comment for violation of rule aws-ec2-add-description-to-security-group in terratest-poc/terratest-poc/main.tf
There is nothing to comment on at line [9] in file [terratest-poc/terratest-poc/main.tf]
Preparing comment for violation of rule aws-ec2-add-description-to-security-group-rule in terratest-poc/terratest-poc/main.tf
There is nothing to comment on at line [12] in file [terratest-poc/terratest-poc/main.tf]
Preparing comment for violation of rule aws-ec2-enable-at-rest-encryption in terratest-poc/terratest-poc/main.tf
There is nothing to comment on at line [1] in file [terratest-poc/terratest-poc/main.tf]
Preparing comment for violation of rule aws-ec2-enforce-http-token-imds in terratest-poc/terratest-poc/main.tf
There is nothing to comment on at line [1] in file [terratest-poc/terratest-poc/main.tf]

I'm seeing the same error, and to my horror realised recently it's been happening silently for a while and I've just been assuming all is good with the sec scan!

+1

any updates on this?

I've been diving the code and I think the problem will be fixed if in the file setup.sh the env variable GITHUB_WORKSPACE is setup from:
export GITHUB_WORKSPACE=/github/workspace
to
export GITHUB_WORKSPACE=/github/workspace/./
Becouse adding "./" this will be removed from result.Range.Filename
	workspacePath := fmt.Sprintf("%s/", os.Getenv("GITHUB_WORKSPACE"))
	fmt.Printf("Working in GITHUB_WORKSPACE %s\n", workspacePath)

	workingDir := os.Getenv("INPUT_WORKING_DIRECTORY")
	if workingDir != "" {
		workingDir = strings.TrimPrefix(workingDir, "./")
		workingDir = strings.TrimSuffix(workingDir, "/") + "/"
	}

	var errMessages []string
	var validCommentWritten bool
	for _, result := range results {
		result.Range.Filename = workingDir + strings.ReplaceAll(result.Range.Filename, workspacePath, "")

and then will match with filepath in the PR. Because in the PR the files haven't ./ prefix like tfsec write into the results.json
Resuls:
Working in GITHUB_WORKSPACE /github/workspace/
Preparing comment for violation of rule google-compute-no-default-service-account in ./terraform

I was also facing the same issue.
Thanks @mario-fernandez-mm , it seems that adding an empty working_directory does works.

The suggested fix of adding a blank working_directory doesn't work when using --force-all-dirs.

@sysophost It seems to be working with --force-all-dirs with my setup. I already have --force-all-dirs in my workflow.

@mandeeps13k I'm still getting the behaviour described by the OP.
Starting the github commenter
Working in repository my-repo
Working in PR 12
TFSec found 1 issues
Working in GITHUB_WORKSPACE /github/workspace/
Preparing comment for violation of rule custom-custom-001 in modules/security_group/rules.tf
Ignoring - change not part of the current PR

With this action definition
name: Run tfsec and comment on PR
on:
  - pull_request
jobs:
  tfsec:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.ref }}

      - name: tfsec
        uses: aquasecurity/tfsec-pr-commenter-action@main
        with:
          github_token: ${{ github.token }}
          working_directory: ''
          tfsec_args: --concise-output --config-file=.tfsec.yml --custom-check-dir=.tfsec --force-all-dirs

.tfsec.yml contains a few entries in the exclude block. There is a single custom rule in .tfsec which is the rule that is being picked up for violation (custom-custom-001).

I am seeing the same issue, I checked the code and the problem is related with commenter. Commenter checks if the comment line fits the changed lines. If the changed file contains multiple changed locations (ie changes -> from line 3 to line 10 and line 20 to line 30) commenter only checks the first hunk. And the tfsec problem is in between line 20 and line 30 it throws the exception Ignoring - change not part of the current PR

@sysophost @ahmetrehaseker
What version of tfsec-commenter-action are you guys using?
Try forcing the use of the 1.2.0 version, it might solve this problem.
This error was already described here.

Thanks for the suggestion @saerosV, I had tried pinning to an older version but was getting the same behaviour.
I think my issue comes from the fact I'm creating a resource from a module where the templated module is not being changed in the PR, just the place where it's instantiated.
Preparing comment for violation of rule custom-001 in modules/security_group/rules.tf
Ignoring - change not part of the current PR

In this case my tf is in a totally separate dir to the module, but as the resource is referencing the template in modules I assume it's being treated as a resource that is actually created in the modules dir rather than the dir where I'm creating an instance of the module.

@sysophost @ahmetrehaseker
What version of the tfsec-commenter-action are you guys using?
Try forcing the use of the 1.2.0 version, it might solve this problem.
A similar error was described here.

I did not test with that version but checked out the repo and debugged it myself, the problem is with the library you are using for commenting on the pr, as I wrote in the comment if you have multiple changed parts in the file library only checks for first hunk and ignores the other changes because of that return change is not part of the current PR
I created an issue for that repository Issue

Having the same this issue here. Certainly was working on these files in my PR.
Starting the github commenter
Working in repository terraform-modules
Working in PR 18
TFSec found 108 issues
Working in GITHUB_WORKSPACE /github/workspace/
Preparing comment for violation of rule aws-iam-no-policy-wildcards in ./global/iam/iam_groups/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-iam-no-policy-wildcards in ./global/iam/iam_groups/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-iam-no-policy-wildcards in ./global/iam/iam_groups/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-iam-no-policy-wildcards in ./global/iam/iam_groups/main.tf
Ignoring - change not part of the current PR
Preparing comment for violation of rule aws-iam-no-policy-wildcards in ./global/iam/iam_groups/main.tf
Ignoring - change not part of the current PR

My actions file:
- name: Run tfsec and write pull request comments
        uses: aquasecurity/tfsec-pr-commenter-action@main
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          tfsec_args: --force-all-dirs

I am also having this issue. My workflow:
name: tfsec
on:
  pull_request:
    paths:
        - 'terraform/modules/**'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment'
        required: true
        default: 'dev'
        type: choice
        options:
          - dev
          - staging
          - preprod
      refToBuild:
        description: 'Branch, tag or commit SHA1 to build (default HEAD)'
        required: false
        default: ""
        type: string

jobs:
  tfsec:
    name: tfsec PR commenter
    runs-on: ubuntu-latest

    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Clone repo
        uses: actions/checkout@v3
      - name: tfsec
        uses<EMAIL_ADDRESS>        with:
          github_token: ${{ github.token }}
          working_directory: terraform/modules

+1

I am also getting this error, even when using the --no-ignores argument.

+1

+1

+1

similar issue here
`name: tfsec-pr-commenter
on:
pull_request:
permissions:
pull-requests: write
actions: write
contents: write
id-token: write
jobs:
tfsec:
name: tfsec PR commenter
runs-on: ubuntu-latest
steps:
- name: Clone repo
uses: actions/checkout@v3
- name: tfsec
uses<EMAIL_ADDRESS>with:
github_token: ${{ secrets.GITHUB_TOKEN }}
tfsec_args: --force-all-dirs`
      I did changes to test it , it recognize the "error" in the change but say that it does not belong to PR , when it does.

just add working_directory empty:
- name: tfsec
        uses: aquasecurity/tfsec-pr-commenter-action@main
        with:
          working_directory: ''

This works for me!

Thank you so much. Its save me week

Also hitting this issue.  A fix has been suggested on the commenter action that is used but has not yet been merged https://github.com/owenrumney/go-github-pr-commenter/issues/14

The repository go-github-pr-commenter has been archived :(


Typing mistake in doc Documentation/concepts/networking/ipam/azure.rst

Documentation Error:
If you check this Documentation/concepts/networking/ipam/azure.rst I can see --aws-release-excess-ips=true. Shouldn't this be --azure-release-excess-ips=true

@fristonio Am I correct here I can give a small PR :)

Hey @chowmean, thanks for opening an issue for this. Taking a quick look at the operator code, it looks like we don't have the same option for Azure. Azure IPAM is relatively new for Cilium and I think the documentation is not very mature. I believe that we need to remove these docs related to the releasing of excess IPs as for Azure we don't provide this functionality. See the last option here for azure allocator.
cc/ @tgraf

Got it. Should I remove this part of the Documentation then?

Yeah, I think we should remove those lines as they don't apply to Azure allocator.


The action starts to fail to setup stable version with error: unknown function: help.print_and_exit

it was running nicely till today CI start to fail to install v, here are the logs
Run vlang/setup-v@v1
Resolved  as
Downloading vlang ...
Determining the default branch
Retrieving the default branch name
Default branch 'master'
Creating directory: /home/runner/work/vgrid/vgrid/vlang/vlang_linux_x64
Downloading the archive
Downloaded archive 'https://codeload.github.com/vlang/v/legacy.tar.gz/refs/heads/master'
Writing archive to disk
Extracting the archive
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/vgrid/vgrid/vlang/vlang_linux_x64/282aca22-2b51-4431-85ad-c768b[27](https://github.com/threefoldtech/vgrid/runs/7091283045?check_suite_focus=true#step:3:28)baf4c -f /home/runner/work/vgrid/vgrid/vlang/vlang_linux_x64/[28](https://github.com/threefoldtech/vgrid/runs/7091283045?check_suite_focus=true#step:3:29)2aca22-2b51-44[31](https://github.com/threefoldtech/vgrid/runs/7091283045?check_suite_focus=true#step:3:32)-85ad-c768b27baf4c.tar.gz
Resolved version vlang-v-64b8284
Running make...
cmd/v/v.v:149:15: error: unknown function: help.print_and_exit
  147 | fn invoke_help_and_exit(remaining []string) {
  148 |     match remaining.len {
  149 |         0, 1 { help.print_and_exit('default') }
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~
  150 |         2 { help.print_and_exit(remaining[1]) }
  151 |         else {}
cmd/v/v.v:150:12: error: unknown function: help.print_and_exit
  148 |     match remaining.len {
  149 |         0, 1 { help.print_and_exit('default') }
  150 |         2 { help.print_and_exit(remaining[1]) }
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  151 |         else {}
  152 |     }
make: *** [GNUmakefile:97: all] Error 1
Error: Command failed: make
cmd/v/v.v:149:15: error: unknown function: help.print_and_exit
  147 | fn invoke_help_and_exit(remaining []string) {
  148 |     match remaining.len {
  149 |         0, 1 { help.print_and_exit('default') }
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~
  150 |         2 { help.print_and_exit(remaining[1]) }
  151 |         else {}
cmd/v/v.v:150:12: error: unknown function: help.print_and_exit
  148 |     match remaining.len {
  149 |         0, 1 { help.print_and_exit('default') }
  150 |         2 { help.print_and_exit(remaining[1]) }
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  151 |         else {}
  152 |     }
make: *** [GNUmakefile:97: all] Error 1

my workflow file wasn't changed recently, I'm using the action with the below configuration:
- name: Set up V version latest
      uses: vlang/setup-v@v1
      with:
        stable: true

Any updates?

@ulises-jeremias can you have a look please?

Working on this right now. I'll send an update soon!

@sameh-farouk as it is mentioned in the README. It is needed to use check-latest: true when using the stable property.
# Set this option if you want the action to use the stable version of V.
# Will be ignored if `check-latest` is false.
stable: false

can you try with this instead?
- name: Set up V version latest
  uses: vlang/setup-v@v1
    with:
      check-latest: true
      stable: true

what is mentioned in the docs is that stable will be ignored if I use both check-latest and stable. I should be able to use the stable release in my pipeline, and this is what needs to be fixed.

Hi, @ulises-jeremias @sameh-farouk
I was getting the same error, but by running vlang/setup-v@v1 before actions/checkout as in the linked commit, it stopped failing.
https://github.com/zztkm/microcms-v-sdk/commit/70b48dedcb41acbc7b51b53e56ce43d4d86d0a70
I hope this helps.

The error does not seem to be specific to setup-v, I got the same error on a Windows runner using shell commands, not setup-v . Fixed by moving V build to start of job as suggested by @zztkm. The preceding step that seemed to trigger the error was:
- uses: msys2/setup-msys2@v2
  with:
    msystem: MINGW64
    update: true
    install: mingw-w64-x86_64-pcre2

Related to what I'm noticing here: https://github.com/ArtemkaKun/v-project-basement/pull/1#issuecomment-1519070727
Can run without adding a version. As soon as I add a version it runs into a make error.


Switch from 'categories' field to 'category' in model and serializers

@kshefchek

I thought I got all these over in https://github.com/biolink/biolink-api/pull/330#issuecomment-543967657 but I'll have to take a closer look

here is that commit - https://github.com/biolink/biolink-api/pull/330/commits/b2105e9161097bb40791e2c9439d8e0497c1e120

Any chance we can get #330 merged in soon?
If not, I would vote for this PR to be merged as I can get the latest change running off of api-dev
This is a dependency for a demo for STAGE that isn't working anymore because category ends up as null

is stage running off of api prod or dev?  api-dev is running off of #330 so perhaps I missed one

Oh! I was not aware of that.
The demo was using prod until now. I switched to dev.
I'll close this PR since what I want is already in dev.
Confirmed by the following query: https://api-dev.monarchinitiative.org/api/bioentity/HP:0002099


Add StaFiHub assets

Description

Checklist

Adding Chains

If adding a new chain, please ensure the following:

[x] Chain is registered to the Cosmos Chain Registry.
Note:

Chain's registration must have staking defined, with at least one staking_token denom specified.
Chain's registration must have fees defined; at least one fee token has low, average, and high gas prices defined.

[x] Add chain to bottom of zone_chains.json
Note:

RPC and REST must not have any CORS blocking of the Osmosis domain.
RCP node must have WSS enabled.

Adding Assets

If adding a new asset, please ensure the following:

[x] Asset is registered to the Cosmos Chain Registry.

[x] Add asset to bottom of zone_assets.json.
Note:

The IBC channel referenced in path must be registered to the Chain Registry.
osmosis_main defaults to false (or else cite the listing rule that justifies enlisting to the Main app)

On-chain liquidity
If adding a new asset, please provide the plan for on-chain liquidity of the asset: (choose one)

[x] The submitting team will ensure a pool will be created soon. Please hold off until further communication, which will notify of the Pool ID. (Please set the PR's status to Draft until the pool has been created)

OR

[ ] A preview link is requested for the submitting team to be able to create a pool using the Osmosis Frontend.

OR

[ ] The token is, or will be, going through a StreamSwap stream; thus, the token should be listed without requiring on-chain liquidity. A Pool ID will be provided in this PR following the stream's completion once the team has had a chance to create a pool using the earned funds.

Validation Testing
If adding or updating a chain or asset, the changes must be validated from the updated build of the Osmosis Frontend. Testers will need a small amount of tokens to validate.

[x] Validaters can buy a small amount of this asset from Pool ID: 1227, for rATOM, 1230 for FIS.
[ ] Preview build has been validated.

Can we add a trace under rATOM in the Chain Registry pointing it to ATOM so we can properly identify it as a related asset?
https://github.com/cosmos/chain-registry/blob/master/stafihub/assetlist.json#L46
Is it a liquid-stake ATOM?

Can we add a trace under rATOM in the Chain Registry pointing it to ATOM so we can properly identify it as a related asset? https://github.com/cosmos/chain-registry/blob/master/stafihub/assetlist.json#L46 Is it a liquid-stake ATOM?

Sure, will make a PR for this.

Can we add a trace under rATOM in the Chain Registry pointing it to ATOM so we can properly identify it as a related asset? https://github.com/cosmos/chain-registry/blob/master/stafihub/assetlist.json#L46 Is it a liquid-stake ATOM?

Added the traces. https://github.com/cosmos/chain-registry/pull/3050/files.


Make the processes that collect muscle results resumable so clustal omega doesn't get re-run at every workflow resume

Question asked in the Seqera discourse forum here: https://community.seqera.io/t/making-processes-that-collect-a-large-number-of-input-files-resumable/1578


logo in readme

adding logo that was accepted into the project's readme
related to https://github.com/kubernetes-sigs/descheduler/issues/978 and https://github.com/cncf/artwork/pull/374
Closes #978

Preview here: https://github.com/kubernetes-sigs/descheduler/blob/97c22e1e4b03b25b6e858aed1a5d202db589d08c/README.md
Please let me know if bigger or smaller or not centered is of your preference

Awesome!
/lgtm