NIFI-8609: Added unit test that is ignored so that it can be manually…

… run for testing performance before/after changes to AvroTypeUtil. Updated AvroTypeUtil to be more efficient by not using Record.getValue() and instead iterating over the Map of values directly. getValue() is less efficient here because we know the RecordField's we are iterating over exist in the schema since they are retrieved from there directly; as a result, any null values still have be looked up by aliaases, but that step can be skipped in this situation. Also avoided looking for fields that exist in Avro Schema and not in RecordSchema just to set default values on GenericRecord - there's no need to set them if they are default values.

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Thanks for the review @gresockj . Not sure that we really need to encourage developers to use it, to be honest. I created the unit test to verify that the changes that I was going to make would in fact result in better performance. And I figured it may be useful to me and others in the future so I left it. Is intended to simply be a tool to aid in that process when desirable.


fix misusage of dimension as string instead of dimensionSchema in DataSchema

The return type of dimensionsSpec.getDimensions() is DImensionSchema, timestampColumn is String, so dimensionsSpec.getDimensions().contains(timestampColumn)
always return false

@kaijianding can we add a test?

There is no need to add new test, the DataSchemaTest has all the needed test.
Actually, this bug isn't really a bug, the later code, especially the Sets.difference(dimensionExclusions, dimSet)
            inputRowParser.getParseSpec()
                          .withDimensionsSpec(
                              dimensionsSpec
                                  .withDimensionExclusions(
                                      Sets.difference(dimensionExclusions, dimSet)
                                  )
                          )

makes things right even when the dimensionExclusions is incorrect, that's the reason why it passed the UT before I fixed this bug.

👍


BUG: Integrated pandas can't Read CSV while latest pandas can

Describe the bug

Problem 1: a 25G csv file, latest pandas can load properly, however,  "import xorbits.pandas as pd" can't, xorbits gives out EOF error
Problem 2:  a data frame data loaded from latest pandas can't be send to dup function (from xorbits.experimental import dedup)
Problem 3: dedup function can handle a str with very long str, e.g. length between 4000-100,000, it gives out error "too many open files"

To Reproduce
To help us to reproduce this bug, please provide information below:

Your Python version: 3.10
The version of Xorbits you use: 0.6.3
Versions of crucial packages, such as numpy, scipy and pandas: numpy 1.26.0, scipy 1.11.3, pandas 2.1.1
Full stack of the error.
Minimized code to reproduce the error.

Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

Problem 1: Is your csv file located in local disk or remote (by a url)?
Probelm 2: Are you using pandas to load the csv and constructing a xorbit Dataframe by the pandas Dataframe? If so, it could be out of memory crash, because the full data will be serilialized to worker.
Problem 3: The too many open files can be fixed by configure the ulimit.


ansible: poor performance with multiple targets

Per user report in #249, now that >100 targets is possible growing pains are starting to rear their heads.
Likely low hanging fruit:

1-syscall-per-msg overhead that can be coalesced easily
5-syscalls-per-Latch.get() overhead due to the work in #249
1 connection multiplexer per CPU core on the controller
ModuleFinder inefficiencies that are hidden by Ansiballz caching
dict(hostvars) in connection.py

Hard mode:

VariableManager is one of the most expensive parts of Ansible (the code is from hell itself), we call it repeatedly to form the chain of configurations for connection delegation.

Hi @sharph, if you're feeling adventurous, I have 3 Ansible patches in https://github.com/dw/ansible/tree/devel that reduce runtime almost 40% for 10 host runs. These don't help explain why Mitogen is slower, but I'm struggling to profile that yet because Ansible sucks so much in these cases :)
Still "work in progress", but if you're interested in kicking the tires, it would be very useful.

Hi @sharph, can you please let me know the latency between your controller machine and the target machines. Thanks

Hi @sharph,
Long time no update :)  I have not stopped working on very large run performance, but unfortunately most of the important work cannot exist in a third-party extension.
Meanwhile, over the past months quite a few big fixes have been found -- two in particular were addressed in 0.2.3, relating to context switching overhead due to the use of threading, and one that concentrated a lot of redundant CPU usage in the connection multiplexer process.
I also have a separate branch that supports per-CPU multiplexer processes, however it is a little stale at this point. If you are still interested in this kind of work, and the newer fixes don't help much -- please say! I can at least see if the per-CPU work helps in your case, and if so, we could see about merging it into the stable series.


Feature request: styles for deeper shadows

Is this a new component, or a missing feature in an existing one?
A missing feature.
Component name:
Card (sort of -- shadows aren't a "component" but the MDL shadow styles are documented on the MDL Card documentation page).
Material Design spec URL for the component (if any):
https://www.google.com/design/spec/what-is-material/elevation-shadows.html
What does this component or feature allow you to do which isn’t possible at the moment?
Deeper shadows.  MDL provides mdl-shadow--2dp through mdl-shadow--16dp, which is great as far as it goes, but the MD spec calls for shadows between 1dp and 24dp.
Please provide any URLs or screenshots of good examples of usage of this component or feature that you may have come across.
The Material Design "Dialogs" page shows dialog boxes with "Dialog elevation: 24dp": https://www.google.com/design/spec/components/dialogs.html#dialogs-specs -- there's no dialog boxes in MDL yet (I know about #781) but if we had the shadow mixins we could make our own easily enough.

I think a mixin to generate any arbitrary DP of shadow may be a bit complicated to pull off. Probably best we focus on adding the 24dp shadow directly since it seems to be the major missing one from the spec.

+1 to @Garbee's suggestion.


[AddOn] Helm values are not merged

When deploying an Helm Chart Addon, I would like to be able to configure it with the use of additional values parameters.
The AddOn constructs offer me to to this with a values filed, but It seams that the default values are not merged with the one I specify and that only the specified values are taken into account.
This can be difficult for applications where we could have hundreds of goo default parameters, we should be able to merge our values with default ones.
example:
                new ssp.ArgoCDAddOn( {
                    bootstrapRepo: {
                        repoUrl: gitUrl,
                        targetRevision: "main",
                        path: 'envs/dev'
                    },
                    adminPasswordSecretName: MultiRegionConstruct.SECRET_ARGO_ADMIN_PWD,
                    namespace: "argocd",
                    values: {
                            ingress: {
                                enabled: true,
                                hosts: [
                                 "argo."+subdomain,
                                ]
                            }
                        }
                    }
                }),

If understanding the problem correctly (others please feel free to correct/address):
Problems
The ArgoCD community has a default non-empty helm chart (and customer may have their own best practice) to deploy
The current SSP utilize the AWS CDK's default HelmChart module to pass the value. The behavior is if values are present, it will use the values pass in to overwrite the entire helm chart
I am thinking is is to solve merging of two helm value files (default vs one created by above parameter passed in)
https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-eks/lib/helm-chart.ts , it is not doing any merge, but directly pass the value as is to custom resource
        Values: (props.values ? stack.toJsonString(props.values) : undefined),

The reason may due to helm v3 by itself does not support merge (still in discuss, but not exist yet) as discussed in this thread (the biggest concern is accidently bad results from merging)
https://github.com/helm/helm/issues/3486
So AWS CDK EKS Helm implementation seem to follow that idea and not performing merging on its own to create a "different behavior" than default helm chart behavior
SSP CDK follows that paractice, and some values are able to pick up as these are "known" to control and can just be override not merge
Few ideas to "bridge" these problems for now (assume Helm will not implement it nor CDK EKS Helm):

Doc this limitation explicitly and provide better tooling to assist users (either or combine below points or other way)
Allow an optional field to let user input values as a helm chart, but customer provide the helm chart location. User then can specify a helm chart with all custom values in one place, and rely on CDK to just pass in and use it. Then keep that helm chart with their SSP git repo code change for audit

Good thing:
There will be centralized way to keep the merging and avoid ambiguous merge
Full control the argocd helmchart without need CDK/SSP to constantly keep up with latest feature
Bad thing
There are now some configuration leaked out from CDK managed code into outside of CDK, which is not ideal
But did see some project with similar approaches to work around some customization around argocd, some may already need to do this way as there argocd is hosted internal git repo instead of public github repo, so they keep a copy along with SSP

[Not recommend] SSP itself will implement a brute force way:
An options to specify default helm chart values location, if so, try to download the pre-exist template and did merge in local SSP (which is like 2, but did merge automatic). Not recommend as above helm git topic points out, that merge of yaml may produce some unexpected problems, and not sure if really want to rely on that.

I think 1 and 2 probably is best we can do now (not sure 2 is okay by the team or it is totally outside of realm and only will suggest 1).

@allamand this has been fixed a few months ago. If you get a chance, do you mind confirming?

@danilko somehow I missed your original comment. The issue is more about how our add-on is structured. We allow passing values directly to the helm add-on as well as providing shortcuts to some of the values. When merging shortcuts with the passed values in some cases the values side was not retained and was overwritten. I believe it is resolved now, but looking for feedback.

Yes sorry forgot about the issue this is solved for me


[Bug]: Can't set assistant avatar

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch name
main
Commit ID
6ff63ee
Actual behavior

Fixed


Playground: Cannot handle multi-file imports in a single code sample in the Playground

Summary
TypeScript Twoslash says it can “Handle multi-file imports in a single code sample”
https://github.com/microsoft/TypeScript-Website/blob/v2/packages/ts-twoslasher/test/fixtures/importsModules.ts
// @filename: Component.tsx
import React from "react"

export function Hello() {
  return (

Hello World

  )
}

// @filename: index.ts
import { Hello } from "./Component"
console.log(Hello)

Try it in the playground and find it doesn't work.

Screenshot

Repro: https://www.typescriptlang.org/play/#code/PTAEAEDMEsBsFMB2BDAtvAXKAwge1QA66JIAuAdKQM4AeAUNIbgE6mgBK8yAxm5M-lAAiZl15C6deDSKtQkAK6Je0YqAAS8WLFwAKAJSgA3nVChRpBc0Shdps6AA8AE2gA3AHz2HTgBYBGD01tXFAAdRZYZ0dgAK8fGNdPe306AF9JEAgYBBR0LGhEZ2lKKgYmOSMNLR1QNPkBVGFyYDwmEkRSCW5iKlwEch0Ac11gnVSgA

Expected behavior: It works normally.
Actual behavior:  Can't work.

Twoslash isn't actually running in the playground, there's just some specific features like // ^? which were replicated within the playground (I did get multi-file running, but it wasn't stable enough to warrant shipping permanently )

I did get multi-file running, but it wasn't stable enough to warrant shipping permanently

Hope it arrives soon, this issue can be closed (or kept open until it works).


Cannot install 1.1.0

https://github.com/openSUSE/spec-cleaner/blob/d7e49579e7c523340ef4944d84dcd0890b4e8e6f/spec_cleaner/rpmpreamble.py#L5
It's called during setup via the following.
2018-07-03 04:24:37.456998 | primary | Collecting spec-cleaner!=0.9.3,>=0.8.2 (from -r /opt/stack/new/requirements/global-requirements.txt (line 413))
2018-07-03 04:24:38.017423 | primary |   Downloading http://mirror.mtl01.inap.openstack.org/pypi/packages/81/95/129d8eaed80d2ae6892a3d9b1e6f2abc833c0a7f9cc5be89882aed7ffc29/spec_cleaner-1.1.0.tar.gz (495kB)
2018-07-03 04:24:38.361448 | primary |     Complete output from command python setup.py egg_info:
2018-07-03 04:24:38.361967 | primary |     Traceback (most recent call last):
2018-07-03 04:24:38.362087 | primary |       File "<string>", line 1, in <module>
2018-07-03 04:24:38.362255 | primary |       File "/tmp/pip-install-L5Km70/spec-cleaner/setup.py", line 7, in <module>
2018-07-03 04:24:38.362436 | primary |         from spec_cleaner import __version__
2018-07-03 04:24:38.362557 | primary |       File "spec_cleaner/__init__.py", line 13, in <module>
2018-07-03 04:24:38.362650 | primary |         from .rpmcleaner import RpmSpecCleaner
2018-07-03 04:24:38.362764 | primary |       File "spec_cleaner/rpmcleaner.py", line 16, in <module>
2018-07-03 04:24:38.362853 | primary |         from .rpmpreamble import RpmPreamble
2018-07-03 04:24:38.362967 | primary |       File "spec_cleaner/rpmpreamble.py", line 5, in <module>
2018-07-03 04:24:38.363061 | primary |         from urllib import parse
2018-07-03 04:24:38.363152 | primary |     ImportError: cannot import name parse

I dropped py2 support for easier unicode support at:
https://github.com/openSUSE/spec-cleaner/commit/1252176826d2fda4365b881149a45425a4b420aa
I guess you are installing it in python2 right?

ya, looks like it

so, looks like devstack (openstack) is installing spec-cleaner unconstrained in python2, which is wrong.  we could change the requirement to the following though.
-spec-cleaner>=0.8.2,!=0.9.3  # BSD
+spec-cleaner>=0.8.2,!=0.9.3,<1.0.9;python_version<'3.0'  # BSD
+spec-cleaner>=0.8.2,!=0.9.3;python_version>='3.0'  # BSD

If you guys can do that, it would be lovely, alternatively I would have to revert the py2 drop...

I think it's within policy because of the type of breakage it is.
https://docs.openstack.org/project-team-guide/dependency-management.html#global-requirements
https://review.openstack.org/579936

we simply backported the removal of the requirement.  closing this, thanks for the quick response


Event was commented notification

As a user I want to be notified when an event I posted was commented so that I can reply people's questions soon
Preconditions

User created an event and another user left a comment

Acceptance Criteria

When a new comment is left under the user's event, the user must get a new notification about it

A notification counter value is increased by one

User should see a corresponding notification record when he views the most recent notifications popup or All notifications page:

[Username] commented on your event [Event name]. [Date and Time]

(ex. "Maria commented on your event Eco Day. Today 10:00 a.m")
[Username] - user who left a message

If more than one user left a comment by the moment the user sees this notification, then the usernames of 2 last commentators and "other users" should be instead (ex, "Maria, Olha and other users commented on your event Eco Day. Today 10:00 a.m")
If the same user left more than one comment on the same event by the moment the user sees this notification, then user should see only the latest one in the notification record

[Event name] - name of the event. Should be a link, should be cut if too long (?)
[Date and time] - Date and time when the comment was left.

Date format is DD.MM.YYYY
Time format is  HH:MM XM

If the day = Today, then use "Today" word instead of calendar date
If the day = Today - 1, then use "Yesterday" word instead of calendar date

Epic link
E.g.: Epic #6332
Labels to be added
"User story"

Already done in EventCommentServiceImpl 123 line.


How to support Oauth2 flow

I am trying to add "AddSecurityDefinition" with type "SecuritySchemeType.OAuth2". It can redirect to IDP (Azure B2C) to login. But it fails when callback "http://localhost:7071/oauth2-redirect.html". Could you please let me know how to configure it?

Hi, currently it's not supported, but planned.
I guess I will look into implementing it somewhere in July (after my project release) I guess I should have more time.

Hi, @tungphuong
First of all I need to see your configuration and what you are trying to achieve.
Better to have a repro of the problem, if possible.
For instance, configuration for oAuth 2.0 Implicit flow might look something like this:
                x.AddSecurityDefinition("oauth2", new OpenApiSecurityScheme
                {
                    Description = "JWT Authorization header",
                    Name = "Authorization",
                    In = ParameterLocation.Header,
                    Type = SecuritySchemeType.OAuth2,
                    Scheme = "Bearer",
                    Flows = new OpenApiOAuthFlows()
                    {
                        Implicit = new OpenApiOAuthFlow()
                        {
                            TokenUrl = new Uri($"someurl"),
                            AuthorizationUrl = new Uri($"someurl"),
                            Scopes = { }
                        }
                    }
                });
                x.AddSecurityRequirement(new OpenApiSecurityRequirement
                {
                    {
                        new OpenApiSecurityScheme
                        {
                            Reference = new OpenApiReference
                            {
                                Type = ReferenceType.SecurityScheme,
                                Id = "Bearer"
                            }
                        },
                        Array.Empty<string>()
                    }
                });

Closed as stale

Hi, I am facing the same issue as @tungphuong mentioned using @vitalybibikov proposed code snippet from the above.
It error when trying to post back to oauth2-redirect.html:
No webpage was found for the web address: http://localhost:7071/oauth2-redirect.html#access_token=xxxxxx

My swagger ui's url is: http://localhost:7071/api/Swagger/ui


TypeError if set prototype of a function in some situations

A reproducible example:
package main

import (
	"fmt"

	"github.com/dop251/goja"
)

const code = `
"use strict";
function A() {}
A.__proto__ = Object;
// void A.prototype;
A.prototype = {};
`

func main() {
	fmt.Println(goja.New().RunString(code))
}

// <nil> TypeError at <eval>:6:15(11)

Since a function's prototype is lazy initialized at getPropStr, the code above does not trigger an initialization.
When executing A.prototype = {};, goja would throw a TypeError with empty message as it detects A.__proto__.prototype is not writable. Uncommenting // void A.prototype; can avoid this.
I fixed the issue temporarily by moving addPrototype to Runtime#newFunc. Maybe there would be a better solution?

Hi @lujjjh,
Thanks for reporting this. There is a better solution and I am going to push it shortly.


removing duplicate def reference

Summary of the Pull Request

PR Checklist

[ ] Closes Issue #xxx
[ ] Code of Conduct signed. If not, go to Code of Conduct.

Validation Steps Performed

@sasever , @uday31in , can you please update the default parameter for secret and key expiration etc.?


Upgrade to latest openssl

Currently, lettre's Cargo.toml specifies using Rust-OpenSSL version 0.7.x.  At the time of this writing, the latest release is version 0.8.2, which includes a number of bug fixes that, in some applications has resolved some functional failings.
I'd like to ask for lettre to adopt these changes and upgrade to the latest version.

Thank you, done in b4603b4fbccc071d5c77b6723f4636c914fdd51c, closing.


Add an argument for skipping secret verification

In some environments, e.g. behind a Kong gateway, you may be doing secret verification in a different place and want to skip it in your app.

Thanks for the pull, I've implemented something similar to this in the latest release.


how to get densepose videos without shutering andd blurry flicker

as the title ,i use vid2densepose to generate densepose of driving videos, it happens to be shuttering and blurry flicker while i found that there are no such matters in densepose you provide in demo .
as the blur and flicker in densepose ,there are blur and flicker in the output video .
so how to get densepose videos without shutering andd blurry flicker
https://github.com/magic-research/magic-animate/assets/111668189/d70d2fdd-cbe7-45d3-8bc9-ceb9981ee088
https://github.com/magic-research/magic-animate/assets/111668189/2a9c77fc-e480-4806-b6c9-f73a8da8de7b
https://github.com/magic-research/magic-animate/assets/111668189/4d15c5b4-dbc0-4956-aff6-b9796f8c23e0
https://github.com/magic-research/magic-animate/assets/111668189/d4b1b972-f866-49c2-88c1-408972ff6c17

Hi, can you please upload the video again. i am facing similar problem. and where did you get the code to create densepose. the one i am using seems to have different color coding.


<EMAIL_ADDRESS>decompress error

Updating one outdated app:
Updating 'paint.net' (4.2.11 -> 4.2.12)
Downloading new version
paint.net.4.2.12.install.zip (9,8 MB) [=======================================================================] 100%
Checking hash of paint.net.4.2.12.install.zip ... ok.
Uninstalling 'paint.net' (4.2.11)
Removing shim for 'paintdotnet'.
Unlinking ~\scoop\apps\paint.net\current
Installing 'paint.net' (4.2.12) [64bit]
Loading paint.net.4.2.12.install.zip from cache
Extracting paint.net.4.2.12.install.zip ... done.
Running pre-install script...
Running installer script...
ERROR Exit code was 1618!
Failed to extract files from C:\Users\XXX\scoop\apps\paint.net\4.2.12\tmp\PaintDotNet_x64.msi.
Log file:
  ~\scoop\apps\paint.net\4.2.12\tmp\msi.log

Please try again or create a new issue by using the following link and paste your console output:
https://github.com/lukesampson/scoop-extras/issues/new?title=paint.net%404.2.12%3a+decompress+error

msi.log is empty

Use lessmsi instead of msiexec. scoop install lessmsi; scoop config MSIEXTRACT_USE_LESSMSI $true

thx.


translate directive messes with buttons

Hi,
I have issues when using the translate directive with buttons, in particular when I use ng-show/hide and translate: some buttons are shown / hidden as they should but others don't.
This is clearly related to angular-translate because when I remove the translate directive, it works as expected.
Using filters instead of a directive works, but angular's website states that it's better to use the directive...
Here's an example that shows my issue: http://jsfiddle.net/nxrhpxuy/2/
Note that I had this problem on my website which uses angular 1.3.14 and angular-translate 2.6.1.
Thanks,
Laurian

btw: this fiddle is using Angular JS 1.2

That's because fiddle doesn't allow me to use a newer version. But I have the issue on my website as well and it uses angular 1.3.14.

You can inject your own JS, do not have to use the predefined ones. I've migrated it already to a plunkr, it's okay.

@lverre please you have the following choices for it to work correctly - it's more an issue of the combination / scope-creation for angular-translate and the ng-click-handling...:
prefix the scope variables with a model var and initialize it beforehand
<button ng-hide="mainmodel.isEditingName" type="button" class="btn btn-primary" ng-click="mainmodel.isEditingName = true" translate="SHARED_EDIT">Edit</button>

and in the controller, init it
var mainmodel = {};

Or, beginning from AngularJS 1.3, the controllerAs syntax
<div ng-controller="EditCtrl as editCtl">
...
	<button ng-show="editCtl.edit" type="button" class="btn btn-primary" ng-click="save();" translate="SHARED_SAVE">Save</button>
	<button ng-show="editCtl.edit" type="button" class="btn btn-default" data-translate="SHARED_CANCEL">Cancel</button>
	<button ng-hide="editCtl.edit" type="button" class="btn btn-primary" ng-click="editCtl.edit = true" translate="SHARED_EDIT">Edit</button>

And also don't forget to update / change the corresponding variable name in the save - function :-)
Hope that helps...

And just for the reference, the plnkr.

Thanks!


⚠ Bofrost.de has degraded performance

In 9993edb, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1448 ms

Resolved: Bofrost.de performance has improved in 29ea96c.


Responsive images

What does this PR do? (please provide any background)

Responsive images are here!
This change allows you to specify a range of images for display at different widths using the srcset attribute.

What tests does this PR have?
None yet - this will not be merged
How can this be tested?

Clone the UI Toolkit
Run grunt build
Run grunt docs
When the docs have loaded up, go to the image component and view the responsive section (a cute panda)
Shrink your browser window to a small size (smaller... smaller still) and hard refresh the page. You should see a green panda image with 1000w written on it.
As you expand the window this image will change to blue (1500w) and red (2000w). This is swapping out images in a responsive manner.
If you now shrink your browser back down to mobile, it will stay red (2000w) as that asset has already been downloaded.

if you need me to demo this, please ask :)
Screenshots / Screencast

What gif best describes how you feel about this work?

Developer Definition of Done/Quality Checklist (for PR author to complete BEFORE code review):

[x] I have checked the contributing document and I'm happy for this to be reviewed.

Software Engineer or Developer review:

[x] I’ve witnessed the work behaving as expected (this could be on the authors machine or screencast).
[x] I’ve checked for coding anti-patterns.
[x] I’ve checked for appropriate test coverage.
[x] I’ve checked all the tests are passing.

Software Engineer or project guru review:

[ ] I’ve witnessed the work behaving as expected (this could be on the authors machine or screencast).
[ ] I’ve checked for coding anti-patterns.
[ ] I’ve checked for appropriate test coverage.
[ ] I’ve checked all the tests are passing.

Taking DEV

Looks awesome :+1:

:)

Taking SE/Guru


Rule 921170 always flags in error (PL 3 rule)

Rule 921170 is not a detection rule: it is an admin/scaffold rule designed to make the HTTP parameter pollution rules.
SecRule ARGS_NAMES "@rx ." \
    "id:921170,\
    phase:2,\
    ⋮
    setvar:'TX.paramcounter_%{MATCHED_VAR_NAME}=+1'"

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --paranoia-level=4
11:09AM INF ⏳Running quantitative tests with 10 goroutines
Run 10000 payloads in 9.657304833s
Total False positive ratio: 26481/10000 = 2.6481
False positives per rule id:
  920220: 130 false positives
  920221: 130 false positives
  920272: 3299 false positives
  920273: 10000 false positives
  921170: 10000 false positives
  ⋮

It looks like go-ftw also cannot correctly handle rule 920273 (from the results above).


#155550446 - implement rate idea feature

What does this PR do?

Implement rate idea feature on the front-end

Description of Task to be completed?

create rate idea section in idea details page
create actions and reducers

How should this be manually tested?

clone and setup this repo locally
checkout to this branch feature/155550446/rate-idea
run npm start:client to start the app.
click on idea title to navigate to idea details page.

Any background context you want to provide?

None

What are the relevant pivotal tracker stories?
https://www.pivotaltracker.com/story/show/155550446
Screenshots (if appropriate)

#### Questions:
N/A

Pull Request Test Coverage Report for Build 144

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.951%

Totals

Change from base Build 142:
0.0%

Covered Lines:
302

Relevant Lines:
342

💛  - Coveralls


Can't upload file more than 1MB suddenly

Dear All,
Today I found that some of my files that more than 1MB can't be upload to my s3 bucket. It worked a week ago. Please Help! Thank you!
.... function createClient() {         return require('s3').createClient({             maxAsyncS3: 20,              s3RetryCount: 30,              s3RetryDelay: 60000,             multipartUploadThreshold: 209715200,             multipartUploadSize: 157286400,              s3Options: options         });     } var client = createClient(20); var params = { localDir: lambdaTmp + '/content', s3Params: { Bucket: exportBucket, Prefix: "content/" } }; console.log('start'); var uploader = client.uploadDir(params); console.log(uploader); uploader.on('error', function(err) { console.log(err);     reject(); }); uploader.on('progress', function() { console.log("progress", uploader.progressMd5Amount, uploader.progressAmount, uploader.progressTotal); }); uploader.on('end', function() { console.log('end');     resolve(); }); ....

This does not seem related to node_redis. Please submit your issue to a related repository (like the s3 repo)


SQL executor to use library list

This might be possible now we don't use db2util anymore. Will look into it

tried and failed


SaveInputAnimationAsync never completes

Describe the bug
If I record an animation and save it via SaveInputAnimation, a .bin file is saved in the respective folder as expected.
If I however try to record an animation and save via SaveInputAnimationAsync, only an empty .bin file is created and the async saving task is never completed.
I prefer to use SaveInputAnimationAsync, since SaveInputAnimation leads to a noticable interruption of the app.
To reproduce
Replace RecordingService.SaveInputAnimation(); in InputRecordingControls with RecordingService.SaveInputAnimationAsync();.
Expected behavior
SaveInputAnimationAsync should also create a not-empty .bin file.
Screenshots
If applicable, add screenshots to help explain your problem.
Your setup (please complete the following information)

Unity Version 2020.3.6f1
MRTK Version 2.7.0

Target platform (please complete the following information)

HoloLens 2

Hi @cwule, just to make sure could you confirm that you used the await keyword before RecordingService.SaveInputAnimationAsync()? If not please make sure to do that and mark the SaveRecordedInput function as async. Let me know if that works for you.

@MaxWang-MS Thanks for the quick reply, that was it!


Actions

Added action to run all tests on Merge Requests and merges.

CI/CD doesn't need secondary review .


[tokenizer] decouple tokenizer from the main scheduling thread

This PR tries to remove detokenization off the critical path of engine execution. We achieve so by create an abstraction of Detokenizer, which provides functionality of detokenizing output token_ids into text.

Some interesting findings:
We calculate how much time spend on overall time, step time (time take to run step functions), and execute time (time take to execute on the model), the benchmark is run on A100 40G  * tp 4 with Llama 70b, 512 context length and 128 prompt. We run 500 requests and measure the time spend on each stage:
baseline:

model execution time takes 183.7 seconds. step takes 187.5 seconds, overall it takes 185.8 seconds
-- the overhead of tokenizing + other scheduling takes 3.8 seconds in step function
-- the overhead of other scheduling takes 2.1 seconds

if we use dummy tokenizer (noop)
-- we reduce the tokenizing + other scheduling overhead from 3.8 -> 1.4 seconds
fi we use ArgMax sampler
-- we reduce the model execution time from 183.7 seconds to 167.8 seoncds.

So overall:
tokenizer is around 1.27% overhead
sampler is around 8.47% overhead
other scheduling is around 1.12% overhead
remaining are model execution


Update sponsoring-an-open-source-contributora.md

Why:
Closes [issue link]

What's being changed (if available, include any code snippets, screenshots, or gifs):

Check off the following:

[x] I have reviewed my changes in staging (look for the "Automatically generated comment" and click the links in the "Preview" column to view your latest changes).
[x] For content changes, I have completed the self-review checklist.

Writer impact (This section is for GitHub staff members only):

[x] This pull request impacts the contribution experience

[x] I have added the 'writer impact' label
[x] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

Sponsoring changes

@AbhijeetSharma1 Thanks for opening this PR! However after speaking with the team, we are not going to make this change. 💛
Please take a look at our help wanted section to find open issues you can work on.
Thank you for interest and passion for improving the docs! ✨

😝.🤣


Support to electron-updater

Hi guys, i would like to know if has electron-updater in project.
thanks

Hi @nicholasess. No, has not.


Use official FSharp.Core package

Any chance you could use https://www.nuget.org/packages/FSharp.Core.Microsoft.Signed/?
We're getting method missing exceptions downstream in paket.

The problem for me with not using the package is that I don't feel comfortable adding Assemly redirects to NUnit runner and/or Visual Studio.


[KYUUBI #3052] Support to get the real client ip address for http connection when using VIP as kyuubi server load balancer

Why are the changes needed?

To close #3052
If your server is behind a load balancer or other proxy, the server will see this  load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

this is odd, shall it be forwardedAddresses/X_FORWARDED_FOR_HEADER

I tried to

this is odd, shall it be forwardedAddresses/X_FORWARDED_FOR_HEADER

X-Forwarded-For
The X-Forwarded-For (XFF) request header is a de-facto standard header for identifying the originating IP address of a client connecting to a web server through a proxy server.

In fact, it depends on the proxy setting.
For the ebay HLB, there is no  X-Forwarded-For header.
And the header to present the real client ip is X-eBay-Web-Tier-IP instead of X-Real-IP.

FYI:
https://docs.fastly.com/signalsciences/faq/real-client-ip-addresses/#setting-alternative-headers-directly-in-the-agent
Because the LB I am using does not have X-Forwarded-For, so I can not make manual testing for that.
How about do it in another pr.

how about making X-Forwarded-For/X-Real-IP in one and delivering with forwardedAddresses
we need the IP of the direct accessor if it is not only proxying address but also proxying user

on second thought, it makes sense. please add a ut

ok, I will add UT for rest connection.

Codecov Report

Merging #3053 (b375d02) into master (1bc3916) will increase coverage by 1.05%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #3053      +/-   ##
============================================
+ Coverage     50.44%   51.50%   +1.05%
  Complexity        6        6
============================================
  Files           454      453       -1
  Lines         25480    25166     -314
  Branches       3576     3506      -70
============================================
+ Hits          12854    12961     +107
+ Misses        11406    10967     -439
- Partials       1220     1238      +18

Impacted Files
Coverage Δ

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
97.25% <100.00%> (-0.08%)
:arrow_down:

.../apache/kyuubi/server/http/ThriftHttpServlet.scala
69.76% <100.00%> (+0.23%)
:arrow_up:

...ver/http/authentication/AuthenticationFilter.scala
91.66% <100.00%> (+0.14%)
:arrow_up:

...uubi/engine/spark/events/SparkOperationEvent.scala
88.88% <0.00%> (-5.56%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
80.00% <0.00%> (-2.00%)
:arrow_down:

...ache/kyuubi/operation/KyuubiOperationManager.scala
80.82% <0.00%> (-1.37%)
:arrow_down:

...g/apache/kyuubi/server/KinitAuxiliaryService.scala
65.78% <0.00%> (-0.88%)
:arrow_down:

...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala
98.23% <0.00%> (-0.84%)
:arrow_down:

...he/kyuubi/jdbc/hive/ZooKeeperHiveClientHelper.java
2.88% <0.00%> (-0.76%)
:arrow_down:

...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala
89.27% <0.00%> (-0.70%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bc3916...b375d02. Read the comment docs.

thanks @yaooqinn @pan3793
merged to master


[Feature Request] populate indexes for has-Many associations

I want to propose a method that would populate hasMany associations with an array of primary keys instead of fully fetched objects.
Reason why?

Performance: Using relational databases we only have to look up the keys in the join table without performing an expensive full join
Standards: A lot of popular JSON API implementations accept or even expect relationships to be expressed as an array of primary keys, see JSON API specs or Ember Data's RESTAdapter

Current situation:
When using associations in Sails / Waterline and we query for records, we currently have two choices:
with populate: associated records are retrieved and embedded
without populate: belongsTo (model) associations are represented by their primary key and hasMany (collection) associations are ignored (in fact the JSON representation is even missing the properties)
Use case:
In order to build a performant solution for sideloading records, I want to query my Models with associations included as primary keys. Then I want to filter duplicated associations before fetching the full records for the associations.
Example:
I would like Group.find().populate("members") to return something like { topic: "...", members: [1,2,3,4] } instead of the fully fetched objects.
I am not sure if Waterline is already optimizing for duplicated records, but if it is, I couldn't find any exposure of this functionality to tap into.

@mphasize I think handling the association should go exactly after that _.each here which copies the attributes (but does nothing to associations).
Anyone knows if some work had already been done on this issue or not?
cc @particlebanana

I am guessing this would simplify the deep populate, since foreign keys of 2nd level associations would be available.. it will become a recursion of populate much simpler than the current setup.

isn't this related to (and would be closed by) the toObject issue https://github.com/balderdashy/waterline/issues/317#issuecomment-87274671

@mphasize, @anasyb, I like this. :+1:
Have you thought about how to actually request find() to only provide the fkeys? Something like:
Group.find().populate("members", { fKeys: true } );

// or

Group.find().populateKeys("members");

Comes to mind//. I'm not a 100% sure if this won't need changes or additions in the adapter API to make it more performant.

I definitely vote for this. This will solve multiple use cases.
Traversing relationships is a pain at the moment.. populating the whole record just to get the PK of a relationship is too much. Not to mention the freaky performance issue https://github.com/balderdashy/waterline/issues/944 that I never got the chance to look into. I had to access the join collections directly at multiple occasions (yuk!).
I was thinking more of the second notation Group.find().populateKeys("members");. For the first notation I think that it should be skipped in favor of https://github.com/balderdashy/waterline/pull/952 ... which on the other hand should only access the join table in case the only picked attribute is the id.

@anasyb, great point regarding skipping the first notation in favor of select/pick which I agree with. Lets stick with the below if no one disagrees:
Group.find().populateKeys("members");

I like this.  I'd just like to add, it'd be nice to see adapters dealing with this in an efficient way.  For example, sequel-interfaced adapters could get away with one join in many-to-many relationships rather than two.  Any suggestions how to introduce a shim for this feature within waterline and simultaneously allow adapters to provide their own performant solutions?

Totally agree! I think this should be done in the same manner as .populate() / adapter.join() works:

If the adapter doesn't provide a custom .joinKeys() (or whatever) waterline core will provide the results, probably using some subset logic of the existing .populate;
If the adapter implements .joinKeys() waterline will call it.

Okeedokes, sounds good!

Couldn't agree more and I'm also voting for the .populateKeys() style api. :+1:

Hello guys, any news about the .populateKeys feature?
Best,

i have the same question, any news about the .populateKeys feature ?

This is clearly desired, so I'm reopening until someone PRs this into the roadmap file.  Anybody feel free!

this is interresting, not only for primaryKey ... it would be faster than a populate, i vote for :
model.find(1).pluck('id')

This should be re-opened (and that damned bot should be shut down).
I love sails, but this is my number one complaint. If you're exposing a REST API, you really need the one-to-one and many-to-one ids that are populated directly on the record. You don't need to populate the associated data because clients can then make a second API call to fetch the data for the association.
My current workaround is to populate the relationships twice on ever model: once as a 'model' association (for use in custom controller logic) and again as an 'integer' so that the id gets populated without the enormous overhead of populating the associations.
Before I go and fork Waterline and try to hack together a fix, has anyone else solved this?

@jcowley you could use populate with select ['id'], but the way waterline and adapters are wrote doesn't let you populate childs ids without the "enormous overhead of populating the associations" ...
the problem is that sequel use union for populate and the cursor is too generic to be fast...
we forked waterline and wrote an adapter who use left join instead of union and a sql optimized cursor to optimize performance see https://github.com/Atlantis-Software/sails-hook-orm-offshore/issues/5

@atiertant: That's helpful, thanks. I'll check it out.
The thing that bothers me is that if you're only interested in the foreign keys on the record itself, you don't need to do a union or a join. There should be an optimization (like the default?) that allows you to forgo any population of relationships but still get the values of any foreign keys that are populated as columns on the records you fetch.

@jcowley in a hasMany relation, the foreign keys are in the other side, you can't select parent record and child pk on the same table, so union, join, or a second request...

@atiertant: Yes, of course. As I mentioned, I'm talking only about one-to-one or many-to-one relationships, e.g.:

@jcowley one-to-one and many-to-one are already showing id without need to populate

@atiertant : I'm using @mphasize sails-generate-ember-blueprints for my blueprints (which is what led me to this issue). It's not populating the ids. Mistakenly thought it was a limitation of sails/waterline because of this issue, but it must be a bug in the blueprints I'm using.

@jcowley The sails-ember-blueprints are pretty much out of date, I can't recommend to use them anymore. I added a custom populateIndexes function to the _actionUtils  in conjunction with a hacky "many-to-many through" association described here https://gist.github.com/mphasize/aeaff696132357797e0b in order to make indexing work. Not sure if this is still compatible with the latest Sails.

@mphasize: Yeah, I keep seeing your comments on the blueprints being "out of date", but many of us are still happily using them! The reality is that there is no sails JSON API blueprint yet that's production ready. So I've stayed with the REST adapter in ember and your blueprints — and you can see from the download stats on npm that others have as well. (The benefits of upgrading to a JSON API don't really justify the additional work and risk at this point IMHO.)
Appreciate your creating those blueprints. With the exception of the foreign key issue and a another minor issue that I patched in my fork own of the blueprints, they've been great. And by the way, they do work fine with the latest version of sails (0.12.4).
Thanks for the tip on populateIndexes. I'll take a look!

@jcowley Nice to hear, that the blueprints are still appreciated and honestly I feel kinda sad, that I don't have time to work on them, but I've moved away from Sails and Node as a backend for the time being. One of the main reasons for that were issues like the one we're talking about here, which I feel are important for a strong API story.

In desperate need of having only the ids in an array, we wrote a fairly efficient function to only get the ids via the junction table (a bit different to populateIndexes as we went directly for mongo native aggregate).
However, when trying to set these arrays of ids to the model's property, we still get the embedded records magically. @mphasize also ran into this back then:
// get rid of the record's prototype ( otherwise the .toJSON called in res.send would re-insert embedded records) record = create( {}, record.toJSON() );

_actionUtils
Now we clone all the parent records via parse/stringify (_.cloneDeep now clones functions as well!). This of course is a bit blocking once we get a number of bigger records to clone.
Is there a better way to set/modify association fields (one and many) without cloning them?
This would be an intermediate fix without that performance penalty.

You could query the association table/collection itself.
To get the list of association models that you can use through sails:
run sails console
Object.keys(sails.models)

You get the list of all the models, including association models. You can query them as any other model, and make the join yourself.

The query is not the problem. More setting the results to the model property.
Example:

Models Users and Pets, having a many-to-many relationship designed in their models, like in http://sailsjs.com/documentation/concepts/models-and-orm/associations/many-to-many.
We use our custom populateIds function to get all Pets for User "John". The result is something likevar populatedPetsIds = ["id1","id2","id3"];
When I try to set this result array to the association property of the User record of "John", it is not there when I log it to console or other functions where it is converted to JSON.

So, johnsUserRecord.pets =  populatedPetsIds; does not work.
Instead, we do:
var clonedJohnsUserRecord = JSON.parse( JSON.stringify( johnsUserRecord )); // _.cloneDeep does now clone functions as well, so won't work
clonedJohnsUserRecord.pets =  populatedPetsIds;

That works of course, but doing this for 50 or more heavy records is blocking and more memory and GC-intense.

The issue why pets does not work is because it has associated getters and setters. You have to call toObject() on each object to remove that. And then you can assign anything to them. But, if you use model functions, then they will not work.


Investigate caching issue on Dreamhost deployment

Expected Behavior
When a page or its dependencies is updated, it should be reflected after an amount of time.
Current Behavior
On Dreamhost, prior responses seem to be cached indefinitely.
Possible Solution
Probably a matter of setting response.headers['Cache-Control']; see these potential solutions. It is unclear if this caching is occurring in Flask or in Passenger (probably the latter).

Verified that it's definitely a problem after the Flask stack; I created a new entry which should appear as a sibling to the image rendition test. While the entry itself renders, it doesn't appear in the index or in the previous test.
However, I verified that the entry is available to the templating system from the command line:
bibb[~/publ.beesbuzz.biz] (master) plaidfluff$ pipenv run python3
Python 3.6.4 (default, Feb  6 2018, 23:06:37)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from publ import entry
>>> entry.Entry(336)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/plaidfluff/publ.beesbuzz.biz/publ/entry.py", line 29, in __init__
    self.date = arrow.get(record.entry_date)
AttributeError: 'int' object has no attribute 'entry_date'
>>> from publ import model
>>> e=entry.Entry(model.Entry.get(model.Entry.id==336))
>>> e
<publ.entry.Entry object at 0x2d228ca9c88>
>>> e.title
'Test of image renditions'
>>> e.id
336
>>> e.next
<publ.utils.CallableProxy object at 0x2d23761a438>
>>> e.next()
<publ.entry.Entry object at 0x2d228c46128>
>>> e.next.title
'How long does it take this to show up?'
>>> e.next.id
345
>>>

So obviously if the template were being rendered it would be getting the new sibling.
I created the new entry at 2018-04-07 23:38:23-07:00; it is not yet showing up in the tests index or feed. No cache-control header is being sent:
HTTP/1.1 200 OK
Date: Sun, 08 Apr 2018 06:54:54 GMT
Server: Apache
X-Powered-By: Phusion Passenger 4.0.60
Content-Length: 6242
Status: 200 OK
Content-Type: application/xml

<?xml version="1.0" encoding="utf-8"?>

<feed xmlns="http://www.w3.org/2005/Atom">

    <title>Publ</title>
    <subtitle>A personal publishing system for the modern web</subtitle>
    <link href="http://publ.beesbuzz.biz/tests/feed" rel="self" />
    <link href="http://publ.beesbuzz.biz/tests/" />
    <id>tag:publ.beesbuzz.biz,2018-04-05:tests</id>
    <updated>2018-04-05T12:40:17.674931-07:00</updated>

    <entry>
        <title>Test of image renditions</title>
        <link href="http://publ.beesbuzz.biz/tests/336-Test-of-image-renditions" rel="alternate" type="text/html" />
        <updated>2018-04-05T12:40:17.674931-07:00</updated>
        <id>urn:uuid:a28a68eb-c668-4fb8-9d3a-6cbcc3920772</id>
        <content type="html">
<![CDATA[

Image rendition tests

            Read more...

]]>
        </content>
        <author>
            <name>fluffy</name>
        </author>
    </entry>

so I am guessing that Passenger is deciding that lack of cache-control means "cache it for as long as the process stays active." In the morning I will see if the feed and/or category index have updated.

I didn't do any pinging to see when it started appearing but at 01:49-07:00 the next day it did appear, so the default cache duration is no more than 2 hours, for whatever that's worth.
In any case, the actual cache time should probably be configurable (ideally at the template level - perhaps that's something that could be put into a Jinja tag?), and there should definitely be some sort of cachebusting thing.

Ah, turns out I had FeedOnFeeds' debug log enabled, so I know that the item appeared sometime between 08:00+0000 and 08:10+0000, so that's around 01:10-0700, i.e. a 90-minute TTL on the cache. Good to know.

Looks like the cache duration is just random, possibly indefinite. I added another entry which didn't appear until I manually restarted Passenger.

Just adding any cache-control header seems to make Passenger happy. Well then.

And it turns out that while it was technically a caching bug, it had absolutely nothing to do with the fronting service. Oops.


Fix for issue 377 - tcp or http bind on service with accepts matching Enterprise shield socks host and port cause binding conflict

This is a partial fix for issue #377. Although this fixes the issue, some other tests fails because of this change, so it is still in progress.

Closing this as it comes from develop branch.


Potential buffer overflow in  http_header_generate_string() in file http_header.c (IDFGH-2529)

Environment

Development Kit:      custom
Kit version
Module or chip used:   ESP32-WROOM-32D
IDF version  v3.3.1
Build System:         Make
Compiler version 1.22.0-80-g6c4433a5
Operating System:     Windows
(Windows only) environment type: MSYS2 mingw32
Using an IDE?:     No
Power Supply:         Battery

Problem Description
Potential buffer overflow in  http_header_generate_string() in file http_header.c at line 190.
Expected Behavior
loop should break here and also should print error here that buffer_size is smaller than required
Actual Behavior
loop is not breaking here and buffer gets overflow and write data in undesirable location of memory
Steps to reproduce
init http client
esp_http_client_handle_t client = esp_http_client_init(&config);
config.buffer_size = 100;  (default size is 512)
set multiple header here so that size of headers (strings) gets bigger than  config.buffer_size
esp_http_client_set_header(client, "authorization", accessToken);
code will not issue any warning and http request gets failed
I think issue is fixed in v4.x and  that patch needs to be added here too in next release of v3.3.x

@umer-ilyas The issue has been fixed in our internal repository for v3.3.x as well. When the changes are synced with Github, your issue will be fixed.

@umer-ilyas Fix is available on release/v3.3 branch as well (https://github.com/espressif/esp-idf/commit/b7a9f5e115ca8d58186baad061251365c9dededd#diff-c7c711c57aa3482655dd39165cf29087). Closing this issue.


HTTP 401 on failed login

Gitea version (or commit ref): 1.1.3
Git version: 2.11
Operating system: Debian Stretch
Database (use [x]):

[ ] PostgreSQL
[x] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[x] Yes (provide example URL) https://try.gitea.io/user/login
[ ] No
[ ] Not relevant

Log gist:

Description
When purposely causing a login error on Gitea, I can see (using web inspector and/or logfiles) that the HTTP return code is 200, i.e. "everything is ok", for the page that presents the error to the user.
It would be great if Gitea would return a 401 ("Unauthorized", see here).
Indeed, I think a 401 code in the webserver logs has great security value, and makes it easy to integrate with solutions such as Fail2Ban and others.
If this change were made, there would be absolutely no impact on the user.
Screenshots

I agree with that.
@lunny Can I work on that as a starter for contribution? ;-)

@daviian sure, go ahead but this should be tested on all supported browsers if this works as expected and displays correct content not generic browser error page

This should be easy to start (should be near https://github.com/go-gitea/gitea/blob/master/routers/user/auth.go#L133).
Don't forget to read : CONTRIBUTING.
Feel free to add integration test (this will maybe brake some)
And don't hesitate to ask any question here or on discord
Have fun :smile:

After having a look into the code I found out that the HTTP status code is wrong in multiple cases.
The function used for prompting errors in form validation should generally return the 400 Bad Request status code, but instead it returns 200..
But this is a major change because it is used throughout gitea.
I think the most beautiful solution would be to extend this function with a status code parameter, but that would have a huge impact on other parts of the code.
In a first step I could remove the usage of this function for the authentication errors to return a 401 code.

@lafriks Where can I find a list about the supported browsers?

I'm not sure returning a 401 on failed login is the correct thing to do; see https://stackoverflow.com/questions/8273757/should-a-failed-login-attempt-result-in-a-http-401-response
I've investigated several public-facing sites (Google, Amazon, Instagram, Github, Gitlab), and none of them returned 401 on failed login.

To my surprise, I've seen that they don't. I guess they don't apply Fail2Ban rules on an Apache reverse proxy's log file like I'm currently doing!
Maybe it was an XY-problem and what we should have is a log file with the IP of the offenders (but this won't solve the issue if the reverse proxy is on another machine).
But I know that Jenkins CI app does return a 401 on failed logins, and beside infringing the RFC quoted on StackOverflow, it's damn pratical :)

Fail2Ban can never be as good at catching intrusion attemts as the application itself can. So most sites just returns 200 because that at least slows the intrusion down (the bot has to parse and check that they got in or not)

I do not share this point of view entirely. I did not thought about the benefit of a 200 code you stated, though.
But I see that relevant logging information on the offending intrusion attempts will be added to Gitea, so it's OK for me :)

Issue can be closed. Is fixed by https://github.com/go-gitea/gitea/pull/2334


LanguageDetector does not respect browser's preferred language list

🐛 Bug Report
LanguageDetector does not respect browser's preferred language list. I know this seems like a duplicate, but issue persists (or I am doing something wrong!).
To Reproduce
The browser's ordered language list is de-AT, es-AR.

In the app, i18n.options.supportedLng = ["en", "es"] and i18n.options.fallbackLng = "en".
See below:
i18next
    .use(ChainedBackend)
    .use(LanguageDetector)
    .use(initReactI18next)
    .init({
       fallbackLng: "en",
       supportedLngs: ["es", "en"],
...

Expected behavior
The app should render with i18n.language == "es-AR" and i18n.resolvedLanguage == "es", instead, the result is i18n.language == "de-AT and i18n.resolvedLanguage == "en". However, if "Spanish (Argentina)" is at the top of the browser's preferred language list, then the language is detected correctly (with i18n.language == "es-AR" and i18n.resolvedLanguage == "es").
After some choice log statements in LanguageDetector.detect() I found it puzzling that services.languageUtils.getBestMatchFromCodes(detected) returns "es-AR", but this value does not seemed to be used downstream.
Environment
browser: Version 96.0.4664.55
i18next: Version 21.6.11
os: macOS BigSur

Closing, this was due to user (me) error

hey @NashBaughn  what was your error? Can you explain what you did wrong? thanks


Will the Triton branch of GPTQ with parameters --true-sequential --act-order --groupsize 128 be supported?

These parameters give the best quantization results currently, but I'm not sure if they are compatible with the current 4-bit LoRA.

@luxtiasco do you have a set of parameters that works?... I'm struggling getting a quantized model that I've created working.    My last failed attempts used all three params... --true-sequential, --act-order and --groupsize 128
... I'll try removing the first two to see if I can get something to train.

Now triton backend support v2 model with --true-sequential, --act-order.
Will add cuda backend support in the future.

I see that the cuda branch is not stable, and even behaves differently from triton.
What version of gptq-llama is supported by alpaca_lora_4bit now?

I thought cuda supported --true-sequential and --act-order.. just not with group size.
This is the gptq that is supposed to be used: https://github.com/sterlind/GPTQ-for-LLaMa/tree/lora_4bit

Yes I think add g_idx passing into cuda kernel would solve the problem

Yes I think add g_idx passing into cuda kernel would solve the problem

However I find it behaves differently on triton and cuda branch. The current version of cuda branch didn't work as expected. Only triton branch works well, and it seems that qwopqwop200 will work mainly on this branch.
Either to use a cuda branch with no act-order and groupsize together, or to adapt the triton branch.
See https://github.com/qwopqwop200/GPTQ-for-LLaMa/issues/128

Thanks I'll look into it


Cannot relate HasManyRelation using insertGraph or upsertGraph

upsertGraph uses insertGraph and expects it to do the job, but insertGraph cannot relate a HasManyRelation because it is an update operation, not an insert. insertGraph only inserts stuff.
insertGraph should throw an exception in this case and advice the user to use upsertGraph instead. upsertGraph should just work.
Thank you @Allam76 for bringing this to our attention!
Reproduction provided by @Allam76
const {up, down} = require('./migrations/20150613161239_initial_schema');

async function test(){
    await up(knex);
    var person = await Person.query().insert({id:1,firstName:"Peter",lastName:"Jackson"});
    var pet = await Animal.query().insert({id:1, name:"Mila"});
    var test = await Person.query().upsertGraph({
        id:1,
        pets:[{
            id:1
        }]
    },{relate:true}).debug();
}

test();

Ok. I will test now.

It works now in the example model. Thank you for fixing this so quickly!!


Use fully qualified name for container images

Describe the proposal
Dapr CLI supports two container runtimes, Docker and Podman. For Podman to work, the fully qualified names for the container registries are required. See https://unix.stackexchange.com/questions/701784/podman-no-longer-searches-dockerhub-error-short-name-did-not-resolve-to-an
When dapr init is done using Podman, result: FAILURE
time="2022-10-10T16:22:58Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"

Error: error getting default registries to try: short-name "redis:6" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

When redis is installed with explicit docker.io, result: SUCCESS
podman pull docker.io/redis:6
...
Trying to pull docker.io/library/redis:6...
Getting image source signatures
Copying blob bd159e379b3b done
Copying blob 7fade13fa484 done
Copying blob 729b630784ac done
Copying blob 7e8175a58591 done
Copying blob 10a73736cdfb done
Copying blob 065c77bf222a done
Copying config 136fbb0456 done
Writing manifest to image destination
...

Note
This was not consistent across different Operating Systems. It worked on Windows, Mac, and Ubuntu 20, but not in vanilla Debian Linux.
I tested it on Podman v4.2.1 in Windows 11 Pro 10.0.22000 N/A Build 22000.
Also, the current workaround is to update containers-registries.conf
unqualified-search-registries = ["docker.io"]

Release Note

RELEASE NOTE: ADD: fully qualified names for container images

Right, I tested with macos[brew installs v4.2.1] and Ubuntu 20[podman v3.4.2] it worked there but with podman v3.0 it did not work. IMO we can take a look at this for 1.10 as there is another way to make it work on older Podman versions as mentioned in above description.

/assign

/assign


errorhub { [Error: connect ECONNREFUSED]

Hi--
I've been running into a problem with the API and hoping someone might be able to offer some assistance. It began when I installed a second hub. I've been successfully running it in Docker on a Synology DiskStation but now when I run it, the hubs are both refusing connections. This error occurs on both hubs:

errorhub { [Error: connect ECONNREFUSED 192.168.1.XX:5222]
05:49:07	  code: 'ECONNREFUSED',
05:49:07	  errno: 'ECONNREFUSED',
05:49:07	  syscall: 'connect',
05:49:07	  address: '192.168.1.XX',
05:49:07	  port: 5222 }

A screenshot of the log is attached.
What I've tried:
I've created multiple new containers in an attempt to start fresh, disconnected the new hub and still get the error on the original one which worked correctly prior to this and tried adding the username/password parameters to the config.json file and the environment variables (at separate times), all resulting in the same error.
Please, if anyone can help ... I would really appreciate if someone might shed some light on what's going on and how I may go about correcting this. Thanks so much.

Logitech has pushed a new firmware, blocking all access to the hubs.
See https://github.com/maddox/harmony-api/issues/101#issuecomment-447817869

THANK YOU!!!

Hi everyone. Unfortunately Logitech has decided to close the local API available on the hub.
https://community.logitech.com/s/question/0D55A00008D4bZ4SAJ/harmony-hub-firmware-update-fixes-vulnerabilities
This means you should either downgrade, or find another means to control your media stuff. Home assistant has a patch out to switch to the new found API, if you use Home Assistant, I suggest you switch that from Harmony API.
This effectively ends Harmony API. I'll write a note on the README later today. Thanks for everyone's support!

Logitech has changed their mind!
https://community.logitech.com/s/question/0D55A00008D4bZ4SAJ/harmony-hub-firmware-update-fixes-vulnerabilities
Update to beta .210 to restore XMPP and harmony-api should start working again. :)

https://github.com/JordanMartin/harmonyhub-api
This library uses the websocket api and works well!


Fix the folder hierarchy problem in v12 and remove a deprecation warning

In v12 the folder hierarchy is not created properly, this PR fixes that, it also removes 1 deprecation warning, marks it as verified in v12 and bumps the version.
I tested it on my vault all works fine but I don't use some features such as merging.

Do you think you might include this PR at some point?

Do you think you might include this PR at some point?

With no response in a ocuple of months, might you consider forking the module at this point @matt-duffy ?

Do you think you might include this PR at some point?

With no response in a couple of months, might you consider forking the module at this point @matt-duffy ?

Sure, here is my work, release 3.1.1 is the one I am using in my v12 setup at the moment and it works fine for me:
https://github.com/matt-duffy/lava-flow/
Hopefully the PR will get back into this repo at some point as well.

https://github.com/matt-duffy/lava-flow/

Thank you. I hope it does get merged in.
The original author created something awesome and I am grateful to both of you.

There an issue in the main project addressing a major issue with the module regarding editing the journal entries created when importing the vault notes. https://github.com/Praxxian/lava-flow/issues/64
Could this issue be solved in this fork, since the original project? Thank you for the fork, by the way. This module is great and hopefully it can be completely functional again.
The issue as posted by 4everGM:
Noticed that any changes I make in Foundry are not saved, but only for journals that were imported from Obsidian. It looks like I can edit/save if I set the sheet configuration to Markdown. I don't believe it behaved this way before, THINKING it is a v12 issue. A warning error in the console shows this:

Trying:
Host(s): both Forge and Sqyre
Version V12 Build 331
System: All I've tested.

I'm not too sure what is going on here, that deprecation warning is fixed in this fork, but the save does fail if you use the standard journal entry editor, annoyingly no error/warning is thrown.
I don't believe that the module interacts with the save process at all, so I think it must be something to do with the way the journal page is being generated by the import process, as I can add a new page to the journal and modify that fine.
It might take me a while to work oit what is going on here.

I'm not too sure what is going on here, that deprecation warning is fixed in this fork, but the save does fail if you use the standard journal entry editor, annoyingly no error/warning is thrown.
I don't believe that the module interacts with the save process at all, so I think it must be something to do with the way the journal page is being generated by the import process, as I can add a new page to the journal and modify that fine.
It might take me a while to work oit what is going on here.

In fact I have just tested this in base foundry and I think it is working as expected and not to do with Obsidian/LavaFlow.
If you create a journal page and initially set the content using the markdown editor then it is saved as markdown.  You then have to use the markdown editor, you can't use the standard editor.  This makes sense to me as otherwise Foundry would have to convert the HTML generated in the normal editor back into markdown when you save.
So I don't think this is an issue with this module.
To re-create:

Start a foundry world using the DND 4.x system
Don't add any modules
Create a new Journal
Add a page to the journal, immediately switch to the markdown editor
Add some text and save
Edit and change some text in the markdown editor, save
See that the changes have been applied
Edit again and switch to the dedault text editor, make a change and save
See that the changes have not been applied

I see. I guess it makes sense that Foundry behaves like that yes, but it also makes certain things more difficult. Thanks for looking into this!
I was looking to import my Obsidian notes for characters, locations and so on, and from there export those journal entries into compendiums, using Scene Packer to relink the new compendium entries properly. But that module can't edit these entries, since they're markdown. I'll contact that author and see what can be done. Thanks for the help!

@matt-duffy thank you for your contributions! I am going to include these and some other stuff to resolve v12 compatibility, but I have that in a different branch. I am also going to create a major release to keep it consistent with Foundry's major releases and because the permission change is not backwards compatible.


SPARK-4843 [YARN] Squash ExecutorRunnableUtil and ExecutorRunnable

ExecutorRunnableUtil is a parent of ExecutorRunnable because of the yarn-alpha and yarn-stable split. Now that yarn-alpha is gone, this commit squashes the unnecessary hierarchy. The methods from ExecutorRunnableUtil are added as private.

@JoshRosen yup, moved lazy val env = prepareEnvironment to be last statement so it is the same as before.

Test build #25078 has started for   PR 3696 at commit 486716f.

This patch merges cleanly.

Test build #25078 has finished for   PR 3696 at commit 486716f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25078/
Test PASSed.

LGTM, so I'm going to merge this into master (1.3.0).  Thanks!


dep: Replace deprecaetd github.com/golang/protobuf

This commit is to replace deprecated package github.com/golang/protobuf by google.golang.org/protobuf as mentioned in upstream README.

/test


Block comments regex alternative

The current grammar does not support block comments properly, the below is not recognized as a block comment:
/**
 * @comment
 */

Since Solidity essentially uses C-style comments I copied the comment regex from tree-sitter-c which seems to be working.

Below is the fixed regex working in nvim with tree-sitter playground, you can see the comment block range identified properly:

@ibhagwan Thanks for the contribution! I was on vacation, but I'll definately have time to get this merged this week!

@ibhagwan does this branch work for you? https://github.com/JoranHonig/tree-sitter-solidity/tree/feat/block-comments

I think the main problem was that . doesn't match on new lines.

@ibhagwan does this branch work for you? https://github.com/JoranHonig/tree-sitter-solidity/tree/feat/block-comments

Works great, ty!

Cool! Merged it!
I got some small things to fix but will push a release soon!

Ty!

Regex is complex; looking at it some more I think that the c patterns would work here too, merged 🚀

Works great, ty!


poll_for_frames limiting refresh rate to slowest stream

Required Info

Camera Model
D435i

Firmware Version
<IP_ADDRESS>

Operating System & Version
Ubuntu 20

Kernel Version (Linux Only)
5.10.104-tegra

Platform
Jetson Orin Nano

SDK Version
2.x.x

Language
Python

Segment
Robot

When coding in Pyrealsense2, I have the depth, color, accel, and gyro streams turned on. When using pipeline.poll_for_frames(), the rate that it returns a valid frameset is limited to the slowest stream even though I am using poll instead of wait. This ends up returning a frameset with all the streams instead of individual frames of each stream. Is there any way to get each individual frame instead of framesets? I'm not seeing the difference between wait_for_frames and poll_for_frames when both of them essentially have the same refresh rate.
This is running on a ROS2 Humble node refreshing at 500Hz. I have confirmed it is the different streams causing this because when I only enable accel and gyro, it runs at 200Hz but if I enable color (30Hz), it will go down to 30Hz even though I am looking at the accel/gyro frames.
def timer_callback(self):
    frames = self.pipeline.poll_for_frames()

    accel_frame = frames.first_or_default(rs.stream.accel)
    gyro_frame = frames.first_or_default(rs.stream.gyro)

    if accel_frame.is_motion_frame() and gyro_frame.is_motion_frame():

        self.counter += 1

        timer = Clock().now().to_msg()
        time = timer.sec + (timer.nanosec /<PHONE_NUMBER>)
        time -= self.start_time

        self.get_logger().info(f'{self.counter / time}')

def __initialize_camera():
    # Initializes the camera for all the frames being analyzed
        pipeline = rs.pipeline()
        config = rs.config()
        config.enable_stream(rs.stream.accel, rs.format.motion_xyz32f, 200)
        config.enable_stream(rs.stream.gyro, rs.format.motion_xyz32f, 200)
        config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 90)
        config.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 60)
        pipeline.start(config)

        return pipeline

This isnt everything, but assume that pipeline in initialize_camera is the same as the one in timer_callback. This code counts the amount of valid motion frames and prints the Hz.

Hi @Starblazer75  Having depth, color and IMU simultaneously enabled can cause problems in the RealSense SDK.  A workaround is to set up two separate pipelines, with IMU on one pipeline and depth & color on the other pipeline.  The best example of a Python script for implementing this is at https://github.com/IntelRealSense/librealsense/issues/5628#issuecomment-575943238

Hi @Starblazer75  Do you require further assistance with this case, please?  Thanks!

Hi! Thanks for the help, the problem has been resolved with the comment.

It's excellent to hear that your problem is solved.  Thanks very much for the update!


Using multiple modules directories

I keep modules in two different sub directories: site for mine code and modules for modules downloaded from Forge. As far as Puppet goes it can use both directories to look for modules. With Bolt, trying to pass them in modulepath-way (as --modules ./site:./modules) or with two parameters (--modules ./site --modules ./modules) ends in either my plan not being found, or dependencies not being met.

Hi [@therek], thanks for filing this. We're currently tracking this in https://tickets.puppetlabs.com/browse/BOLT-122 and somewhat related https://tickets.puppetlabs.com/browse/BOLT-121.

BTW. I think by default Bold should try to use puppet config print modulemath if path is not set explicite.

FWIW - the puppet PAL API (used by bolt) already supports being given a modulepath, and if given a configured puppet environment, will use the modulepath defined for that environment.

@therek The fix for multiple module directories was merged in https://github.com/puppetlabs/bolt/commit/c59edaa3eeca8348c0bdd1413c5e958ebde1a172. Note the option has changed to modulepath instead of modules, so I'm going to close this.
There is on-going discussion/work planned to default the modulepath option to whatever puppet is using. That work is being tracked in https://tickets.puppetlabs.com/browse/BOLT-121. Feel free to comment/watch that ticket.


Set up Contact me section on Portafoilo project (html&css only)

In this PR, I fulfilled the following tasks:

[x] Create a Contact me section
[x] Create a Contact Form with an endpoint at https://formspree.io/html
[x] Set Name field accepts up to 30 chars.
[x] Set Email field accepts only emails.
[x] Set Message field accepts up to 500 chars.
[x] Set all fields with placeholder text.
[x] Set all fields "required"
[x] Update README file with new features
[x] Contact me section use FlexBox
[x] Set Hover and Active effects to the buttons

Hello @huseyinbiyik, Could you please check my readme file I think it is now complete

Hello @huseyinbiyik, Could you please check my readme file I think it is now complete

Hey @chelobotix, kindly submit a new request from the dashboard 👍
Invalid Code Review Request does not count into the code reviews limit.

All the observations mentioned in the above comments have been implemented.
Thank you


Use simpler and proper way to get headless kafka name

Signed-off-by: Keyvan Hedayati<EMAIL_ADDRESS>What changes were proposed in this pull request?
In case the release name is cp-kafka the current logic will break and Kafka will not be deployed successfully. This commit fixes that in a simple way.
How was this patch tested?
Helm upgrade on minikube

@confluentinc It looks like @k1-hedayati just signed our Contributor License Agreement. :+1:
Always at your service,
clabot

This is causing our Jenkins CI to break due to the "unknown repository" - This needs to be closed to get around that. If this change is still required, please re-do it in a fork or branch.

This is causing our Jenkins CI to break due to the "unknown repository" - This needs to be closed to get around that. If this change is still required, please re-do it in a fork or branch.


Question: Do we still need deleteTCStringIfOutdated?

Do we still need deleteTCStringIfOutdated? Or was it fixed already?
I'm talking about this:
https://github.com/bocops/UMP-workarounds/blob/main/detect_outdated_consent/android/kotlin/detect_outdated_consent.kt

I haven't followed all conversations about this in detail, but this is something that might have been solved in the meantime. I will check if I can find a reliable source for this, and would update README.md if that is the case, but probably still keep the code around for the time being.

How do you check it out? You do reverse-engineering?


Cannot find module 'xyz'

Hi!
Maybe it's not an issue but a problem only for me, but I am not able make atom-typescript not to report Cannot find module 'xyz' errors in the diagnostics pane.
To reproduce, you can git clone https://github.com/DanielRosenwasser/typescript-vue-todomvc which comes with a tsconfig.json file, I did not change.
I already understand that atom-typescript utilizes sourcegraph/javascript-typescript-langserver, but I have no clue, why it does not resolve the modules - all modules, whether implicit node modules like pathor explicit node_modules like e.g. Vue.
I really would appreciate some help with this.
Br,
@matths

We do not utilize javascript-typescript-langserver. You sure you're not confusing us with ide-typescript?
Typescript will report that it cannot find a module if it can't find definitions for said module, and not all modules bundle (or even have) TypeScript definitions. For popular packages, you can usually find definitions in https://www.npmjs.com/~types (install via npm i @types/..., a good idea is to save those into devDependencies)

@lierdakil you're absolutely right, I confused atom-typescript with ide-typescript! And thanks for explaining @types/* to me, which I suddenly understand much better. I close this non-issue. ;)


Feature/producer new

GitHub Issue
#194
Changes

Add producer job with corresponding helm charts
Migrate nx to latest verison

Packages Added
Brief list of any packages added, and if folks need to rerun yarn install to run locally
Checks
Before making your PR, please check the following:

[ ] Critical lint errors are resolved
[ ] App runs locally
[ ] App builds locally (run the build command for any impacted package and check for any errors before the PR)

looks good, left one question. im wondeirng if alot of these files (minus the nx migration files) are generated by helm?
if so, is there a reference to the generator commands for adding jobs?

No most of the files were from the nx migration