ARROW-10068: [C++] Add bundled external project for aws-sdk-cpp

To do:

Patch/upstream so that git is not required for awssdk's dependencies (c-common et al.)
Handle openssl dependency (add bundled external project and/or add to all necessary docker images)
curl dependency?
Take over/test with macOS wheel building (#7997)
With ARROW_DEPENDENCY_SOURCE=AUTO, cmake fails unless you set AWSSDK_SOURCE=BUNDLED because it can't "find" it. We should fix that.

@kou since you're looking at this now, wanted to let you know I'm debugging locally how on the rstudio/r-base:3.6-centos7 image, it can't find openssl even though I've yum installed it (debugging and failing, that is, and my next step is the bundled openssl build, the nuclear option I'm afraid). I'm currently skipping that problem though and working on the patch to aws-sdk-cpp so that git isn't required.

Thanks for sharing the current status.
I'll also take a look the OpenSSL problem.

I got past that particular openssl problem by installing openssl-static as @pitrou suggested, though I suspect there's more to the issue.

I know that this is not quite related, but since we're looking upstream now... is there any thought to when/if we'll update the AWS SDK version?  I ask because as of 1.8.0, the default credentials resolution, etc, was enhanced to more fully match what you might expect from awscli/boto3.

I know that this is not quite related, but since we're looking upstream now... is there any thought to when/if we'll update the AWS SDK version? I ask because as of 1.8.0, the default credentials resolution, etc, was enhanced to more fully match what you might expect from awscli/boto3.

Actually, ignore me, I see the version change in this PR now.  That's exciting!  Given these changes it might be worth (at some point) taking another look at how credentials specification is handled to see if there's anything that can be streamlined.

@corleyma for R on macOS and Windows we're still using binaries for aws-sdk-cpp that are 1.7, so we can't assume 1.8 in the Arrow code just yet. cf. https://github.com/autobrew/homebrew-core/pull/25

The latest R failures from the CentOS 7 builds are compiler warnings being converted to errors in the awssdk_ep. As it turns out, aws-sdk-cpp requires gcc >= 4.9, and these builds are on 4.8. So somewhere we're going to need to add a if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.9") check.

@github-actions crossbow submit test-r-*

@github-actions crossbow submit test-r-r*

Latest R issue(s):

The R Linux builds do a static Arrow C++ build, which gets pulled into a shared arrow.so library with the R bindings when the R package builds. AFAICT, because the C++ build is static, it insists on the static version of openssl. So IIUC when R makes arrow.so, the openssl symbols will be pulled into the shared library--exactly what folks on https://issues.apache.org/jira/browse/ARROW-10138 were concerned to prevent.
aws-sdk-cpp doesn't build on gcc 4.8, so we need to catch that in cmake. I'd like to have a way to build like ARROW_S3=IF_POSSIBLE, like build with S3 support if we can but fall back to ARROW_S3=OFF rather than fail the build if some prerequisite is not met (similar to ARROW-8155). If there's no interest in supporting that in cmake itself, I'll find some way to hack that into the R Linux build script.

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

Linux:

We need MinIO for testing S3 filesystem

There is no package for MinIO in system package manager
MinIO provides binary only for amd64
We need to download it in our Dockerfiles for amd64
We need to build MinIO on non-amd64 platforms such as arm64

  * We need to build MinIO on non-amd64 platforms such as arm64

It's not true: https://dl.min.io/server/minio/release/linux-arm64/archive/

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic test-r-rstudio-r-base-3.6-centos8

@github-actions crossbow submit test-r-rstudio-r-base-3.6-opensuse15

@github-actions crossbow submit test-r-r*

Current R Linux static build status (using shared libs for curl and openssl/crypto, provided by the system):

test-r-rhub-ubuntu-gcc-release (ubuntu 16.04, gcc 5.4.0): FAIL. C++ build with S3 succeeds but R package link error: undefined symbol: __cpu_model
test-r-rocker-r-base-latest (debian:testing, gcc 9.3.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-bionic (ubuntu 18.04, gcc 7.5.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-centos6 (gcc 8.3.1): FAIL because system openssl is stale: Found unsuitable version "1.0.1e", but required is at least "1.0.2"
test-r-rstudio-r-base-3.6-centos7 (gcc 4.8.5): SUCCESS without S3 (gcc too old)
test-r-rstudio-r-base-3.6-centos8 (gcc 8.3.1): FAIL (missing: OPENSSL_CRYPTO_LIBRARY) (found suitable version "1.1.1c", minimum required is "1.0.2")
test-r-rstudio-r-base-3.6-opensuse15 (gcc 7.4.1): FAIL (missing: OPENSSL_CRYPTO_LIBRARY) (found suitable version "1.1.0i", minimum required is "1.0.2")
test-r-rstudio-r-base-3.6-opensuse42 (gcc 4.8): SUCCESS without S3 (gcc too old)

Summary: 2 success with S3, 2 successfully built without S3 because it detected that gcc was too old, 2 fail because they are finding openssl but not crypto for some reason, 1 fails because it has openssl but the version provided by the package manager is too old, one fails with undefined symbol: __cpu_model (a gcc 5 bug?).
Among the reasons this is frustrating:

I've installed openssl but it says it can't find it (sometimes); I've installed it from the system package manager but it's still too old; these are both easily solvable by bundling openssl like we already do in the wheels, but I've gotten strong pushback on that.
I've tried to add logic in the R build script to detect whether we can build aws-sdk-cpp (https://github.com/apache/arrow/pull/8304/files#diff-3875fa5e75833c426b36487b25892bd8R370-R398) but that's apparently not enough to catch these deviant (yet common) cases. It would be nice to move this logic to cmake, an ARROW_S3=PLEASE_IF_YOU_CAN option: is there objection to that?
Given how fragile this is, I'm fearing the onslaught of bug reports after the release.

@github-actions crossbow submit test-r-rstudio-r-base-3.6-centos8 test-r-rstudio-r-base-3.6-opensuse15

@github-actions crossbow submit test-r-rstudio-r-base-3.6-centos8 test-r-rstudio-r-base-3.6-centos6

@github-actions crossbow submit test-r-rstudio-r-base-3.6-centos8 test-r-rstudio-r-base-3.6-centos6

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit test-r-rstudio-r-base-3.6-opensuse15

Latest, after fixing openssl shared library finding and adding extra version detection to the R build script:

test-r-rhub-ubuntu-gcc-release (ubuntu 16.04, gcc 5.4.0): FAIL. C++ build with S3 succeeds but R package link error: undefined symbol: __cpu_model
test-r-rocker-r-base-latest (debian:testing, gcc 9.3.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-bionic (ubuntu 18.04, gcc 7.5.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-centos6 (gcc 8.3.1): SUCCESS without S3 (system openssl is 1.0.1e, but we require 1.0.2)
test-r-rstudio-r-base-3.6-centos7 (gcc 4.8.5): SUCCESS without S3 (gcc too old)
test-r-rstudio-r-base-3.6-centos8 (gcc 8.3.1): FAIL. error building arrow_bundled_dependencies.a because awssdk_ep-install/lib/libaws-c-event-stream.a: No such file or directory
test-r-rstudio-r-base-3.6-opensuse15 (gcc 7.4.1): FAIL. error building arrow_bundled_dependencies.a because awssdk_ep-install/lib/libaws-c-event-stream.a: No such file or directory
test-r-rstudio-r-base-3.6-opensuse42 (gcc 4.8): SUCCESS without S3 (gcc too old)

I'm trying to reproduce the arrow_bundled_dependencies issue locally. I have no idea how to handle __cpu_model.

It appears that the aws-sdk dependencies are installing to a lib64 directory on these platforms instead of lib:
[root@add4b8c61e56 r]# ls /build/awssdk_ep-install/lib
cmake				   libaws-cpp-sdk-core.a		 libaws-cpp-sdk-sts.a
libaws-cpp-sdk-cognito-identity.a  libaws-cpp-sdk-identity-management.a  libaws-cpp-sdk-transfer.a
libaws-cpp-sdk-config.a		   libaws-cpp-sdk-s3.a
[root@add4b8c61e56 r]# ls /build/awssdk_ep-install/
include  lib  lib64
[root@add4b8c61e56 r]# ls /build/awssdk_ep-install/lib64
aws-c-common	    aws-checksums  libaws-c-common.a	    libaws-checksums.a
aws-c-event-stream  cmake	   libaws-c-event-stream.a

We are already providing CMAKE_INSTALL_DIR and the docs (and source) say this should get passed on to the dependencies. Debugging further.

Lol, @pitrou reported this over a year ago: https://github.com/aws/aws-sdk-cpp/issues/1215

@github-actions crossbow submit test-r-r*

Ok, with that latest patch,

test-r-rhub-ubuntu-gcc-release (ubuntu 16.04, gcc 5.4.0): FAIL. C++ build with S3 succeeds but R package link error: undefined symbol: __cpu_model
test-r-rocker-r-base-latest (debian:testing, gcc 9.3.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-bionic (ubuntu 18.04, gcc 7.5.0): SUCCESS with S3
test-r-rstudio-r-base-3.6-centos6 (gcc 8.3.1): SUCCESS without S3 (system openssl is 1.0.1e, but we require 1.0.2)
test-r-rstudio-r-base-3.6-centos7 (gcc 4.8.5): SUCCESS without S3 (gcc too old)
test-r-rstudio-r-base-3.6-centos8 (gcc 8.3.1): SUCCESS with S3
test-r-rstudio-r-base-3.6-opensuse15 (gcc 7.4.1): SUCCESS with S3
test-r-rstudio-r-base-3.6-opensuse42 (gcc 4.8): SUCCESS without S3 (gcc too old)

That's 4 successful builds with S3, 3 successful builds but without S3 because system requirements aren't met, and 1 strange link error.
Someone who knows better should revise my cmake hackery in https://github.com/apache/arrow/pull/8304/commits/d3f13d4ca2a63af5d758e900a981c27251a75a90

Great!
I'll remove all patches by building aws-c-common, aws-checksums and aws-c-event-stream ourselves.
If we use auto build by aws-swk-cpp, we break our offline build by cpp/thirdparty/download_dependencies.sh.
It almost works on local, I'll push it in a few days.

undefined symbol: __cpu_model is caused by aws-c-common.
And it has been fixed by https://github.com/awslabs/aws-c-common/pull/645 .
It means that we can fix it by using the latest aws-c-common. It's easy with my change because we build aws-c-common by ourselves.

@github-actions crossbow submit test-r-r*

@github-actions crossbow submit -g nightly

@github-actions crossbow submit wheel-osx-*

@github-actions crossbow submit conda-osx-*

@github-actions crossbow submit conda-linux-gcc-*

@github-actions crossbow submit wheel-osx-*

@xhochy Do you know how to fix the following in conda-linux-gcc-py36-cpu?
https://dev.azure.com/ursa-labs/crossbow/_build/results?buildId=18312&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181&l=2205
WARNING (arrow-cpp,lib/libgandiva.so.200.0.0): Needed DSO lib/liborc.so found in ['orc']
WARNING (arrow-cpp,lib/libgandiva.so.200.0.0): .. but ['orc'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

@github-actions crossbow submit conda-linux-gcc-*

@xhochy Do you know how to fix the following in conda-linux-gcc-py36-cpu?
https://dev.azure.com/ursa-labs/crossbow/_build/results?buildId=18312&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181&l=2205
WARNING (arrow-cpp,lib/libgandiva.so.200.0.0): Needed DSO lib/liborc.so found in ['orc']
WARNING (arrow-cpp,lib/libgandiva.so.200.0.0): .. but ['orc'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

This has been fixed on master. Sorry.

We can complete #7997 after we merge this.

We can complete #7997 after we merge this.

I have a working PR https://github.com/apache/arrow/pull/8315 based on this branch. I'm going to rebase that to check the builds are still passing.

@github-actions crossbow submit *-fedora-cpp -ubuntu-20.04-cpp *-debian-10-cpp

OK. I closed #7997.

@github-actions crossbow submit *-fedora-cpp

@github-actions crossbow submit test-fedora-32-cpp test-debian-10-cpp test-ubuntu-20.04-cpp

@github-actions crossbow submit -g r

I am revising the R installation guide to mention S3 support and the required system dependencies. Will push that today.
@kszucs @kou are there other developer guides that need revision here? I still think it's odd that we now have a setup with ARROW_DEPENDENCY_SOURCE=BUNDLED but that still requires system dependencies, so I'm guessing there are docs that need to be clarified.

@github-actions crossbow submit -g r

@github-actions crossbow submit test-r-rhub-ubuntu-gcc-release test-r-rocker-r-base-latest

Thanks!
We have https://arrow.apache.org/docs/developers/cpp/building.html#build-dependency-management for C++.
(I think that updating the documentation can be worked as a follow-up task.)

(I think that updating the documentation can be worked as a follow-up task.)

Ok. I'll make a followup JIRA and merge this. Thanks again!

https://issues.apache.org/jira/browse/ARROW-10180 for the docs


Feature: square-media-grid

Feature request, see: http://stackoverflow.com/questions/8809652/twitter-bootstrap-media-grid-square-box-with-images

Sorry, we won't be supporting this in the near term. It should be easy enough for folks to style those classes on an as-needed basis to enforce a fixed height for whatever implementation they require.


minor typo correction in readme.md

"We respect you .editorconfig ... " to "We respect your .editorconfig ..."

:man_facepalming: Thank you!


Tests for Models

Issue Reference
This PR addresses the Issue : Fixes #
Summarize
Please, describe what the PR does, the changes you have made.
Added few more Tests for models.

Also, mention the key points if any to look into while code reviews

@Kavit900 This failed on local
File "/home/biswas/Projects/floodrelief/mainapp/tests/test_models.py", line 30
expected_requestee_name = f'{request.requestee}'
^
SyntaxError: invalid syntax

@biswaz it runs successfully on my local system, taking a look what could have caused failure on ur system.


Added bulk select to infrastructure details page

Added bulk select to AWS and Azure (infrastructure) details pages

Codecov Report

Merging #1644 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1644   +/-   ##
=======================================
  Coverage   78.88%   78.88%
=======================================
  Files         240      240
  Lines        4006     4006
  Branches      720      720
=======================================
  Hits         3160     3160
  Misses        747      747
  Partials       99       99

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f00ce9...de05801. Read the comment docs.


SQS Message Attributes

SQS message attributes are not populated from SNS message attributes.
When publishing a message to an SNS topic with message attributes, the message attributes are not passed into the message that is sent to the subscribed SQS queue.

I can confirm this issue. The message attributes are wrapped inside Body field which is not compliant with the documented response format for boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message
{
    "MessageId": "51c05a3e-e01f-436c-9fa5-2ae7e2e8cf8c",
    "ReceiptHandle": "51c05a3e-e01f-436c-9fa5-2ae7e2e8cf8c#2eec18b8-ff9e-4da7-813e-2a872ccd30ef",
    "MD5OfBody": "549cfca570cf3ba67bedffe8645e528d",
    "Body": {
        "Type": "Notification",
        "Token": "",
        "MessageId": "25a5abd8-8d0e-4167-b6f0-0f84f49f737c",
        "TopicArn": "arn:aws:sns:ap-southeast-1:100010001000:test-topic",
        "Subject": "",
        "Message": "data",
        "Timestamp": "2021-05-28T07:06:31Z",
        "SignatureVersion": "1",
        "Signature": "Q+URKFSue8KEbyWyatY6rwrFCkqucgJqoju/FBoqWzKMxSDBejshXIA/wPDRdLHqLuKgkzuMON584AKHqsK8s16cFsS4h2Uznv9ODQ+xnNMGyhAjPzBBpQBxe7mLzFw+J41GXyRKLrq9v2XOr7BtduGFJ62NEZbMkgpoZpAl7xKxevoh4VL0rrOIjQBl6R1V7nwW+cVngbJZDqV4tYZ8lj1pJ5w0InwfEKIiZRL7brsqgIjfUG/+o09E4DczTQ7IYkoQrrh7MXCCijCDR0FqE3lBfUMV8agOpcclx0lVLfuL2NEqXydPs0ttvbhwKcolBcrsB8J33rcqKNk63ddQ7w==",
        "SigningCertURL": "http://goaws:4100/SimpleNotificationService/25a5abd8-8d0e-4167-b6f0-0f84f49f737c.pem",
        "UnsubscribeURL": "http://goaws:4100/?Action=Unsubscribe\\u0026SubscriptionArn=arn:aws:sns:ap-southeast-1:100010001000:test-topic:2b1d0649-2c8e-46cc-8a25-dd3232bd1635",
        "SubscribeURL": "",
        "MessageAttributes": {
            "data": {
                "Type": "BinaryValue",
                "Value": "AMh7InN0YXR1cyI6InN1Y2Nlc3MiLCJkYXRhIjp7InJlc3VsdFR5cGUiOiJ2ZWN0b3IiLCJyZXN1bHQiOltdfX0="
            },
            "request_id": {
                "Type": "StringValue",
                "Value": "b27b35f7-4f9c-4998-bc36-6076e0f18f56"
            }
        }
    },
    "Attributes": {
        "ApproximateFirstReceiveTimestamp": "1622185591",
        "SenderId": "100010001000",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1622185591"
    }
}

Turns out the docker hub image is outdated. It's now working as expected with an image built from source.


Phase Shift Seed

Adding the ability to use a seed for the random phase shifts need for null hypothesis testing. The seed is added to an integer derived using the first TOA of the pulsar, so that each pulsar has a separate phase shift.

So the tests won't pass, but they pass locally on my machine. I've pasted the two kinds of error messages below. The only reference I can find to this is for an issue with cython, fixed in later version.
@vallis, @stevertaylor and @paulthebaker Is there a specific reason that all of the dependencies have strict ==, or just for safety's sake? I'm not sure which one could be the issue here, maybe it is cython. I might start adding>= to requirements one at a time, if that seems reasonable.
======================================================================
ERROR: test_ecorr_sparse_ipta (tests.test_white_signals.TestWhiteSignalsPint)
Test of sparse ecorr signal and solve methods.
----------------------------------------------------------------------
ValueError: Cannot set the NumPy array 'base' dependency more than once
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/travis/build/nanograv/enterprise/tests/test_white_signals.py", line 413, in test_ecorr_sparse_ipta
    self._ecorr_test_ipta(method="sparse")
  File "/home/travis/build/nanograv/enterprise/tests/test_white_signals.py", line 378, in _ecorr_test_ipta
    assert np.allclose(N.solve(self.ipsr.residuals, logdet=True)[1], wd.logdet(), rtol=1e-8), msg
  File "/home/travis/build/nanograv/enterprise/enterprise/signals/signal_base.py", line 951, in solve
    mult = cf(other)
SystemError: <sksparse.cholmod.Factor object at 0x7f29bcf95780> returned a result with an error set
======================================================================
FAIL: test_like_corr (tests.test_likelihood.TestLikelihood)
Test likelihood with spatial correlations.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/nanograv/enterprise/tests/test_likelihood.py", line 266, in test_like_corr
    self.compute_like(npsrs=2, inc_corr=True)
  File "/home/travis/build/nanograv/enterprise/tests/test_likelihood.py", line 257, in compute_like
    assert np.allclose(eloglike, loglike), msg
AssertionError: Incorrect like for npsr=2, phiinv=partition

I changed the == in both Cython and scikit-sparse to >= and the checks passed. I'm now reverting back to using numpy.random instead of the Python module random. If checks pass I'll merge.


Integrate utf8 validation module and expose it as a primitive

This will be required for websockets validation (currently missing from weblit-websocket, if that is to be integrated). Also generally useful, I guess, and since proven code exists that is written in C there's no need to reinvent the wheel.
Implementation: https://github.com/websockets/utf-8-validate/blob/master/src/validation.c and https://www.cl.cam.ac.uk/~mgk25/ucs/utf8_check.c
Tests: https://github.com/websockets/utf-8-validate/blob/master/test/test.js
If this turns out to have issues, I'm sure other implementations could be borrowed (V8?).

Isn't there already one? Saw something in the sample apps, though it may be from one of the dependencies. Just putting this reminder here to investigate later.


Search feature

The main option in the menu is a search feature described briefly in #17.
It should:

take a phrase from an user
search database/datacache and filter out all matching records
print out results

For this task you can simulate database/datacache in prepared file. Other tasks will cover preparing real data.

Yep, it looks like done.


Numeric columns do not have a numeric type

The final data frame is not type consistent. Numeric columns, confidence, support and lift, have the char type. It is problem if I want to sort data after pruning (I have to transform columns to numeric explicitly).

Columns confidence, support and lift are provided as numeric.


does not split files

does not split files in all formats

Could you be more specific? What format, when splitting does not work? Please, post svar command line invocation.

i try in WAV and other format (mp3, ogg)
command line:
svar --verbose --sig-level=0 --split-time=30 --sig-meter
svar -s 10 -o MP3
svar --split-time=30
svar --sig-level=0 --split-time=30
svar -v -l 0 -s 30
but it does not split files
maybe I'm doing something wrong?

svar --sig-level=0 --split-time=30

This will not split, because volume will be never below 0. New file is created if incoming signal is below signal level for more than the split-time, e.g.:
$ svar -s 10 -o mp3 -v
SVAR - Simple Voice Activated Recorder
Selected PCM device: default
Hardware parameters: 44100 Hz, S16LE, 1 channel
Output file format: mp3
Output bit rate [min, max]: 32, 128 kbit/s
info: Creating new output file: rec-06-21:34:28.mp3
info: Creating new output file: rec-06-21:35:00.mp3
info: Creating new output file: rec-06-21:35:22.mp3

Use --sig-level=1. If you need better signal sensitivity, I think that it will be required to allow fractional values for signal level (code change), or boost microphone gain.

I understood you. I want to record continuously, but file was split by time.
Is it possible?

No, it's not possible. The whole idea of VOR, is to trigger recording by signal level. Then, in order to save disk space and still know when the event has occurred, svar has output splitting (not interesting part of data, which is below signal level, will not be stored), so file creation time will tell the beginning of event.

Thanks!


Make omniauth_openid_connect gem compatible with omniauth v2.0

As the omniauth is now at version > 2.0, I have updated omniauth_openid_connect so that it is compatible with the latest version. I also updated the other dependencies so that they use the latest versions as well.
There were two major changes I found that needed to be fixed:

In omniauth the following methods request_path was updated to include script_name, The method for script_name is:

def script_name
  @env['SCRIPT_NAME'] || ''
end

and it would raise an error whenever request_path was called as @env would be nil. Therefore, I added a method into lib/omniauth/strategies/openid_connect.rb which will return '' if @env is nil and will use the method in lib/omniauth/strategy.rb as normal if @env is not nil. This is similar to what was done for session already.

There was a change in the omniauth gem for the current_path in lib/omniauth/strategy.rb which changed from using path_info to the path method as path_info has been depreciated. This did not affect the code in the library but it did impact on the tests. This is because path_info was being stubbed. Therefore, I changed the places in the tests where path_info was being stubbed to stub path instead.

After these changes were made, I was able to run all the test successfully.

I hope this one get merged.
I have a PR open since April (#88) could not make it pass the specs on github even though they passed local.
There was another on (#84) closed because of lack of activity.
Yours at least is green, I am crossing my fingers to get this one in.

I hope this one get merged.
I have a PR open since April (#88) could not make it pass the specs on github even though they passed local.
There was another on (#84) closed because of lack of activity.
~Yours at least is green~(edit: They disabled rubocop check), I am crossing my fingers to get this one in.

It's been a long time since there were any updates on this gem. I think at this point you're better off just using your own fork of it, that's what we are doing. I can't see this PR making any more progress than yours was able to I'm afraid.

What is blocking from merging this one ?

The project is not being supported anymore it seems.
This fix can be used however so you will need to target the branch.

What is blocking from merging this one ?

The project is not being supported anymore it seems. This fix can be used however so you will need to target the branch.

Time to have a new project maintainer so ?!

Time to have a new project maintainer so ?!

It looks like someone has already done this https://github.com/m0n9oose/omniauth_openid_connect/issues/99:
https://github.com/netsphere-labs/omniauth-openid-connect/

Sorry for this huge delay. You're right, i could use some help with maintaining this project.
Back to subject: just merged very similar PR. Could you please rebase your working branch and resolve conflicts, so i could merge some of these changes.
@tim-s-ccs if you interested in helping me with with maintaining this repo - please let me know

Sorry for this huge delay. You're right, i could use some help with maintaining this project.
Back to subject: just merged very similar PR. Could you please rebase your working branch and resolve conflicts, so i could merge some of these changes.
@tim-s-ccs & @formigarafa if you interested in helping me with with maintaining this repo - please let me know

If I had better knowledge of how it all worked I would but unfortunately I don't. My knowledge doesn't go much beyond 'what changes do I need to make this compatible with omniauth v2.0' 😞 . If there is anything else I can do please ask.

I'm going to close this as a similar PR was merged which does everything this PR does already, just in a slightly different way regarding one scenario.

Sorry for this huge delay. You're right, i could use some help with maintaining this project.
Back to subject: just merged very similar PR. Could you please rebase your working branch and resolve conflicts, so i could merge some of these changes.
@tim-s-ccs & @formigarafa if you interested in helping me with with maintaining this repo - please let me know

I wouldn't mind giving you a hand with that. I can try.

will testing this branch and feedback later.


Presto support in Luigi

Description
I've added presto support in luigig
Motivation and Context
Presto is a distributed sql engine that supports different DBs such as Hive, Postrgres, Cassandra etc
https://prestodb.io/
Have you tested this? If so, how?
I have included unit tests.
I ran my jobs with this code and it works for me.

@honnix
@dlstadther
@Tarrasch
PTAL

@drowoseque Sorry for replying late. I cannot give more constructive comments than "please add docstring to publish method/properties". Is there anyone from your side who can take a look?

maybe @b2arn can take a look

@honnix
can't figure out why codecov thinks I have unexpected cov changes

@honnix @dlstadther @Tarrasch
all checks turned into green
can we go on with merge?

I honestly didn't look at the code. But I'm happy you've fixed the initial feedback and @b2arn looked at this.


Trending shapes section @ landing page

Is your feature request related to a problem? Please describe.
User doesn't the shapes that are currently trending
Describe the solution you'd like

Keeping the user informed about the shapes that are currently most liked and re-edited quite often.
each of the trending shape should have the owner info and when it was created will motivate the owners to come up with more trending shapes

@all-contributors please add @nirmalkc  for design

@all-contributors please add @atapas  for blog ideas mentoring platform projectManagement


[medSegmentationSelectorToolBox] setWorkspace on toolbox

Was done in filtering and registration but not in segmentation.

It's good for me


added Ateliersulpherpool-light

New scheme, light version, part of the Atelier-schemes series. Each has it's own base (background-)color, here purple-blue.

Thanks!

Thanks for the merge, I understand if this could be annoying, but apparently I can spell nor type, hence the renaming of the files in a new pull request ...


React-boilerplate + react-toolbox: Cannot resolve 'file' or 'directory' ./style

I'm using react-boilerplate as base to my project, since it have support do Redux.
When I access my app I get the following error in the terminal:
ERROR in ./~/react-toolbox/lib/button/Button.js
Module not found: Error: Can't resolve './style' in 'E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\react-toolbox\lib\button'
 @ ./~/react-toolbox/lib/button/Button.js 28:13-31

ERROR in ./~/react-toolbox/lib/button/IconButton.js
Module not found: Error: Can't resolve './style' in 'E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\react-toolbox\lib\button'
 @ ./~/react-toolbox/lib/button/IconButton.js 28:13-31

ERROR in ./~/react-toolbox/lib/ripple/Ripple.js
Module not found: Error: Can't resolve './style' in 'E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\react-toolbox\lib\ripple'
 @ ./~/react-toolbox/lib/ripple/Ripple.js 23:13-31
Child html-webpack-plugin for "index.html":
         Asset     Size  Chunks       Chunk Names
    index.html  20.5 kB       0
    chunk    {0} index.html 1.03 kB [rendered]
        [0] ./~/html-webpack-plugin/lib/loader.js!./app/index.html 1.03 kB {0} [built]
Error: no such file or directory
    at MemoryFileSystem.readFileSync (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\webpack-dev-middleware\node_modules\memory-fs\lib\MemoryFileSystem.js:107:10)
    at app.get (E:\ReactLaravel\Boilerplate\react-boilerplate-master\server\middlewares\frontendMiddleware.js:25:21)
    at Layer.handle [as handle_request] (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\layer.js:95:5)
    at next (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\route.js:131:13)
    at Route.dispatch (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\route.js:112:3)
    at Layer.handle [as handle_request] (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\layer.js:95:5)
    at E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:277:22
    at param (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:349:14)
    at param (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:365:14)
    at Function.process_params (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:410:3)
    at next (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:271:10)
    at middleware (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\webpack-hot-middleware\middleware.js:39:48)
    at Layer.handle [as handle_request] (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\layer.js:95:5)
    at trim_prefix (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:312:13)
    at E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:280:7
    at Function.process_params (E:\ReactLaravel\Boilerplate\react-boilerplate-master\node_modules\express\lib\router\index.js:330:12)

My App/containers/MyContainer/index.js which is the main entry for this part of the application:
index.zip
My package.json:
package.json.zip
The file I believe is the equivalent to webpack.config.js in this boilerplate template:
webpack.test.babel.zip

I have exactly same problem with same boilerplate

I made this and not solved:
https://github.com/mxstbr/react-boilerplate/blob/master/docs/css/sass.md

Same issue here

Try 1.0.0!

lol, just add .scss extension to webpack.base.babel.js

What is the status of this issue? I have the same problem and @DenisNeustroev solution, unfortunately didint work :(

@jhardin293 try this:
webpack.base.babel.js

      // Transform react-toolbox styles with SASS
      test: /\.scss$/,
      include: /node_modules\/react-toolbox/,
      loader: options.sassLoaders,

webpack.dev.babel.js

  sassLoaders: 'style-loader!css-loader?localIdentName=[local]__[path][name]__[hash:base64:5]&modules&importLoaders=1&sourceMap!postcss-loader!sass-loader',

Although this is a common issue, it is not a project issue. The setup may not look issue but you just need a module bundler able to load SASS files. I'm closing this issue though, any of the solutions posted in this thread should work and, in other case, refer to the [react-toolbox-example])(github.com/react-toolbox/react-toolbox-example)


Fixed for V15 SDK

Hi @erwinvanhunen, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;


Fishchimp/features/fetch unavailability testing

Describe your changes

Pulled changes from John's branch
Made unit tests for the unavailability fetch (get) method.

Test 1: Success Retrieval of Unavailability Records
Objective: Verify that the system can successfully retrieve unavailability records for a volunteer with existing records.
Criteria:

The response status code must be 200, indicating a successful request.
The response must contain an 'unavailability' key, showing that unavailability data is being returned.
The 'unavailability' key must be associated with a list, indicating one or more unavailability records are present.

Test 2: No Unavailability Records Found
Objective: Confirm the system's response when a valid volunteer has no unavailability records.
Criteria:

The response status code should be 400, indicating a client-side error due to the absence of records.
To inform the client about the lack of records, the response body must explicitly indicate failure, with a structure matching {'userID': user_id, 'success': False}.

Test  3: Invalid Volunteer User ID
Objective: Ensure the system correctly handles requests for non-existent users.
Criteria:
The response status code should be 404, indicating that the requested resource (user or unavailability record) was not found.
The response should include a message indicating that the user was not found, making the error clear to the client.
Test 4: Unauthorized Access Attempt
Objective: Test the system's response to an unauthorized request, simulating a scenario where the requestor is not authenticated.
Criteria:

The response status code must be 401, signaling unauthorized access due to lack of or invalid authentication.
The response must contain an error message, providing feedback that authentication is required.

Test 5: Forbidden Access Attempt
Objective: Validate the system's handling of requests from authenticated users who do not have permission to access the requested data.
Criteria:

The response status code should be 403, indicating that the requestor is authenticated but lacks the necessary permissions to access the requested resource.
The response must include an error message, indicating the requestor's insufficient permissions.

Issue ticket number and link
Issue FE-146
NOTE: I'm a bit unsure how to do the tests for test 4 and test 5 as the original method has the @requires_auth decorative

Why are pushes from main showing up in your diff? Please merge main normally and I will review your work

Hi Ben, apologies, I have a habit of pulling from the main branch to merge incoming changes. I'll make sure to only do merges without pulling from now on. I can't merge from main anymore as the branch is already up to date

I understand, but I can't merge or review your branch in its current state as GitHub is reporting conflicts and the pipelines aren't running. Besides, it is difficult for me to assess your changes when everyone else's are mixed in as I'm not sure what exactly you did. Please roll back to before you pulled main and do a merge commit from main.

Should I delete this branch and create a new one? I think that would make it a lot easier

Should I delete this PR and redo the process on a new branch? That would make it easier.

I would prefer if you didn't, but I don't mind

@fishchimp Please resolve the issues on this branch. This PR has been open for a week and I will not merge your other work until this one is complete

Hi @BenMMcLean , I'm currently trying to solve the merge conflicts and am still running into some git merging and rebasing issues. In the meantime, the features I've implemented here and on my other branch are the same. I would appreciate it if you could take a look at that one. I'll keep trying to solve this merge conflict issue and meet up with John.

Hi @BenMMcLean,
This Git issue is getting too complicated and taking too much time. Would you agree that we close this PR? My other PR has the same changes without all other people's commits junked in.

yep that's fine


fix: remove null items while authentication is not specified

Exported collection was like this:
...
"header": [
	{
		"key": "Accept",
		"value": "application\/json"
	},
	{
		"key": "Content-Type",
		"value": "application\/json"
	},
	null
],
...

Postman error:
Error while importing Postman Collection v2.1: Cannot read properties of null (reading 'disabled')

Thank you for your contribution!


New selector component

:tada: This PR is included in version 2.9.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Clarify allowEmptyValue further

[EDIT: See #3803 for an alternative based on discussion below.]

Fixes #3792

Make it clear that the default value does not do anything, and the non-default value (true) changes the normal expected behavior of the "schema" or "content" fields.  This also explains why the keyword is safe to deprecate, as "schema" and "content" can be used to allow empty values in a more typical way.

@ralfhandl Everyone is encouraged to review/approve this and other draft PRs.  But with this much confusion on what is supposed to happen, I want to make sure that we do not merge it yet.  See policy issue #3802 which I filed because at least 3 different people have merged PRs that I thought needed more attention, but we don't have any well-defined way to express that (discussions in TDC calls and comments in the PRs did not work).


⚠ ClamAV has degraded performance

In a59c548, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 417 ms

Resolved: ClamAV performance has improved in 90268b6.


README: Format code example, whitespace [ci skip]

This PR is a tiny whiny change to the README code example: add a space between hash keys.

Thanks!


Team Meeting - 2024-09-25 16:30-17:30

Date and Time
SEPTEMBER 25, 2024
16:30pm-17:30pm
Attendance

[x] Rafey
[x] Russell
[x] Avanish
[x] Abdul

Agenda

Discuss requirements
Discuss our vision for the user experience

Action items:

[ ] consider requirements templates
[ ] think about possible specific requirements


⚠ 366.ru has degraded performance

In e4a1792, 366.ru (https://366.ru) experienced degraded performance:

HTTP code: 200
Response time: 1512 ms

Resolved: 366.ru performance has improved in b4acda4.


SW-24821 / Disable submit button to prevent multiple form submits

1. Why is this change necessary?
This change is necessary to prevent multiple submit from the registration form. This change fixes the issue https://issues.shopware.com/issues/SW-24821.
2. What does this change do, exactly?
This change disables the submit button after the first form submit was triggered.
3. Describe each step to reproduce the issue or behaviour.
Click the registration submit button more than one time, the user is created on the first request but the request will be canceled after the next click. This results in an error, because the user email address now already exists in the database, created by the first request.
4. Please link to the relevant issues (if any).
https://issues.shopware.com/issues/SW-24821
5. Which documentation changes (if any) need to be made because of this PR?
6. Checklist

[ ] I have written tests and verified that they fail without my change
[ ] I have squashed any insignificant commits
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

Is that form ajaxified? You then might want to reenable if the request fails because of network problems etc.

Maybe use data-preloader-button instead. Like in confirm page

Is that form ajaxified? You then might want to reenable if the request fails because of network problems etc.

No it isn't.

Maybe use data-preloader-button instead. Like in confirm page

That solution also works. Both disabling the submit button.

I would prefer then the data-preloader-button :blush:

I would prefer then the data-preloader-button

Done

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/SW-24825
Please use this issue to track the state of your pull request.

Thanks for your Contribution! :blue_heart:


Auth0 - JWT Token Validation - Request for authenticated route was unauthenticated

I am having issues validating an Auth0 access token when sent to Ocelot. I want to wrap Ocelot behind Auth0 so only authenticated users can call Ocelot.
Workflow is:
--issue--
SPA -- ( Authorization: Bearer {access_token from Auth0} ) --> Ocelot
--working--
Ocelot -- ( Authorization: Bearer {basic auth} --> External service
Expected Behavior / New Feature
Sending an Auth0 JWT access token in the request header to Ocelot should authenticate in same way as a regular .NET Core Web API.
Actual Behavior / Motivation for New Feature
Ocelot does not validate token, and logs the following:
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
requestId: 0HLH2U1L1TJ2L:00000001, previousRequestId: no previous request id, message: /api/v1/users/getByUserId is an authenticated route. AuthenticationMiddleware checking if client is authenticated
warn: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
requestId: 0HLH2U1L1TJ2L:00000001, previousRequestId: no previous request id, message: Client has NOT been authenticated for /api/v1/users/getByUserId and pipeline error set. Request for authenticated route /api/v1/users/getByUserId by  was unauthenticated
Steps to Reproduce the Problem
Startup.cs
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
using Microsoft.IdentityModel.Tokens;
using Ocelot.Configuration;
using Ocelot.DependencyInjection;
using Ocelot.Logging;
using Ocelot.Middleware;
using Ocelot.Tracing.Butterfly;
using System;
using System.IdentityModel.Tokens.Jwt;
using System.Threading;
using System.Threading.Tasks;

namespace APIGateway
{
    public class Startup
    {
        private readonly IConfiguration _configuration;

        public Startup(IHostingEnvironment env)
        {
            var builder = new ConfigurationBuilder();
            builder
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", true, true)
                .AddJsonFile($"ocelot.{env.EnvironmentName}.json", optional: false, reloadOnChange: true)
                .AddEnvironmentVariables();

            _configuration = builder.Build();
        }

        // This method gets called by the runtime. Use this method to add services to the container.
        // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddAuthentication(options =>
            {
                options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
                options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;

            }).AddJwtBearer("Auth0", options =>
            {
                options.Authority = "https://adamsproul-test-ta.auth0.com/";
                options.Audience = "https://adamsproul.test.com/apigateway";
            });

            services.AddCors(options =>
            {
                options.AddPolicy("CorsPolicy",
                    builder => builder
                    .AllowAnyOrigin()
                    .AllowAnyMethod()
                    .AllowAnyHeader()
                    .AllowCredentials()
                );
            });

            services.AddOcelot(_configuration);
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddApplicationInsights(app.ApplicationServices, LogLevel.Trace);
            loggerFactory.AddConsole(_configuration.GetSection("Logging"));

            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseCors("CorsPolicy");

            app.UseAuthentication();

            app.UseOcelot().Wait();
        }
    }
}

ocelot.development.json
{
"ReRoutes": [
{
"DownstreamPathTemplate": "/rpc/AdministrationUsers/GetUser",
"DownstreamScheme": "https",
"DownstreamHostAndPorts": [
{
"Host": "svc.wam-dev.ta.com",
"Port": 443
}
],
"UpstreamHeaderTransform": {
"Authorization": "Basic {basic_auth credentials}"
},
"UpstreamPathTemplate": "/api/v1/users/getByUserId",
"UpstreamHttpMethod": [ "Post", "Options" ],
"HttpHandlerOptions": {
"UseTracing": true
},
"AuthenticationOptions": {
"AuthenticationProviderKey": "Auth0",
"AllowedScopes": []
}
}
]
}
Any help would be greatly appreciated

As a side note, if I remove the "AuthenticationOptions" from the ReRoute then everything flows through as intended.
But for security I want Ocelot to validate the Auth0 JWT tokens when an SPA makes a request to the API gateway.

Code was transforming the request header and replacing Auth0 token with the basic token intended for the downstream.
"UpstreamHeaderTransform": {
"Authorization": "Basic {basic_auth credentials}"
},
Removed this code and will look at transforming the request header after auth.

DownstreamHeaderTransform wouldn't replace the existing Auth0 authorization header. Maybe because the header was already set.
Thank you for your quick response @TomPallister. Very much appreciate the hard work you have put in to this repo.
In the end I added a DelegatingHandler to do the work, feels cleaner and more control, as below:
namespace APIGateway.Handlers
{
    public class WamDelegatingHandler : DelegatingHandler
    {
        private readonly string Username;
        private readonly string Password;

        public WamDelegatingHandler(IConfiguration configuration)
        {
            Username = configuration["WamService:Username"];
            Password = configuration["WamService:Password"];
        }

        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            // update the headers for WAM - uses basic auth
            request.Headers.Remove("Authorization");
            request.Headers.Add("Authorization", $"Basic {GetAuthorizationHeader()}");

            return await base.SendAsync(request, cancellationToken);
        }

        private string GetAuthorizationHeader() => Convert.ToBase64String(Encoding.ASCII.GetBytes($"{Username}:{Password}"));
    }
}

Startup.cs
...
services.AddOcelot(_configuration)
            .AddDelegatingHandler<WamDelegatingHandler>();

ocelot.development.json
{
  "DownstreamPathTemplate": "/rpc/AdministrationUsers/GetUser",
  "DownstreamScheme": "http",
  "DownstreamHostAndPorts": [
    {
      "Host": "localhost",
      "Port": 52608
    }
  ],
  "UpstreamPathTemplate": "/api/v1/users/getByUserId",
  "UpstreamHttpMethod": [ "Post", "Options" ],
  "HttpHandlerOptions": {
    "UseTracing": true
  },
  "AuthenticationOptions": {
    "AuthenticationProviderKey": "Auth0",
    "AllowedScopes": []
  },
  "DelegatingHandlers": [
    "WamDelegatingHandler"
  ],
}

Yeh that looks good!! Thanks for your interest in Ocelot :)

Hello guys,
Hi @TomPallister, i'm having the same issue as described, but not sure if the solution provided by @adamsproul fits.
Ocelot.Startup.cs: (I used both, neither worked)

And not sure how the TestKey works as I assumed it had nothing to do with the Bearer used on my webAPI (resource i'm securing), as long as it appears on my reroute (ocelot.json)
      `services.AddAuthentication("Bearer")
          .AddIdentityServerAuthentication("TestKey", opt =>
          {
              opt.Authority = "https://localhost:5001";
              opt.RequireHttpsMetadata = false;
              opt.ApiName = "weatherForecastApi";
              opt.SupportedTokens = SupportedTokens.Both;
          });
      //services.AddAuthentication("Bearer")
      //    .AddJwtBearer("TestKey", opt =>
      //    {
      //        opt.Authority = "https://localhost:5001";
      //        opt.TokenValidationParameters = new TokenValidationParameters
      //        {
      //            ValidateAudience = false
      //        };
      //    });
      services.AddOcelot();`

And my ocelot config looks like this (Identity is just another webAPI, nothing to do with IS4)
"Routes":
    //Weather Api
    {
      "DownstreamPathTemplate": "/WeatherForecast",
      "DownstreamScheme": "https",

      "DownstreamHostAndPorts": [
        {
          "Host": "localhost",
          "Port": 7001
        }
      ],
      //"DownstreamHeaderTransform": {
      //  "Authorization": "Basic {basic_auth credentials}"
      //},
      "UpstreamPathTemplate": "/gw/WeatherForecast",
        "UpstreamHttpMethod": [ "Get" ],
        "AuthenticationOptions": {
          "AuthenticationProviderKey": "TestKey",
          "AllowedScopes": [ "weatherForecastApi" ]
        }
      },
      //Identity Api
      {
        "DownstreamPathTemplate": "/Identity",

And my IS4 config
public static IEnumerable<ApiScope> ApiScopes =>
            new List<ApiScope>
            {
                new ApiScope("weatherForecastApi", "Weather Forecast API"),
                new ApiScope("identityApi", "Identity Api")
            };
        public static IEnumerable<Client> Clients =>
            new List<Client>
            {
                new Client
                {
                    ClientId = "MoBe",

                    // no interactive user, use the clientid/secret for authentication
                    AllowedGrantTypes = GrantTypes.ClientCredentials,

                    // secret for authentication
                    ClientSecrets =
                    {
                        new Secret("MoBe".Sha256())
                    },

                    // scopes that client has access to
                    AllowedScopes = { "weatherForecastApi", "identityApi" }
                }
            };

i'm just using a console app, which does access the secured weather api, the re-routing works just fine (I tested),  but when I add AuthenticationOptions in ocelot.json, I get 401, unautorized:
Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
requestId: 0HMERCKN2Q182:00000001, previousRequestId: no previous request id, message: /gw/WeatherForecast is an authenticated route. AuthenticationMiddleware checking if client is authenticated
warn: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
requestId: 0HMERCKN2Q182:00000001, previousRequestId: no previous request id, message: Client has NOT been authenticated for /gw/WeatherForecast and pipeline error set. Request for authenticated route /gw/WeatherForecast by  was unauthenticated
warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]
requestId: 0HMERCKN2Q182:00000001, previousRequestId: no previous request id, message: Error Code: UnauthenticatedError Message: Request for authenticated route /gw/WeatherForecast by  was unauthenticated errors found in ResponderMiddleware. Setting error response for request path:/gw/WeatherForecast, request method: GET
Which is ... not normal because, just right before, I make a direct call to the secured API with the same httpClient, and gets the response I want (which means, the client has been authenticated)
Any chance you spot something I've missed?
I'm on .net core 3.1, and use the appororiate version packages:
is4 v<IP_ADDRESS> / ocelot v16.0.1 / Microsoft.AspNetCore.Authentication.JwtBearer v3.1.22

Ok, the problem seems to be coming from my console client as I instantiated 2 http clients, one for getting the IS4 discovery document and the other making calls to my APIs (and therefore, passing by ocelot)
Which is considered as a bad practice even though it's just 2 instances (not much), so just as a reminder:
there should be only one client per project if I understood well (related to sockets connections and stuff ... not my expertise, but that what was messing with my requests)


Add truncate util

Implement the truncate utility.
call_orig and call_new were added to the TestRig to allow finer-grained tests to give the original and new utility slightly different arguments (for example, different output files). This is helpful to test non-idempotent cases like file creation or deletion.
The ability to determine block_size might be useful for other system tools and warrant moving to os in the future.
PS I think README.md was behind, thus the jump from 48 to 50.

Hmm... the numbers in the test failure look suspiciously like you may need to see if you're handling \r\n vs \n properly.


Filter Pushdown

I can't get filter pushdown work.
The table SSEmployeeCountUnion has about 20 million rows. CompanyId is the partition key and RecordHash is the sort key. When I run
  val unionDataset = spark.read.dynamodbAs[CountRecordEx]("SSEmployeeCountUnion")
  unionDataset.filter($"CompanyId" === 209963).foreach(x => println(x))

I expect the result which is 19 records returned immediately but it looks like the full scan is being performed (it runs for a few minutes until I abort the job).
I also tried setting filterPushdown explicitely but it also doesn't work:
  val unionDataset = spark.read.option("filterPushdown", "true").dynamodbAs[CountRecordEx]("SSEmployeeCountUnion")
  unionDataset.filter($"CompanyId" === 209963).foreach(x => println(x))

Could you help?

@jacobfi , can you please provide some insights into this?
If I remember correctly, we do a scan no matter what. The "predicate pushdown" functionality means that data is not brought into Spark, but is handled by the underlying data store, in this case DynamoDB.

That is correct.
"Intelligent" filter pushdown is currently not a feature.
The datasource always performs a full scan, however in this case a query would have been much more efficient. I hope to get around to implementing this in the future.

Thanks for quick reply, guys. Do you mean that currently there is no way to rewrite this piece of code so that a full scan is avoided?

Unfortunately yes.
If it is a big table I suggest you do a direct query outside of Spark.

Got it, Jacob. Thanks for the quick reply.

This issue will have the same solution as #62


Several dataset tests fail instead of skip

When you write a test for a dataset, it should skip rather than fail if the user hasn't installed that dataset.
I'm not totally sure how these tests are passing on Travis but tons of them fail on my non-LISA machine:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/local/google/home/goodfellow/pylearn2/pylearn2/datasets/tests/test_wiskott.py", line 12, in     test_wiskott
    data = Wiskott()
  File "/usr/local/google/home/goodfellow/pylearn2/pylearn2/datasets/wiskott.py", line 30, in __init__
    X = 1. - load(path)
  File "/usr/local/google/home/goodfellow/pylearn2/pylearn2/utils/serial.py", line 107, in load
    return np.load(filepath)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 370, in load
    fid = open(file, "rb")
IOError: [Errno 2] No such file or directory:        '/usr/local/google/home/goodfellow/data/wiskott/wiskott_fish_layer0_15_standard_64x64_shuffled.npy'

Looks like 41 tests fail in this way for me right now

The tests are skipped if PYLEARN2_DATA_PATH is not defined, that is how they are passing on Travis.
Indeed, the individual tests should also be skipped if the corresponding dataset is not installed.


Program Crashes when building con

branch: CmdRepl
> build Peter Pan
Building the concordance. This may take a moment for large books.
|=======  | Stage 7 of 9Exception in thread "main" java.lang.NullPointerException
	at java.io.FileInputStream.<init>(FileInputStream.java:130)
	at java.io.FileReader.<init>(FileReader.java:72)
	at ClassLibrary.Concord.set_common_words(Concord.java:403)
	at ClassLibrary.Concord.<init>(Concord.java:54)
	at ClassLibrary.CmdRepl.buildConcordance(CmdRepl.java:440)
	at ClassLibrary.CmdRepl.evalCmd(CmdRepl.java:205)
	at ClassLibrary.CmdRepl.startRepl(CmdRepl.java:127)
	at Concordanator.main(Concordanator.java:23)

I'm not sure why this is happening, could @Metalaxe1 or @askewseth try to reproduce this?
I will dig into the stack track tomorrow, it's rather late now.

I just tried it using Peter Pan and it worked fine for me.  It's throwing an error in the set_common_words part, and its a null pointer for the file reader, so I'm guessing that it isn't able to find 'commonwords.txt' for some reason.

Cool, thanks man. Can you continue to test around and use the program. I have to some more coding shortly to implement some summary commands and such. As well as handle large output to the screen.

Yeah sure, I'm going through and fixing everything in concord.java right now.

Alright. I've added a method to Concord in my CmdRepl branch which will help with generating summaries, so tonight or some time tomorrow we can do a big merge and get it submission ready.

@askewseth Correct, on not being able to find commonwords.txt
*nix build script now copies over critical files to the build folder.
FIXED


Dev to main

Welcome to Codecov :tada:
Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Prerelease/migration-fixers

this is a /prerelease branch so you can test via yarn add --dev eslint-plugin-sf-plugin@beta

Review and QA, but we need to need to remove the -beta and set the proper version in the pjson before merging
don't return rules for files not in src/commands (previously, rules ran but did this check later)

recommended ruleset

short-char collision rules now looks at aliases and flags names and shortchar for collisions
warning when using -o on a non-org/hub flag
the type argument of SfCommand<foo> must match the return type of the run method
if your flag doesn't have a summary, a description becomes the summary
if your flag doesn't have a description, a longDescription becomes the description
calls to getConnnection warn when they don't receive an api version
imports of oclif Flags/Command are not allowed by commands that extend SfCommand

new migration rules for sfdxCommand=>sfCommand

see the readme for how to enable the migration ruleset
migrate imports and base class
remove deprecated properties (ex: requiresusername)
builtin flags (flags.verbose) are converted to Flags.boolean
flag parsing will happen in a run method where there are flags
this org and this.flags can't be automatically fixed, but have suggestions to help resolve
flags are converted to sf's syntax

CI
enable prereleases from branches
runs rules against many sf plugins (think external nuts, but for linter rules)

Testing
I often have to use VSCode's reload feature to get the linter rules to pick up.  You can always run yarn lint or even yarn lint --fix to do it without the editor.
@W-12181659@

@mshanemc only ran into one issue (see below)
Untitled
recommended ruleset
✅ short-char collision rules now looks at aliases and flags names and shortchar for collisions
✅ warning when using -o on a non-org/hub flag
✅ the type argument of SfCommand must match the return type of the run method
✅ if your flag doesn't have a summary, a description becomes the summary
✅ if your flag doesn't have a description, a longDescription becomes the description
✅ calls to getConnnection warn when they don't receive an api version
✅ imports of oclif Flags/Command are not allowed by commands that extend SfCommand
new migration rules for sfdxCommand=>sfCommand
✅ see the readme for how to enable the migration ruleset
✅ migrate imports and base class
✅ remove deprecated properties (ex: requiresusername)
❓builtin flags (flags.verbose) are converted to Flags.boolean (having trouble getting the state of source “ready” to see the eslint error)
👎flag parsing will happen in a run method where there are flags
The eslint error is displayed and the fix can be applied, but the inserted parse line is not formatted properly
before
[Image: Image.jpg][Image: Image.jpg]
after
[Image: Image.jpg]✅ this org and this.flags can't be automatically fixed, but have suggestions to help resolve
✅ flags are converted to sf's syntax


mistral qlora fine tuning

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
https://github.com/OpenAccess-AI-Collective/axolotl/tree/main/examples/mistral
I'm fine tuning with the mistral-7B model, but there's a problem that the loss is really big. What's the reason? Another similar llama2 model(13b, 70b), a 2k test dataset in .yml, converges with a loss of 1.0 under, and under the same condition, the mistral is stamped with 5.xx.
Current behaviour
A loss such as llama2 should be taken, but it outputs more than 5 times larger (with qlora)
Steps to reproduce
I ran the file in the readme. Please check.
Config yaml
No response
Possible solution
No response
Which Operating Systems are you using?

[X] Linux
[ ] macOS
[ ] Windows

Python Version
3.10
axolotl branch-commit
x
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

What is your config?

@winglian Omg! Can you share your config?

I find this is usually caused by a bad learning rate.


Failed to install on Heroku 6.9.0

Hello,
I just tried to deploy our app on a fresh jump to 6.90 from 0.12 using latest mmmagic 0.4.5, all went well both on local dev and CI (circleci, ubuntu) however deployment broke on heroku.
This is the npm log https://gist.github.com/thanpolas/4bb8d751b5f17396dee700a3bbf72328
Let me know if you have any suggestions

upgrading to the latest heroku's buildpack solved the problem


fix: filemetadata available on dragstart

Hi
This should fix the issue #584 .
I changed it so that when a file is dragged into the browser, it will create the nativesource together with the file informations. Before it simply created placeholders telling that browser wouldn´t allow it. That was only partially true. The browser allows to acces the file´s metadata like name, size, ... . However, it allows not to access its (binary) data. The binary data is available after the ondrop-event. Therefore on the ondrop event the nativedragsoure will be updated again with new DataTransfer-object.
However, this happens only on DragEnter into the browser and also only for the File-Nativetype. All the other ones stay the same. I was not really able to test them out correctly and the security that prevents from accessing the file´s binary data might effect here more, as the Link and escpecially the text don´t have any metadata at all.
I needed this for a rebuild of the "DropZone" with react-dnd. And the only issue was, that I was not able to react on the file´s metainfo (eg. filetype) prior to droping it (eg. marking the dropzone "green" when it is ok).
Pls have a look at it.
Regards
Leo

Do you have an idea when this will be available via npm?

Hopefully this weekend, I have some things to get done at home today to prepare for the Holiday next week. I'll try to cut a release tonight


Update python-in-visual-studio.md

Typo. (Line 189 is correct, I don't know why it's here.)

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot


reflected in other places

Hi @isteven ,
i am using this library in multiple locations,when i select the friends in the upper area that friends also showing in the below model where i am using.
i am unable to produce the plunker ,am attaching the screen

from the drop down i can select the individuals,from individuals when i select the members ,these members are reflected in other place(where i use this library)

Hi,
You need to provide separate object to both directives, like doing a deep copy of the input-model you are using:
var userList1 = [ { name: A, selected: false },  { name: B, selected: false },  { name: C, selected: false } ]
var userList2 = angular.copy( userList1 )

Note that you CANNOT use simple variable assignment like var userList2 = userList1, because js objects by default assign by reference.

Hi,
@isteven  i given two different names to directive objects.

I see.
Can you replicate the problem in Plunker or JsFiddler, or any other online JS collaboration tool so I can have a look?

Hi,
@isteven  unable replicate the problem.could you please help me out

How can I help you if I cannot see your code to find out where the problem is?

Hi,
you can login into website and observe the bug ,while posting the post and commenting the post.
www.telekha.com

Please show me your JS & HTML codes instead either in Plunker or JsFiddler, that's the only way I can check.
I also clearly stated it here that you need to replicate your problem so at least I can have a look.
https://github.com/isteven/angular-multi-select#bug-reporting

$scope.userFriends = [];
  FollowsService.getFriends($scope.loggedInUserId, function(res) {
    $scope.userFriends = res;
    for(var i = 0 ; i < $scope.userFriends.length ; i++) {
      $scope.userFriends[i].icon = '';
      $scope.userFriends[i].label = $scope.userFriends[i].fullName;
    }
    angular.forEach( $scope.userFriends, function( value, key ) {
      value['ticked'] = false;
    });

  }, function() {});

                                       <div
                                isteven-multi-select
                                input-model="userFriends"
                                output-model="outputUserFriends"
                                button-label="icon fullName"
                                item-label="icon fullName"
                                tick-property="ticked"
                                max-labels="5"
                                max-height="200px"
                                translation="localLang">

this is code i am using in two methods.

OK, so that's the code for the first multi-select.
Where's the code for the second multi-select?

{{'public'|translate}}
{{'followers'|translate}}
{{'individuals'|translate}}

``

$scope.loadFriends = function() {
if($scope.shareOptions != 'INDIVIDUAL') {
return;
}
FollowsService.getFriends(loggedInUser.id, function(res) {
  $scope.userFriends = res;
  for(var i = 0 ; i < $scope.userFriends.length ; i++) {
    $scope.userFriends[i].icon = ' ';
    $scope.userFriends[i].name = $scope.userFriends[i].firstName + ' ' + $scope.userFriends[i].lastName;
  }
}, function() {});

setTimeout(function(){
angular.element(document.querySelectorAll('.multiSelect .reset'))[0].click();
},50);
};``

earlier you said ,change the directive name .Ans i given two different names to the input model and as well as output model

Hi,
Your code is not displayed properly; please follow Github markdown syntax requirement to display codes.

<select class="m-t custom-select custom-select-sm pull-left" ng-model="$parent.shareOptions" ng-change="loadFriends()">
            <option value="PUBLIC" selected>{{'public'|translate}}</option>
            <option value="FOLLOWERS">{{'followers'|translate}}</option>
            <option value="INDIVIDUAL">{{'individuals'|translate}}</option>
          </select>

$scope.loadFriends = function() {
  if($scope.shareOptions != 'INDIVIDUAL') {
    return;
  }

  FollowsService.getFriends(loggedInUser.id, function(res) {
    $scope.userFriends = res;
    for(var i = 0 ; i < $scope.userFriends.length ; i++) {
      $scope.userFriends[i].icon = ' ';
      $scope.userFriends[i].name = $scope.userFriends[i].firstName + ' ' + $scope.userFriends[i].lastName;
    }
  }, function() {});

 setTimeout(function(){
  angular.element(document.querySelectorAll('.multiSelect .reset'))[0].click();
  },50);

};

updated with the rules please check it

OK, I still don't know HOW you have 2 multi-selects but only supplying one HTML code.. I'll just make assumption then.
In short, each multi-select needs to have a unique input-model variable. You cannot use userFriends for all of the multi-select instances.
WRONG:
$scope.userFriends = [ {...}, {...}, ... ];

WRONG:
$scope.userFriends1 = [ {...}, {...}, ... ];
$scope.userFriends2 = $scope.userFriends1;

CORRECT::
$scope.userFriends1 = [ {...}, {...}, ... ];
$scope.userFriends2 = angular.copy( $scope.userFriends1 );

If it's still doesn't work, please show the whole JS & HTML files.

userfriends1 coming from one file and userfriends2 coming from other js file ,how can i copy ?

This is as far as I can help without the whole JS & HTML file. Please copy the whole JS & HTML file in Plunker or JsFiddler and give me the URL - if you want to.
Please read https://github.com/isteven/angular-multi-select#bug-reporting, point number 3.

added html and js files here  https://plnkr.co/edit/dTolp3cy20dwek9cgMOQ?p=preview

How does itemcontrol.html get loaded in dashboard.html? I didn't see any ng-view or ng-include loading the itemcontrol.html.
Anyway, from what I see, there might be a scope conflict.
To be safe, use different objects for the input-model, AND the output-model (especially because you use $parent in the output-model).
Example:

(Adjust the controllers accordingly as well)


Xaml pages and FreshMVVM

I have a question about XAML pages and FreshMVVM.
In the n=1 tutorial video, there isn't any changes of the ContactListPage.xaml.cs file from it's default of deriving from ContentPage.  But in the git sample code the ContactListPage.xaml.cs derives from BasePage.cs.  BasePage isn't included in the FreshMVVM nuget package that I can see.
The FreshMVVM nuget package does have a FreshBaseContentPage but it isn't used in the samples - they all derive from the file BasePage.cs.
Should XAML pages derive from FreshBaseContentPage, ContentPage or BasePage?  I am asking this because starting from File..New Project I run into issues if I follow the samples example and the deriving from BasePage.

This is more a question, let's close this as a issue and move over to the
Xamarin.Forms forums please?
On Sat., 31 Dec. 2016 at 7:43 am, wyldebill<EMAIL_ADDRESS>wrote:

I have a question about XAML pages and FreshMVVM.
In the n=1 tutorial video, there isn't any changes of the
ContactListPage.xaml.cs file from it's default of deriving from
ContentPage. But in the git sample code the ContactListPage.xaml.cs derives
from BasePage.cs. BasePage isn't included in the FreshMVVM nuget package
that I can see.
The FreshMVVM nuget package does have a FreshBaseContentPage but it isn't
used in the samples - they all derive from the file BasePage.cs.
Should XAML pages derive from FreshBaseContentPage, ContentPage or
BasePage? I am asking this because starting from File..New Project I run
into issues if I follow the samples example and the deriving from BasePage.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rid00z/FreshMvvm/issues/133, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABeXvX6DzzyuWrXbVJSIdYFIkGuI_87Zks5rNWz2gaJpZM4LYYZ-
.

Will do. Sorry, I am new to open source.
Thanks for the responses.
William Austin
Consultant, Improving - Twin Cities
On Dec 30, 2016, at 3:13 PM, Michael Ridland <notifications<EMAIL_ADDRESS>wrote:
This is more a question, let's close this as a issue and move over to the
Xamarin.Forms forums please?
On Sat., 31 Dec. 2016 at 7:43 am, wyldebill <notifications<EMAIL_ADDRESS>wrote:

I have a question about XAML pages and FreshMVVM.
In the n=1 tutorial video, there isn't any changes of the
ContactListPage.xaml.cs file from it's default of deriving from
ContentPage. But in the git sample code the ContactListPage.xaml.cs derives
from BasePage.cs. BasePage isn't included in the FreshMVVM nuget package
that I can see.
The FreshMVVM nuget package does have a FreshBaseContentPage but it isn't
used in the samples - they all derive from the file BasePage.cs.
Should XAML pages derive from FreshBaseContentPage, ContentPage or
BasePage? I am asking this because starting from File..New Project I run
into issues if I follow the samples example and the deriving from BasePage.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rid00z/FreshMvvm/issues/133, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABeXvX6DzzyuWrXbVJSIdYFIkGuI_87Zks5rNWz2gaJpZM4LYYZ-
.

You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/rid00z/FreshMvvm/issues/133#issuecomment-269822586, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADjvXFyGxbZ2semN7t3_IcM7YpxDZ5iPks5rNXPugaJpZM4LYYZ-.