Embending node-red but "can't establish a connection to the server at ws://localhost:3000/red/comms.

Hi,
I try to install node-red as simple embed app with express 4. With simple/basic intégration node-red work normaly, but when i run the embed code I allways have the folowing error can't establish a connection to the server at ws://localhost:3000/red/comms. and after some minutes i have the alert message "Lost connection to server, reconnecting in ..."
Repo : https://github.com/ethyde/heroku_node-red
Local config :
26 May 10:12:02 - [info] Node-RED version: v0.16.2
26 May 10:12:02 - [info] Node.js  version: v6.10.3

Thanks.

Hi,
what have you done to try to debug this? I'm not going to dig through your repository to find the relevant bits of your code - please link to where you are actually embedding Node-RED.

Right now, I'm starting to install and use it by following the doc.
Code for embeding with ExpressJS are here and it use this settings files.
NPM Script : "dev:server": "cross-env NODE_ENV=development nodemon boot.js",
Normal (non-embed) settings file are here it's the same as previous but with module.export instead of ES2015 syntax.
NPM Script : "node-red": "./node_modules/.bin/node-red --settings ./app/node-red.settings.es5.js"
Server listening are here
I have read about some issue with param `userDir: '/home/nol/.node-red/', but it is well commented, so what can i do ?

The problem is here: https://github.com/ethyde/heroku_node-red/blob/master/app/app.js#L21
The first argument you pass to RED.init must be an HTTP Server instance, not an Express app instance.

Hi thanks for your answer !
I have made the correction, but issue still existe : https://github.com/ethyde/heroku_node-red/blob/master/app/app.js#L24

You pass a server to RED.init, but then you call app.listen which causes express to create its own server instance and not use the one you provided.
You should call server.listen on the instance of server you pass RED.init.

Ok, it's not a real embending : node-red use it's own server instance ? And even embed, to interact with it, it's like requesting an any other external API ?

It does not use its own server instance. You provide the server instance.
We haven't formally documented the runtime api yet, but what is available is listed here: http://nodered.org/docs/api/runtime/api Any questions, please use the node-red mailing list etc.
So I assume your immediate issue is resolved?

My first configuration is similar/same to the one I can have in production, and node-red can not use it.
So ATM, I can not use node-red in embed, because it can not use an existing server instance, so node-red can't be embed in another app. I have to use node-red with embed code as a Core of an app, it's not the same things.
But I think yes, immediate issue is solved.

In my case, it goes on pending state:

Is there any extra thing to do?


no-extraneous-deps globs don't also check against process.cwd() + glob

I want to set my glob to tests/**, so that only the top-level "tests" dir can contain things that require dev deps.
However, this rule compares against the full absolute path of the file, which includes the path to the project root - which isn't really useful, and forces people to start 100% of their globs with "**/".
I'd like the rule to check both against the full filename, and also against path.join(process.cwd(), glob), to remove the need for this boilerplate in airbnb's shared config.

Sounds like a good idea.
The one point I'm not familiar with is whether using process.cwd() is the right way to do this. Because that would mean you would get different results based on the directory we run ESLint in, right?
I think it would be more correct to replace process.cwd() by the root of of the project (probably where package.json is located). But as we saw recently, especially with this rule, that's more complicated than expected with monorepos. So I'd probably fall back to the use of process.cwd().
Do you think that running ESLint not from the root folder is a sufficiently rare case to use process.cwd()? I always run from the root, but others may do it differently.
cc @benmosher if you have time to give your opinion on this

Yes, I don't think any command should ever be run anywhere except from the project root.
There's a thread on eslint about locating the project root, and they concluded that process.cwd was best.

Ok, that sounds good to me then. Help welcome!
(cc @knpwrs, if you feel like helping out again :smile:)

@ljharb Can you link to that thread for reference?

@knpwrs https://github.com/eslint/eslint/issues/3974#issuecomment-144197496

So... speaking of multiple .eslintrc files how should this behave in that context? An .eslintrc file in the test directory would still execute process.cwd() in the root directory if I understand that discussion correctly.
I realize your situation is a single .eslintrc file but I'd want to support all usages.

@knpwrs yes, that's the intention

Using process.cwd makes sense to me. A number of things already do this.
It can cause problems with editor integration, but I think the workarounds in those cases are well understood.


Are there plans to support touch events?

Hello,
This library seems very useful.  Are there any plans to support touch events in the future?  I think that would really help the adoption if you were able to support both HTML5 and touch seamlessly.
Cool library, looking forward to playing around with it.
Thanks,
-Todd

This is already supported, refer to the Sensor and TouchSensor docs.

This is already supported, refer to the Sensor and TouchSensor docs.


Add filter node

Purpose of this pull request
This pull request adds documentation for the filter node to these articles:

https://help.sumologic.com/docs/platform-services/automation-service/automation-service-playbooks/
https://help.sumologic.com/docs/cloud-soar/automation/

Select the type of change

[ ] Minor Changes - Typos, formatting, slight revisions
[ ] Update Content - Revisions, updating sections
[x] New Content - New features, sections, pages, tutorials
[ ] Site and Tools - .clabot, version updates, maintenance, dependencies, new packages for the site (Docusaurus, Gatsby, React, etc.)

Ticket (if applicable)
CSOAR-1807

Thanks for the review, @kimsauce!


Request wasn't successful, there is probably a bug in the API code.

Hi, I'm facing following issue w/ latest packit. Am I doing something wrong?
Version Packit 0.7.2.dev131+gc678c7f
It seems that Packit has hard time to speak with COPR. Not sure what is the problem as we are getting just an exception.
Debug output
➜   packit --debug copr-build
09:43:08.325 config.py         DEBUG  Loading user config from directory: /home/mastyk/.config
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/.packit.yaml
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/.packit.yml
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/.packit.json
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/packit.yaml
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/packit.yml
09:43:08.325 config.py         DEBUG  Trying to load user config from: /home/mastyk/.config/packit.json
WARNING  Please, use 'authentication' key in the user configuration to set tokens for GitHub and Pagure. New method supports more services and direct keys will be removed in the future.
Example:
authentication:
    github.com:
        token: GITHUB_TOKEN
    pagure:
        token: PAGURE_TOKEN
        instance_url: "https://src.fedoraproject.org"

09:43:08.326 utils.py          DEBUG  logging set to DEBUG
Packit 0.7.2.dev131+gc678c7f is being used.
Input is a directory: /home/mastyk/beaker
09:43:08.327 local_project.py  DEBUG  Arguments received in the init method of the LocalProject class:
git_repo: None
working_dir: /home/mastyk/beaker
ref: None
git_project: None
git_service: None
git_url:
full_name:
namespace:
repo_name:
offline: False
refresh True
remote: None
pr_id: None

09:43:08.327 local_project.py  DEBUG  working_dir is set and git_repo is not: let's discover...
09:43:08.327 local_project.py  DEBUG  It's a git repo!
09:43:08.328 local_project.py  DEBUG  Parsed ref develop from the repo <git.Repo "/home/mastyk/beaker/.git">.
09:43:08.330 local_project.py  DEBUG  Parsed remote url https://gerrit.beaker-project.org/beaker from the repo <git.Repo "/home/mastyk/beaker/.git">.
09:43:08.330 local_project.py  DEBUG  Parsed namespace and repo name (None, beaker) from url https://gerrit.beaker-project.org/beaker.
09:43:08.330 package_config.py DEBUG  Local package config found: /home/mastyk/beaker/.packit.yaml
09:43:08.332 package_config.py DEBUG  Package config:
{
    "specfile_path": "beaker.spec",
    "synced_files": [
        "beaker.spec",
        ".packit.yaml"
    ]
}
Input directory is an upstream repository.
09:43:08.922 base_git.py       DEBUG  command handler = RunCommandType.local
09:43:08.922 utils.py          DEBUG  Provided input https://gerrit.beaker-project.org/beaker is an url.
09:43:08.923 base_git.py       DEBUG  Running ActionName.post_upstream_clone hook.
09:43:08.923 base_git.py       DEBUG  Running ActionName.post_upstream_clone.
09:43:08.923 base_git.py       DEBUG  Running default implementation for ActionName.post_upstream_clone.
09:43:08.923 utils.py          DEBUG  cmd = 'git describe --tags --match *'
09:43:08.960 utils.py          DEBUG  b'beaker-27.0-2-g1b01e5055'
09:43:08.961 upstream.py       DEBUG  version = beaker-27.0-2-g1b01e5055
09:43:08.961 upstream.py       DEBUG  sanitized version = beaker.27.0.2.g1b01e5055
09:43:08.968 upstream.py       DEBUG  name + version = beaker-beaker.27.0.2.g1b01e5055
09:43:09.275 utils.py          DEBUG  cmd = 'git archive -o beaker-beaker.27.0.2.g1b01e5055.tar.xz --prefix beaker-beaker.27.0.2.g1b01e5055/ HEAD'
09:43:09.364 base_git.py       DEBUG  Running ActionName.fix_spec.
09:43:09.364 base_git.py       DEBUG  Running default implementation for ActionName.fix_spec.
09:43:09.624 upstream.py       DEBUG  new %setup line:
%setup -q -n beaker-beaker.27.0.2.g1b01e5055
09:43:09.882 specfile.py       DEBUG  The specfile doesn't have any %changelog, will not set it.
09:43:09.884 upstream.py       DEBUG  present srpms = {PosixPath('/home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm')}
09:43:09.884 utils.py          DEBUG  cmd = 'rpmbuild -bs --define _sourcedir . --define _specdir /home/mastyk/beaker --define _srcrpmdir /home/mastyk/beaker --define _topdir /home/mastyk/beaker --define _builddir /home/mastyk/beaker --define _rpmdir /home/mastyk/beaker --define _buildrootdir /home/mastyk/beaker beaker.spec'
09:43:11.505 utils.py          DEBUG  b'Wrote: /home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm'
09:43:11.512 upstream.py       DEBUG  Wrote: /home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm
SRPM is /home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm
09:43:11.512 api.py            DEBUG  owner=None, project=packit-cli-beaker-develop, path=/home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm
ERROR    Request wasn't successful, there is probably a bug in the API code.
Traceback (most recent call last):
  File "/home/mastyk/.local/lib/python3.7/site-packages/packit/cli/utils.py", line 61, in covered_func
    func(config=config, *args, **kwargs)
  File "/home/mastyk/.local/lib/python3.7/site-packages/packit/cli/copr_build.py", line 82, in copr_build
    instructions=instructions,
  File "/home/mastyk/.local/lib/python3.7/site-packages/packit/api.py", line 692, in run_copr_build
    owner, project, srpm_path
  File "/usr/lib/python3.7/site-packages/copr/v3/helpers.py", line 59, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/copr/v3/proxies/build.py", line 141, in create_from_file
    return self._create(endpoint, data, files=files, buildopts=buildopts)
  File "/usr/lib/python3.7/site-packages/copr/v3/helpers.py", line 59, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/copr/v3/proxies/build.py", line 266, in _create
    response = request.send()
  File "/usr/lib/python3.7/site-packages/copr/v3/requests.py", line 52, in send
    handle_errors(response)
  File "/usr/lib/python3.7/site-packages/copr/v3/requests.py", line 108, in handle_errors
    raise CoprRequestException(response_json["error"], response=response)
copr.v3.exceptions.CoprRequestException: Request wasn't successful, there is probably a bug in the API code.

09:43:11.512 api.py            DEBUG  owner=None, project=packit-cli-beaker-develop, path=/home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm
I just noticed owner=None.

I just noticed owner=None.

Good catch. Looks like you have owner not configured in the copr config and you don't set it as an option. Can you try setting the owner using CLI option?
$ packit copr-build --help
Packit 0.7.2.dev126+g07b65dc.d20200102 is being used.
Usage: packit copr-build [OPTIONS] [PATH_OR_URL]

  Build selected upstream project in COPR.

  PATH_OR_URL argument is a local path or a URL to the upstream git
  repository, it defaults to the current working directory.

Options:
  --nowait             Don't wait for build
  --owner TEXT         Copr user, owner of the project. (defaults to username
                       from copr config)
  :
  :

We need to rise a better error message.

Ok, this is a debug from the API so it means you don't specify that as a CLI option. It can still load the owner from copr config.
09:43:11.512 api.py            DEBUG  owner=None, project=packit-cli-beaker-develop, path=/home/mastyk/beaker/beaker-beaker.27.0.2.g1b01e5055-1.fc31.src.rpm

Since #648 is merged, can this be closed now?

The #648 only checks the owner, the problem can still be there.

This should be really fixed now (#713). Can we close it?


Change unstable to v0.3

Release v0.2 has already been done. Unstable should be pointing to
v0.3 as of now.

@r8d8 Cargo.lock was put there on purpose -- we have library crates, those crates will ignore Cargo.lock. However we also have executable crates, like regtests. The Cargo guideline recommends to include Cargo.lock for executable crates, as different machines will produce different results, otherwise.


Test Purchase Debug

Hello author,
I want to test debug with id test: "android.test.purchased"
I tried adding it to the list but when I made the purchase, getProductDetails returned null value

Sounds like you need to setup application licensing first.


Normalize test parameters when parsing results

#41 - See discussion around whitespace and inline test parameters

Closing as this it will be automatically resolved by #90, and has been resolved for xUnit via #95


Readme consists primarily of self-congratulation

Seriously, like 60% of the text is just talking about how awesome stuff is. Show how awesome it is by showing the API, and providing some good docs. Look at @substack's stuff for examples of good readmes.

Please feel free to pitch in.


No Console Logging

In the  latest build all console logging of errors is gone, and my queueTrigger do not start .
Using microsoft/azure-functions-node8:2.0

The python version has similar issues, I can't get any type of function to work.

This issue was moved to Azure/azure-functions-host#3469

The console log is still missing for the python docker image, when the envvar AzureFunctionsJobHost__Logging__Console__IsEnabled is set to true.
I am using docker image base: mcr.microsoft.com/azure-functions/python:3.0.13901-python3.7.


Can't finsh install (only get to 3/4 steps)

Just tried installing this using your curl command, it got to 3/4 steps and just hangs.  I notice in the kalm-system namespace, the   kalm pod fails gets the following errors before going into crashloopbackoff:
2020-09-09T03:39:42.229Z        ERROR   Error updating metrics  {"error": "Get https://<IP_ADDRESS>:443/apis/metrics.k8s.io/v1beta1/pods: dial tcp <IP_ADDRESS>:443: connect: connection refused"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
github.com/kalmhq/kalm/api/log.Error
        /workspace/api/log/logger.go:42
github.com/kalmhq/kalm/api/resources.StartMetricScraper
        /workspace/api/resources/metric_scraper.go:64
main.startMetricServer
        /workspace/api/main.go:137
2020-09-09T03:39:47.229Z        ERROR   Error scraping pod metrics      {"error": "Get https://<IP_ADDRESS>:443/apis/metrics.k8s.io/v1beta1/pods: dial tcp <IP_ADDRESS>:443: connect: connection refused"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
github.com/kalmhq/kalm/api/log.Error
        /workspace/api/log/logger.go:42
github.com/kalmhq/kalm/api/resources.update
        /workspace/api/resources/metric_scraper.go:73
github.com/kalmhq/kalm/api/resources.StartMetricScraper
        /workspace/api/resources/metric_scraper.go:62
main.startMetricServer
        /workspace/api/main.go:137
2020-09-09T03:39:47.229Z        ERROR   Error updating metrics  {"error": "Get https://<IP_ADDRESS>:443/apis/metrics.k8s.io/v1beta1/pods: dial tcp <IP_ADDRESS>:443: connect: connection refused"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
github.com/kalmhq/kalm/api/log.Error
        /workspace/api/log/logger.go:42
github.com/kalmhq/kalm/api/resources.StartMetricScraper
        /workspace/api/resources/metric_scraper.go:64
main.startMetricServer
        /workspace/api/main.go:137

if i kill the pod and let it restart, sometimes it fails with crashloopbackoff, sometimes the istio-proxy container fails with OOM:
2020-09-09T03:39:45.136643Z     info    sds     resource:default new connection
2020-09-09T03:39:45.136857Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T03:39:48.545442Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2020-09-09T03:39:48.545524Z     info    cache   GenerateSecret default
2020-09-09T03:39:48.545766Z     info    sds     resource:default pushed key/cert pair to proxy
2020-09-09T03:39:51.237322Z     info    sds     resource:ROOTCA new connection
2020-09-09T03:39:51.237468Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T03:39:51.237512Z     info    cache   Loaded root cert from certificate ROOTCA
2020-09-09T03:39:51.237670Z     info    sds     resource:ROOTCA pushed root cert to proxy
2020-09-09T03:39:57.831084Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T03:39:57.834243Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T03:39:58.531777Z     info    sds     resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T03:39:58.531765Z     info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-09T03:39:58.531780Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T03:39:58.531880Z     error   sds     Remote side closed connection
2020-09-09T03:39:58.531847Z     error   sds     Remote side closed connection
2020-09-09T03:39:58.532186Z     warn    Envoy may have been out of memory killed. Check memory usage and limits.
2020-09-09T03:39:58.532231Z     error   Epoch 0 exited with error: signal: killed
2020-09-09T03:39:58.532241Z     info    No more active epochs, terminating

The kalm pod then goes into a CrashLoopBackoff.
My system has 64GB of ram free, so its not a system issue
Normal   Scheduled  <unknown>             default-scheduler   Successfully assigned kalm-system/kalm-5f58d8bd9-6pqzt to homelab-a
  Normal   Pulling    10m                   kubelet, homelab-a  Pulling image "docker.io/istio/proxyv2:1.6.1"
  Normal   Pulled     10m                   kubelet, homelab-a  Successfully pulled image "docker.io/istio/proxyv2:1.6.1"
  Normal   Created    10m                   kubelet, homelab-a  Created container istio-init
  Normal   Started    10m                   kubelet, homelab-a  Started container istio-init
  Normal   Created    10m                   kubelet, homelab-a  Created container kalm
  Normal   Pulled     10m                   kubelet, homelab-a  Container image "kalmhq/kalm:v0.1.0-alpha.5" already present on machine
  Normal   Pulling    10m                   kubelet, homelab-a  Pulling image "docker.io/istio/proxyv2:1.6.1"
  Normal   Started    10m                   kubelet, homelab-a  Started container kalm
  Normal   Pulled     10m                   kubelet, homelab-a  Successfully pulled image "docker.io/istio/proxyv2:1.6.1"
  Normal   Created    10m                   kubelet, homelab-a  Created container istio-proxy
  Normal   Started    10m                   kubelet, homelab-a  Started container istio-proxy
  Warning  Unhealthy  9m51s (x14 over 10m)  kubelet, homelab-a  Readiness probe failed: Get http://<IP_ADDRESS>:15021/healthz/ready: dial tcp <IP_ADDRESS>:15021: connect: connection refused
  Warning  BackOff    21s (x35 over 9m21s)  kubelet, homelab-a  Back-off restarting failed container

free -m
              total        used        free      shared  buff/cache   available
Mem:         128713       61142        1289        2558       66281       65946
Swap:             0           0           0

kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:10:16Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Metric errors can now be ignored temporarily, which does not affect the installation process. Can you provide a list of pods when the installation hangs? I need to know which part is not installed correctly. Also please provide description and logs of the Error pods.

✔ kalm-operator
✔ cert-manager
✔ istio-system

NAME                               READY   STATUS             RESTARTS   AGE
kalm-5f58d8bd9-6pqzt               1/2     CrashLoopBackOff   44         3h45m
kalm-controller-85dd8545b6-4b6fs   3/3     Running            1          4h36m

it seems to be  the istio-proxy container:
2020-09-09T07:19:06.738310Z     info    parsed scheme: ""
2020-09-09T07:19:06.738333Z     info    scheme "" not registered, fallback to default scheme
2020-09-09T07:19:06.738380Z     info    ccResolverWrapper: sending update to cc: {[{istiod.istio-system.svc:15012  <nil> 0 <nil>}] <nil> <nil>}
2020-09-09T07:19:06.738397Z     info    ClientConn switching balancer to "pick_first"
2020-09-09T07:19:06.738405Z     info    Channel switches to new LB policy "pick_first"
2020-09-09T07:19:06.738453Z     info    Subchannel Connectivity change to CONNECTING
2020-09-09T07:19:06.738616Z     info    Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-09-09T07:19:06.738777Z     info    sds     SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"

2020-09-09T07:19:06.738755Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc00133f2e0, {CONNECTING <nil>}
2020-09-09T07:19:06.738864Z     info    Starting proxy agent
2020-09-09T07:19:06.738857Z     info    Channel Connectivity change to CONNECTING
2020-09-09T07:19:06.830785Z     info    Opening status port 15020

2020-09-09T07:19:06.738805Z     info    sds     Start SDS grpc server
2020-09-09T07:19:06.831412Z     info    Received new config, creating new Envoy epoch 0
2020-09-09T07:19:06.930693Z     info    Epoch 0 starting
2020-09-09T07:19:07.332942Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster kalm.kalm-system --service-node sidecar~<IP_ADDRESS>~kalm-5f58d8bd9-6pqzt.kalm-system~kalm-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format %Y-%m-%dT%T.%fZ %l      envoy %n        %v -l warning --component-log-level misc:error --concurrency 2]
2020-09-09T07:19:07.430678Z     info    Subchannel Connectivity change to READY
2020-09-09T07:19:07.430783Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc00133f2e0, {READY <nil>}
2020-09-09T07:19:07.430803Z     info    Channel Connectivity change to READY
2020-09-09T07:19:08.939811Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T07:19:08.939915Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T07:19:09.141609Z     info    sds     resource:default new connection
2020-09-09T07:19:09.141754Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T07:19:09.936981Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T07:19:09.937028Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T07:19:11.807408Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T07:19:11.807458Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T07:19:16.589210Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T07:19:16.589294Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T07:19:17.444999Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2020-09-09T07:19:17.530341Z     info    cache   GenerateSecret default
2020-09-09T07:19:17.530835Z     info    sds     resource:default pushed key/cert pair to proxy
2020-09-09T07:19:26.737991Z     info    sds     resource:ROOTCA new connection
2020-09-09T07:19:26.738201Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T07:19:26.738370Z     info    cache   Loaded root cert from certificate ROOTCA
2020-09-09T07:19:26.738545Z     info    sds     resource:ROOTCA pushed root cert to proxy
2020-09-09T07:19:34.234210Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T07:19:34.238624Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T07:19:35.131065Z     info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-09T07:19:35.131104Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T07:19:35.131073Z     info    sds     resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T07:19:35.131132Z     error   sds     Remote side closed connection
2020-09-09T07:19:35.131148Z     error   sds     Remote side closed connection
2020-09-09T07:19:35.131598Z     warn    Envoy may have been out of memory killed. Check memory usage and limits.
2020-09-09T07:19:35.131627Z     error   Epoch 0 exited with error: signal: killed
2020-09-09T07:19:35.131636Z     info    No more active epochs, terminating

here's the kubectl describe on the pod:
 istio-proxy:
    Container ID:  docker://3979a63a71358d5721f47cb01d1d0e0ac54772f38af8adf549d253dd174d55d8
    Image:         docker.io/istio/proxyv2:1.6.1
    Image ID:      docker-pullable://istio/proxyv2@sha256:84e3afe9b4404ca94fd2e6e0277c642eb29b8b37ca46deff49dbe1f5e1b7fdc3
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      kalm.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --trust-domain=cluster.local
      --concurrency
      2
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    0
      Started:      Wed, 09 Sep 2020 00:19:04 -0700
      Finished:     Wed, 09 Sep 2020 00:19:35 -0700
    Ready:          False
    Restart Count:  44
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:      100m
      memory:   50Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30

Based on the information you provided, I guess it is caused by the istio sidecar resource usage exceeding the limit. Since the limit of the current alpha5 version is hard-coded, I submitted a new commit to remove this limit. Please try whether this version works normally. You can update by the following command.
kubectl patch kalmoperatorconfigs.install.kalm.dev -n kalm-operator kalmoperatorconfig-sample --type json -p'[{"op": "replace", "path": "/spec/kalmVersion ", "value": "2ced9fd0780dc4757730e9157a0da227df967d86"}]'

that didnt seem to work.  As a test, I deleted the kalmoperatorconfgs resource, then re-ran setup.  After re-running setup, i applied the patch:
 kubectl get kalmoperatorconfigs -n kalm-operator kalmoperatorconfig-sample -o yaml                                                   apiVersion: install.kalm.dev/v1alpha1
kind: KalmOperatorConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"install.kalm.dev/v1alpha1","kind":"KalmOperatorConfig","metadata":{"annotations":{},"name":"kalmoperatorconfig-sample","namespace":"kalm-operator"},"spec":{"kalmVersion":"v0.1.0-alpha.5"}}
  creationTimestamp: "2020-09-09T23:09:14Z"
  generation: 2
  managedFields:
  - apiVersion: install.kalm.dev/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:kalmVersion: {}
        'f:kalmVersion ': {}
    manager: kubectl
    operation: Update
    time: "2020-09-09T23:09:28Z"
  name: kalmoperatorconfig-sample
  namespace: kalm-operator
  resourceVersion: "53432854"
  selfLink: /apis/install.kalm.dev/v1alpha1/namespaces/kalm-operator/kalmoperatorconfigs/kalmoperatorconfig-sample
  uid: 215c96ca-fabe-4d90-a0ce-fb92e640a729
spec:
  kalmVersion: v0.1.0-alpha.5
  'kalmVersion ': 2ced9fd0780dc4757730e9157a0da227df967d86

but when i delete the running/crashed pod, it runs into the same issue:
kubectl get pods -n kalm-system
NAME                               READY   STATUS      RESTARTS   AGE
kalm-5f58d8bd9-5bsdj               1/2     OOMKilled   4          4m
kalm-controller-85dd8545b6-4b6fs   3/3     Running     1          20h

Events:
  Type     Reason     Age                    From                Message
  ----     ------     ----                   ----                -------
  Normal   Scheduled  <unknown>              default-scheduler   Successfully assigned kalm-system/kalm-5f58d8bd9-5bsdj to homelab-a
  Normal   Pulling    5m34s                  kubelet, homelab-a  Pulling image "docker.io/istio/proxyv2:1.6.1"
  Normal   Pulled     5m33s                  kubelet, homelab-a  Successfully pulled image "docker.io/istio/proxyv2:1.6.1"
  Normal   Created    5m33s                  kubelet, homelab-a  Created container istio-init
  Normal   Started    5m32s                  kubelet, homelab-a  Started container istio-init
  Normal   Pulled     5m30s                  kubelet, homelab-a  Container image "kalmhq/kalm:v0.1.0-alpha.5" already present on machine
  Normal   Started    5m30s                  kubelet, homelab-a  Started container kalm
  Normal   Created    5m30s                  kubelet, homelab-a  Created container kalm
  Normal   Started    5m28s                  kubelet, homelab-a  Started container istio-proxy
  Warning  Unhealthy  5m7s (x11 over 5m27s)  kubelet, homelab-a  Readiness probe failed: Get http://<IP_ADDRESS>:15021/healthz/ready: dial tcp <IP_ADDRESS>:15021: connect: connection refused
  Normal   Pulled     5m5s (x2 over 5m29s)   kubelet, homelab-a  Successfully pulled image "docker.io/istio/proxyv2:1.6.1"
  Normal   Created    5m5s (x2 over 5m29s)   kubelet, homelab-a  Created container istio-proxy
  Normal   Pulling    26s (x6 over 5m30s)    kubelet, homelab-a  Pulling image "docker.io/istio/proxyv2:1.6.1"

logs of the istio-proxy container:

2020-09-09T23:11:42.235705Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T23:11:43.011340Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T23:11:43.011385Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T23:11:43.182912Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T23:11:43.182964Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T23:11:43.905267Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-09T23:11:43.905311Z     warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-09-09T23:11:50.730930Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2020-09-09T23:11:50.731048Z     info    cache   GenerateSecret default
2020-09-09T23:11:50.731315Z     info    sds     resource:default pushed key/cert pair to proxy
2020-09-09T23:12:06.236061Z     info    sds     resource:ROOTCA new connection
2020-09-09T23:12:06.236188Z     info    sds     Skipping waiting for ingress gateway secret
2020-09-09T23:12:06.236222Z     info    cache   Loaded root cert from certificate ROOTCA
2020-09-09T23:12:06.330400Z     info    sds     resource:ROOTCA pushed root cert to proxy
2020-09-09T23:12:12.837496Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T23:12:12.930421Z     warning envoy filter    [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-09-09T23:12:13.731544Z     info    sds     resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T23:12:13.731558Z     info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-09T23:12:13.731597Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-09T23:12:13.731603Z     error   sds     Remote side closed connection
2020-09-09T23:12:13.731625Z     error   sds     Remote side closed connection
2020-09-09T23:12:13.831751Z     warn    Envoy may have been out of memory killed. Check memory usage and limits.
2020-09-09T23:12:13.831794Z     error   Epoch 0 exited with error: signal: killed
2020-09-09T23:12:13.831804Z     info    No more active epochs, terminating

It seems you are still using kalmhq/kalm:v0.1.0-alpha.5 image. The resources limit is still there.
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:      100m
      memory:   50Mi

If you re-run the install script, it will always use v0.1.0-alpha5 version. You need to update kalmVersion in your kalmoperatorconfgs CRD after running the install script. Please use 2ced9fd0780dc4757730e9157a0da227df967d86 instead of v0.1.0-alpha.5.   The command I provided in the last commit can help you do that.
And also, If the OOM error still exists. Please also provide the other kalm pod containers log as well, maybe useful.

@davidqhr  thanks but I did run your command:
 kubectl patch kalmoperatorconfigs.install.kalm.dev -n kalm-operator kalmoperatorconfig-sample --type json -p'[{"op": "replace", "path": "/spec/kalmVersion ", "value": "2ced9fd0780dc4757730e9157a0da227df967d86"}]'
I0915 00:13:31.769487   29137 request.go:621] Throttling request took 1.196948821s, request: GET:https://<IP_ADDRESS>:6443/apis/batch/v1beta1?timeout=32s
kalmoperatorconfig.install.kalm.dev/kalmoperatorconfig-sample patched (no change)

it seems to do something strange to the kalmconfig:
evan@homelab-a:~$ kubectl get kalmoperatorconfigs -n kalm-operator kalmoperatorconfig-sample -o yaml                                                                    apiVersion: install.kalm.dev/v1alpha1
kind: KalmOperatorConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"install.kalm.dev/v1alpha1","kind":"KalmOperatorConfig","metadata":{"annotations":{},"name":"kalmoperatorconfig-sample","namespace":"kalm-operator"},"spec":{"kalmVersion":"v0.1.0-alpha.5"}}
  creationTimestamp: "2020-09-09T23:09:14Z"
  generation: 2
  managedFields:
  - apiVersion: install.kalm.dev/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:kalmVersion: {}
        'f:kalmVersion ': {}
    manager: kubectl
    operation: Update
    time: "2020-09-09T23:09:28Z"
  name: kalmoperatorconfig-sample
  namespace: kalm-operator
  resourceVersion: "53432854"
  selfLink: /apis/install.kalm.dev/v1alpha1/namespaces/kalm-operator/kalmoperatorconfigs/kalmoperatorconfig-sample
  uid: 215c96ca-fabe-4d90-a0ce-fb92e640a729
spec:
  kalmVersion: v0.1.0-alpha.5
  'kalmVersion ': 2ced9fd0780dc4757730e9157a0da227df967d86

there are now 2 verisons of "kalmVersion"

Sorry the command above has a type. "/spec/kalmVersion "should not contain the tailing white space. It seems you have addressed this issue and fix the wrong patch result. Glad to hear your cluster is finialy working. So the problem is the unreasonable resource limitation of the sidecar container. I will fix that in next release.

Thanks!  Btw, I guess I might open a seperate ticket, but it doesn't seem that kalm picks up existing namespaces, is this expected?   I'm deploying kalm into an existing cluster, although just for home automation stuff.   I plan on deploying a new cluster from scratch soon, so will probably through Kalm on from the start so it can handle things.  Thanks again for your help @davidqhr


Add Bubble Sort feature and associated test cases

Fixes #42
Changes proposed in this PR:

LAB 2 SORTING
Bubble Sort Feature
Insertion Sort Feature
Associated Test Cases

Codecov Report

Merging #54 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #54   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           2      3    +1
  Lines           9     18    +9
=====================================
+ Hits            9     18    +9

Impacted Files
Coverage Δ

02_LabJat/bubble_sort.go
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f263dc...3bd4541. Read the comment docs.

@madhavansu  insertion sort feature integrated

Click Update branch and locally git pull origin JLab2 before adding new commits.

Hi,
There has been  no activity in the last 20 days so I will close this PR.
Feel free to open it again if you have something to add.
Thanks!
Julia


[EDQ-418][EDQ-433] Created 'foreign key' notebook. Modified retrospective scripts.

Includes a notebook that calculates the percentage of missing/invalid 'person_id' fields for the following tables: condition_occurrence, procedure_occurrence, visit_occurrence, drug_exposure, measurement, observation.

Changed the metrics_over_time library to accommodate for this new metric (part of EDQ-425)

Validated the output for the resulting metrics_over_time script

Modified the 'non-weekly' scripts that produce the data quality metrics. Modifications typically involved removing the unioned_ehr prefix and intentionally excluding RDR from the output. This allows for 'apples-to-apples' comparisons of data quality over time.

@nishanthpp93 rock 'n roll.

@noahgengel I've approved it for now, but in the future try to create separate PRs for separate Jira tickets. We will be enforcing this soon.

read it loud and clear


Macos Performance Issue

I have a M1 mac air, which is more than capable of running base ship of harkinian at 120 fps (back when it supported it) but i cant hold a consistant 60 in 2ship or new soh. is this known?

Added some details to this in Discord, I'll put them here as well so it's documented.
When a game is crackling / slowing down :

Hovering the mouse to the top of the screen to activate the MacOS Menu Bar, performance jumps from crackly to a perfect 120fps.   Bizarre.
Game Mode does not activate, which should help with task prioritization / bluetooth latency etc

Will wait for the next version to compare!  But yeah, things are really crackly on 2021 14" M1 Macbook Pro, a machine that should be able to run this in its sleep.

Can confirm the same issues on my Macbook Pro M1 14' 16GB RAM


pkg_resources.DistributionNotFound: botocore

I am following steps are per link for setting up ElasticAlert
http://elastalert.readthedocs.io/en/latest/elastalert_status.html#metadata
OS: Redhat 6.7
Python 2.6
When I executing below command I get botocore error can you please check and help me on this
#elastalert-create-index
Traceback (most recent call last):
File "/usr/bin/elastalert-create-index", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/distribute-0.6.14-py2.6.egg/pkg_resources.py", line 2671, in
working_set.require(requires)
File "/usr/lib/python2.6/site-packages/distribute-0.6.14-py2.6.egg/pkg_resources.py", line 654, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/distribute-0.6.14-py2.6.egg/pkg_resources.py", line 552, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: botocore

You'll need to pip install botocore to make sure it's installed.  Looks like you have an open issue over at https://github.com/Yelp/elastalert/issues/677.  Let's track that issue over there.


Improve common data configuration check

This fixes #10498.
This was really simple, I thought that it was not implemented but actually an small check improvement is enough for this be easier.
@javitonino CR this, please.
cc @rporres

@javitonino present? refactor done.

:shipit:

Those unrelated NEWS :laughing:


Bluetooth: Mesh: Change the default value to Kconfig

Move default value of segment retransmit and timeout value
to Kconfig.
Signed-off-by: Lingao Meng<EMAIL_ADDRESS>

@jhedberg could you review?


'lastname is not defined' but lastname is defined

Challenge Waypoint: Use Bracket Notation to Find the Last Character in a String has an issue.
User Agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:
var firstName = "Ada";

var lastLetterOfFirstName = firstName[firstName.length - 1];

var lastName = "Lovelace";

var lastLetterOfLastName = lastname[lastName.length-1];

// Only change code above this line.
// We use this function to show you the value of your variable in your output box.
// You'll learn about functions soon.
(function(v){return v;})(lastLetterOfLastName);

@yourboyvic thanks for posting this issue!
You should always remember that JavaScirpt is case sensitive, so lastname is not the same as lastName. Replace lastname[lastName.length-1] with lastName[lastName.length-1] and you'll be able to pass the test.
Closing this issue. If you have problems with your code, please, post a message in our Gitter chat room first, thanks.

Please review the Guidelines for Contributing, thank you!.


Environment Variable Template vs File

I'm attempting to load environment variables from a file based on the instructions in the README. Unfortunately, unless I have the variables defined in the template the contents of the file are never recognized.
I would expect that the file approach would work regardless of whether the template has the environment configuration set.

I'm seeing a similar problem.  I attempted to load the environment variables a separate file per the docs, but wasn't able to get it to work.  I also attempted to load the environment variables by prefixing them before the command, and didn't have any luck either.

UPDATE:  I was able to get the environment variables to pull in from the host once I defined empty environment variables in the template.yaml that matched the names of my host environment variables.

UPDATE: I was able to get the environment variables to pull in from the host once I defined empty environment variables in the template.yaml that matched the names of my host environment variables.

@MoMenne can you show an example of how you got it to work, please?

@danilopopeye Sorry for not getting back to you sooner.  He's an example of what I did
`AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: |
Sends a text message as a reminder
Resources:
Place your project resources here (AWS Lambda functions, API Gateway) etc
SendText:
Type: AWS::Serverless::Function
Properties:
Timeout: 10
Runtime: nodejs6.10
Handler: index.handler
Environment:
Variables:
AUTH_TOKEN:
ACCOUNT_SID:
I then setACCOUNT_TOKENandACCOUNT_SID` env variables and it seemed to pull those two env vars in correctly.

I then set ACCOUNT_TOKEN and ACCOUNT_SID env variables

@MoMenne you mean SHELL env variables or using the env-vars.json file?

Sorry about that.  Right.  I set environment variables in the shell.
export ACCOUNT_TOKEN=xxxxxxx
On Fri, Oct 6, 2017 at 2:54 PM, Danilo Sousa<EMAIL_ADDRESS>wrote:

I then set ACCOUNT_TOKEN and ACCOUNT_SID env variables
@MoMenne https://github.com/momenne you mean SHELL env variables or
using the env-vars.json file?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-sam-local/issues/139#issuecomment-334854248,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANTMA6rbrhMLzlqDc6XorajTvlme_xLks5spoVlgaJpZM4PjjS5
.

--
Mike Menne
Technology Director
4811 Delmar Blvd, St. Louis, MO 63108
LaunchCode.org https://www.launchcode.org/  |  Facebook
https://www.facebook.com/launchcodestl/  |  Twitter
https://twitter.com/Launch_Code
<EMAIL_ADDRESS>314-252-8921 <314-477-9816>

This is expected behavior where SAM Local only parses environment variables that are defined in the SAM template. This is exactly how it will work when you push the Lambda to cloud. So as long as you define the env-var in SAM template it should show up in your function

@sanathkr - I feel like we should be an option to have a separate file for environment variables so we can choose whether or not to commit the file. As of right now we have to commit the env variables.

I found this from google when looking for a reference to using environment variables. According to the docs about sam local environment variables:
The Environment key defines the variables and the default values. One can use the --env-vars cli argument to define a json file to override the values.

@frob - the link to those docs is broken

Here is the updated link: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html
Move to the Environment variable file section.


Interop with async? (Tokio/Futures)

Hi -
Have you put any thought into how this could be used with an async IO model, such as with Tokio? Right now it seems like it's all implicitly synchronous, but perhaps it's just a matter of making the various functions take/return Futures?
Thanks

The big thing here is that this would also enable an implementation of something like dataloader, which Facebook has deemed the best practice for handling remote data fetching for a GraphQL query.

I've made a few attempts locally at integrating futures-rs into Juniper, but there are a lot of stumbling blocks that need to be overcome. There are also some nice properties the current synchronous implementation has that we'll probably lose, particularly around memory usage and heap allocations. This might not be a big issue, but I like the "you don't pay for what you don't use" mentality of Rust and C++ in general.
So, these are some issues that have prevented me from building this. There are probably more :)

Many AST nodes, particularly identifiers, use references to the original query string to avoid copying. Rust statically guarantees that no AST nodes will outlive the query string, and it's easy to reason about when execute just parses and executes the query synchronously.
GraphQLType::resolve* would need to return a Box<Future>, which means that all fields will cause extra heap allocations, even if it's just field like_count() -> i32 { self.like_count }. Now that I think about it, maybe you could change FieldResult to something like enum { Immediate(Value), Err(String), Deferred(Box<Future<Item=Value, Error=String>>) }.
The core execution logic would need to be transformed into something that joins multiple futures together and emits a result. No fundamental problem here, it was just a very difficult programming task :)
The futures-rs crate changing around a bit while working on this. This shouldn't be the case anymore, but I would be weary releasing Juniper 1.0 while depending on a pre-1.0 release of futures-rs.

I've been dabbling with futures-rs and Tokio in another project that I'm working on, and the general feeling I get is that's it's pretty unergonomic to use. I've stumbled upon cases where I couldn't break out a big and_then callback to a separate function because of various reasons: either the type was "untypeable" - i.e. containing closures, BoxFuture requires Send despite the name, Box<Future> did not work because of lifetime issues, and even when switching to nightly and returning impl Future there have been cases where returning lifetimes have been an issue.
Despite all of this, I still think this a feature we want to have! :) However, there are some constraints on the implementation:

Minimal impact on the non-async path. Ideally, Juniper should only create a new heap-allocated future for each object to resolve, not each field. Scalars should not cause any overhead at all.
No reliance on unstable compiler features. Juniper should not require a nightly compiler.

This became a long response with little amount of actual content :) I might open up a branch to do some work on this, but it's been hard trying work in incremental pieces since it's such a cross-cutting change.

Juniper should only create a new heap-allocated future for each object to
resolve, not each field.

What about fields that require an expensive calculation that you'd rather
define with an async method? After all, fields are how you get objects in
the first place. :-)
Den ons 12 juli 2017 11:37Magnus Hallin<EMAIL_ADDRESS>skrev:

I've made a few attempts locally at integrating futures-rs into Juniper,
but there are a lot of stumbling blocks that need to be overcome. There are
also some nice properties the current synchronous implementation has that
we'll probably lose, particularly around memory usage and heap allocations.
This might not be a big issue, but I like the "you don't pay for what you
don't use" mentality of Rust and C++ in general.
So, these are some issues that have prevented me from building this. There
are probably more :)

Many AST nodes, particularly identifiers, use references to the
original query string to avoid copying. Rust statically guarantees that no
AST nodes will outlive the query string, and it's easy to reason about when
execute just parses and executes the query synchronously.
GraphQLType::resolve* would need to return a Box, which
means that all fields will cause extra heap allocations, even if
it's just field like_count() -> i32 { self.like_count }. Now that I
think about it, maybe you could change FieldResult to something like enum
{ Immediate(Value), Err(String), Deferred(Box<Future<Item=Value,
Error=String>>) }.
The core execution logic
https://github.com/mhallin/juniper/blob/master/src/types/base.rs#L291
would need to be transformed into something that joins multiple futures
together and emits a result. No fundamental problem here, it was just a
very difficult programming task :)
The futures-rs crate changing around a bit while working on this.
This shouldn't be the case anymore, but I would be weary releasing Juniper
1.0 while depending on a pre-1.0 release of futures-rs.

I've been dabbling with futures-rs and Tokio in another project that I'm
working on, and the general feeling I get is that's it's pretty unergonomic
to use. I've stumbled upon cases where I couldn't break out a big and_then
callback to a separate function because of various reasons: either the type
was "untypeable" - i.e. containing closures, BoxFuture requires Send
despite the name, Box did not work because of lifetime issues,
and even when switching to nightly and returning impl Future there have
been cases where returning lifetimes have been an issue.
Despite all of this, I still think this a feature we want to have! :)
However, there are some constraints on the implementation:

Minimal impact on the non-async path. Ideally, Juniper should only
create a new heap-allocated future for each object to resolve, not
each field. Scalars should not cause any overhead at all.
No reliance on unstable compiler features. Juniper should not
require a nightly compiler.

This became a long response with little amount of actual content :) I
might open up a branch to do some work on this, but it's been hard trying
work in incremental pieces since it's such a cross-cutting change.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mhallin/juniper/issues/2#issuecomment-314710027, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAGP7F4ZyFLEmO6FWhUxK2m_aCAzKzLks5sNJPqgaJpZM4K0V7G
.

I maybe expressed myself a bit ambiguous there - if a user defines an async field, than it should obviously allocate a future for that field. I meant that no futures should be allocated for synchronous fields.

I agree that tokio is really cumbersome to use on stable right now.
impl trait helps a lot and async/await will help even more, but that's a long way from being stable.
Regarding the return type, an enum sounds reasonable. And with impl Trait it would actually also be very convenient to write.

@mhallin @theduke In your opinion, how feasible is it to ship this?
Lack of async support is currently the thing that prevents me from using Juniper for more projects. I appreciate that all in all it would be a huge refactor, but maybe this is something that could be shipped in increments?
Would you be motivated to help me get the PRs reviewed and merged if I started on this? Or do you feel like it's not the right time for this feature?

@srijs Just out of curiosity, do you already have a futures-based Rust codebase that you want to expose with Juniper, or are you looking at GraphQL servers with async support in general?
To be honest, the more i work with Promises/futures-rs and other concurrency systems such as Erlang/Go, the less interested I am working on this. Just compare the work with integrating futures-rs with say Rayon: it would be trivial making the execution logic parallel without running into any of the problems I listed in my first reply here. If the Rust community's async efforts were directed to something with that kind of API, I'd be more interested.
That said, I will of course help you out if you decide to tackle this! I'm not sure even where to begin, but keeping AST nodes alive for the duration of the entire execution without using references with lifetimes is something that needs to be solved first. That might require putting all nodes under Arc, which is kind of unfortunate...

@mhallin I do have existing Rust codebases (that use tokio extensively) where I would love to be able to use Juniper.
It seems with generator functions/coroutines we're moving into the direction you're describing, but  of course that will not be usable in stable Rust for quite a while. So while I agree that it would be simpler by leveraging coroutines, I don't think it's practical to wait. At least personally I'd like to see Juniper support async before coroutines land in Rust stable.
I have started to play around with the AST/parser parts, to use reference-counted string objects as you suggested (although I'm not 100% they would need to be Arc; Rc might be sufficient). Servo is using tendril for this purpose, but it seems rather unstable at the moment. I wonder if there is a crate like Bytes just for text...

Hi, can someone give me an update on this, I recently started a project using Juniper but the data to fulfil the GraphQL request comes for an external web api so I need to take advantage of futures for performance. Is futures support being worked on, if so by when will it be ready? if not, how do people go about doing what I am trying to do without using async I/O.

Mostly in response to, or in addition to, @mhallin's comment above, here are some thoughts based on the recent advances on the rust futures & async front, especially drawing off of these sources:

http://aturon.github.io/2018/04/06/futures2/
https://boats.gitlab.io/blog/post/2018-04-06-async-await-final/

heap allocations
With conservative impl trait and, less importantly, universal impl trait slated for rust 1.26, we could experiment with field signatures having a return type of impl Future<...> as opposed to Box<Future<...>>, to save on heap allocations (we'll see when it actually lands on stable).
Per some comments above about reference issues even inside of a return impl Future<...>, perhaps the upcoming Pin API will be useful.
AST sharing
With futures-rs 0.3, the Pin API will be leveraged for futures, which will allow reference sharing between futures &c. I haven't looked at the code in Juniper where the AST sharing is taking place, so the Pin API may or may not make any difference. Need to investigate a bit more.
futures stability
futures-rs has just recently gone through a pretty massive revamp with the 0.2 release. Apparently the design is more long term, with a 0.3 coming soon:

we anticipate a 0.3 release relatively soon. That release will set a stable foundation for futures-core, after which we can focus on iterating the rest of the stack to take full advantage of async/await!

So, once 0.3 lands, that may be a good time to start experimenting more aggressively.
Thoughts?

FYI, futures 0.3.0-alpha.1 has been released.

By the way, I'm not sure if this has been proposed discussed, but other libraries implemented support for dataloader by using thunks and a breadth-first resolution, without the need to introduce async code into the graphql executor.
The golang graphql library PR explains the approach and contains other references as well: https://github.com/graphql-go/graphql/pull/388
Would this be worth pursuing? Should this be added as a separate issue?

Will this async juniper comming soon? :joy_cat:

I looked into how this could be implemented a bit. My understanding is that we would need to return futures instead of results in a few places if we want to keep the changes minimal, for example resolve_field. This is a good example because it is a trait method, so we currently would have to box the returned future, which means one allocation per field. As far as I understand, returning unboxed futures from trait methods is blocked on GAT and existential types/async trait methods.
With async/await and async methods in traits, it looks like it would be possible to generate code for async resolution without boxing a future per field, but it blocks this feature until at least a few months after async_await lands (existential types are not working at the moment, even on nightly, as far as I know).

True, we will also need to figure out how to deal with the query AST. We will need to either put it behind an Arc or clone the sub-AST when neccessary.
The boxing issue may not be that bad if we can come up with an API that's transparently upgradable to a non-boxing solution, which should be possible.
Also, considering that async resolvers will only be necessary for fields that do DB or api requests the boxing should not be significant.
The bigger challenge may  be tuning the tokio runtime to handle the workload correctly. (EG: always use blocking for non-async resolvers, or having a dedicated thread pool for the sync resolvers, etc.

@theduke yea, I would imagine that cloning an Arc would be more performant than cloning the query AST or subsections thereof (could be wrong on that though).
Agreed on the boxing issue.
Do you mind expounding a bit on "possibly restricting the concurrency for certain fields"?
As far as tokio runtime tuning, using blocking and such, I do have a few thoughts for discussion:

I don't know if we should do anything other than strongly encourage folks to understand that if they are not using an async return value (a future) then they must be very careful not to block the runtime.
For folks that understand this, then the overhead of delegating to a background thread via the blocking API may be overkill and will add some performance hits (as minimal as they may be).
For folks that do not understand this, they can still easily block the runtime even in async handlers, by using thread blocking locks or making synchronous network calls. So arguably the only real solution is to teach and help folks understand.

So, perhaps a nice balance may be to teach non-blocking in the docs fairly well, include examples of how to use the blocking API for situations where such a thing is needed (eg, dealing with legacy crates), and then just encourage folks not to do any blocking operations when resolving scalar values &c.
Also, +1 on futures 0.3. The compatibility layer is pretty solid as well, so that shouldn't cause any issues.

Futures just stabilized in 1.36 - https://github.com/rust-lang/rust/pull/59739 (as 0.3)
Are we confident we now have an API to start building out juniper resolvers as Futures?

@jkoudys & @theduke looks like with the 1.36 stabilization & the fact that tokio has interop and a strong compatibility layer between Futures 01 and std futures, now might be a great time.

I don't think await will be all that useful for the code in Juniper itself, since we will mostly be dealing with joins and a few manually written futures, so I think it's fine to go ahead now.
The big questions we need to answer are:
the FieldResult type
We certainly want to allow a mix of async and non async resolvers. EG it makes no sense to resolve fields that are just a struct field access with a future. In that light, the FieldResult type will have to become an enum that can contain both a sync response and a async one. And both the macro auto-generated and other plumbing  will have to deal with resolving sync resolvers directly and async ones via a future, and then wrap everything in a future when appropriate.
Runtime
There is also the question of how and when a threadpool will need to come into play. Preferably we would remain runtime agnostic, and not rely on anything provided by tokio et al. I'm not sure that's possible but it probably should be. But that would also mean no threadpool and letting the user decide what things are expensive and need to be delegated to a threadpool.
With futures we can now add timeouts to resolvers in a sensible way, but that might require a specific runtime.
If we need a runtime, we should wait until tokio is refactored to `0.3.
Concurrency limits
Currently everything happens one after the other, synchronously.
With futures we could have a lot of work items fired off simultaneously.
Just imagine a API request that fires of 10 database requests and 5 network fetches.
Currently, they happen one after the other in a thread.
With futures we might have 30 requests arriving at once, all started at the same time, leading to 300 db  and 150 network requests. And new requests arriving all the time. I guess this could overwhelm the runtime pretty quickly and cause a ballooning of workload because every single request takes longer.
So I think some limiting system will be essential. Eg: no more than X actively processed requests at a time. Additional ones get put in a queue and are only resumed once another request finishes.
Also timeouts come into play here.
I think this will be trial and error. Let's whip something up and see how it works out with benchmarking. But it is something to consider.
I'm pretty certain the limit/queuing will need to happen.
Introspection / tracing
I don't know much about the tracing functionality of tokio (https://github.com/tokio-rs/tokio/tree/master/tokio-trace), but this might be a very nice thing to have to provide insight into your workload and point out slow / problematic resolvers. Investigating this would be nice.

The threadpool part could be hard to abstract, so it would make sense to leave setting up the runtime and handling blocking tasks to users of the library in my opinion. For example the tokio threadpool handles blocking tasks differently from most others (the blocking function).

But that would also mean no threadpool and letting the user decide what things are expensive and need to be delegated to a threadpool.

@theduke & @tomhoule yea, I definitely agree. As long as their blocking/background threadpool operation returns a future, then we should be good. We shouldn't have to do anything else to handle it.
As far as Concurrency Limits:

on one hand I'm tempted to say that we should allow users to horizontally scale out their instances as they need to on their own. Allow them to load balance traffic across their instances, and then we just operate on a best effort basis. If we impose limitations on our side, then we might actually inhibit scalability for folks that have other ways of managing this sort of stuff (like using kubernetes & HPA).
on the other hand, perhaps we could introduce some sort of complexity limitations. Something that is opt-in. A few other GraphQL frameworks in other languages have something like this. We could offer depth complexity limits and maybe offer parallel resolution limits. However, once again, the tokio runtime can handle a pretty massive number of concurrent futures/tasks. Perhaps introducing a limitation is an over-optimization here which might actually dwarf the capabilities of the underlying system. Maybe something optional would be best.

Thoughts?

For concurrency limits, does juniper already batch requests within a query?  If not, it might be useful to see how the Sangria graphql server does it with what they call deferred values, deferred resolvers, and fetchers:
https://sangria-graphql.org/learn/#deferred-value-resolution
https://sangria-graphql.org/learn/#high-level-fetch-api

I'm all for limiting the number of DB + network queries it could fire off, but is it really juniper's job to worry about this? If I'm making DB queries, then I have a DB connection pool that my resolver functions will query from. I could limit that size, which means that those 300 DB requests will wait on the pool, so while I have 300 requests I could still only have 10 or so active DB connections. This is  exactly what I want, since it maximizes how much can run in parallel and could scale up really easily.
And do we really need an enum of FieldResult? We already say impl IntoFuture for Result, because anything sync can be treated as async, and then we can easily pass results back in closures in a chain of and_thens, e.g. .and_then(|i| Ok(i + 5)). If there's aFieldResult, why not a FieldFuture, that impl Future, so we could impl IntoFuture for FieldResult? You could still use any non-blocking FieldResult-returning functions the same as before.

Relevant: https://www.reddit.com/r/rust/comments/bi2puk/help_rate_limiting_in_async_rust_and_tokio/

And do we really need an enum of FieldResult?

We definitely need a efficient way of dealing with sync results like a simple field access.
Just consider a list query that returns 100 items and gets 10 fields for each item. Most objects will have mixed sync and aynsc resolvers (eg a BlogPost with a bunch of sync fields and a author async field).
A naive approach would mean that for each post, we need to join over the 10 fields, which means allocating a Vec and boxing each of the field result futures. That's a lot of redundant allocations.
Sync values would be represented by an immediately ready future (https://docs.rs/futures/0.1.26/src/futures/future/result.rs.html#13-15) which has little overhead, just an extra Option.
We can check if the future is ready by calling poll right in the resolver and prevent the boxing and joining in the generated code. The compiler should inline the poll call and leave us with relatively little overhead, if the poll call is inlined. And it still leaves us with a redundant  self.inner.take().expect("cannot poll Result twice").map(Async::Ready) which can till add up, and there's no guarantee that this will be inlined since the expect produces quite a few instructions.
I think a enum will be the better solution.

but is it really juniper's job to worry about this?

Probably not, and hopefully not.
But it's the classic problem in network services: let's say your server can maintain equilibrium with 10 db pool connections and serving 20 queries per second. Now the query load doubles, and you have a lot more code fighting over those DB connections, driving up latency for each query. Which leads to even more active queries as more and more come in, resulting eventually making the server totally unresponsive.
At some point, you really want to reject additional requests instead of leaving them hanging for a minute.
Of course there are plenty of ways to deal with it, load balancers, reserving a single db connection for each query, etc. It's just something we should keep in mind and benchmark how tokio will deal with it under high load.
Wrapping the execution into some kind of rate limiter should be easy, if necessary.

One thing to consider is the path of another project: actix-web. They are preparing for their 1.0 release, and have actually removed their internal Async enum, going with a futures-first interface and using generics to differentiate between the sync and async cases. Going with an enumeration means that every field access incurs the cost of branch during runtime for every request, regardless of whether they use both. I think that this can be better handled with a trait that has different impls for the sync and async cases (and you can probably have things returning impl Future, so no boxing necessary, again, look to actix-web for inspiration). This way you get the zero-cost and only-pay-for-what-you-use benefits.
Worrying about the cost of allocating a Vec in this case seems to me like a premature optimization. If allocating a Vec means that I can handle double the requests per second per instance, that might be a tradeoff I'm happy to make. This cost seems outweighed by other bigger costs, e.g. no way to cache parsed and validated queries means that every request must be re-parsed and re-validated, even for repeated requests with the same query text (albeit with different variables).
On the rate-limiting/balancing, that seems outside of the scope of juniper. There are other tools that do it very well, and tower-qos is a great example, using tokio-sync under the covers. If someone wants to limit the number of concurrent query executions, they can do that at the layer above juniper. If they want to limit the number of concurrent operations on an underlying service, they can do that at the layer below. Timeouts or other policies to apply backpressure feel like cross-cutting concerns, so I see these as possible extensions, but not part of the core concern for this library. Everyone is going to want to configure it their own way, and my belief is that adding an core internal concurrency limit is more likely to lead to suboptimal configuration by users in the real world than the default of letting tokio handle it.

@neoeinstein the problem for GraphQL are the object level resolvers. They need to resolve each field and return the aggregated response. Returning impl Future doesn't help there, you still have to put all individual field futures into a Vec of boxed trait objects and call join_all on them, which means dynamic dispatch etc.
Now like I said this can be prevented by manually polling the future to see if it's async, and that probably won't be much worse in terms of overhead than a enum if the poll is inlined, but it still involves a branch even for an immediately ready future, and actually a bit more code than just a enum match. (See my last post).
In the end this is all transparent to the user and an implementation detail that can be changed up pretty easily, users only have to return something that can be converted into a field result.

@neoeinstein The actix-web is exactly the approach I was imagining. impl Future has really cleaned out a lot of cases where Boxing was necessary, and is doubly good because of the sync case with Results you describe.
I'll always want rate-limiting, but I have a hard time thinking of a case when I'd want my graphql lib to be the one doing it. Often I'll have resolvers that query psql, redis, and/or a couple 3rd-party RESTful services. Seems mad that I'd leave my redis + psql sitting idle, when there are queries that only need those to resolve, because I have other queries waiting on a RESTful service that's slowed to a crawl. Plenty of places where I could limit that, but no idea why juniper would be involved in the decision. In the case where I'd want to reject the query, I'm happy to have my pool's request for a connection throw back the Err for me.

So, it sounds like folks are pretty on-board with the impl Future pattern, and given that 03 futures don't require an error value to be present, just an Output which could be whatever, we have a few nice patterns that we could try out. Perhaps something like the following would be nice.
First from a Juniper user's perspective:

Resolvers may return all of the standard scalar values which are available in Juniper today as well as results and options as normal.
Resolvers may return Futures of some output type which falls under the category defined above.

From the Juniper framework perspective:

Standard object derivations behave pretty much the same way. Fields will be resolved as scalar values &c.
Resolver functions will need a little love. Perhaps, in order for proper Futures03 interop, we will need to define a type, say JuniperFuture, with some From impls.

If the resolver returns a scalar value, then the From impl will just create a resolved future of the returned value.
If the resolver returns a future, then the From impl will simply wrap that future for seamless interop.

This has obvious implications for code generation and the executor. As far as the executor is concerned, perhaps we use something like FuturesUnordered to be able to poll all the futures of a top-level query, and then also be able to push additional futures into the pool for resolution as more fields with resolvers are traversed.
The benefit of taking an approach like this is that the end-user doesn't need to worry about wrapping their resolved values in an enum or anything like that. Things will stay pretty consistent.
Any and all feedback welcome. I am not a maintainer of this crate, so there are probably plenty of oversights here on my part. Thoughts?

Just a little update: making the AST owned/shareable requires a lot of changes. Since it needs work anyway, it's a good opportunity to switch the parser to graphql-parser. To make that happen it needs a little work though.
I'm working on one element ( #19 ), but it also needs quite a few improvements to the spanning infrastructure for error reporting. That's not too challenging but quite a bit of work, if anyone has some spare time..
Regarding the resolver topic:

So, it sounds like folks are pretty on-board with the impl Future pattern

I'm not on board with fully moving to Futures only internally.
First, we don't want to force users to go async, juniper should continue to work in a sync world and provide tokio integration with a feature. (sensibly a default feature, I reckon)
This won't be too difficult to accomplish, mostly internal plumbing.
Note that regarding the discussion around enums, I was never talking about a user-visible enum like actix-web had, just on an internal level. We would need this anyway since otherwise we would introduce a lot of extra work due to the nested nature of how the resolving internally works - even for completely sync resolving like for types with #[derive(GraphQLObject)].

Looks like await is on its way to stabilization:
https://boats.gitlab.io/blog/post/await-decision-ii/

I'm still learning futures and async await...if we go impl Future route, that means we can't have a dyn trait object without boxing / using futures::future::BoxFuture.

We will need to use BoxFuture.
Traits can neither have async functions nor functions that return -> impl Trait. Enabling those is blocked on existential types in traits, which doesn't have an RFC yet and is quite a ways off.
Even if it was ready, we probably wouldn't use it across the whole query: It could create a huge generator that blows the stack or just slows us down because it get's so large.
Existential type aliases do have an RFC, which is currently blocked on a few other things though and also not close to being ready.
What we could do now is change the way we generate code for objects  and use async/await inside GraphQLType::resolve by skipping GraphQLType::resolve_field.
This is somewhat difficult due to the dynamic nature of the resolvers though: we'd have to create no-op futures for all fields that are not included, and then use something like try_join! on all futures and collect the values into an object. This could also become problematic for the stack: what if a object has a lot of fields or ones that result in large generators...
It's an interesting area to explore, but for now I think sticking to boxing and helpers like FuturesUnordered is the right choice.
Since most of this is internal plumbing, we can hopefully play around with improvements in the future.

I just noticed: where we definitely can use await though: at the top level of resolving a query.
This should enable us to use a borrowed AST and Context in all nested resolvers, which is great.

FWIW this is what GraphQL.js does:
"The resolve function can return a value, a promise, or an array of promises"

Any issues that need to be fixed before this work can go in? Are we waiting on anything in the Futures spec, tokio, etc. or is this ready to be built? Is it going to be BoxFutures? Happy to send in some PRs myself, but I need to know the design direction.

I have a working branch. I'll try to get a PR up in the next 1-2 weeks.

Hey @theduke, I couldn't find anything on your fork: is there any chance we could have early access to that work of yours? It'd be really really helpful even if it's not all polished yet.

Seems like @theduke may have abandoned us. We should probably consider this unimplemented again.

Not abandoned, but an unexpectedly long vacation.
I'm back this weekend and I should have enough time to at least post a WIP PR.

Things are quite a bit better now since multiple reference support has landed.

Almost there!
Expect a working branch tomorrow.

Alright, here we go.
I finished a very rough prototype.
It has a bunch of issues which are listed below, so don't use this for anything serious yet.
It is published as the async-await branch.
Caveats

feature support is unfinished: unions and interfaces don't work yet, and I'm sure a few other impls are missing
only async fn resolvers are supported right now. this will be extended to resolvers that return -> impl Future soon
the async_closure feature is required in addition to the async_await feature. This will be fixed soon
completely unoptimized, there is a lot of redundant boxing right now
Some of it will be reduced soon, but we really need a few accepted (impl trait type aliases) and up in the air (impl trait in trait methods) language features
for a stable release, the trait architecture will need quite a bit of refactoring
this shouldn't impact the public interface much though, if you are not implementing GraphQLType yourself

Example
I plan to fix up hyper integration with hyper master and tokio 0.2 tomorrow.
So far I only updated the warp integration, based on tokio 0.1, so beware of the compatability and integration issues.
There is a complete usage that shows how to use it here.
There is an example resolver that uses reqwest to fetch a remote url in a async function, but it looks a bit awkward due to the compatibility layer.
How to help

PRs against the async-await branch are very welcome
if you find issues that are not yet mentioned, please report them with a reproduction repo

Benchmarks:
Sadly we don't have a benchmark suite yet.
For improving and tuning the performance, I will really need some good, realistic benchmarks to work with.
I will add a benchmark harness to the branch tomorrow, and would then be very happy about contributed benchmark code that is extensive and realistic with a large schema.
The benchmarks should not use real external dependencies like a DB/redis/etc, but should mock the behaviour in a realistic fashion. (eg with randomized delays using tokio-timers).

Awesome, hoping to take a look at it later this week!

Initial benchmarking indicates that the async machinery overhead is below 10%, even with the current suboptimal implementation.

I've updated juniper_rocket to support async-await & the async branch of rocket here if anyone's interested.

@obmarg awesome, happy to get a PR against the async-await branch.

Done: https://github.com/graphql-rust/juniper/pull/417

It looks like with the introspection query async might be on par with sync:

Yeah I also have a bunch of benchmarks and the overhead is monomalt, even with the current suboptimal implementation.

Ah but the introspection query is a bad benchmark for this since the introspection lookup can (and do) skip the async logic

@theduke I plan to work on this later in the week / this weekend. Can you taskify things that you aren't planning to work on so we don't duplicate work?

@LegNeato I'll only have time to continue next week so the most important tasks right now:

Implement union and interface support in the macros. This is rather awkward at the moment (which is why I haven't done it yet) because they still use macros instead of a proc macro. The best path would probably be to rewrite them as proc macros, which we wanted to do for a while anyway.

Write tests

Is there anything I can do to help with this project? I have a personal and professional interest in moving this forward, and potentially some time to spend on it.

Hello @bsmedberg-xometry , long time no see! Would love help as we have been a bit swamped lately. This work is happening on the async-await branch.
The ideal short-term help would be to implement union and interface support there. That entails porting logic over from the macros into proc_macros. It's a bit hairy...not because the work is hard but because macros are annoying to work with.
There is a WIP progress PR for async subscription support but it isn't touching the union and interface stuff so you shouldn't really conflict.

Is there a need for more examples? I was able to successfully use the async-await branch with tide: https://github.com/dfrankland/tokei-aas
I'd be happy to add an example using that as the basis.

Small update: I refactored the graphql_union! macro to a juniper::union proc macro.
graphql_interface! will follow soon.
No async support yet, that's the next step, but a fairly small one.

@theduke Will graphql_union! be deprecated? I'm currently using it in juniper-from-schema.

Yes, I'm replacing all the macros with proc macro variants.
The proc macros have equivalent functionality.

Just an additional update: I've been traveling a lot for the past few months, but starting this Saturday, I will have 2 weeks downtime.
I will push hard to have a alpha release with async await by next Friday.

Is there a need for more examples? I was able to successfully use the async-await branch with tide: https://github.com/dfrankland/tokei-aas
I'd be happy to add an example using that as the basis.

@dfrankland what would be extremely helpful is building out a full-featured benchmark with a complex schema in https://github.com/graphql-rust/juniper/tree/async-await/juniper_benchmarks . ( this is currently very rudimentary )

@theduke Seeing as we are telling folks to base their PRs on the aync-await branch, let's just get CI passing and merge that into master so we can jam on it. I don't think we'll do another sync-only release and if we need to we can re-branch at the pre-async point.

:+1: , that's what I wanted to suggest.

Should this issue not be closed then?

I believe that it wouldn't make sense until async support is in a release on crates.io

Interfaces still don't work and we need to rip out the sync code. Keeping this open until those are done and a release is made.

Hey! Thanks for the awesome work! I managed to integrate master with dataloader-rs! I noticed that the batch loading isn't working on deeper levels of the tree though. The following example will make it clear:
query {
  recipes {
    id
    name
    ingredients {
      ingredient { name }
      amount
    }
  }
}

This will result in the following db queries (all of which are done by dataloader batch loading functions).
SELECT name FROM recipes
SELECT ingredient_id, amount FROM recipe_ingredients WHERE recipe_id IN ($1, $2, $3)
parameters: $1 = '3', $2 = '1', $3 = '2'
SELECT name FROM ingredients WHERE id IN ($1)
parameters: $1 = '1'
SELECT name FROM ingredients WHERE id IN ($1)
parameters: $1 = '2'
SELECT name FROM ingredients WHERE id IN ($1)
parameters: $1 = '4'

So the the first level of resolvers is batched but the third isn't. This could just as well be a bug in dataloader-rs or even more likely just my incompetence, but i thought i'd post it here if someone has come across this and solved it already.

So you have a link to the code?

@LegNeato sure: https://p.jokke.space/5l2FU/

With #682 landed, we're fully async compatible now!

@tyranron Are there any example?

@wongjiahau check the book on master and examples in the repository.

I found the fix by using juniper = { git = "https://github.com/graphql-rust/juniper", rev = "68210f5"} instead of juniper = 0.14.2.

crates.io has been updated with juniper's async support, sorry for the delay. Any future bugs or api changes can get their own issues.
Note that we still support synchronous execution via juniper::execute_sync.
Thank you to all the contributors who made this possible, especially @nWacky, @tyranron , @theduke , and @davidpdrsn  🍻 🎉 🥇


Minor corrections to READMEs

Relates to https://github.com/grahampugh/jamf-upload/issues/106.

Many thanks for this thorough review!


⚠ Mobitopup has degraded performance

In 1f0587c, Mobitopup (https://napcard.voz.tech) experienced degraded performance:

HTTP code: 200
Response time: 14706 ms

Resolved: Mobitopup performance has improved in ae4469d.


TODO: Syncing new debian packaging files with master branch

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/10596773-todo-syncing-new-debian-packaging-files-with-master-branch?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github).

https://github.com/kivy/kivy/issues/3593


Array index out of bound when testing

In python/train_snli.py , line 246
The code gives index out of bound error, because results is a tuple with 2 elements.
     results = evaluate_final(classifier.restore, classifier.classify, [test_matched, test_mismatched,     test_snli], FIXED_PARAMETERS["batch_size"])
     logger.Log("Acc on multiNLI matched dev-set: %s" %(results[0]))
     logger.Log("Acc on multiNLI mismatched dev-set: %s" %(results[1]))
     logger.Log("Acc on SNLI test set: %s" %(results[2]))

I guess what you mean is the following:
     results = evaluate_final(classifier.restore, classifier.classify, [test_matched, test_mismatched,     test_snli], FIXED_PARAMETERS["batch_size"])
     logger.Log("Acc on multiNLI matched dev-set: %s" %(results[0][0]))
     logger.Log("Acc on multiNLI mismatched dev-set: %s" %(results[0][1]))
     logger.Log("Acc on SNLI test set: %s" %(results[0][2]))
     logger.Log(" correct count | total by length: %s" %(results[1]))

I changed it and it works fine on my machine.
Thanks.

I get same error for train_mnli.py also. However, my issue was that I had to uncomment some lines which replaced test files with dev files (lines 300-303).


OpenMP examples freeze for 30 seconds and cause kernel soft lockup

When I build and run some of the examples that get installed into /usr/lib/aomp/examples/openmp, they execute correctly but then freeze for about 30 seconds before returning to the shell prompt. For example:
$ time make run
Makefile:27: AOMP not found at /home/jayfoad2/rocm/aomp
/usr/lib/aomp/bin/clang -O3 -target x86_64-pc-linux-gnu -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx900    reduction.c -o reduction
./reduction
The result is correct on target =<PHONE_NUMBER>00!

OS: Ubuntu 18.04
CPU: AMD Ryzen 9 3900X
GPU: Sapphire RX VEGA 64 Nitro+ 8GB

Similar freezes have been reported before: #45, RadeonOpenCompute/ROCT-Thunk-Interface#52

[AMD Official Use Only - Internal Distribution Only]
Jay and I have been chatting over teams on this.   ROCm is not installed. So I suggested Jay install 3.5.  Also, @Chesterfield<EMAIL_ADDRESS>might have suggestion on how to run without installing ROCm.
From: jayfoad<EMAIL_ADDRESS>Sent: Monday, June 15, 2020 10:36 AM
To: ROCm-Developer-Tools/aomp<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: Re: [ROCm-Developer-Tools/aomp] OpenMP examples freeze for 30 seconds and cause kernel soft lockup (#111)
[CAUTION: External Email]
Similar freezes have been reported before: #45https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FROCm-Developer-Tools%2Faomp%2Fissues%2F45&data=02|01|ron.lieberman%40amd.com|320c851a651343cd4cfd08d81141dbf9|3dd8961fe4884e608e11a82d994e183d|0|0|637278321848695635&sdata=4pJeiqM%2BO5xq8AoXHouZACvsBnZKb2%2F%2BHH6i7T5kRGI%3D&reserved=0, RadeonOpenCompute/ROCT-Thunk-Interface#52https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCT-Thunk-Interface%2Fissues%2F52&data=02|01|ron.lieberman%40amd.com|320c851a651343cd4cfd08d81141dbf9|3dd8961fe4884e608e11a82d994e183d|0|0|637278321848695635&sdata=iWMg7mZYB64ea%2FfQ0wP%2B%2BNMT09E8UwfUMTkoDXXykkA%3D&reserved=0

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FROCm-Developer-Tools%2Faomp%2Fissues%2F111%23issuecomment-644209207&data=02|01|ron.lieberman%40amd.com|320c851a651343cd4cfd08d81141dbf9|3dd8961fe4884e608e11a82d994e183d|0|0|637278321848705590&sdata=o33%2BGtkXlWUKLDB1ai%2B%2BPkea487L50fSVycTyKx8rzs%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD3EYZZCASENCJUKVDX6YY3RWY5XPANCNFSM4N6I3VPA&data=02|01|ron.lieberman%40amd.com|320c851a651343cd4cfd08d81141dbf9|3dd8961fe4884e608e11a82d994e183d|0|0|637278321848705590&sdata=cJbcMrjCQGM8jiTQSjcr2tavnwXNpMhiDeRaN7WB9J0%3D&reserved=0.

I don't have much to add here. I build aomp from source on a system which doesn't have rocm or rock-dkms installed.
I'm inclined to blame kfd here on the basis that userspace shouldn't be able to hang the kernel for that long whatever it does. I'm running the same kernel and have some similar hardware available (570X/gfx906), will see if I can reproduce.

I believe the issue stems from using the amdgpu-pro driver instead of the rock-dkms (kfd) that comes with ROCm.

Have you installed ROCK-DKMS or are you using the kfd that comes with the linux kernel?

I've never heard of ROCK-DKMS (I just installed a big aomp package from here) so I guess the latter.

Jayfoad, Please install rock-dkms from ROCM 4.1 and AOMP 13.0-2 .
Ethan, do we have a smoke test for this?

Our examples are working now with 13.0-2.   Reopen or create new issue if this fails again.


⚠ Empornium has degraded performance

In 3376d4f, Empornium ($EMP) experienced degraded performance:

HTTP code: 500
Response time: 1488 ms

Resolved: Empornium performance has improved in cc95707 after 52 minutes.


Move narrowed_msg_list to message_list.js

Automated message from Dropbox CLA bot
@hackerkid, it looks like you've already signed the Dropbox CLA. Thanks!

I think maybe you should initialize message_list.narrowed somewhere?

@timabbott Updated the PR.

@hackerkid a bunch of the changes in this PR has the indentation messed up -- can you fix those?

@timabbott I have update the PR.

Merged, thanks @hackerkid !


Added New Feature In the App.

Added New Feature In the App.
Added the Universal Home Button in all Screens in app,When we press on the app icon that is present on top of every screen, it navigates the user to the HomeScreen.

@KartikChippalkati Please post some screenshots

The above image shows that "The App Logo image which is present in the top left of the screen acts as a Home button for this Screen".
When we press on that logo image,the below homeScreen appears.

n.

@KartikChippalkati The screen in the second screenshot should be shown only when the user is not logged in.


Gstreamer component error

Home Assistant release with the issue:
0.95.4
Operating environment (HassOS/Generic):
HassOS
Supervisor logs:
Error while setting up platform gstreamer
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 126, in _async_setup_platform
SLOW_SETUP_MAX_WAIT)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
return fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/gstreamer/media_player.py", line 31, in setup_platform
from gsp import GstreamerPlayer
File "/usr/local/lib/python3.7/site-packages/gsp/init.py", line 11, in
import gi  # pylint: disable=import-error
ModuleNotFoundError: No module named 'gi'
Description of problem:
When adding gstreamer to the configuration.yaml on HassIO, the above error is generated.
Seems the module gi is not being installed

Yes, we have no gstream installed. Please use vlc and remote-vlc add-on

"VLC currently not supported with Hass.io"


Expose IR stream as a webRTC video stream.

Camera Model: D415
Firmware Version: <IP_ADDRESS>
Operating System & Version: Windows 10 Enterprise 1809 |
SDK Version: 2.38.1
Language: JavaScript
I am building a webRTC sample application and would like to consume IR stream when lighting conditions is low.  This article illustrate how to consume the depth sensor https://github.com/intel/depth-camera-web-demo/blob/master/depth-camera.js .  Is it possible to consume IR stream and any code samples?

Hi @rajrajeswaran  Do you require further assistance with this case, please?  Thanks!

@MartyG-RealSense I apologize for not closing this question.  I will do it now.

@rajrajeswaran  It's no problem at all.  Thanks very much for the update!