Can't verify Blue Box apt mirrors

Here's the ansible trace when attempting to run an allinone with Ursula:
TASK: [apt-repos | add any dependent repository keys from url] ****************
Wednesday 28 October 2015  19:22:39 +0000 (0:00:02.021)       0:00:13.959 *****
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
ok: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/sensu/apt sensu main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/sensu.key', 'validate_certs': False})

FATAL: all hosts have already failed -- aborting

The task fails to validate the SSL certificates for the blue box apt mirrors. If I wget anything from the mirror, it succeeds. I tried copying the certificate from the mirror into /etc/ssl/certs, but the error still occurred.
I noticed the last repo succeeds, but that is because validate_certs is False.

did you install libssl-dev before running?

@epwn yep, libssl-dev is installed. I just singled out the second repo to test it out. Doing a curl to the repo doesn't have any sort of SSL errors, I can connect to it just fine. So it's only the playbook/ansible that is having problems with this site.

I just re-ran my allinone this morning and didnt hit the issue you are having. So its not reproducible, at least not on my end. I guess you could try (although it wouldnt be best practice) to set validate_certs to false on those two repos and see what happens?

When we set validate_certs to False things worked fine. After looking at https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/urls.py#L569, Python 2.7.10 was tried, and that got through without SSL errors. Does there need to be something in the readme to put a requirement on python >= 2.7.9?

@j2sol Should we add a line to requirements.txt that requires python >= 2.7.9 based on this issue?

I'm not sure we want to start expressing python version requirements in requirements.txt. We can document that it is preferred to run Ursula, but we don't know all the ways people install Ursula, and expressing python version requirements there seems antisocial.

HI,
I am seeing a similar issue with the allinone configuration, the first time I run "ursula envs/example/allinone site.yml", it ends with the error below:
TASK: [common | install raid utilities] ***************************************
Wednesday 03 February 2016  04:36:37 -0500 (0:00:00.111)       0:06:37.590 ****
failed: [allinone] => (item=tw-cli,megacli) => {"failed": true, "item": "tw-cli,megacli"}
stderr: E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/megacli_8.07.14-1_amd64.deb

E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/tw-cli_<IP_ADDRESS>+10.2.1-1_amd64.deb

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  megacli tw-cli
0 upgraded, 2 newly installed, 0 to remove and 73 not upgraded.
Need to get 5913 kB of archives.
After this operation, 10.4 MB of additional disk space will be used.
Err https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/ precise/main megacli amd64 8.07.14-1

Err https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/ precise/main tw-cli amd64 2:<IP_ADDRESS>+10.2.1-1

msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"   install 'tw-cli' 'megacli'' failed: E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/megacli_8.07.14-1_amd64.deb

E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/tw-cli_<IP_ADDRESS>+10.2.1-1_amd64.deb

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

FATAL: all hosts have already failed -- aborting

If I reran "ursula envs/example/allinone site.yml", then got the same error addressed in this issue topic:
(ursula) root@c910f04x19k02:~/ursula# ursula envs/example/allinone site.yml

PLAY [common role for all hosts] **********************************************
Wednesday 03 February 2016  04:37:36 -0500 (0:00:14.782)       0:00:14.782 ****
===============================================================================

GATHERING FACTS ***************************************************************
Wednesday 03 February 2016  04:37:36 -0500 (0:00:00.000)       0:00:14.782 ****
ok: [allinone]

TASK: [apt-repos | add any dependent repository keys from url] ****************
Wednesday 03 February 2016  04:37:37 -0500 (0:00:00.589)       0:00:15.372 ****
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu trusty main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu trusty main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
ok: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/sensu/apt sensu main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/sensu.key', 'validate_certs': False})

FATAL: all hosts have already failed -- aborting

I am running Ubuntu 14.04.3 on x86_64 platforms. Should I update Python on the allinone node?
Thank you for any suggestion.

That looks like a different issue - perhaps the apt mirror was in the middle of updating, and had got the new package indexes but not all the debs. Can you rerun and see if it completes? Those debs work for me now.

Guang, I have been having the same error.  That is: the Ursula playbook goes through, all apt things work, then changes the repos, then things apt-related fail. As the message from Nov 3 above says, it's due to Ubuntu 14.04-3 LTS using Python 2.7.6, which doesn't come with support for SNI. (Python 2.7.9 adds that).
From what I understand, Ursula uses 2 methods to install packages: (a) the apt module from Ansible with a url, and (b) a url get followed by an install.  If you are having a failure of the 1st type, then the hack is to add the "validate_certs: false" to the ansible call.  If you are having a failure of the 2nd type (I think your message shows that), then the hack that has worked for me is to create a file (/etc/apt/apt.conf.d/00ursula), that contains the lines:
Acquire::https::apt-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::file-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::gem-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::pypi-mirror.openstack.blueboxgrid.com::Verify-Peer "false";

That hack is based on info from here

I'm going to bet that you're using the example defaults-2.0.yml, which has a purposefully broken SSL cert in it. That cert is getting used and put into the system certs file, which in turn makes the whole thing broken, and breaks your ability to validate SSL connections.
https://github.com/blueboxgroup/ursula/blob/master/envs/example/defaults-2.0.yml#L456

Thanks @j2sol, @jzer7, @notnownikki for your suggestions. Here are the steps I tried:

update to python 2.10 on the allinone node
add the lines suggested by jzer7 to /etc/apt/apt.conf.d/00ursula on the allinone node
Updated the cert and private key in defaults-2.0.yml using the cert and key created locally using openssl commands

But these above steps did not help.
I then edited the ./roles/apt-repos/tasks/main.yml, changed the validate_certs line to validate_certs: "no", then the "ursula envs/example/allinone site.yml" goes much much further, but finally failed with another SSL cert verification problem:
TASK: [common | ursula_monitoring directories] ********************************
Wednesday 03 February 2016  23:56:53 -0500 (0:00:00.616)       0:01:28.697 ****
ok: [allinone] => (item=/opt/ursula-monitoring/archive)

TASK: [common | fetch ursula_monitoring when install method is tar] ***********
Wednesday 03 February 2016  23:56:53 -0500 (0:00:00.704)       0:01:29.403 ****
failed: [allinone] => {"failed": true}
msg: Failed to validate the SSL certificate for file-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
Wednesday 03 February 2016  23:56:54 -0500 (0:00:00.807)       0:01:30.210 ****
===============================================================================
Total tasks: 138
===============================================================================
Slowest 25 Tasks
===============================================================================
common | install lm-sensors package ------------------------------------ 11.39s
common | install sensu plugin pip modules ------------------------------- 1.74s
logging | install logstash ---------------------------------------------- 1.73s
common | install sensu -------------------------------------------------- 1.57s
common | update ca certificates ----------------------------------------- 1.46s
common | implement password controls for pam.d login -------------------- 1.24s
common | install shade for ansible modules ------------------------------ 1.22s
common | Permit SSH ----------------------------------------------------- 1.12s
common | pip install eventlet ------------------------------------------- 1.10s
common | pin alembic to 0.7.0 ------------------------------------------- 1.10s
logging | configure rsyslog --------------------------------------------- 1.02s
common | install raid utilities ----------------------------------------- 0.94s
common | python dependencies -------------------------------------------- 0.94s
common | install system tools ------------------------------------------- 0.94s
common | remove unwanted packages --------------------------------------- 0.94s
common | httplib2 and request Python modules that already use system CA certs --- 0.93s
common | update apt index ----------------------------------------------- 0.93s
common | remove older libruby ------------------------------------------- 0.93s
common | ubuntu apt source list ----------------------------------------- 0.93s
common | (possibly self-signed) ssl cert -------------------------------- 0.92s
common | Install prerequisites for acquiring crash dumps ---------------- 0.92s
common | install ufw ---------------------------------------------------- 0.92s
common | sensu monitoring credentials ----------------------------------- 0.88s
common | openstack cert ------------------------------------------------- 0.85s
common | drop ursula release file --------------------------------------- 0.83s
           to retry, use: --limit @/root/site.retry

allinone                   : ok=111  changed=10   unreachable=0    failed=1

(ursula) root@c910f04x19k02:~/ursula#

I am new with Ursula and Ansible, any futher suggestion will be highly appreciated.

@ligc: I'm not sure how to solve this issue, but one thing that might be helpful in he future is to run w/ the -vv flag for more verbose output.

At this point your system's certs file is probably messed up. I'd suggest starting over with a clean OS install and not using the purposefully broken SSL cert in envs/

I'm going to bet that you're using the example defaults-2.0.yml, which has a purposefully broken SSL cert in it. That cert is getting used and put into the system certs file, which in turn makes the whole thing broken, and breaks your ability to validate SSL connections.
https://github.com/blueboxgroup/ursula/blob/master/envs/example/defaults-2.0.yml#L456


need a little help understanding and solving signal errors

we have the goal to compile and run holochain natively on ios, and it's almost working. Zome calls are working, but gossip isn't. here's what we're now seeing in the logs. We are now using wss://signal.holotest.net, which corrected the initial error that we were previously having.
try connect addr=<IP_ADDRESS>:443
err=Custom { kind: InvalidData, error: InvalidCertificateData("invalid peer certificate: UnknownIssuer") }
failed all sig dns addr connects

is the cause of this obvious to you @neonphog ?

@Connoropolous - can you go to https://signal.holotest.net/metrics on the device you're testing on and see what it says about the cert?
In the mean time, I'll look into the go lib to see if it's using some alternate TLS validation other than the native system one.

There's very little information, possibly because it looks like it's just built-in that it uses the native device certs:
https://pkg.go.dev/crypto/tls#Config
// RootCAs defines the set of root certificate authorities
// that clients use when verifying server certificates.
// If RootCAs is nil, TLS uses the host's root CA set.
RootCAs *[x509](https://pkg.go.dev/crypto/x509).[CertPool](https://pkg.go.dev/crypto/x509#CertPool)

Is it possible the device you are testing on hasn't had an update in long enough that Let's Encrypt root CA is not up-to-date on it?

Also looks like you need iOS >= 9 (~2015) - https://www.ssllabs.com/ssltest/analyze.html?d=signal.holotest.net

I visited, and it says 'connection is secure' from an iPhone XR, with iOS 16.4.1

can you give a bit more background.
what server is this? <IP_ADDRESS>:443 (from try connect addr=<IP_ADDRESS>:443)

that is the signal server ip I guess, based on using ping signal.holotest.net

@Connoropolous - Double check my logic - it looks like rustls-native-certs doesn't support ios, and the recommendation is to instead use https://github.com/rustls/rustls-platform-verifier... but that crate seems to have been deleted from crates.io.
I will look into it further tomorrow.

If nothing else, I guess we could use webpki-roots on android and ios.

Your logic seems right. It seems to me that this particular comment has a simple solution that could work. https://github.com/rustls/rustls-native-certs/issues/3#issuecomment-1494166114

The same solution that is being used for macOS should work with iOS, so just including it through the targets
https://github.com/rustls/rustls-native-certs/blob/88066f5eebeabf034a5a1deda83c6214b3d024c9/src/lib.rs#L36

@Connoropolous - as I don't have a good way to directly test such a fix, I wouldn't be comfortable submitting a PR to their repo.
I'll work on the webpki-roots fallback in our code, since it looks like we'll need that for android anyways. If you'd like to submit a PR for rustls-native-certs, then I can remove ios from the fallback set if/when it gets merged.

Ok thankyou. We will await your solution and try it before doing anything else

Will you give me a heads up here?

@Connoropolous - can you give this a try? https://github.com/holochain/tx5/pull/35 Thanks!

Yes absolutely, asap

Thanks!

Updating the stack to use this branch you posted worked to resolve the error!
Now attempting hands on testing

PR to potentially backport to 0.2: https://github.com/holochain/holochain/pull/2415

k. as I've done hands on testing, I've had inconclusive results. @neonphog I am wondering if its mandatory that tx2 still be enabled. Why is it?
Here's my writeup on how things are going, and where we're stuck.
https://hackmd.io/_CFnGaWvQg6Ys3DxZblmug

@Connoropolous - To get some additional hints into the tx5 state machine: RUST_LOG=info,tx5::state=debug

thanks

@neonphog I've just tried updating to 0.3.0-beta-dev.0
Here's my update, can you take a peek, does the error make any sense to you?
https://hackmd.io/_CFnGaWvQg6Ys3DxZblmug?view#May-23

@Connoropolous - I just created this issue: https://github.com/holochain/holochain/issues/2422

Fantastic thanks


Combine .filter() and .map() to create atomic selective DELETEs

In many cases you would want to delete only a very specific row in the database. Take the following code for an example:
DELETE * FROM `user` WHERE `id` = 5 AND `avatar` IS NULL;

We want to delete the user with id "5" only if the avatar is set to null. If we do this in Speedment right now, it would look like this:
users.stream()
    .filter(User.ID.equal(5))
    .filter(User.AVATAR.isNull())
    .forEach(users::delete);

The problem is that the generated code will do this in two steps. First, it will download every user that matches the condition and then it will remove those users. This is equivalent to the following:
SELECT * FROM `user` WHERE `id` = 5 AND `avatar` IS NULL;
// Some time passes
DELETE * FROM `user` WHERE `id` = 5;

Here we have a problem. If another client updates the user row so that avatar is no longer null, the user might still be deleted since there will be a race condition. It would be better if the manager could translate the complete stream into one SQL statement and execute it atomically.

Will be fixed by #539

I am closing this now because #539 is fixed


Removed turtlesim key

Removed turtlesim key. Related with the comment from @clalancette in this PR https://github.com/ros/ros_tutorials/pull/153.
If I'm not wrong we should remove the key in rolling and then release the package with the right key, isn't it ?

I think we might get away here with an inline edit rather than having to remove and replace.
Suggestion
  ros_tutorials:
    doc:
      type: git
      url: https://github.com/ros/ros_tutorials.git
      version: rolling
    release:
      packages:
        - turtlesim
      tags:
        release: release/rolling/{package}/{version}
      url: https://github.com/ros2-gbp/ros_tutorials-release.git
      version: 1.7.1-1
    source:
      test_pull_requests: true
      type: git
      url: https://github.com/ros/ros_tutorials.git
      version: rolling
    status: maintained

I think we might get away here with an inline edit rather than having to remove and replace.

Yeah, I think that would be better.  My suggestion is the following:

Make the update as @nuclearsandwich suggests.
Merge this in.
Immediately do a new bloom-release ros_tutorials of the package to verify that this does what we expect.

We may then want to consider backporting this to both humble and iron.

Thank you @clalancette and @nuclearsandwich


hive parameters not settable

Is it possible to set hive parameters like set hive.tez.container.size=8192 for all connections or per model?
I added the statement in a dbt macro like

{% macro hive_settings() %}

{% set sql %}
    set hive.tez.container.size=8192
{% endset %}

{% do run_query(sql) %}

{% do log("Hive settings applied", info=True) %}
{% endmacro %}

But runnning it dbt run-operation hive_settings ends in
Running with dbt=1.3.2
Hive settings applied
Encountered an error while running operation: Runtime Error
Runtime Error
Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line4:8 cannot recognize input near 'set' 'hive' '.' in statement

Additionally, it would be nice to set them via profiles.yml similiar to the server_side_parameters in the dbt-spark module, see https://docs.getdbt.com/reference/warehouse-setups/spark-setup#odbc.

@bijzz please see this https://github.com/cloudera/dbt-hive/issues/85
You can use pre-hooks to use hive parameters.

Thanks, pre-hooks work after building the package from latest source.


Android line thickness

The problem is that I'm setting the pen width to 2 or even 1 and the line doesn't change the size, it is too big

Could you try the demo app on the repo and see if you get the same result? If not, could you provide a sample app to test this with? I have not seen this issue personally.

Demo app is working, if you have an issue still please share the code and I'll help you figure out what's wrong in your setup. 😄


cloud-init resets hostname

Cookbook version
0.4.2
Chef-client version
n/a
Platform Details
AWS, and other cloud images that use cloud-init
Scenario:
Hostname set via cookbook should persist across reboots in EC2 with newer Redhat releases
Steps to Reproduce:
Standard Centos7 and Fedora20 EC2 images, after setting hostname and rebooting will revert to dhcp provided hostname.
Expected Result:
Expecting hostnamectl results to be persistent, and respect user provided configuration after rebooting.
Actual Result:
/etc/hostname and other critical files are reset to the initial hostname of the machine on boot

I have a PR for this, but needs to go through testing.

Can use the https://supermarket.chef.io/cookbooks/cloud_init cookbook to deal with this

that seems like a good way to deal with this issue.


Would you consider support for MSK & IAM integration

Is your feature request related to a problem? Please describe.
Recently, AWS released an integration between Amazon MSK (Amazon Managed Stream for Kafka) and IAM (Identity & Access Management). There is an example of this implementation but they're using a java client and a library named aws-msk-iam-auth in order to match IAM security.
Describe the solution you'd like
Would like to fully support this new integration.
Additional context
Currently, the only security measures that can be added to a Amazon MSK cluster is by protecting it with security groups or TLS. These options are full deny or full allow, there's no way to define a set of access per topic. This new integration with IAM allow this.

Currently the SASL mechanisms are all bundled into KafkaJS, so while I would like for is to support this, the first would need to be to allow for pluggable mechanisms so that we don't have to bundle the AWS-SDK into KafkaJS. Similar to what we do with compression codecs. I opened up an issue for this about a year back: #840
Since the server side implementation is closed-source, properly testing this in CI is gonna be a mess. Either we fall back to just unit testing or we have to provision an MSK cluster during test runs, which we don't really have the funds for (nor time to implement). Yet another reason for this to live outside of KafkaJS core.

@Nevon If I understand this issue correctly, this package does not currently support MSK & IAM integration, right?

That is correct.

@Nevon I see you already started with supporting this feature: https://github.com/aws/aws-msk-iam-auth/issues/10#issuecomment-846599054
We would like to help in developing this. Is this pushed to any branch?

No, I was developing it outside of KafkaJS, and never pushed the repo. It was in a very early stage anyway, so not much to continue on. Like I mentioned in my earlier comment, we would not want to bundle this into KafkaJS itself, but rather keep it as a plugin, so that we don't end up bringing in big dependencies that are useless for 99% of users, and making the KafkaJS testing process far more complicated since the authentication mechanism is proprietary and we can't even use it in a local Kafka cluster. That's one of the reasons I wrote #840, so that these things could be injected as plugins instead of having to bundle everything into KafkaJS.
So feel free to take a stab at it, using the branch in #840 as a peer dependency. I know I ran into some rough spots around the Typescript interface, where it was difficult to export an interface that allowed the authentication mechanism to integrate in a type safe way, so probably there will need to be some tweaking there as well to make it work well - but it does work, even if the interface needs some more love.

@Nevon I created some demo of configuration here: https://github.com/jmaver-plume/kafka-msk-iam-demo
With example you list topics to validate we have permissions.

hi @jmaver-plume , I tested your example with MSK serverless version. I couldn't receive messages from the consumers. But create tpoics, send messages works fine.  when I point the same code to a non-serverless version of MSK, everything works fine. Do you have any idea what has gone wrong?

I'm currently trying to integrate with aws serverless and running into the same issue. It seems I can do everything except consume messages. The application hangs on JoinGroup request.

Injecting custom authentication mechanisms is now available in a beta release. See https://github.com/tulios/kafkajs/issues/840#issuecomment-1177251826 for more information.

https://github.com/jmaver-plume/kafkajs-msk-iam-authentication-mechanism This example is not working with the beta release when using Mechanism as authenticationProvider.
The error is KafkaJSProtocolError: []: Access denied

@Nevon Do we have working example to connect to AWS MSK Cluster using kafkajs library,i tried using https://github.com/jmaver-plume/kafkajs-msk-iam-authentication-mechanism but not able to connect and getting below error.
/home/ubuntu/ext-drive/sampleProject/msk-iam-demo/kafka.js:3
AuthenticationMechanisms[Type] = () => Mechanism
^
TypeError: Cannot set property 'AWS_MSK_IAM' of undefined
at Object. (/home/ubuntu/ext-drive/sampleProject/msk-iam-demo/kafka.js:3:32)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:12)
at Module.require (internal/modules/cjs/loader.js:974:19)
at require (internal/modules/cjs/helpers.js:101:18)
at Object. (/home/ubuntu/ext-drive/sampleProject/msk-iam-demo/server.js:2:39)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
It would be really helpful to provide any specific version of node.js is required to execute the server.js file.

@NikolayPIvanov @bitnitesh I updated the library to work the new interface provided by #840.
https://github.com/jmaver-plume/kafkajs-msk-iam-authentication-mechanism/tree/v2.0.1

@bitnitesh I have verified that the library works using https://github.com/jmaver-plume/kafkajs-msk-iam-authentication-mechanism/blob/main/example/cloudformation.yaml. Is your MSK correctly provisioned for IAM? Library is using https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_credential_provider_node.html to get accessKeyId and sessionToken.

@jmaver-plume Yes for MSK clsuter  IAM role-based authentication is Enabled

Closing this issue, as KafkaJS now provides the capability to inject custom authentication mechanisms. Discussions about specific plugins should be had within those issue trackers.


Mortality tree data disappearing

Hello, we have discovered an issue with some of the mortality trees. With the new mortality layer there is an orange circle indicating the mortality trees and when we mark the circle as complete it turns blue. We have just noticed that with some of the trees where we turned the circle orange to blue, all the data- census and mortality- is gone except for the status. An example of a tree like this is 131080. We are currently investigating this but I wanted to loop you everyone in.
We think we have tracked the date of the error to October 11.

Hello, we could do a short zoom on Thursday at 2 if that works for you.
On Tue, Oct 24, 2023 at 11:36 AM jess-shue @.***> wrote:

@Iriskennedy https://github.com/Iriskennedy Hi Iris, I will be back in
the office tomorrow and can take a look at things. Would you all be
available for a Zoom chat to discuss the issue? I'm wondering if October
11th was a one time issue, or if you've backed off the mortality because of
the lost data. Thank you!
—
Reply to this email directly, view it on GitHub
https://github.com/SCBI-ForestGEO/2023census/issues/53#issuecomment-1777500039,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A66DOHVBCR74HASTUSM545TYA7N4TAVCNFSM6AAAAAA6EMKU2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZXGUYDAMBTHE
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @Iriskennedy,  below is a Zoom link to meet at 2 pm. I'm testing things now and doing some Googling for answers, nothing quite matches this issue though. Looking forward to seeing everyone soon:
Jessica Shue is inviting you to a scheduled Zoom meeting.
Topic: SCBI - FieldMaps data issue
Time: Oct 26, 2023 02:00 PM Eastern Time (US and Canada)
Join Zoom Meeting
https://smithsonian.zoom.us/j/85079491033?pwd=TjZnMGNKdEcraXgwRUtMMUtEbm1CQT09
Meeting ID: 850 7949 1033
Passcode: 065557
One tap mobile
+16469313860,,85079491033#,,,,*065557# US
+13017158592,,85079491033#,,,,*065557# US (Washington DC)

Thank you for discussing things with me yesterday. I wasn't able to recover the lost data as it was cleared before being sent to ArcGIS Online. But, you have things under control and can continue to collect data. Closing this issue.


Replace observers with concerns

rails-observers were removed in Rails 4 but carried on being used in whitehall via an extracted gem. This is no longer supported in Rails 5, so this commit replaces observers with native Rails concerns.
This is a precursor to the full upgrade to Rails 5.

Actually... could we get rid of the rails-observers gem in this PR too?

Yup, missed that since I was doing it in a separate branch to upgrade to Rails 5


Is there any way to skip creating the markdown files and nav manually ?

mkdocstrings supports the injection as said in docs,
::: my_library.my_module.my_class

I also have to create the navigation.
This is great because it gives precise control, but is there any existing way to completely generate all the pages with sane defaults?
Like each module's docs in separate page. And links to sub or super module. Something like pdoc3 but using the material theme.
Many thanks for your help

I've been using this ugly Bash script myself:
#!/usr/bin/env bash
cd src || exit 1
rm -rf ../docs/reference
mkdir ../docs/reference
find your_package | grep -v -e pyc\$ -e __pycache__ -e '^your_package$' -e py.typed -e __init__ -e __main__ | sort | while read -r line; do
    indent="$(echo "${line}" | sed -r 's/\w+\//  /g;s/[^ ]+$//')"
    case "${line}" in
        *.py)
            mod=${line##*/}
            line=${line%.py}
            dot=${line//\//.}
            md=${line#r_pytb/platform/}.md
            dir=${md%/*}
            if [ ! "${dir}" = "${md}" ]; then
                mkdir -p ../docs/reference/${dir}
            fi
            echo "::: ${dot}" > ../docs/reference/${md}
            echo "${indent}- ${mod}: reference/${md}"
        ;;
        *)
            echo "${indent}- ${line##*/}:"
    esac
done

It will regenerate the reference/ directory in the docs/ one, and also output the related nav for mkdocs.yml.
We could consider allowing handlers to provide ways to do this (creating files with ::: and outputing/updating the nav) and provide a common CLI for all handlers.

@pawamoy Thanks a lot.
I was planning to write somekind of script. The script you shared, will perfectly do my work for now.
Having some option in the cli would be great ! ☺

This will be even more relevant for Crystal handler because Crystal programming language tends to have a ton of classes, and the convention for docs currently is to put one class per doc page.
See my script for generation in the commit message here:
https://github.com/oprypin/athena-website/commit/264e549303477a7c374a8c0ca43a7a6742efab08 😩
I think there could even be some generic MkDocs plugin for file generation, and it would actually work very well to just import and use mkdocstrings.handlers.base.get_handler inside a script, as long as it shares the same Python process the instance will be shared.

Another script found in the wild
https://github.com/cgarciae/pypeln/blob/142a9f20c2765b8c751a853b27cace83a275dd01/scripts/generate_docs.py → https://github.com/cgarciae/pypeln/tree/142a9f20c2765b8c751a853b27cace83a275dd01/docs/api

I am tackling this issue in a comprehensive way.

Add a convenient way to access the collector
Develop a generic plugin to generate files

Preview of the feature:
Instead of actually adding the files, I generate them with a script

Oh, not to forget:

A generic plugin that can infer parts of the navigation

This is really interesting @oprypin!
So what your "generate" plugin does is execute the given Python scripts that will create the files themselves, using a collector to know how. In your example, the root attribute of the Crystal collector contains the whole tree, so you can walk it. For Python we could do kind of the same, using pytkdocs to walk the tree searching for modules.
Since users all have different ways of writing the code reference markdown files, I guess it's best to start with something flexible like this, and maybe later add native methods to the handlers themselves for the most common ways (like one page per module in Python, etc.).
Then your plugin inferring the nav is the cherry on top: not only users don't have to write the code reference markdown pages themselves, they don't even have to write the nav either! But if a file is named __init__.md, the item in the nav will be named __init__ right? Then the plugin could likely accept config settings to add suffixes like .py. Just thinking out loud :slightly_smiling_face:
I'll take a look at your PR in the next few days!

Oho, now I am indeed running into trouble with page names in inferred navigation, and I see what you mean.
I'm adding ::: foo.py into foo.md, and the title becomes "Foo". Not ideal. And generating the nav as well is cumbersome, though definitely doable.

Oho, now I am indeed running into trouble with page names in inferred navigation, and I see what you mean.
I'm adding ::: foo.py into foo.md, and the title becomes "Foo". Not ideal. And generating the nav as well is cumbersome, though definitely doable.

I have concluded that the best way forward for Python is to not generate the nav automatically -- only generate the files themselves.
And for that, the solution looks like this:
Via https://github.com/oprypin/mkdocs-gen-files

mkdocs.yml

plugins:
  - gen-files:
      scripts:
        - gen_doc_stubs.py

gen_doc_stubs.py

from pathlib import Path
import mkdocs_gen_files

src_root = Path("my_module")
for path in src_root.glob("**/*.py"):
    doc_path = Path("reference", path.relative_to(src_root)).with_suffix(".md")

    with mkdocs_gen_files.open(doc_path, "w") as f:
        ident = ".".join(path.with_suffix("").parts)
        print("::: " + ident, file=f)

(also applied it in https://github.com/mkdocstrings/mkdocstrings/pull/242)

Could you give more details about that conclusion?

It's not too hard, just more boilerplate than you're getting for it.
Reasoning why it's OK to not generate the nav:
The files themselves will always be auto-updated. And there are rarely many files.
Whenever something gets changed, you'll get a warning from MkDocs that either there's an unused page or a nonexistent page is referenced. So you won't forget.
Reasoning why it's hard to generate the nav:
You need to pull in a heavy plugin like awesome-pages or literate-nav and change your way of using the nav.
Generating the nav itself, including directories, is just quite complicated code logic. More than 2x the complexity of the above script.

I see, thank you for the explanation :slightly_smiling_face:

See an example in #271 :stuck_out_tongue:
That was not so straight-forward indeed! If you have something more elegant, please share :smile:

I'll close this now. Please take a look at how mkdocstrings generates its own pages and nav, using the mkdocs-gen-files and mkdocs-literate-nav plugins from @oprypin :slightly_smiling_face:
We'll document all this in mkdocstrings docs soon.

What's currently the best place to look for docs on this?

Funny timing, I was exactly going to add a "recipe" for this in a new Recipes page (#378) :slightly_smiling_face:


Add v-model support

Currently there is no support to add a default value

Hi @sahapranta, can you give me an example ? and why we should consider having a default value for file input ?
Thanks

@khofaai default value is useful when the component will be also used for updating image.
Like in most case for creating and updating we share same form component.


Pipetest Analytics

Scale CC pipetest Analytics

@ingridhovden  When the report is ready please add it to reports admin and write the global-identifier here so we can create the app for it.

@kjellhaaland Can you please add power bi into Pipetest app so we can test the report.

What is the report id @aman-riaz?

Have you published the report @aman-riaz ?

@aman-riaz, i get 404 error message when i add the Pipetest Analytics Report.
Is it published?

@kjellhaaland, @Gustav-Eikaas The report is now published and ready for test. Can you please try now?

@ingridhovden, i now get it to load, but i dont have access to the report, so i cant view the content.
Am i supposed to be able to view the contents of the report, like the other apps?
I will make a PR, so you can test.

@aman-riaz, @ingridhovden i have added the report top the Pipetest app, but i do not have any rights to view the content of the report.
Can you test?


Update to parity-wasm 0.42 and allow to configure stack limits

This PR updates parity-wasm to 0.42 and allows to use env variable to set the stack limits. This is useful in applications like embedded where the available memory is very limited
Give it a check although the changes are trivial. I just got started with Rust and WASM recently 😄

The change to default value is not strictly required to achieve reasonable stack size on embedded targets.
A custom StackRecycler can be instantiated using StackRecycler::with_limits and the module started with NotStartedModuleRef::run_start_with_stack.
Note that you will likely require https://github.com/paritytech/wasmi/pull/243 to be able to load on a deeply embedded target.

Nice idea thanks! I'll take the chance to ask you how are you managing the stack size for a WASM module in MCU. I'm currently using a fork of wasmi where I limit the max pages to 1, since AFAIK, 1page of 64KB is the minimum size as defined in the standard. But 64KB is taking a lot in my STMF4 with only 128KB RAM in total.

It is indeed a requirement of the webassembly specs that memory instances are sized in unit of page-size defined as 64KiB.
In the proof of concept I worked on, the wasm module was build form a hand crafted wat source and did not require memory page.
I am not actively following wasm's news so there might be some work done or being done around that topic.
In the meantime my bet would be to use in the wasm modules a fork of wee_alloc where the PAGE_SIZE is adjusted to match what you set in your runtime.
The runtime would need to be built against a fork of memory_units to change the page size using cargo's patch feature but this is IMHO a bit too hacky.
I guess the “cleanest” approach to deviate from wasm's specs would be to modify MemoryInstance and ByteBuf so that it:

all operations to ByteBuf::new , ByteBuf::realloc and ByteBuf::len are in Page unit rather than bytes.
MemoryInstance relies on a trait (T: ByteBuf or T: MemoryBuffer) so that we can provide our own implementation of it in the ImporResolver.

But this will probably require some extra changes to ImportResolver to add an associated type for the MemoryInstance new parameter.
Embedded use-cases may be too much of a niche for the extra complexity it requires (see https://github.com/paritytech/parity-wasm/pull/283#issuecomment-762212530).

In the proof of concept I worked on, the wasm module was build form a hand crafted wat source and did not require memory page.

In my case, I was able to reduce the initial stack size by using the compiler flag link-arg=-zstack-size=16 as indicated here https://github.com/rustwasm/wasm-bindgen/issues/1345. I also use the release optimizations:
[profile.release]
lto = true
opt-level = "z"

The final output is quite minimal for a simple module, especially if no heap is used. All rusty stuff like unwinding etc. is removed if not actually used. This allows me to compile from Rust without manually writing the wat file.
About the memory size spec, there's some discussion about it here: https://github.com/WebAssembly/spec/issues/899.

Embedded use-cases may be too much of a niche for the extra complexity it requires (see paritytech/parity-wasm#283 (comment)).

Yup, I guess so. For the time being, I can live allowing 1 page of 64KB. Or one can avoid specifying the number of minimum pages in the WASM module, then the interpreter can allocate any custom default memory size.


Requests from new "desktop app" version aren't part of web session

Due to my particular use of ARC, I think I'll have to abandon the new implementation. I need to first authenticate with a site and get an active session, then I jump over to ARC and make REST calls. (I know this isn't very standard practice.) The new implementation makes requests that aren't part of the same session (I'm guessing it just doesn't send the cookies that the browser has for the given URL.) So, all of my REST requests are getting a 403 back from the services.
The general picture for the services I'm calling is...
browser (or ARC) ---> service proxy that handles OAuth wrapping the call ---> REST endpoints
So, the browser request is to a stateful, session-based service proxy. The service proxy makes an OAuth secured call to the REST endpoint and returns the response to the browser (or ARC). So I normally pop open a Chrome window and login to the site, which tells the service proxy to get an OAuth token for me and store it in session. Then my calls to the proxy are turned into OAuth-secured REST calls to the backend. Since the new ARC doesn't send cookies to the server, I'm not in the same session, so no OAuth token is found for my new session. All of the REST calls return HTTP 403 in that case.

Yeah same problem here. Could have given us a heads up to those who don't follow the project this closely. I have to authenticate as well and for some strange reason I can't even call the login site as it gets 410'd. This is probably do to my departments attempt at a defense against bots from scouting out the admin back page.

Same problem for me. I even tried to setup a local session within ARC by hitting the "user signin" endpoint of my app, but for some reason, the cookies don't get retained in ARC and I'm unable to test any endpoint that requires an authenticated session. Real bummer, ARC was my go-to for this type of thing.

Hi,
Thanks for an issue report.
Right now I'm testing own cookie session management in the app. It is basically implementation of the HTTP spec session management. So it will mimic browser's functionality.
It won't however have an option to synchronise cookies between Chrome browser and the app because there is no API available for apps to do it.
The update will be released in coming days.

I've run into the same issue with java sessions. Now I have to manually add a "Cookie:" request header with JSESSIONID to each request.

Hi,
Today's morning (London's time) I've released an update. This update contains a session management inside the app (a cookie storage with cookie logic described in the spec).
There is no, however, cookie synchronization between Chrome and the app. There's no API for that in Chrome. So all cookies received by the app will be used as usual but they need to be physically received by the app.
Please, let me know if you find an issue in this new session management. I'll be happy to fix it. However it algorithms has been implemented as described in the spec.
More info here: https://restforchrome.blogspot.co.uk/2016/04/session-management-aka-cookies-in.html


Feature request: Notify when the process is done

I believe there are two main ways to achieve this.
1- Using the 'meta_bbb-recording-ready-url' just like described in
https://github.com/bigbluebutton/bigbluebutton/blob/fe403dfd05fe38ca50eb0ef319c22c2cc62f2c43/record-and-playback/core/scripts/post_publish/post_publish_recording_ready_callback.rb
So, to user needs to send the callback URL as a parameter the moment the meeting is created.
The message sent needs to be a little different from the default so we could distinguish which event we are getting. Otherwise the user would get a duplicated notification.
This looks like the best solution and a great part of it is already done.
2- Using bbb-webhooks. For this you'd would need to publish an event on Redis server and let the bbb-webhooks deal with it. I'm not really sure if this would really work right away because the weebhooks probably don't notify unknown events. But I think it's worth mentioning because if you plan to merge this project, would be cool to add this event on the bbb-webhooks too.

Maybe this issue doesn't make sense anymore. Previously the scripts' filename started with "render_*" so everything used to run after the "post_publish_recording_ready_callback.rb".
Now it starts with "export" so one could use the native BBB callback if they feel like it.

@hostbbb shared the following mp4CallBack.sh script to achieve this with me:
#!/bin/bash
# Call back - needs meetingID as shell argument $1
callBack=$(cat /var/bigbluebutton/published/presentation/$1/metadata.xml | grep mp4-recording-ready-url |sed 's/    <mp4-recording-ready-url>//g' | sed 's|</mp4-recording-ready-url>||' | sed 's/\&/&/')
echo "$callBack"
/usr/bin/curl -d "meetingId=$1" -X POST $callBack
#CURL='/usr/bin/curl'
#CURLARGS="-f -s -S -k"
# The result can be stored in a variable
#raw="$CURL $CURLARGS $callBack&meetingId=$1"
#echo $raw


versioned map mock storage for blob-retention unit-testing

This internal mock will help test the blob-retention changes in an upcoming PR. This PR adds a versioned in-memory blob-store to kopia which allows setting retention times and delete-markers on put and delete api calls respectively. This will allow unit-testing various versioned buckets storage features (like blob-retention) without the need for external environmental and infrastructure dependencies.

Closing in favor of https://github.com/kopia/kopia/pull/1590

Commented upstream in kopia#1590


Cant download mods

Traceback (most recent call last):
File "C:\Users\user\Desktop\curseforge-mass-downloader-master\main.py", line 74, in
data = json.loads(rq.get(f"https://curse.nikky.moe/api/addon/{id}").content) # Put the request's data into a Python-readable format
File "C:\Users\user\AppData\Roaming\Applications\VS22\Shared\Python39_64\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\user\AppData\Roaming\Applications\VS22\Shared\Python39_64\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\user\AppData\Roaming\Applications\VS22\Shared\Python39_64\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Unfixable due to a dependency breaking. Closing.


Viewing character: stats font is generic, not LCARS

The font used for stats and inventory items is not the LCARS font. This appears in the random NPCs as well as when I click "view" at the end of regular charcater creation.
(Inspect shows it as helvetica, which is amusing because technically helvetica is an LCARS font in the ultra compressed style, but the app uses its own LCARS font family style elsewhere, that should be used here as well.)
Edit: Actually, it's showing in steps in the creation process. Compare the upbringing vs environment Attributes labels.
Edit: It's the div class "stat-entry-name" vs "attribute-container" or "stat-name".
Edit: Mucking about in the browser developer tools, it looks like it's not a simple "add the font family to the css" type of change.

The upbringing details page is still showing as the generic font. It appears to be https://github.com/bcholmes/StarTrek2d20/blob/sta-complete/WebTools/src/scss/attributes.scss that contains the styles that are causing it for that page.


reset_simulators install only latest iOS simulators

Hi, this may be not an issue but lack of my knowledge of the tool, please do not judge.
My Xcode 6.4 has Simulators 7.1, 8.2, 8.2, 8.3, 8.4 installed.
But hewn when I am running snapshot reset_simulators I am getting simulators only for latest 8.4.
Is is possible to specify somehow to configure all the simulators?

I haven't yet looked into creating the simulators for multiple versions. Right now it only supports the currently used version from xcode-select.

@KrauseFx thanks for your quick response.
I thought I was blocked by this issue, but then found that it is possible to create additional simulators from  "Organiser" in Xcode. Feel free to mark this issue as Enhancement. It will be really nice-to-have.

:+1:

Solved with https://github.com/KrauseFx/snapshot/pull/192, this will be in the next release :+1:

@KrauseFx Perfect! Thank you!


Quote formatting is inconsistent between Markdown and Koenig

Issue Summary

To Reproduce

Add markdown block and type "> Test".
Resulting HTML:

Markdown

Type "> Test" right into Ghost editor.
Resulting HTML:

Not markdown

Technical details:
Ghost Version: Version 2.19.3
Node Version: 8.15.1
Browser/OS: Firefox 66.0.2 (Windows)
Database: MySQL

The difference in HTML output with an inner
 vs no inner
 is a known difference between the markdown renderer output and the rich-text renderer output and is not a bug in the editor.
Themes should be prepared to handle the variation as both are valid HTML, if you are using Casper could you open an issue on that repo or report the issue on the respective theme repo or theme developer. Thanks 🙂

@kevinansfield, Are you really sure that theme developers are aware and properly informed about blockquotes handling? According to current Casper theme which had this big: it's not true.
I think the good result of this issue is a few lines into documentation.
BTW, the link to the "full and extensible documentation" at the https://demo.ghost.io/themes/ article is broken (broken link is https://themes.ghost.org/v2.0.0/docs).


M3-405: Additional Disks and Linode Rescue Test

Description
Test added

Creates a Linode with 3 attached volumes
Verifies disk image and swap disk are present in rescue drive dropdown
Verifies attached volumes are present in rescue drive dropdown
Resizes Linode from 1G to 4G
Attaches an empty disk to Linode
Attaches a disk from image to Linode
Verifies additional disks are present in rescue drive dropdown
Rescues Linode with maximum 7 rescue drive disks, 2 initial disks, 2 added disks, and 3 attached volumes

Additionally some logic was centralized, and false positive tests refactored. Have to temporarily remove the support search test and linode listing test due to failures only in docker container runs, need to revisit in separate ticket.
Type of Change

Integration Test

Note to Reviewers
To Test yarn && yarn start new shell yarn selenium new shell yarn e2e --spec=e2e/specs/linodes/detail/rescue.spec.js
For this test to work I had to remove ramda clamp validation from src/features/linodes/LinodesDetail/LinodeSettings/LinodeDiskDrawer.tsx onSizeChange() function, please let me know if this is an issue.

Coverage remained the same at 33.33% when pulling 356b7243a913d29429ccd884d488149a7c948f32 on nwisnewski:M3-405 into 6fd925b5afc50f062d238ea94a15a37bec73f305 on linode:develop.


Add new exercise: Binary search

This PR adds a new exercise from the xjavascript track.

@rchavarria I am noticing the test order actual/expected is reversed on most of the tests.

You're right. I didn't notice that.


Generic USB Sensor Driver

Create a USB sensor driver. Test on the thermometer until the new sensors arrive.

I have fixed my issue reading data from the Thermometer. It turns out I was having a USB permissions issue, but solving it took quite a lot more effort than I hoped. Here is what worked in the end:

Create a file called /etc/udev/rules.d/50-datalogger.rules
Put the following in it:SUBSYSTEM !="usb_device", ACTION !="add", GOTO="datalogger_rules_end"
SYSFS{idVendor} =="0c45", SYSFS{idProduct} =="7401", SYMLINK+="datalogger"
MODE="0666", OWNER="jack", GROUP="root"
LABEL="datalogger_rules_end"

where idVendor and idProduct are obtained from lsusb, and OWNER is my username. This file will grant the necessary permissions for a single USB device.
Reboot.

I can now use the pcsensor driver I found online. I am still experimenting with writing my own. No luck yet, but solving this issue gives me a lot of confidence.

Tabling this. It is not needed.


Useless error message

Hi,
today I had a problem deploying my aspnet core application on Azure.
The error message was saying:
Application: Gaia.Bmw.Admin.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.Check(Int32)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_bind(Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle, Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.SockAddr ByRef, Int32)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.Bind(Microsoft.AspNetCore.Server.Kestrel.ServerAddress)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListenerPrimary.CreateListenSocket()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener+<>c.<StartAsync>b__6_0(System.Object)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.ListenerPrimary+<StartAsync>d__11.MoveNext()

Exception Info: System.AggregateException
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean)
   at System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.CreateServer(Microsoft.AspNetCore.Server.Kestrel.ServerAddress)
   at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Start[[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context, Microsoft.AspNetCore.Hosting, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60]](Microsoft.AspNetCore.Hosting.Server.IHttpApplication`1<Context>)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Start()
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost, System.Threading.CancellationToken, System.String)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost)
   at Gaia.Bmw.Admin.Program.Main(System.String[])

After few hours I noticed that the order on program.cs file was wrong.
.UseIISIntegration() must be after .UseUrls()
I don't know if it is my fault or not (for me the order wasn't so important, otherwise I should get a compilation error) but the error message wasn't helpful.

This issue was moved to aspnet/KestrelHttpServer#1020

The order is important because the ASP.NET Core module passes the port to Kestrel when running behind IIS. It's the equivalent in node when you do this:
 app.set('port', process.env.PORT || 3000);

In ASP.NET it's the reverse:
.UseUrls(...) -> // defaults
.UseIISIntegration() // override when IIS active

As for the unhelpful error message, that's a kestrel issue:
/cc @cesarbs @halter73

Hi @davidfowl
thanks for the quick answer.
If the order is important, I'm not should be able to do it right?
You can "play" with fluent interfaces to prevent this error. doesn't it?

https://github.com/aspnet/IISIntegration/issues/242


Refactor gin middleware

Added a New() function which will be used as a standardised function for all the middleware instead of NewMiddleware(It will still be present so we don't break existing bindings).
Updated how we were fetching the route name.

PTAL

PTAL

I will change the readme file.

PTAL

The sample application of apex/log will be updated in the separate merge request, I am working on removing the custom severity levels and using the levels provided by apex/log

Ok, maybe we don't need to change it in this PR? :)


Document the technical design of incomplete type metadata

@swift-ci Please smoke test and merge.

@slavapestov @jckarter I'm going on vacation next week, and I'd like to make sure that people understand how this works just in case something comes up before I disappear.  Can you let me know if the new section makes sense enough that you'd feel comfortable with the implementation?


Update BearerTokenAuthenticationPolicy implementations with sync workflow

API change check
API changes are not detected in this pull request.


AccessibilityLabel for Menu

Description
MA-2352
Add a description of your changes here.
Notes
How to test this PR
Reviewers
If you've been tagged for review, please check your corresponding box once you've given the :+1:.

[ ] Code review: @BenjiLee
[x] Code review: @danialzahid94

👍

@saeedbashir  I'm a bit confused by this PR. Looks like only a string change that may or may not happen?

@BenjiLee Yes its just a string change.

Current coverage is 43.93% (diff: 100%)

Merging #793 into master will not change coverage

@@             master       #793   diff @@
==========================================
  Files           360        360
  Lines         25328      25328
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits          11127      11127
  Misses        14201      14201
  Partials          0          0

Powered by Codecov. Last update c1fe969...6888182


Style: Avoid overriding Kernel#p

Tiny change: Kernel#p is never overridden, even for a short while.

Never thought about this being an issue. Thanks!
❤💛💚💙💜

Team p reminds of the amazing powers of log-and-return-self!


Port tests to Python

Motivation for this change
Processing the task list in issue https://github.com/NixOS/nixpkgs/issues/72828.
Now that PR https://github.com/NixOS/nixpkgs/pull/71684 (nixos: add python testing support) by @tfc got merged, I would like to port some of the tests to Python.
Things done
Ported several tests to Python, which also conform the Black code style.
Tests have been run and all of them succeed.

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @tfc @flokli @worldofpeace

@marijanp I forgot this installed test https://github.com/NixOS/nixpkgs/pull/72935/commits/a4cac83b529712d28a6c43aca5174cbc47bf216c in https://github.com/NixOS/nixpkgs/pull/72860.
Feel free to drop the commit as I'll push the port manually for that new code.

@marijanp I forgot this installed test a4cac83 in #72860.
Feel free to drop the commit as I'll push the port manually for that new code.
Edit: pushed that in 85251b3

@worldofpeace I've dropped the commit

@jonringer I just started the build again after a fresh rebase and I couldn't reproduce the fail. It works for me:
machine # [  226.745136] systemd[1]: Started cadvisor.service.
machine # [  226.909883] systemd[1]: Reached target Multi-User System.
machine # [  226.927071] systemd[1]: Startup finished in 33.340s (kernel) + 3min 13.301s (userspace) = 3min 46.641s.
machine: must succeed: curl http://localhost:8080/containers/
(0.38 seconds)
influxdb: waiting for the VM to finish booting
influxdb: connected to guest root shell
influxdb: (connecting took 0.00 seconds)
(0.00 seconds)
influxdb: must succeed: curl -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE root"
influxdb # [  229.255593] influxd[659]: ts=2019-11-08T08:32:45.647545Z lvl=info msg="Executing query" log_id=0IzY5sfW000 service=query query="CREATE DATABASE root"
influxdb # [  229.345082] influxd[659]: [httpd] ::1 - - [08/Nov/2019:08:32:45 +0000] "POST /query HTTP/1.1" 200 33 "-" "curl/7.66.0" 56b5d1ab-0202-11ea-8002-525400123456 150944
(0.61 seconds)
influxdb: must succeed: curl http://localhost:8080/containers/
(0.50 seconds)
(232.83 seconds)
test script finished in 233.11s
cleaning up
killing influxdb (pid 6)
killing machine (pid 18)
(0.00 seconds)
/nix/store/g0kvxg9sj61ccyvdmq3lyincncy33jis-vm-test-run-cadvisor

I was also running a lot of other reviews at the time, could have been cpu starved and then timedout

@GrahamcOfBorg test cadvisor

Ran all the updated tests manually, and they did succeed. Thanks :-)


Typed JWS/JWE

What do you think, is this worth it?
See also https://github.com/a-sit-plus/vck/pull/163

also: maybe provide a default shorthand for ByteArray? also handy wrt. backwards-compatibility


Gruntfile not found (again)

Installed Atom on Windows 10, add package grunt-runner and got the error: Gruntfile not found.
Searched the web, searched though all related issues here, tried all offered solution but to no avail.
In Atom I added a project folder which I used to work on using Netbeans which shows and executes all tasks in my Gruntfile.js. Grunt was installed locally in the subfolder of the project.
I tried many Path variations in the settings for this package. I've installed grunt globally and tried more Path variations...
I'm lost.

idk but i created a file named Gruntfile without .js then it worked.
he need to add Gruntfile.js & gruntfile.js to this array

+1

@REJack the array seems ok to me... Those values are used inside a require() and as far as I understand you should not include file extensions when you use require. So in fact the values on that array should be able to resolve for Gruntfile.js & gruntfile.js.
But I can't reproduce this is issue again. But I will keep an eye open for it.

@ealves-pt I've added you as a contributor btw, as a little thanks for your great work <3

@kokarn glad to help mate :+1:

Duplicate #10. Several issues cause Error loading Gruntfile: gruntfile not found. I will try to gather all the reported issues in one place and go through the code to try to understand a bit more the root of all evil.

Hey @ealves-pt and @kokarn https://github.com/kokarn/atom-grunt-runner/pull/91

@pamtbaau @REJack @kwhc  Would you mind updating to the latest version (0.13.6) and see if that solves your issues?

Sorry @kokarn, I've uninstalled and stopped using Atom.

No problem, somebody else can probably reproduce :)

I just checked last version, this seems to be resolved :+1:

Thanks @RoverWire. I'll close the issue and we can keep an eye open for this bug on the other similar issues.


mismatch between json config file and displayed data at aprs.fi

Hey Richon, thank you so much for your effots in programmin LoRa APRS.
I use TTGO  T22_V1.1 which is recognized by VSC as 1.2
I attach the configuration json file and reception screenshots of aprs.fi
As you can see there is a little difference between config and reception.
May be this information helps you to improve your software.
Hope to hear from you.
best regards and 73s de DG1RUE (aka Holger)

tracker_config.json

All looks good in configuration
It seems DG1RUE-15 may be joining two packets and this is appearing as faulty comment
Can you update it to my last igate firmware?

could we start all over again?
do you have updated the igate?


TS-4896: TSHttpClientAddrGet/TSHttpIncomingAddrGet may return NULL

Several of our plugins would experience crashes once we tightened up clean up on session shutdown.  They assumed that the TSHttp{Txn|Ssn}ClientAddrGet and TSHttp{Txn|Ssn}IncomingAddrGet calls would never return NULL.   If the client initiates the shutdown, the underlying netvc may get cleaned up before the sessions, transactions, and state machines were completely shutdown.  The original code would return NULL in that case.
In our version, we added this logic to cache the address information and use that data if the vc has been cleaned up.  This avoided spinning changes in the plugins.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1372/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1265/ for details.

I think it should be backported to 7.1.x because returning NULL seems to be a sort of API breakage if old versions didn't return NULL in the same situation.

Replaced NULL references with nullptr

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1299/ for details.

Nit picky, and too late, but I kinda wish these two new variables would have been named "saved_XXX" instead of "cached_XXX". Cached make it sounds like it's an optimization of some sort, but as far as I can tell, this is to save way the IP addresses since the VC goes away.


ParallelAssignment false positive

The autofix for ParallelAssignment leads to an error in logic.
Expected behavior
@item.position, @item2.position = @item2.position, @item.position

Should resolve to:
original_item_position = @item.position
@item.position = @item2.position
@item2.position = original_item_position

The autofix should introduce a temporary variable here or another approach, where the original value isn't overwritten.
Actual behavior
Currently, the above resolves to:
@item.position = @item2.position
@item2.position = @item.position

Steps to reproduce the problem
An important context to consider for reproduction: @item and @item2 are active record instances, position is a numeric value.
RuboCop version
$ [bundle exec] rubocop -V
0.86.0 (using Parser <IP_ADDRESS>, rubocop-ast 0.1.0, running on jruby 2.5.7 java)

Might also be interesting, additional gems I have linked in the contet of rubocop:
  gem 'rubocop', '0.86.0', require: false
  gem 'rubocop-rspec', '1.40.0', require: false
  gem 'rubocop-performance', '1.6.1', require: false
  gem 'rubocop-rails', '2.6.0', require: false

Agree with the bug report, except the cop should simply accept it, as it already accepts a, b = b, a

Interesting point, that is true, that with a simple parallel assignment statement (default settings), no violation is shown, only in this object context. In that case I think it would make sense to extend the bug report so that not only the autofix needs to be adjusted, but the cop as well.

Yes (except that the autofix doesn't need to be adjusted, just the cop). I've taken the liberty of editing your title.

How so? Could you elaborate? Let's say in my rubocop YAML - File I decide to disallow any kind of Parallel Assignment. The autofix provided doesn't "fix" it to correct form and leads to a bug in code. I have a certain location in code where this is used to swap page ordering and due to this, sooner or later, all pages move towards one and the same position number, which doesn't happen in the "before fix" form, ie. parallel assignment. The fix leads to a code, where instead of a swap, an overwrite occurs. What am I missing?

Let's say in my rubocop YAML - File I decide to disallow any kind of Parallel Assignment

Unless I'm mistaken, there is no such setting. I don't think there should be one either; such parallel assignment is both convenient and expressive.

Hmm, O.K., I see how the cop works then. In that case fixing the cop should suffice. Thanks for the clarification.
Whether or not there should be an option to disallow parallel assignment or not: I think it would be interesting to have it, if one might deem use of parallel assignment not convenient / not expressive for their code base, but I assume that's too much of an edge case to provide as an option.

Hmm, this issue sounds familiar. This might be an edge case of an existing condition that we check for. This seems related to this commit 00b4c9034 2016-10-01 17:39:31 +0200 Jonas Arvidsson | [Fix #3286] Handle self.a, self.b = b, a in ParallelAssignment. I agree with @marcandre interpretation that this cop should be modified to allow the case of variable reassignment (variable swapping?).

Am I misunderstand something, but the current implementation of Style/ParallelAssignment already accepts the variable reassignment at https://github.com/rubocop-hq/rubocop/blob/master/lib/rubocop/cop/style/parallel_assignment.rb#L56 by using topolocial sort. It's also stated in the test. I tested the sample code but found no offenses in the latest rubocop version.

@marcandre As a previous commenter, I can't reproduce this on a stated version (0.86) nor on master.
I think, this should be closed.


Favicon Causes DNS issues.

How are you hosting the Favicon for the chameleon favicon that site defaults to when no favicon is set ?
It seems when you don't set a favicon it tries to use that default icon and messes with the DNS and seems to inject the hosted favicon host into the URLS inside the application.
Example of Injection
It seems that upon setting a favicon it eliminates the problem but I think this might be an issue with the core of the application. @owen2345

@ajmyers01
I tried to find the problem but without success, could you please give me steps to reproduce the problem?


decide how to compute "g2v score" from new v2g table

My proposed scoring system is here https://github.com/opentargets/v2g_data/tree/master/v2g_score_prototype
We need to look at implementing this in SQL.
The above will only calculate per variant assignments (v2g score). We also want an additional layer that, for a given index variant, aggregates over tag variants to give a locus-to-gene assignment (l2g score).

Can you write some pseudocode of the scoring logic for me to look at?

Our cis-regulatory data is structured as:
gs://genetics-portal-staging/v2g/{data type}/{experiment type}/{source}/{version}/{tissue or cell line}/{chroms}.processed.tsv.gz
E.g. here's one tissue from each of our data sources
gs://genetics-portal-staging/v2g/interval/dhscor/thurman2012/180703/unspecified/1-23.processed.tsv.gz
gs://genetics-portal-staging/v2g/interval/fantom5/andersson2014/180703/unspecified/1-23.processed.tsv.gz
gs://genetics-portal-staging/v2g/interval/pchic/javierre2016/180703/Endothelial_precursors/1-23.processed.tsv.gz
gs://genetics-portal-staging/v2g/qtl/eqtl/gtex_v7/180703/Adipose_Subcutaneous/1-23.cis_reg.processed.tsv.gz
gs://genetics-portal-staging/v2g/qtl/pqtl/sun2018/180703/Blood_plasma/1-22.pval2.5e-05.cis_reg.processed.tsv.gz

We would like to maintain our ability to quickly add new datasets without changing the scoring algorithm too much (except maybe adding new weights). We therefore want something generalisable and non-parametric, as scores between the different experiment types are not necessarily comparable.
# Pseudocode

This is assuming we've fixed for a single variant. In practice, we'll calculate for all variants in a region.

## Preprocessing (applied across whole dataset)
- Create a score column
  * QTL datasets: 1 - pvalue
  * Interval datasets: interval score
  * VEP: map to https://github.com/opentargets/postgap-api/blob/master/data-prep/eco_scores.tsv
- Group by (source, tissue), transform into quantiles

## Aggregate across tissues (per source)
- Group by (source, gene)
- Calculate features: (i) max score, (ii) non-null tissue count
- Scale feature (ii) to 0-1, so that its on a similar scale to (i)
- Take weighted mean of features (i) and (ii) using ft_weights (see below)
- This gives a score per gene for each source

## Aggregate across sources (per variant)
- Group by (gene)
- Caluclate weighted mean over sources using source weights (see below)
- This gives a score per gene for a given variant

ft_weights = {
    'vep':           {'ft_max': 1,   'ft_count_scaled': 0},
    'javierre2016':  {'ft_max': 0.8, 'ft_count_scaled': 0.2},
    'andersson2014': {'ft_max': 0.8, 'ft_count_scaled': 0.2},
    'thurman2012':   {'ft_max': 0.8, 'ft_count_scaled': 0.2},
    'gtex_v7':       {'ft_max': 0.8, 'ft_count_scaled': 0.2},
    'sun2018':       {'ft_max': 0.8, 'ft_count_scaled': 0.2}
}

source_weights = {
    'vep': 1,
    'javierre2016': 1,
    'andersson2014': 1,
    'thurman2012': 1,
    'gtex_v7': 1,
    'sun2018': 1
}

(Above weights are just placeholders)

As mentioned above, we may actually want to calculate a 2nd score (locus-to-gene) which will be used for the main platform (as well as places in the genetics portal). In this, for a given index variant we will aggregate over tag variants in the LD or finemapping set using another weighted mean.

i have been working on translating this plan in sql.
a few things worth pointing out:

the goal is to calculate the score locally and dynamically, not precompute it for all variant-2-gene combos
when we translate this into sql there is probably no need to take (1- pvalue), we just sort in the opposite way.
the quantile transform currently used by @edm1 may be complicated to translate, but we can calculate quantiles with clickhouse and bigquery approximations and assign each value in the buckets ( as integers)

OK this looks good. The one thing we might want to consider is protein coding change consequences and whether these should have a preferential higher priority. This was the idea in the other OT score. Otherwise you will have occurrences where there is a protein coding chane in one gene being assigned to another gene perhaps fro technical reasons e.g. there is a nearby intronic enhancer, and the functional genomics data captures the exon because of a restriction enzyme cut, or similar.

In VEP, the consequences are assigned at the transcript level, which we have then mapped to gene. We haven't filter out on gene-type at all.
Would the solution just be a matter of re-jigging the consequence scores outlined here?
Or can one consequence term be found in both a protein-coding and non-protein-coding gene, meaning we would have to take into account the gene-type as well as the consequence term?

we are planning to include gene_type in the next iteration of the v2g table, which will allow us to determine if we have cases in which severe consequences land on non-coding genes.
To clarify, we store all VEP consequences (and the gene each maps to, through the transcript), not the output of --most_severe, because we grab the data from the dump and not the tool/api. However, from the VEP docs:

--most_severe |   Output only the most severe consequence per variant. Transcript-specific columns will be left blank. Consequence ranks are given in this table.To include regulatory consequences, use the --regulatory option in combination with this flag.Not used by default

we can tell that --most_severe outputs the HIGH consequences from this table
Our options:

use the scoring you have been using in postgap
score the consequences as 1 to 4 from HIGH to MODIFIER as in the consequence table from the original VEP
start with 2 and then further refine from HIGH to HIGH and SLAMDUNKS

Let's compare the scoring that's in the postgap score with the VEP Impacts - I think these were developed in parallel. It looks to me like we can go with either just HIGH as being the ones where VEP dominates, or HIGH plus MODERATE

thought: would the fact that a qtl is found in a single tissue make the connection stronger, rather than weaker?
in that case we might want to invert the ranking based on tissue count

If it's the "right" tissue. Ideally you'd want to know whether the tissue profile fits with the tissue profile for the disease. For instance if you were looking at height and there was just an eqtl in liver, say, you might not be so interested, but if it was NASH then you would.
Are we rewarding more eqtls across more tissues? Might be better to reward the min p value across all tissues

We currently calculate two features for the QTLs: max score (which equates to min p-value across tissues), and 'number of tissues' found in. Personally, I don't think the 'number of tissues' is very useful as a predictor. It depends entirely on the architecture of each disease.

I agree - max score is the useful one. Later if we could identify the expectation for the disease in terms of tissues we could try to match that up with the eQTL tissues

@mkarmona here is the updated pseudocode taking into account the above discussion
# Pseudocode v2

This is assuming we've fixed for a single variant. In practice, we'll calculate for all variants in a region.

## Preprocessing (applied across whole dataset)
- Create a score column
  * QTL datasets: 1 - pvalue
  * Interval datasets: interval score
  * VEP: map to https://github.com/opentargets/postgap-api/blob/master/data-prep/eco_scores.tsv
- Group by (source, tissue), transform into quantiles (I suggest 10 bins [deciles] but this is flexible)

## Aggregate across tissues (per source)
- Group by (source, gene)
- Calculate max score across tissues
- This gives a score per gene for each source

## Aggregate across sources (per variant)
- Group by (gene)
- Caluclate weighted mean over sources using source weights (see below)
- This gives a score per gene for a given variant

source_weights = {
    'vep': 1,
    'javierre2016': 1,
    'andersson2014': 1,
    'thurman2012': 1,
    'gtex_v7': 1,
    'sun2018': 1
}

This is a temporal quick hack as I didnt properly format vep so I am not taking it into account at this moment
create materialized view ot.v2g_quantiles engine=Memory populate as
select source_id, feature,
  quantilesIf(0.10, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)(1 - qtl_pval, qtl_pval
> 0) as qtl_quantiles,
  quantilesIf(0.10, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)(interval_score, qtl_pval = 0) as interval_quantiles
from ot.v2g
where source_id <> 'vep'
group by source_id, feature

To continue with next step we must fix a variant, so by example 16_53800954_T_C which is the FTO example

WIP
select *from (select source_id, gene_id, feature, qtl_score, interval_score, qtl_quantiles, interval_quantiles from ( select source_id, gene_id, feature, maxIf(1 - qtl_pval, qtl_pval > 0) as qtl_score , maxIf(interval_score, qtl_pval = 0) as interval_score from ot.v2g prewhere chr_id = '16' and position = 53800954 and variant_id = '16_53800954_T_C' and source_id <> 'vep' group by source_id, gene_id, feature) all inner join (select source_id, feature, qtl_quantiles, interval_quantiles from ot.v2g_quantiles) using source_id, feature) array join interval_quantiles, qtl_quantiles, arrayEnumerate(interval_quantiles) as interval_quantiles_pos

pretty printed
SELECT *
FROM
(
    SELECT
        source_id,
        gene_id,
        feature,
        qtl_score,
        interval_score,
        qtl_quantiles,
        interval_quantiles
    FROM
    (
        SELECT
            source_id,
            gene_id,
            feature,
            maxIf(1 - qtl_pval, qtl_pval > 0) AS qtl_score,
            maxIf(interval_score, qtl_pval = 0) AS interval_score
        FROM ot.v2g
        PREWHERE (chr_id = '16') AND (position = 53800954) AND (variant_id = '16_53800954_T_C') AND (source_id != 'vep')
        GROUP BY
            source_id,
            gene_id,
            feature
    )
    ALL INNER JOIN
    (
        SELECT
            source_id,
            feature,
            qtl_quantiles,
            interval_quantiles
        FROM ot.v2g_quantiles
    ) USING (source_id, feature)
)
ARRAY JOIN
    interval_quantiles,
    qtl_quantiles,
    arrayEnumerate(interval_quantiles) AS interval_quantiles_pos

a new drafted query with precomputed percentiles with a fixed variant 16_53800954_T_C
SELECT
    gene_id,
    ifNull(max(interval_score_q) * 1., 0.) AS mi,
    ifNull(max(qtl_score_q) * 1., 0.) AS mq,
    ifNull(max(fpred_max_score) * 1., 0.) AS mf,
    ((mi + mq) + mf) / 3 AS score
FROM ot.v2g
PREWHERE (chr_id = '16') AND (position = 53800954) AND (ref_allele = 'T') AND (alt_allele = 'C')
GROUP BY gene_id
ORDER BY score DESC

┌─gene_id─────────┬──mi─┬──mq─┬─mf─┬───────────────score─┐
│ ENSG00000176842 │ 0.7 │   0 │  0 │  0.2333333333333333 │
│ ENSG00000140718 │   0 │ 0.2 │  0 │ 0.06666666666666667 │
└─────────────────┴─────┴─────┴────┴─────────────────────┘

2 rows in set. Elapsed: 0.009 sec. Processed 73.73 thousand rows, 2.59 MB (8.04 million rows/s., 282.77 MB/s.)

one of the problems associated with the defined way to score is the triple aggregation: across tissues, then across sources and everything across genes

Temporally solved as
create table if not exists ot.v2g_score_by_source
engine MergeTree partition by (source_id, chr_id) order by (variant_id, gene_id)
as select
  chr_id,
  variant_id,
  gene_id,
  source_id,
  max(ifNull(qtl_score_q, 0.)) AS max_qtl,
  max(ifNull(interval_score_q, 0.)) AS max_int,
  max(ifNull(fpred_max_score, 0.)) AS max_fpred,
  max_qtl + max_int + max_fpred as source_score
from ot.v2g
group by source_id, chr_id, variant_id, gene_id

across all sources
create table if not exists ot.v2g_score_by_overall
engine MergeTree partition by (chr_id) order by (variant_id, gene_id)
as select
  chr_id,
  variant_id,
  gene_id,
  avg(source_score) as overall_score
from ot.v2g_score_by_source
group by chr_id, variant_id, gene_id

And a query using the overall score
select
 index_variant_id,
 top_genes,
from
 (
   select
     index_variant_id
   from ot.v2d_by_stchr
   prewhere stid = 'NEALEUKB_50'
   group by index_variant_id
 )
   all inner join
 (
   select
     variant_id as index_variant_id,
     groupArray(tuple(gene_id,overall_score)) as top_genes,
   from ot.d2v2g_score_by_overall
   prewhere
     variant_id = index_variant_id and
     overall_score >= 0.9
   group by variant_id
 )
using index_variant_id
limit 10

returns
┌─index_variant_id─┬─top_genes─────────────────┐
│ 10_126290161_C_T │ [('ENSG00000189319',0.9)] │
└──────────────────┴───────────────────────────┘
┌─index_variant_id─┬─top_genes───────────────┐
│ 10_104285594_A_T │ [('ENSG00000138111',1)] │
│ 4_1341553_A_G    │ [('ENSG00000179979',1)] │
└──────────────────┴─────────────────────────┘
┌─index_variant_id─┬─top_genes─────────────────────────────────────────┐
│ 17_46123004_G_A  │ [('ENSG00000005243',0.9)]                         │
│ 11_75275479_T_G  │ [('ENSG00000149273',0.9)]                         │
│ 12_110016450_G_A │ [('ENSG00000135093',0.9),('ENSG00000256351',0.9)] │
│ 20_34025756_A_G  │ [('ENSG00000101019',1),('ENSG00000242372',0.9)]   │
│ 17_76790279_C_T  │ [('ENSG00000108669',1)]                           │
└──────────────────┴───────────────────────────────────────────────────┘
┌─index_variant_id─┬─top_genes───────────────────────────────────────────────────────────────────────────────────────┐
│ 17_7204078_A_T   │ [('ENSG00000004975',0.9),('ENSG00000161956',0.9),('ENSG00000132507',1),('ENSG00000161960',0.9)] │
│ 1_218520995_A_G  │ [('ENSG00000143353',1),('ENSG00000067533',1)]                                                   │
└──────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘

10 rows in set. Elapsed: 0.163 sec. Processed 3.29 million rows, 116.25 MB (20.18 million rows/s., 713.55 MB/s.)

I have added a file containing source_id weights for the scoring algorithm to gs://genetics-portal-data/lut/v2g_scoring_source_weights.tsv.
These weights are completely arbitrary. The weighted average is:
( Σ xi * wi ) / C
where C = sum(weights in the file) is constant across variants. This should allow scores to be compared across variants, e.g. when calculating an index-to-gene score (in the future).

Done at clickhouse code level. @edm1 I already pushed my changes to the clickhouse loader scripts. It is running for the current data and it will be finished in a couple of hours


fix(WebSocket): add "client.id" property

When implementing broadcasting, there has to be an identifier for each client connection. The client.id string serves as the identifier.
On broadcasting API in the Interceptors
I believe that the Interceptors library should remain low-level. This means that higher-level features such as event broadcasting have to be implemented by higher-level libraries like MSW.

Released: v0.26.4 🎉
This has been released in v0.26.4!

📄 Release notes
📦 npm package

Make sure to always update to the latest version (npm i @mswjs/interceptors@latest) to get the newest features and bug fixes.

Predictable release automation by @ossjs/release.


Tracing extension: take currently active span as parent

Description of the feature
We're using tracing to build distributed tracing across a few services. We would like everything that happens for a request to appear inside a single trace. To do that service A calls service B and sends across tracing headers. We then extract those headers and create a new span with trace and span id from the headers. We're using a combination of async-graphql, tide, opentelemetry,tracing, and tracing-opentelemetry to achieve that.
The Tracing extension is super useful. It's very easy to use and reports everything we could every want through tracing. But at the moment it forces a new trace for every GraphQL query:
https://github.com/async-graphql/async-graphql/blob/79ab41487c914d9f5c7d4e72dbdeecf9d6dc05a0/src/extensions/tracing.rs#L42
Without parent: None the GraphQL spans would be part of the current trace.
It would be amazing if we could either emove parent: None altogether or make this behaviour configurable. I'm not sure what the best practice is for libraries with tracing support.
I'd be happy to submit a PR if we can agree on the solution 🙂

Released v2.0.14.

Thanks for working on this so quickly. Using Tracing::with_parent, is it possible to have a different parent span for each HTTP request?
A minimal sample of our code looks like this at the moment:
use async_graphql::{Context, EmptyMutation, EmptySubscription, Object, Schema};
use async_std::task;
use tide::{Middleware, Next, Request};
use tracing_futures::Instrument;
type Result<T> = std::result::Result<T, Box<dyn std::error::Error + Send + Sync>>;

struct TracingMiddleware;

#[async_trait::async_trait]
impl<State: Clone + Send + Sync + 'static> Middleware<State> for TracingMiddleware {
    async fn handle(&self, req: Request<State>, next: Mext<'_, State>) -> tide::Result {
        let span = tracing::info_span!(
            "request",
            http.method = req.method().as_ref(),
            http.target = req.url().as_ref()
        );
        let res = next.run(req).instrument(span).await;
        Ok(res)
    }
}

struct QueryRoot;

#[Object]
impl QueryRoot {
    async fn hello(&self, ctx: &Context<'_>) -> String {
        "World"
    }
}

#[derive(Clone)]
struct AppState {
    schema: Schema<QueryRoot, EmptyMutation, EmptySubscription>,
}

fn main() -> Result<()> {
    task::block_on(run())
}

async fn run() -> Result<()> {
    let schema = Schema::build(QueryRoot, EmptyMutation, EmptySubscription)
        .extension(async_graphql::extension::Tracing)
        .finish();

    let mut app = tide::with_state(AppState { schema });
    app.with(TracingMiddleware);
    app.at("/graphql").post(handle_graphql);
    app.listen("localhost:8000").await?;

    Ok(())
}

async fn handle_graphql(req: Request<AppState>) -> tide::Result {
    let schema = req.state().schema.clone();
    let req = async_graphql_tide::receive_request(req).await?;
    async_graphql_tide::respond(schema.execute(req).await)
}

We could move the schema builder into handle_graphql and construct a new tracing middleware. Would that be the right approach?
async fn handle_graphql(req: Request<AppState>) -> tide::Result {
    let schema = Schema::build(QueryRoot, EmptyMutation, EmptySubscription)
        .extension(async_graphql::extension::Tracing::with_parent(tracing::Span::current())
        .finish();
    let req = async_graphql_tide::receive_request(req).await?;
    async_graphql_tide::respond(schema.execute(req).await)
}

Haha, I didn't think about it clearly, let me change it again.

I added TracingConfig to the master branch, please help to see if it is what you want. @frigus02

Yes, that looks great. Thanks again for working on this so quickly. 👍


Update to new build system

Note: PR is opened as draft, this PR depends on changes that have yet to be merged into DuckDB. Opened for reviewing purposes. The PR should not be merged before the submodule is updated to DuckDB master.
This PR changes the sqlite scanner repo to the updates that have been made to DuckDBs extension build system. These changes will make sure:

duckdblabs/sqlite_scanner CI is passing again and actually tests the architectures we deploy this extension for
sqlite_scanner test can be run from DuckDB's CI to prevent breaking the extension with upstream changes
sqlite_scanner can be statically linked (its currently disabled though, as it breaks on windows)

Follow ups:

open issue for TODO: fix needing to specify the schema here that i added to the tests

Looks great, thanks!