Instrumental Variables - Justifying Instruments

Just a draft PR for the moment.
Pretty happy with the example. Need to add some more write up and discuss if we want to add JAX/Numpyro as a dependency to the package.

For discussion @drbenvincent , @juanitorduz  how do you think we should handle the Jax/Numpyro install. I've modified the IV class here to sample the PPC using the experimental JAX flag @jessegrabowski recomended, but i'm unsure if you want to make this a default.

Just a quick comment based on an initial look on my phone... This seems to remove sampling from the prior altogether?

It did, but need not stay that way...   Jessie's trick only seemed to work for posterior sampling, so prior sampling of mvNormal remains slow on prior checks.
However, I could keep the prior sampling but only sample the beta parameters which is fast.
Could also just default to only sample for the IV class and show how to do prior and posterior checks in the notebook after the original fit...
Options sort of depend on what you want to do with Jax and how integral it should be to CausalPy installation?

You can do it for prior as well, but you have to freeze the model before you sample, as in:
with pm.Model() as m:
    ...

from pymc.model.transform.optimization import freeze_dims_and_data

with freeze_dims_and_data(m):
    prior = pm.sample_prior_predictive(compile_kwargs={'mode':'JAX'})

I am going to open an issue for JAX forward sampling support, because it's really nice and all these hoops are silly.

Failing remote tests will be fixed when #346 is merged

Options sort of depend on what you want to do with Jax and how integral it should be to CausalPy installation?

What would be the main con's of adding that as a dependency?

I guess it's just heavier and @jessegrabowski 's trick to speed up the mvNormal ppc, just seems hacky and maybe things change down the road...
On the plus side you enable numpyro sampling which is great.
I think we should enable it but not bake the dependency into the model fit step. Instead, code defensively. Keep the IV fit method light but demonstrate usage in the notebook docs....

The preference is to stay light, but if we get nifty new functionality then I don't see a fundamental problem with adding dependencies.
So in your proposal you'd use a cell magic which conda installs numpyro so that it's just there locally. That should work.
It's also worth remembering that in the tests/doctests the IV ones are the slowest. If they can be sped up then that would be great. But that probably would require adding dependencies?

PS if you update from main all the checks should pass now

Trying to play with the freeze dim approach for prior predictive sampling and getting a JAX not implmented error:

Think I'm in a bit of a catch 22 re: failing tests. Added new test fails in github action because we require Jax installed in the deployment environment. Remove thar function call and test coverage fails...

Added further discussion on why IV regression is particularly well-suited to bayesian inference methods

@NathanielF Just checking about the failing remote tests. Seems pretty clear that it's failing because of the new jax dependency. Not sure I explicitly thought about it before, but I assumed the remote tests would be based on the code in the PR, so should therefore pass? But it looks like it's not - so any PR that adds a new dependency will fail remote checks until it's merged? That seems a bit odd?
So just checking on this. Do you know either way, and do the tests pass for you locally?

@NathanielF Just checking about the failing remote tests. Seems pretty clear that it's failing because of the new jax dependency. Not sure I explicitly thought about it before, but I assumed the remote tests would be based on the code in the PR, so should therefore pass? But it looks like it's not - so any PR that adds a new dependency will fail remote checks until it's merged? That seems a bit odd?
So just checking on this. Do you know either way, and do the tests pass for you locally?

Yeah, I thought this strange too.  Wasn't sure how to handle it.

Before I do a proper review, can we try to rasterize many of the plots in the notebook. 6.1 MB is a bit heavy.
See info here https://matplotlib.org/stable/gallery/misc/rasterization_demo.html
It should just come down to setting a kwarg rasterized=True

Didn't know you could do that. Will adapt.

Locally I re-build environment, ran tests, and get a failing test
FAILED causalpy/tests/test_integration_pymc_examples.py::test_iv_reg - TypeError: No model on context stack.

Odd. I think it worked for me. Will rebuild environment and try myself.

Tests passing now.

Tests pass locally for me too. Still a bit confused by failing remote test.

Summary of discussion outside of GitHub... Remote test environment is currently pip installed with run: pip install -e .[test] in ci.yml, so it's looking at the pyproject.toml not the environment.yml

Yeah I kind of agree with @maresb that the jax install is heavy and a lot to add as a default. I tried to extract the jax calls here to a separate optional function, but when I removed that function call from the test, the code coverage fails... so i was caught in a catch 22.

Ideal world we'd keep the optional function call available as I have it. But up the code coverage elsewhere... so Jax doesn't need to be invoked when running the tests.

WOOOP! Works now. Removed Jax dependency and have a function call to the ppc sampler that is optional for the IV class @drbenvincent

I'm more convinced now that the pre-commit checks are not applying ruff to notebooks (see https://github.com/pymc-labs/CausalPy/issues/340). I'll see if I (or someone) can get that issue sorted

Seemed to work on pre-commit for me. At least it stopped me a few times
when e.g. I had an import statement not in the first cell
On Thu 13 Jun 2024, 21:02 Benjamin T. Vincent, @.***>
wrote:

I'm more convinced now that the pre-commit checks are not applying ruff
to notebooks (see #340 https://github.com/pymc-labs/CausalPy/issues/340).
I'll see if I (or someone) can get that issue sorted
—
Reply to this email directly, view it on GitHub
https://github.com/pymc-labs/CausalPy/pull/345#issuecomment-2166667985,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACQZAB43KNXFVPPF7VNV5N3ZHH3FXAVCNFSM6AAAAABIXJB7F2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGY3DOOJYGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

If you could review/approve https://github.com/pymc-labs/CausalPy/pull/352, once it's merged then you can update from main and it should fix up the minor formatting issues in the notebook.

Thanks adjusted. Added link to wikipedia article on the credibility revolution. It's useful language to have.

View entire conversation on ReviewNB

Removed LATE from title and added glossary link

View entire conversation on ReviewNB

Hid code and expanded explanation

View entire conversation on ReviewNB

Hid code

View entire conversation on ReviewNB

Hid code and expanded explanation

View entire conversation on ReviewNB

Done.

View entire conversation on ReviewNB

Added more signposting

View entire conversation on ReviewNB

Checked headers

View entire conversation on ReviewNB

Yep. Added

View entire conversation on ReviewNB

Added a one-liner

View entire conversation on ReviewNB

yep, removed.

View entire conversation on ReviewNB

added!

View entire conversation on ReviewNB

Hid code

View entire conversation on ReviewNB

I wasn't sure about changing the DAG here. I'm not sure it adds more as it's still the NEAR variable just adding more species of college... I think it's fine without a specific DAG.

View entire conversation on ReviewNB

Added in the detail

View entire conversation on ReviewNB

Adjusted

View entire conversation on ReviewNB

Updated

View entire conversation on ReviewNB


On import from GCS, default skill cause unspent points, and skill points to be empty

If a skill is added to a character sheet in GCS with no points spent on it (i.e. just to keep the default skill level of a commonly used skill easily accessable) the unspent points and skill points fields seem to be empty. Not sure if there is any other effects.

Interesting.    Fortunately, no, those fields are only for display.  They have no other effect on the character sheet.


Emit events not working on Vue3 +3.0.5 versions

Hi! There is a problem trying to handle revogrid events on current vue 3 versions.
I prepared two codesandbox working / not working.
Working on vue 3.0.5 or lower versions:
https://codesandbox.io/s/nd9g2?file=/src/App.vue
Not working on 3.0.6 or newer versions:
https://codesandbox.io/s/o37hj?file=/src/App.vue
Steps to reproduce:

Click on cell for trigger beforeCellFocus component event or click on column header for trigger beforeSorting component event.
Then look at to console messages.

Waiting for your response. Thank you so much.

@beforesorting works


Resource creation fails when having circular dependencies using referencers

What happened?
It does not seem possible to create objects with circular dependencies. A practical use case for this is an AWS security group which allows inbound traffic from other members of the same security group:
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: self-referential
spec:
  forProvider:
    region: eu-central-1
    vpcIdRef:
      name: sample-vpc
    groupName: self-referential
    description: self-referential security group
    ingress:
      - fromPort: 0
        toPort: 65535
        ipProtocol: tcp
        userIdGroupPairs:
          - groupIdRef: self-referential
  providerConfigRef:
    name: default

How can we reproduce it?
Create object above
Name:         self-referential
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  ec2.aws.crossplane.io/v1beta1
Kind:         SecurityGroup
...
Status:
  At Provider:
    Owner Id:
    Security Group ID:
  Conditions:
    Last Transition Time:  2022-01-21T20:36:38Z
    Message:               cannot resolve references: spec.forProvider.ingress[0].userIdGroupPairs[0].groupId: referenced field was empty (referenced resource may not yet be ready)
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                           Age                From                                         Message
  ----     ------                           ----               ----                                         -------
  Warning  CannotResolveResourceReferences  14s (x7 over 75s)  managed/securitygroup.ec2.aws.crossplane.io  cannot resolve references: spec.forProvider.ingress[0].userIdGroupPairs[0].groupId: referenced field was empty (referenced resource may not yet be ready)

What environment did it happen in?
Crossplane version: 1.6.1, provider-aws 0.23, EKS

The reason it gets stuck at resolving references is that resolver function call has to succeed for the rest of the reconciliation pass to continue. One option could to make it work in such cases is to add a policy field to reference type that will let users toggle the behavior. For example, the execution could continue even if the following reference cannot be resolved:
        userIdGroupPairs:
          - groupIdRef:
              name: self-referential
              policy: Optional

This way, it would move on to call Create which will populate the field that the reference resolver is looking for.

@negz What are your thoughts? @chlunde would you be interested in implementing this?

Some kind of 'policy' field for references makes sense. Not sure whether it should be under Ref or next to it. I think https://github.com/crossplane/crossplane-runtime/issues/250 is another place a policy could apply (always resolve or resolve only if unset) so we might want to think about both in unison when designing the API.

Would it be possible to workaround this issue, in the specific case of SecurityGroup.spec.forProvider.ingress.userIdGroupPairs, to use groupIdSelector instead of groupIdRef?
doc link<EMAIL_ADDRESS>Or even when using a selector instead of a direct ref, the reference has to be fulfilled at object creation time?
Is there any workaround to this scenario?

I am not sure if there is a work around. Here we resolve the references and if any of them fails, it returns with an error. The failure would mean that the final value could not be retrieved.


Crash from Sentinel issue on arm 32-bit

In what area(s)?

/area runtime

What version of Dapr?

1.2.2

Expected Behavior
No crash at startup.
Actual Behavior
dapr v1.2.2 crashes at startup on arm 32-bit due to an integer overflow issue in Sentinel v1.0.2, as I reported upstream, including a stack trace. What's odd is that I have not configured dapr to actually use Sentinel -- no component or configuration yaml.
I have found two ways to workaround the error. First, comment out references to Sentinel in cmd/daprd/main.go. Second, use a version of Sentinel patched for the overflow. See Sentinel patch and use.
Steps to Reproduce the Problem
You should be able to reproduce just by running an app with dapr v1.2.2 on an arm 32-bit platform, like a Raspberry Pi 3.
Release Note
RELEASE NOTE: UPDATE Runtime dependency.

Seems to need a fix in upstream, just use this patch directly.

/assign
@yaron2 if not emergency, will wait for next release of upstream.

Not an emergency from my perspective. Thanks for resolving.

Assigned to @daixiang0

@daixiang0 Can you test it on an ARM32 bits.

@artursouza I am afraid that I can not since no ARM32 machine, just run it within hacked container.

@daixiang0 No worries.
@kb2ma Can you validate the fix from master branch?

@artursouza  Sure, I'd be happy to test.

Seems like this was fixed, but is once again breaking on :latest

@daixiang0  Can this be fixed again and validated as part of the build? I see that we are skipping running tests in arm32.


Props not available with Vue + TSX

Describe the bug
I am using Vue with TSX. The props object does not contain the property I expect. It is available on attrs.foo, but this is not what I was expecting (maybe I misunderstood how the TSX plugin works)?
import { defineComponent } from "vue";

const Bar = defineComponent<{ foo: string }>({
  setup(props) {
    return () => <h1>{props.foo}</h1>
  }
})

export const App = defineComponent({
  setup() {
    return () => <Bar foo='foo' />
  }
})

Reproduction
https://github.com/lmiller1990/vue-tsx-props-bug
Steps to reproduce
yarn
yarn dev

Code is in src/App.tsx.
System Info
System:
    OS: macOS 12.2.1
    CPU: (8) x64 Apple M1 Pro
    Memory: 147.48 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v16.16.0/bin/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm
  Browsers:
    Chrome: 106.0.5249.119
    Firefox: 100.0
    Safari: 15.3
  npmPackages:
    @vitejs/plugin-vue: ^3.1.0 => 3.1.2
    @vitejs/plugin-vue-jsx: ^2.0.1 => 2.0.1
    vite: ^3.1.0 => 3.1.8

### Used Package Manager

yarn

### Logs

_No response_

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

Argh dup of https://github.com/vuejs/babel-plugin-jsx/issues/607 I guess


Hairline separators made thinner on course dashboard screen

https://openedx.atlassian.net/browse/MA-1268
I've removed some unused attributes and standardized some styles as well in this PR.
Looks like this now:

@aleffert @bguertin @1zaman plz review

:+1:


Add SalModernization plugin

Add SalAnalyzer plugin, rule 105

"ValidatorsAssemblyName": "SalAnalyzer.dll",

since we don't have a validator, we can remove this line

Refers to: Src/Plugins/SalAnalyzer/SEC105.CorrectUseOfSal.json:2 in a49d86a. [](commit_id = a49d86a8fcdf9e64ce04d0e555dde164324502a3, deletion_comment = False)

"ValidatorsAssemblyName": "SalAnalyzer.dll",

I suggest that we create a stub analyzer anyway, in the event that we'll need it.

In reply to: 810627200 [](ancestors = 810627200)

Refers to: Src/Plugins/SalAnalyzer/SEC105.CorrectUseOfSal.json:2 in a49d86a. [](commit_id = a49d86a8fcdf9e64ce04d0e555dde164324502a3, deletion_comment = False)

rename the pr as well :)

Refers to: Src/Plugins/SalAnalyzer/SalAnalyzer.csproj:1 in a49d86a. [](commit_id = a49d86a8fcdf9e64ce04d0e555dde164324502a3, deletion_comment = False)

@jameswinkler , change is good. Let's make one more test with Yong. If that works fine, we can merge :)
Well done!

#define __allocator                         SAL_L_Source(__allocator, (), __inner_allocator)

I don't think an example that itself defines the use of _allocator is a great example. Am I missing something in this test??

Refers to: Src/Plugins/Tests.SalModernization/TestData/UpdateSalToCurrentVersion/Inputs/SEC105_001.AnalysisAssume.cpp:1 in 23024c2. [](commit_id = 23024c2f4b0cf4a8ac8aaf8858788fbab0c7e15f, deletion_comment = False)

#define __allocator                         SAL_L_Source(__allocator, (), __inner_allocator)

Not missing anything, I just googled some of the expressions mentioned and copy and pasted the first example code I could find.  Perhaps these are counter-examples if the fixes we were given don't work?  Some of them seemed real suspect, to be honest.  I'm not sure what to do about bad data!

In reply to: 812933478 [](ancestors = 812933478)

Refers to: Src/Plugins/Tests.SalModernization/TestData/UpdateSalToCurrentVersion/Inputs/SEC105_001.AnalysisAssume.cpp:1 in 23024c2. [](commit_id = 23024c2f4b0cf4a8ac8aaf8858788fbab0c7e15f, deletion_comment = False)


Automation of Multiple Activities on a Single Test.

From<EMAIL_ADDRESS>on April 22, 2013 05:52:49
Hi,
I am working on automating an application with robotium, which has a Facebook button, clicking on the button opens the pre-installed android mms application
I am able to automate till clicking on the facebook button, but robotium does not recognize the mms application.
I have followed the instructions "Robotium for Preinstalled apps" to re sign the Mms.apk file with debug key and push it to the device(emulator)
Please let me know if it is possible to automate activities which are opened when clicking on some button in the test activity. I have attached the test script for your reference.
Also, specifying the activity name "com.android.mms.ui.ConversationList" returns NoClassDefFound Exception (at line 72 in the attached file which is commented)
Appreciate your help
Attachment: TestAnd.java
Original issue: http://code.google.com/p/robotium/issues/detail?id=441

From<EMAIL_ADDRESS>on April 22, 2013 23:45:34
You are only allowed to interact with activities that belong to the same process as the start activity. Please see Q&A for more information.
Status: Invalid
Owner<EMAIL_ADDRESS>


Fix #1104: Improve lakeinfo structure and reporting

Closes #1104. Cherry picked from #1129.
TODO: In the meantime, we added tests for our dashboards. I think we should add Dash testing to this dashboard as well, but I would prefer doing it in a different PR.

Decided with Trent to add some basic unit tests to LakeInfo.

@trentmc @calina-c
I'm not sure why all of this was separated from the lake module?
we now have 3 clis that are poorly named... structured
cli/cli_module.py (main cli)
cli/cli_module_lake.py (lake cli)
lake_info/cli.py (lakeinfo cli)
the core problem is that these tools exist to serve the lake module and help operate the lake along with drop, update, and many other functionalities...
the validations/info/debuggin/operational commands should remain inside of lake module... not moved into something that is to serve FE
I believe all frontend/webapp tools should exist in a separate module
/pdr_backend/ -> serves all the core engine systems, apis, etc..
/pdr_frontend/ -> provides all implementations for all webapp requirements
all core functionality for the web apps should then be referenced from pdr_backend
pytest should first run all coverage inside pdr_backend/ and then pdr_frontend/

I have created a ticket requesting that we address this, including separating the webapp logic from engine logic, so we don't run into these problems in the future
https://github.com/oceanprotocol/pdr-backend/issues/1419

The "pdr_backend/" directory is meant to house all the code, frontend and backend and everything in between. It's meant to mirror the name of the repo itself. The way that most well structured python repos work. And, needed for pypi. For historical reasons it has "backend" in the name. Let's not change that name right now, it would be a huge pain.
All cli stuff should be in pdr_backend/cli/. It can be cli to run lake, sim, sim plots, predictoor bots, trader bots, or dashboard.
And we have many support directories. Building blocks, if you will. Ppss, util, lake, aimodel, etc etc. They interlink though each has fairly well-defined responsibility.
We should 100% definitely NOT add a new directory parallel to pdr_backend. It would totally mess with the overall structure of the repo, including compromising how it needs to behave for pypi.
And there should definitely not be a directory named frontend/, anywhere. We have sim plots, we have the feed analysis app, and now we have the dashboard app. Each gets their own subdirectory of pdr_backend/, with an appropriate label.


Remove travis-cargo from Travis build

travis-cargo is only used in the coveralls upload currently, which to my knowledge is not actually being used, as that step was also being removed as part of the original adding of JSON services PR.
Hopefully removing these dependencies can speed up the CI a little bit. 😄

This seems like as good a place to ask as any...
Should we remove doc generation/upload from the Travis CI? Now that all the crates are individually shipped, docs.rs does a good job of generating and hosting them. Removing that step from CI would shorten each CI run by ~280s (based on the most recent CI run).

So it doesn't look like it sped up CI significantly. Still, reduces dependencies and simplifies the Travis configuration.

👍 thanks!


feat(discord): webhook message

Codecov Report

Merging #75 (337ff13) into master (a40b25d) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##           master      #75   +/-   ##
=======================================
  Coverage   82.80%   82.80%
=======================================
  Files           1        1
  Lines         407      407
  Branches       85       85
=======================================
  Hits          337      337
  Misses         70       70

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

<message forward>
  <message id="114514">
    <author
      nickname={"_custom_ nickname"}
      avatar={
        "https://s.gravatar.com/avatar/1dc454fa994913deea45584c0ea74084.png"
      }
    />
    <image url="https://koishi.chat/logo.png" />
    我是<b>笨蛋</b> **你是笨蛋**
  </message>
  <message id="7355608">
    <author avatar={"https://koishi.chat/logo.png"} />
    消息
  </message>
  <message>
    <quote id={session.messageId}></quote>
    <image url="https://s.gravatar.com/avatar/1dc454fa994913deea45584c0ea74084.png" />
    quote 到原 channel, 带图片 quote
  </message>
  <message>
    <quote id="114514" />
    <author nickname={"aaaa"} />
    quote 到原 channel 的第一条
  </message>
  <message>
    <author nickname={"bbbb"} />
    <at id={session.userId} />
    <quote id="7355608" />
    quote 到内子区
    @everyone
  </message>
  <message>没 author 的普通消息</message>
</message>

一条 XSS 测试
<audio url="https://huggingface.co/datasets/HuanLin/Own/resolve/main/never.mp3">

[CQ:record,file=]

一条 XSS 测试
hello world

草，怎么还有个 conflict

https://github.com/satorijs/satori/blob/93263d81eb32af8833c701e759907bab7f3bfdca/adapters/discord/src/utils.ts#L161-L163
不加这里的话，如果发一条链接，discord 会抓取预览信息，加到 embed，然后发一个 MESSAGE_UPDATE

payload 和 channel message
{
  id: '1083765668144500856',
  embeds: [
    {
      url: 'https://google.com/?123',
      type: 'link',
      title: 'Google',
      description: "Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."
    }
  ],
  channel_id: '816619123890716694',
  guild_id: '814065827053043743'
}

{
  id: '1083765668144500856',
  type: 0,
  content: 'https://google.com/?123',
  channel_id: '816619123890716694',
  author: {
      url: 'https://google.com/?123',
      title: 'Google',
      description: "Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."
    }
  ],
  mentions: [],
  mention_roles: [],
  pinned: false,
  mention_everyone: false,
  tts: false,
  timestamp: '2023-03-10T14:57:50.678000+00:00',
  edited_timestamp: null,
  flags: 0,
  components: [],
  application_id: '814066234282475520',
  webhook_id: '1083763306390638693'
}

不知道作为 adapter 要不要处理掉，上面 MESSAGE_CREATE 是允许单独出现 embeds 的，先删了看看（

上面修完合一下主分支跑下 lint 就差不多了。


How to accumulate QC files over multiple bcbio runs?

Hi Brad,
We want to accumulate the QC files of the increasing cohorts of samples that we sequence.
We re-variant call all samples when our cohorts expands (we get new samples)  to get a square genotype table over all samples.
This leads to fastqc and samtools stats being run for samples for which we already have fastq and samtools stats files. See https://github.com/chapmanb/bcbio-nextgen/issues/1807
When a (re)variant calling is finished we would like to copy the samtools and fastq zip files to another directory were we accumulate them so we later can generate on-demand (aggregated) reports on those files with multi-qc or other tools.
The only way we current see to do this is to do a find exec on the stats files in the bcbio final directory to copy them to the accumulation directory. E.g.
find /bcbio_run_123/final/ -name *zip -exec cp {} /accumulatated_QC_files/fastq \;
find /bcbio_run_123/final/ -name *txt -exec cp {} /accumulatated_QC_files/samtools \;

This would be easier if the samtools and fastq file names include not only the sample name but also samtools and  fastq in the name. These find execs currently select not just the intended QC files but also other files.
Another option would be that bcbio outputs a list of QC files per sample. I am not sure how easy / difficult it would be to implement that but it sure would be useful for accumulating the QC files over multiple bcbio analysis runs.
Thank you.
Neill

Neill;
Thanks for all this discussion, definitely re-running/combining from multiple projects is not a strong point of bcbio and happy to try and improve this.
The simplest way to do this right now is to save the sample qc directories from work/qc/sample-name. These are equivalent to the directories in final/sample-name/qc but named in an easier way for dropping into a new run. Putting them into work/qc before running will skip re-calculating them and include them in the final MultiQC report.
Unfortunately we don't have much flexibility on file naming in many cases since MultiQC needs to use the filename to determine the sample. So if we call a file samplename-samtools.txt it will call is samplename-samtools instead of samplename. This is true for any files where the same name is not embedded in the output report somewhere. So keeping the directory structure is the easiest way to approach it.
There is also #1708 which aims to something similar and make it easier to run a consolidated MultiQC following the initial run. This would be helpful in parallel to explicitly saving the work directories in re-runs, so you can re-combine with MultiQC without a re-run.
Does saving/copying the QC directories work for what you'd need?

Thank you Brad for the quick answer. I think we will try r-syncing the bcbio work QC directory to and from our QC accumulate directory per species and reference genome.

Hi @NeillGibson
I will close this guessing you got it working. We changed a lot the QC in the last months so probably that helped as well. Let us know if you find more issues.
Thanks!


Add release notes for 0.57 - 0.43

No changes needing a change description found.

You can try these changes at https://cadlplayground.z22.web.core.windows.net/typespec-azure/prs/996/
Check the website changes at https://tspwebsitepr.z22.web.core.windows.net/typespec-azure/prs/996/


Compiler API (language service) becomes incredibly slow when noEmitOnError = true

TypeScript Version:  3.0.0-dev.20180526

Search Terms: compiler api noEmitOnError
I'm building an in-memory version of the TypeScript compiler using the language service API. This allows to compile and generate declarations for files that never reach the hard disk. While debugging another issue, I found that setting noEmitOnError to true drastically increases compile times with minor changes to the compiled sources.
I'm not 100% sure that the problem doesn't come from me misusing the API, so forgive me if this is actually not an issue in TypeScript.
Code
This is too much for a short repro, so I created a repro repo here:
https://github.com/AlCalzone/virtual-tsc/tree/ts-repro
To reproduce, npm install and then npm test. The project will run a few simple compilations using the language service API. It contains two identical test cases, where one has noEmitOnError = true and the other one false.
Both test cases correctly compile correct code and both catch syntax errors.
Expected behavior:
Similar results for both.
Actual behavior:
  compiler API repro =>
    √ noEmitOnError = false (2265ms)
    √ noEmitOnError = true (13794ms)
  2 passing (16s)

Related Issues: none found

Didn't see anything obviously wrong with the way things were being called.

The time spent in noEmitOnError is in calculation of declaration diagnostics which isn't done if noEmitOnError is false.
In the test, declaration = true for compiler options (link) is set and that's the reason you are seeing this.

What exactly do you mean by "declaration diagnostics"?
Since I do need the declarations, is there a better way to detect that the compilation had errors than setting noEmitOnError to true and spending an extra 10 seconds?
I still find it weird that the time needed for that almost doubled from 2.5 to 2.7. Also when I try to reproduce the test manually using the tsc CLI, both settings for noEmitOnError take roughly 2 seconds.


fix rf2822 time recognition to deal with sub-seconds

Work for this issue will begin begin on branch work-liquid-labs/string-input/5.


Add ollama documentation

This commit introduces the initial documentation for Ollama:

Write the ollama documentation
Change model defaults to be correct
Explain how to pull model (which then avoid using a dev service)
Explain how to use the dev service

I think the ollama -deployment artifact is missing from docs

Thanks @gsmet !


README.md references non-existent example.config.json file

The Step 3 Details section of README.md instructs developers to copy a file named example.config.json, substituting in their own Discord API token. However, this file does not exist in this repository and cannot be found in this repository's history via git log --all --full-history -- "**/config.*", which can be used to search for deleted files in VCS [according to this Stackoverflow answer](git log --all --full-history -- "**/thefile.*"). This section of README.md should be updated to include updated instructions for config setup.

@jonah-dev was working on this. At some point we switched to environment variables. (I'm recalling something about Dan and Docker) Anyways, I've doubled down on them. Note that in my dev build I don't actually write this in my shell config but I run DISCORD_TOKEN="asdf" python bot.py from the terminal. I do this because I'm usually switching bots and projects often.
There are only two.

DISCORD_TOKEN is the value provided in Discord's developer portal
DISCORD_ADMINS is a string with space-separated Discord IDs

These are also important in our system unit.
Environment=DISCORD_TOKEN="asdf"
Environment=DISCORD_ADMINS="1234 5678"
WorkingDirectory=/home/jd/lfd2-bot
ExecStart=/home/jd/lfd2-bot/env/bin/python /home/jd/lfd2-bot/bot.py

@jonah-dev was working on this. At some point we switched to environment variables. (I'm recalling something about Dan and Docker) Anyways, I've doubled down on them. Note that in my dev build I don't actually write this in my shell config but I run DISCORD_TOKEN="asdf" python bot.py from the terminal. I do this because I'm usually switching bots and projects often.
There are only two.

DISCORD_TOKEN is the value provided in Discord's developer portal
DISCORD_ADMINS is a string with space-separated Discord IDs

These are also important in our system unit.
Environment=DISCORD_TOKEN="asdf"
Environment=DISCORD_ADMINS="1234 5678"
WorkingDirectory=/home/jd/lfd2-bot
ExecStart=/home/jd/lfd2-bot/env/bin/python /home/jd/lfd2-bot/bot.py

There are also two required, none-committed files that authenticate with google sheets. Let me know if you need these.
https://gsheets.readthedocs.io/en/stable/

There are also two required, none-committed files that authenticate with google sheets. Let me know if you need these.
https://gsheets.readthedocs.io/en/stable/

@JoeDailey Awesome, thanks for the explanation! I agree that environment variables are a step in the right direction for sensitive info.

@JoeDailey Awesome, thanks for the explanation! I agree that environment variables are a step in the right direction for sensitive info.


Add test coverage.

Codecov Report

Merging #97 into master will increase coverage by 1.76%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #97      +/-   ##
==========================================
+ Coverage   71.28%   73.04%   +1.76%
==========================================
  Files           9        9
  Lines         794      794
  Branches      139      139
==========================================
+ Hits          566      580      +14
+ Misses        101       86      -15
- Partials      127      128       +1

Impacted Files
Coverage Δ

src/md5.c
87.69% <0%> (-3.08%)
:arrow_down:

src/process_data.c
80.45% <0%> (+4.59%)
:arrow_up:

src/scheduler.c
58.53% <0%> (+4.87%)
:arrow_up:

src/aker_md5.c
79.54% <0%> (+18.18%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a69410a...4968af0. Read the comment docs.