Add new visualization system

The main goal is to add a new visualization system that takes the various outputs from RiboViz and displays them as a unified set of graphs on a webapp using flexdashboard, shiny, or some related software. This will use the much of code @FlicAnderson has already written for making graphs, but may incorporate some new code to make new graphs.
Peter will be part of this, can someone add him to the github? His github ID is p6t.

@john-s-f - thanks for this! Edward has added @p6t to the riboviz developer team, but I think Peter needs to accept the invite before it'll stick and apply permissions.

@john-s-f - thanks for this! Edward has added @p6t to the riboviz developer team, but I think Peter needs to accept the invite before it'll stick and apply permissions.

The issue I was working on which relates most to this issue is #193, but this was a really convoluted and kind of recursive approach to creating a .Rmd from the .R file, then creating the .html file from that .Rmd.  It all fell over a bit when it came to making sure the right argument values got handed through from Nextflow to R, to Rmarkdown step, to .html output.
The flexdashboard/shiny approach looks good!
I've also been working this week on #153 - Document Analysis Code, where I'm adding roxygen2-style comments to the functions in read_count_functions.R to detail their parameters, usage examples, and what's returned by each function.  This should be helpful in letting you guys see what code can be incorporated and what won't be covered by existing code?
An example in development on an issue branch here:
#' GetCDS5start(): Get start coordinate for gene in CDS from GFF
#'
#' Extract start locations for each gene from GFF tidy dataframe for CDS only
#'
#' @param name character; gene name
#' @param gffdf object; riboviz-format GFF in tidy data format, as created by readGFFAsDf()
#' @param ftype character; gene feature to extract start location from GFF object for. Default: "CDS"
#' @param fstrand character; which strand of GFF file data to use. Default: "+"
#'
#' @return Single 'start' value of transcriptome coordinate for named gene from CDS.
#'
#' @examples
#' gff_df <- readGFFAsDf("vignette/input/yeast_YAL_CDS_w_250utrs.gff3")
#' GetCDS5start("YAL068C", gff_df)
#'
#' @export
GetCDS5start <- function(name, gffdf, ftype="CDS", fstrand="+") {
  gffdf %>%
    dplyr::filter(type==ftype, Name == name, strand == fstrand) %>%
    dplyr::pull(start) %>%  # pull() pulls out single variable
    min
}
#TEST: GetCDS5start(): returns 1 integer value = TRUE

The issue I was working on which relates most to this issue is #193, but this was a really convoluted and kind of recursive approach to creating a .Rmd from the .R file, then creating the .html file from that .Rmd.  It all fell over a bit when it came to making sure the right argument values got handed through from Nextflow to R, to Rmarkdown step, to .html output.
The flexdashboard/shiny approach looks good!
I've also been working this week on #153 - Document Analysis Code, where I'm adding roxygen2-style comments to the functions in read_count_functions.R to detail their parameters, usage examples, and what's returned by each function.  This should be helpful in letting you guys see what code can be incorporated and what won't be covered by existing code?
An example in development on an issue branch here:
#' GetCDS5start(): Get start coordinate for gene in CDS from GFF
#'
#' Extract start locations for each gene from GFF tidy dataframe for CDS only
#'
#' @param name character; gene name
#' @param gffdf object; riboviz-format GFF in tidy data format, as created by readGFFAsDf()
#' @param ftype character; gene feature to extract start location from GFF object for. Default: "CDS"
#' @param fstrand character; which strand of GFF file data to use. Default: "+"
#'
#' @return Single 'start' value of transcriptome coordinate for named gene from CDS.
#'
#' @examples
#' gff_df <- readGFFAsDf("vignette/input/yeast_YAL_CDS_w_250utrs.gff3")
#' GetCDS5start("YAL068C", gff_df)
#'
#' @export
GetCDS5start <- function(name, gffdf, ftype="CDS", fstrand="+") {
  gffdf %>%
    dplyr::filter(type==ftype, Name == name, strand == fstrand) %>%
    dplyr::pull(start) %>%  # pull() pulls out single variable
    min
}
#TEST: GetCDS5start(): returns 1 integer value = TRUE

Update: I haven't managed to finish off #153 code comments yet, so haven't got those pushed yet, they're still in the document-anlaysis-code-flic-153 branch as yet. This is what I plan to do for a couple of hours tonight after the Hackathon Day 1 meeting.
@p6t & @john-s-f : I haven't done anything else on this issue so far beyond this update, above, but I have created a git primer guide for the hackathon here on the private bbsrc-nsf repo about the git process.  I hope this might be helpful, but it's not been road-tested yet, so if things don't make sense, @ me!
Next Steps:  It'd be great to get some of (or all of) the existing visualisation code @p6t has written into a riboviz issue branch (e.g visualization-239-p6t as per naming convention?).
Does this seem like a good first step for your side? Is there anything else we need to do first, or especially anything that I need to do first?

Update: I haven't managed to finish off #153 code comments yet, so haven't got those pushed yet, they're still in the document-anlaysis-code-flic-153 branch as yet. This is what I plan to do for a couple of hours tonight after the Hackathon Day 1 meeting.
@p6t & @john-s-f : I haven't done anything else on this issue so far beyond this update, above, but I have created a git primer guide for the hackathon here on the private bbsrc-nsf repo about the git process.  I hope this might be helpful, but it's not been road-tested yet, so if things don't make sense, @ me!
Next Steps:  It'd be great to get some of (or all of) the existing visualisation code @p6t has written into a riboviz issue branch (e.g visualization-239-p6t as per naming convention?).
Does this seem like a good first step for your side? Is there anything else we need to do first, or especially anything that I need to do first?

Sounds good to me! I'll keep in touch with you and John when questions arise.

Sounds good to me! I'll keep in touch with you and John when questions arise.

After discussion in the Day 1 Hackathon meeting, something I can work on would be making a new branch from @p6t 's branch once it's ready, where I can try and get the nextflow process integration for the visualisation step working while @p6t and @john-s-f work on adding new visusalisation features and any refactoring needed.

After discussion in the Day 1 Hackathon meeting, something I can work on would be making a new branch from @p6t 's branch once it's ready, where I can try and get the nextflow process integration for the visualisation step working while @p6t and @john-s-f work on adding new visusalisation features and any refactoring needed.

I've created a new branch off visualization-239-p6t called visualization-239-FlicAnderson to work in on getting the visualisation working as a nextflow step.
I've recently merged develop into it locally and will keep doing so to make sure I get all this good Hackathon content :grin:
I've :crossed_fingers: finished the work on #153 (pending review of PR #249) and can now move over to give this my full attention from this evening/tomorrow morning.

I've created a new branch off visualization-239-p6t called visualization-239-FlicAnderson to work in on getting the visualisation working as a nextflow step.
I've recently merged develop into it locally and will keep doing so to make sure I get all this good Hackathon content :grin:
I've :crossed_fingers: finished the work on #153 (pending review of PR #249) and can now move over to give this my full attention from this evening/tomorrow morning.

Something to do:
Add new required R packages to the docs/user/install.md documentation:
library(flexdashboard)
library(tidyverse)
library(patchwork)
library(scales)

Note, although we DO have a lot of tidyverse packages installed, we do not have the meta-package tidyverse installed.  Raised this as new issue #251.
@john-s-f & @p6t do you want to take this, or shall I add it to my branch?

Something to do:
Add new required R packages to the docs/user/install.md documentation:
library(flexdashboard)
library(tidyverse)
library(patchwork)
library(scales)

Note, although we DO have a lot of tidyverse packages installed, we do not have the meta-package tidyverse installed.  Raised this as new issue #251.
@john-s-f & @p6t do you want to take this, or shall I add it to my branch?

@john-s-f @p6t : trying to run new_visualisation.Rmd but got error::
"Quitting from lines 110-128 (new_visualization.Rmd)
Warning: Error in : Problem with mutate() input Frame 0.
x object 'Ct_fr0' not found
ℹ Input Frame 0 is Ct_fr0/(Ct_fr0 + Ct_fr1 + Ct_fr2).
156: ".
I haven't investigated this yet or looked into other files in your branch, so it might be something daft from my end.

@john-s-f @p6t : trying to run new_visualisation.Rmd but got error::
"Quitting from lines 110-128 (new_visualization.Rmd)
Warning: Error in : Problem with mutate() input Frame 0.
x object 'Ct_fr0' not found
ℹ Input Frame 0 is Ct_fr0/(Ct_fr0 + Ct_fr1 + Ct_fr2).
156: ".
I haven't investigated this yet or looked into other files in your branch, so it might be something daft from my end.

After @john-s-f updated branch -p6t to use the vignette for visualisation instead of Gupta-dataset, this has let me get the .Rmd running after a bit of tinkering with John's super newly added code for getting the paths and sample names (and use of {here} R package to take hassle out of file path issues).

Commit 718f17db contains the here() edits, as well as adding here() to library call block.
This DOES contain a high amount of comments showing structure of data etc, but these can be removed later (also mean useful bits could be pulled out for documentation a la #153 if we go down that route later? hashtag NewIssueRequiredAlert).
:question: @p6t what plans or ideas (if any) do you have for documenting the visualisation code? What do you think would be useful to have? Perhaps a visualization.md file at this level as well as anything built into the new_visualization.Rmd file?  How do you find the embedded code documentation from the #153 work? Do you think something like that might be useful for this?

After @john-s-f updated branch -p6t to use the vignette for visualisation instead of Gupta-dataset, this has let me get the .Rmd running after a bit of tinkering with John's super newly added code for getting the paths and sample names (and use of {here} R package to take hassle out of file path issues).

Commit 718f17db contains the here() edits, as well as adding here() to library call block.
This DOES contain a high amount of comments showing structure of data etc, but these can be removed later (also mean useful bits could be pulled out for documentation a la #153 if we go down that route later? hashtag NewIssueRequiredAlert).
:question: @p6t what plans or ideas (if any) do you have for documenting the visualisation code? What do you think would be useful to have? Perhaps a visualization.md file at this level as well as anything built into the new_visualization.Rmd file?  How do you find the embedded code documentation from the #153 work? Do you think something like that might be useful for this?

My next steps are: adding running new_visualization.Rmd as new process to Nextflow.
Ideally at some point soon replace yaml1 variable in function find_sample_names() in line 89 in visualization-239-p6t branch & equivalent bit in my branch with input from -params-file argument to Nextflow call, and remove yaml1 <- "..." , yaml2 <- "..." part.

My next steps are: adding running new_visualization.Rmd as new process to Nextflow.
Ideally at some point soon replace yaml1 variable in function find_sample_names() in line 89 in visualization-239-p6t branch & equivalent bit in my branch with input from -params-file argument to Nextflow call, and remove yaml1 <- "..." , yaml2 <- "..." part.

@FlicAnderson I thought it would make sense to wait until the .Rmd was relatively stable before detailed documentation, it might be there now--I'd like @john-s-f to give a thumbs-up first, but I'm willing to start. Thank you for both examples, they will be very helpful.

@FlicAnderson I thought it would make sense to wait until the .Rmd was relatively stable before detailed documentation, it might be there now--I'd like @john-s-f to give a thumbs-up first, but I'm willing to start. Thank you for both examples, they will be very helpful.

I think it's stable enough for @FlicAnderson to try and add to nextflow.
I also agree with Flics previous comment, the yaml1 and yaml2 ere simply there for me to make sure it worked with non-multiplexed and multiplexed simdata, I forgot to remove it. Pulling from the command line was my intended goal, eventually.

I think it's stable enough for @FlicAnderson to try and add to nextflow.
I also agree with Flics previous comment, the yaml1 and yaml2 ere simply there for me to make sure it worked with non-multiplexed and multiplexed simdata, I forgot to remove it. Pulling from the command line was my intended goal, eventually.

I'm still having issues trying to get the rmarkdown running from commandline.  Have either of you (@p6t , @john-s-f ) got this working or tried this yet?
Running from commandline is the logical next step I think I need to achieve before I can integrate to nextflow and I'm not having much success at the moment...

I'm still having issues trying to get the rmarkdown running from commandline.  Have either of you (@p6t , @john-s-f ) got this working or tried this yet?
Running from commandline is the logical next step I think I need to achieve before I can integrate to nextflow and I'm not having much success at the moment...

@FlicAnderson neither of us have given it a serious attempt yet. @john-s-f  mentioned converting it from R markdown to a standard R script, which might help but I'm not familiar with the process.

@FlicAnderson neither of us have given it a serious attempt yet. @john-s-f  mentioned converting it from R markdown to a standard R script, which might help but I'm not familiar with the process.

@p6t in commit 822f4766 I've added rscripts/viz.R which calls render() to create new_visualization.Rmd.  I still get an error about Error in output$input_ui <- renderUI({ : object 'output' not found:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ Rscript rscripts/viz.R

processing file: new_visualization.Rmd
  |                                                                       |                                                                       |                                                                       |...                                                                   |   4%
  ordinary text without R code

  |                                                                       |......                                                                |   8%
label: unnamed-chunk-1
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.3     ✔ dplyr   1.0.1
✔ tidyr   1.1.1     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.5.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

Attaching package: 'scales'

The following object is masked from 'package:purrr':

    discard

The following object is masked from 'package:readr':

    col_factor

here() starts at /home/fanders6/riboviz/riboviz
  |                                                                       |........                                                              |  12%
  ordinary text without R code

  |                                                                       |...........                                                           |  16%
label: unnamed-chunk-2
  |                                                                       |..............                                                        |  20%
  ordinary text without R code

  |                                                                       |.................                                                     |  24%
label: unnamed-chunk-3
  |                                                                       |....................                                                  |  28%
  ordinary text without R code

  |                                                                       |......................                                                |  32%
label: unnamed-chunk-4
  |                                                                       |.........................                                             |  36%
  ordinary text without R code

  |                                                                       |............................                                          |  40%
label: unnamed-chunk-5
Parsed with column specification:
cols(
  SampleID = col_character(),
  TagRead = col_character()
)
Parsed with column specification:
cols(
  SampleName = col_character(),
  Program = col_character(),
  File = col_character(),
  NumReads = col_double(),
  Description = col_character()
)
  |                                                                       |...............................                                       |  44%
  ordinary text without R code

  |                                                                       |..................................                                    |  48%
label: unnamed-chunk-6
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
  |                                                                       |....................................                                  |  52%
  ordinary text without R code

  |                                                                       |.......................................                               |  56%
label: unnamed-chunk-7
  |                                                                       |..........................................                            |  60%
  ordinary text without R code

  |                                                                       |.............................................                         |  64%
label: unnamed-chunk-8
  |                                                                       |................................................                      |  68%
  ordinary text without R code

  |                                                                       |..................................................                    |  72%
label: unnamed-chunk-9
  |                                                                       |.....................................................                 |  76%
label: unnamed-chunk-10
  |                                                                       |........................................................              |  80%
  ordinary text without R code

  |                                                                       |...........................................................           |  84%
label: reactive
  |                                                                       |..............................................................        |  88%
  ordinary text without R code

  |                                                                       |................................................................      |  92%
label: input
Quitting from lines 420-439 (new_visualization.Rmd)
Error in output$input_ui <- renderUI({ : object 'output' not found
Calls: <Anonymous> ... handle -> withCallingHandlers -> withVisible -> eval -> eval

Execution halted

I'm not quite sure how to re-write this to try different things.  Does the object (output$input_ui) NEED to be called output? Is that a reserved name and perhaps we're getting issues due to that?  But I think this is maybe a {shiny} thing, so it maybe that it can't be called anything else? I'm not sure...

@p6t in commit 822f4766 I've added rscripts/viz.R which calls render() to create new_visualization.Rmd.  I still get an error about Error in output$input_ui <- renderUI({ : object 'output' not found:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ Rscript rscripts/viz.R

processing file: new_visualization.Rmd
  |                                                                       |                                                                       |                                                                       |...                                                                   |   4%
  ordinary text without R code

  |                                                                       |......                                                                |   8%
label: unnamed-chunk-1
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.3     ✔ dplyr   1.0.1
✔ tidyr   1.1.1     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.5.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

Attaching package: 'scales'

The following object is masked from 'package:purrr':

    discard

The following object is masked from 'package:readr':

    col_factor

here() starts at /home/fanders6/riboviz/riboviz
  |                                                                       |........                                                              |  12%
  ordinary text without R code

  |                                                                       |...........                                                           |  16%
label: unnamed-chunk-2
  |                                                                       |..............                                                        |  20%
  ordinary text without R code

  |                                                                       |.................                                                     |  24%
label: unnamed-chunk-3
  |                                                                       |....................                                                  |  28%
  ordinary text without R code

  |                                                                       |......................                                                |  32%
label: unnamed-chunk-4
  |                                                                       |.........................                                             |  36%
  ordinary text without R code

  |                                                                       |............................                                          |  40%
label: unnamed-chunk-5
Parsed with column specification:
cols(
  SampleID = col_character(),
  TagRead = col_character()
)
Parsed with column specification:
cols(
  SampleName = col_character(),
  Program = col_character(),
  File = col_character(),
  NumReads = col_double(),
  Description = col_character()
)
  |                                                                       |...............................                                       |  44%
  ordinary text without R code

  |                                                                       |..................................                                    |  48%
label: unnamed-chunk-6
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
  |                                                                       |....................................                                  |  52%
  ordinary text without R code

  |                                                                       |.......................................                               |  56%
label: unnamed-chunk-7
  |                                                                       |..........................................                            |  60%
  ordinary text without R code

  |                                                                       |.............................................                         |  64%
label: unnamed-chunk-8
  |                                                                       |................................................                      |  68%
  ordinary text without R code

  |                                                                       |..................................................                    |  72%
label: unnamed-chunk-9
  |                                                                       |.....................................................                 |  76%
label: unnamed-chunk-10
  |                                                                       |........................................................              |  80%
  ordinary text without R code

  |                                                                       |...........................................................           |  84%
label: reactive
  |                                                                       |..............................................................        |  88%
  ordinary text without R code

  |                                                                       |................................................................      |  92%
label: input
Quitting from lines 420-439 (new_visualization.Rmd)
Error in output$input_ui <- renderUI({ : object 'output' not found
Calls: <Anonymous> ... handle -> withCallingHandlers -> withVisible -> eval -> eval

Execution halted

I'm not quite sure how to re-write this to try different things.  Does the object (output$input_ui) NEED to be called output? Is that a reserved name and perhaps we're getting issues due to that?  But I think this is maybe a {shiny} thing, so it maybe that it can't be called anything else? I'm not sure...

@FlicAnderson  output and input are created by Shiny, they're used to transfer variables between client and server. It's possible that Shiny doesn't know it's showing a flexdashboard, and is looking for standard Shiny app syntax.

@FlicAnderson  output and input are created by Shiny, they're used to transfer variables between client and server. It's possible that Shiny doesn't know it's showing a flexdashboard, and is looking for standard Shiny app syntax.

@p6t would something like this work: https://rmarkdown.rstudio.com/flexdashboard/shiny.html#inline_applications or https://rmarkdown.rstudio.com/flexdashboard/shiny.html#shiny-modules
Perhaps keeping the shiny stuff a bit modular within the larger flexdashboard .Rmd?

@p6t would something like this work: https://rmarkdown.rstudio.com/flexdashboard/shiny.html#inline_applications or https://rmarkdown.rstudio.com/flexdashboard/shiny.html#shiny-modules
Perhaps keeping the shiny stuff a bit modular within the larger flexdashboard .Rmd?

@FlicAnderson I'll work on it with @john-s-f. This will be clarified in today's meeting, but I think we'll be aiming for working command line arguments when you next wake up.

@FlicAnderson I'll work on it with @john-s-f. This will be clarified in today's meeting, but I think we'll be aiming for working command line arguments when you next wake up.

I've added a (very simple) process to prep_riboviz.nf (in this commit) which means that I now get the same error (object 'output' not found) by running riboviz via nextflow - success?? :grimacing:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [condescending_mendel] - revision: b0b02a7fe6
[0d/917f0c] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[8f/7693a4] Submitted process > buildIndicesrRNA (yeast_rRNA)
[de/fb2688] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[08/f552e9] Submitted process > dashboard
[e9/c2ef48] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[c7/39ab06] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[60/f2ba2b] Submitted process > hisat2rRNA (Tag2)
[3f/3b52ba] Submitted process > hisat2rRNA (Tag1)
[4b/b6648a] Submitted process > hisat2rRNA (Tag0)
[42/f7af12] Submitted process > hisat2ORF (Tag2)
[09/84e98a] Submitted process > trim5pMismatches (Tag2)
[1b/32e20e] Submitted process > hisat2ORF (Tag1)
[c8/5eb16b] Submitted process > hisat2ORF (Tag0)
[59/8f6326] Submitted process > trim5pMismatches (Tag1)
[cf/0a8cf9] Submitted process > trim5pMismatches (Tag0)
[85/a9f51d] Submitted process > samViewSort (Tag2)
[fc/0213ac] Submitted process > dedupUmis (Tag2)
[f2/46a532] Submitted process > groupUmisPreDedup (Tag2)
[57/641e89] Submitted process > samViewSort (Tag1)
[37/59ef1c] Submitted process > samViewSort (Tag0)
[01/9636fe] Submitted process > dedupUmis (Tag1)
[45/36fb6a] Submitted process > groupUmisPreDedup (Tag1)
[2e/c62556] Submitted process > dedupUmis (Tag0)
[a0/8a8ff8] Submitted process > groupUmisPreDedup (Tag0)
Error executing process > 'dashboard'

Caused by:
  Process `dashboard` terminated with an error exit status (1)

Command executed:

  Rscript --vanilla -e "rmarkdown::render('/home/fanders6/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"

Command exit status:
  1

Command output:

    |
    |....................................                                  |  52%
    ordinary text without R code

    |
    |.......................................                               |  56%
  label: unnamed-chunk-7

    |
    |..........................................                            |  60%
    ordinary text without R code

    |
    |.............................................                         |  64%
  label: unnamed-chunk-8

    |
    |................................................                      |  68%
    ordinary text without R code

    |
    |..................................................                    |  72%
  label: unnamed-chunk-9

    |
    |.....................................................                 |  76%
  label: unnamed-chunk-10

    |
    |........................................................              |  80%
    ordinary text without R code

    |
    |...........................................................           |  84%
  label: reactive

    |
    |..............................................................        |  88%
    ordinary text without R code

    |
    |................................................................      |  92%
  label: input

Command error:
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Quitting from lines 421-440 (new_visualization.Rmd)
  Error in output$input_ui <- renderUI({ : object 'output' not found
  Calls: <Anonymous> ... handle -> withCallingHandlers -> withVisible -> eval -> eval
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/08/f552e9c282097641c7af33c1b8c1e6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Workflow finished! (failed)
WARN: Killing pending tasks (5)

Unfortunately it seems like getting the value of -params-file (ie. the path of the yaml config file riboviz is running on) isn't going to be as simple as I expected, as @mikej888 suggests:

I create a new short YAML file using "ribosome_fqs_yaml = new Yaml().dump(ribosome_fqs)"
It might then be possible to try something like config_copy_yaml = new Yaml().dump(params)
as params has all the parameters.

I haven't tried this suggestion yet, and at least if we can crack the "object output not found" problem, then it might work with our hard-coded yaml1 parameter and we can work from there!

I've added a (very simple) process to prep_riboviz.nf (in this commit) which means that I now get the same error (object 'output' not found) by running riboviz via nextflow - success?? :grimacing:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [condescending_mendel] - revision: b0b02a7fe6
[0d/917f0c] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[8f/7693a4] Submitted process > buildIndicesrRNA (yeast_rRNA)
[de/fb2688] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[08/f552e9] Submitted process > dashboard
[e9/c2ef48] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[c7/39ab06] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[60/f2ba2b] Submitted process > hisat2rRNA (Tag2)
[3f/3b52ba] Submitted process > hisat2rRNA (Tag1)
[4b/b6648a] Submitted process > hisat2rRNA (Tag0)
[42/f7af12] Submitted process > hisat2ORF (Tag2)
[09/84e98a] Submitted process > trim5pMismatches (Tag2)
[1b/32e20e] Submitted process > hisat2ORF (Tag1)
[c8/5eb16b] Submitted process > hisat2ORF (Tag0)
[59/8f6326] Submitted process > trim5pMismatches (Tag1)
[cf/0a8cf9] Submitted process > trim5pMismatches (Tag0)
[85/a9f51d] Submitted process > samViewSort (Tag2)
[fc/0213ac] Submitted process > dedupUmis (Tag2)
[f2/46a532] Submitted process > groupUmisPreDedup (Tag2)
[57/641e89] Submitted process > samViewSort (Tag1)
[37/59ef1c] Submitted process > samViewSort (Tag0)
[01/9636fe] Submitted process > dedupUmis (Tag1)
[45/36fb6a] Submitted process > groupUmisPreDedup (Tag1)
[2e/c62556] Submitted process > dedupUmis (Tag0)
[a0/8a8ff8] Submitted process > groupUmisPreDedup (Tag0)
Error executing process > 'dashboard'

Caused by:
  Process `dashboard` terminated with an error exit status (1)

Command executed:

  Rscript --vanilla -e "rmarkdown::render('/home/fanders6/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"

Command exit status:
  1

Command output:

    |
    |....................................                                  |  52%
    ordinary text without R code

    |
    |.......................................                               |  56%
  label: unnamed-chunk-7

    |
    |..........................................                            |  60%
    ordinary text without R code

    |
    |.............................................                         |  64%
  label: unnamed-chunk-8

    |
    |................................................                      |  68%
    ordinary text without R code

    |
    |..................................................                    |  72%
  label: unnamed-chunk-9

    |
    |.....................................................                 |  76%
  label: unnamed-chunk-10

    |
    |........................................................              |  80%
    ordinary text without R code

    |
    |...........................................................           |  84%
  label: reactive

    |
    |..............................................................        |  88%
    ordinary text without R code

    |
    |................................................................      |  92%
  label: input

Command error:
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    Pos = col_double(),
    Counts = col_double(),
    End = col_character()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Parsed with column specification:
  cols(
    gene = col_character(),
    Ct_fr0 = col_double(),
    Ct_fr1 = col_double(),
    Ct_fr2 = col_double(),
    pval_fr0vs1 = col_double(),
    pval_fr0vs2 = col_double(),
    pval_fr0vsboth = col_double()
  )
  Quitting from lines 421-440 (new_visualization.Rmd)
  Error in output$input_ui <- renderUI({ : object 'output' not found
  Calls: <Anonymous> ... handle -> withCallingHandlers -> withVisible -> eval -> eval
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/08/f552e9c282097641c7af33c1b8c1e6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Workflow finished! (failed)
WARN: Killing pending tasks (5)

Unfortunately it seems like getting the value of -params-file (ie. the path of the yaml config file riboviz is running on) isn't going to be as simple as I expected, as @mikej888 suggests:

I create a new short YAML file using "ribosome_fqs_yaml = new Yaml().dump(ribosome_fqs)"
It might then be possible to try something like config_copy_yaml = new Yaml().dump(params)
as params has all the parameters.

I haven't tried this suggestion yet, and at least if we can crack the "object output not found" problem, then it might work with our hard-coded yaml1 parameter and we can work from there!

Your process is
process dashboard {
    publishDir "${params.dir_out}", mode: 'copy', overwrite: true
  // input:
  //  file ?

  // this is not the correct output, but I'm still not clear on what output new_visualization.Rmd produces?
    output:
      file x

      shell:
      """
      Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      """
}

Which declares an output file x but the process doesn't create any outputs. Try creating an empty output for now:
    output:
      file 'tmp.txt'

      shell:
      """
      Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      touch tmp.txt
      """

If there are still issues try
    output:
      file 'tmp.txt' into dashboard_output

Then after the task add
dashboard_output.subscribe { println "Received: " + it.text }

Your process is
process dashboard {
    publishDir "${params.dir_out}", mode: 'copy', overwrite: true
  // input:
  //  file ?

  // this is not the correct output, but I'm still not clear on what output new_visualization.Rmd produces?
    output:
      file x

      shell:
      """
      Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      """
}

Which declares an output file x but the process doesn't create any outputs. Try creating an empty output for now:
    output:
      file 'tmp.txt'

      shell:
      """
      Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      touch tmp.txt
      """

If there are still issues try
    output:
      file 'tmp.txt' into dashboard_output

Then after the task add
dashboard_output.subscribe { println "Received: " + it.text }

Things the new process ought to have:

better output info (possibly set to optional for now?)
when: - a flag to know if/when this process should be run. REF

Later things to do:

potentially integrate the sample_id values from nextflow instead of find_sample_names() method?

Things the new process ought to have:

better output info (possibly set to optional for now?)
when: - a flag to know if/when this process should be run. REF

Later things to do:

potentially integrate the sample_id values from nextflow instead of find_sample_names() method?

Then after the task add:
dashboard_output.subscribe { println "Received: " + it.text }

@mikej888 thanks! Just to clarify, "after the task" means "outside the dashboard process curly brackets"? Or, after the shell script section but within the braces?

Then after the task add:
dashboard_output.subscribe { println "Received: " + it.text }

@mikej888 thanks! Just to clarify, "after the task" means "outside the dashboard process curly brackets"? Or, after the shell script section but within the braces?

Yes, the former, outwith the scope of the process {...}

Yes, the former, outwith the scope of the process {...}

For creating a new YAML file for input into riboviz.tools.count_reads, the Nextflow workflow does:
// Create YAML fragment including 'params.fq_files' and
// 'params.multiplex_fq_files' to serve as a configuration file for
// riboviz.tools.count_reads. There is no way to get the location
// of the YAML configuration file itself (i.e. the value of
// '-param-file') from within Nextflow so this is a workaround.
Map ribosome_fqs = [:]
ribosome_fqs.fq_files = params.fq_files
ribosome_fqs.multiplex_fq_files = params.multiplex_fq_files
ribosome_fqs_yaml = new Yaml().dump(ribosome_fqs)

This creates a new YAML string with the fq_files and multiplex_fq_files values from the original configuration file. This is then used in process countReads which needs this configuration information to find out the location of the input files. The relevant parts of countReads are below.
process countReads {
    input:

        val ribosome_fqs_yaml from ribosome_fqs_yaml

    shell:
        """
        echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
        python -m riboviz.tools.count_reads \
           -c ribosome_fqs.yaml \
           -i ${file(params.dir_in).toAbsolutePath()} \
           -t ${file(params.dir_tmp).toAbsolutePath()} \
           -o ${file(params.dir_out).toAbsolutePath()} \
           -r read_counts.tsv
        """
}

`countReads` receives the YAML-compliant string from the `ribosome_fqs_yaml` channel. In the `shell` section this value is then dumped, using `echo`, into a local `ribosome_fqs.yaml` file in the process's `work` directory. An example of this file, from a Nextflow run, is:

$ cat work/a7/8f7dc5d0a1ff690cac939a5701d99d/ribosome_fqs.yaml
fq_files: {WTnone: SRR1042855_s1mi.fastq.gz, WT3AT: SRR1042864_s1mi.fastq.gz, N
otHere: example_missing_file.fastq.gz}
multiplex_fq_files: null

Like all Nextflow channels, the channel `ribosome_fqs_yaml` can only be used as an input by a single process. To use the channel in multiple processes needs a split:

// Split channel for use in multiple downstream processes.
ribosome_fqs_yaml.into {
count_reads_ribosome_fqs_yaml; dashboard_ribosome_fqs_yaml
}

I use a naming convention whereby the split channel names have the process that uses them within the name to make it easier to identify what channels are used where.

`process countReads` then needs updated to use one of these outputs from the `split`. The `input` line:

    val ribosome_fqs_yaml from ribosome_fqs_yaml

needs changed to:

    val ribosome_fqs_yaml from count_reads_ribosome_fqs_yaml

...

`process dashboards` can then use the other output from the `split`:

process dashboard {
...
input:
val ribosome_fqs_yaml from dashboard_ribosome_fqs_yaml
...
shell:
"""
echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
...
"""
Hope that helps. We can discuss more tomorrow!

For creating a new YAML file for input into riboviz.tools.count_reads, the Nextflow workflow does:
// Create YAML fragment including 'params.fq_files' and
// 'params.multiplex_fq_files' to serve as a configuration file for
// riboviz.tools.count_reads. There is no way to get the location
// of the YAML configuration file itself (i.e. the value of
// '-param-file') from within Nextflow so this is a workaround.
Map ribosome_fqs = [:]
ribosome_fqs.fq_files = params.fq_files
ribosome_fqs.multiplex_fq_files = params.multiplex_fq_files
ribosome_fqs_yaml = new Yaml().dump(ribosome_fqs)

This creates a new YAML string with the fq_files and multiplex_fq_files values from the original configuration file. This is then used in process countReads which needs this configuration information to find out the location of the input files. The relevant parts of countReads are below.
process countReads {
    input:

        val ribosome_fqs_yaml from ribosome_fqs_yaml

    shell:
        """
        echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
        python -m riboviz.tools.count_reads \
           -c ribosome_fqs.yaml \
           -i ${file(params.dir_in).toAbsolutePath()} \
           -t ${file(params.dir_tmp).toAbsolutePath()} \
           -o ${file(params.dir_out).toAbsolutePath()} \
           -r read_counts.tsv
        """
}

`countReads` receives the YAML-compliant string from the `ribosome_fqs_yaml` channel. In the `shell` section this value is then dumped, using `echo`, into a local `ribosome_fqs.yaml` file in the process's `work` directory. An example of this file, from a Nextflow run, is:

$ cat work/a7/8f7dc5d0a1ff690cac939a5701d99d/ribosome_fqs.yaml
fq_files: {WTnone: SRR1042855_s1mi.fastq.gz, WT3AT: SRR1042864_s1mi.fastq.gz, N
otHere: example_missing_file.fastq.gz}
multiplex_fq_files: null

Like all Nextflow channels, the channel `ribosome_fqs_yaml` can only be used as an input by a single process. To use the channel in multiple processes needs a split:

// Split channel for use in multiple downstream processes.
ribosome_fqs_yaml.into {
count_reads_ribosome_fqs_yaml; dashboard_ribosome_fqs_yaml
}

I use a naming convention whereby the split channel names have the process that uses them within the name to make it easier to identify what channels are used where.

`process countReads` then needs updated to use one of these outputs from the `split`. The `input` line:

    val ribosome_fqs_yaml from ribosome_fqs_yaml

needs changed to:

    val ribosome_fqs_yaml from count_reads_ribosome_fqs_yaml

...

`process dashboards` can then use the other output from the `split`:

process dashboard {
...
input:
val ribosome_fqs_yaml from dashboard_ribosome_fqs_yaml
...
shell:
"""
echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
...
"""
Hope that helps. We can discuss more tomorrow!

I made these changes, but get an error at line 354 here
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [naughty_becquerel] - revision: a04b1a63b1
No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
 -- Check script 'prep_riboviz.nf' at line: 354 or see '.nextflow.log' file for more details

From .nextflow.log:
Jan-14 19:10:33.588 [main] DEBUG nextflow.cli.Launcher - $> nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
# ...
Jan-14 19:10:34.955 [main] DEBUG nextflow.Session - Session aborted -- Cause: No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
Jan-14 19:10:34.970 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.MissingMethodException: No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70)
	at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
	at Script_1dd7a5f3.runScript(Script_1dd7a5f3:354)
	at nextflow.script.BaseScript.runDsl1(BaseScript.groovy:164)
	at nextflow.script.BaseScript.run(BaseScript.groovy:200)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:219)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:219)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:127)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:274)
	at nextflow.cli.Launcher.run(Launcher.groovy:466)
	at nextflow.cli.Launcher.main(Launcher.groovy:648)

I made these changes, but get an error at line 354 here
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [naughty_becquerel] - revision: a04b1a63b1
No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
 -- Check script 'prep_riboviz.nf' at line: 354 or see '.nextflow.log' file for more details

From .nextflow.log:
Jan-14 19:10:33.588 [main] DEBUG nextflow.cli.Launcher - $> nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
# ...
Jan-14 19:10:34.955 [main] DEBUG nextflow.Session - Session aborted -- Cause: No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
Jan-14 19:10:34.970 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.MissingMethodException: No signature of method: java.lang.String.into() is applicable for argument types: (Script_1dd7a5f3$_runScript_closure1) values: [Script_1dd7a5f3$_runScript_closure1@7d2a6eac]
Possible solutions: intern(), find(), any(), with(groovy.lang.Closure), is(java.lang.Object), any(groovy.lang.Closure)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70)
	at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
	at Script_1dd7a5f3.runScript(Script_1dd7a5f3:354)
	at nextflow.script.BaseScript.runDsl1(BaseScript.groovy:164)
	at nextflow.script.BaseScript.run(BaseScript.groovy:200)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:219)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:219)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:127)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:274)
	at nextflow.cli.Launcher.run(Launcher.groovy:466)
	at nextflow.cli.Launcher.main(Launcher.groovy:648)

Ah OK. I see what it is. I forgot that ribosome_fqs_yaml is not a channel, it is a value. Whereas channels need to be split (using into) to be used as inputs to multiple processes, values do not. Values can be used as-is as inputs to multiple processes.
Restore countReads to its old version:
    input:
        ...
        val ribosome_fqs_yaml from ribosome_fqs_yaml
        ...

Remove the ribosome_fqs_yaml.into block.
Update dashboard to have:
    input:
        val ribosome_fqs_yaml from ribosome_fqs_yaml

(Note: in Nextflow this essentially says create a channel within this process, ribosome_fqs_yaml, which provides the value ribosome_fqs_yaml. There is no need for the channel and value to have the same name - I do this to keep from having too many new names.)
dashboard is now:
process dashboard {
    input:
        val ribosome_fqs_yaml from ribosome_fqs_yaml
     output:
        file 'tmp.txt' into dashboard_output
    shell:
        """
	echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
	touch tmp.txt
	"""
}

Update the dashboard_output.subscribe command as I noticed it wouldn't print the output file name. It does if changed to:
dashboard_output.subscribe { println "Received from 'dashboard': $it" }

I ran a workflow with this dashboard and dashboard_output.subscribe commands:
$ nextflow run prep_riboviz.nf -params-file vignette/simdata_umi_config.yaml -ansi-log false --publish_index_tmp
N E X T F L O W  ~  version 20.04.1
Launching `prep_riboviz.nf` [pedantic_mandelbrot] - revision: a4a10691e7
[86/17fdef] Submitted process > cutAdapters (umi5_umi3)
[94/d2b5c9] Submitted process > buildIndicesrRNA (yeast_rRNA)
[d1/e85699] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[a3/2fb657] Submitted process > dashboard
[49/78f476] Submitted process > extractUmis (umi5_umi3)
Received from 'dashboard': /home/ubuntu/riboviz/work/a3/2fb657645409e8c110cdb63673be78/tmp.txt
[a1/574afb] Submitted process > hisat2rRNA (umi5_umi3)
[ae/3f68e5] Submitted process > hisat2ORF (umi5_umi3)
[aa/b1a5ce] Submitted process > trim5pMismatches (umi5_umi3)
[6b/4b7e20] Submitted process > samViewSort (umi5_umi3)
[e4/cc3d39] Submitted process > dedupUmis (umi5_umi3)
[ef/e6e442] Submitted process > groupUmisPreDedup (umi5_umi3)
[75/5cb629] Submitted process > groupUmisPostDedup (umi5_umi3)
[ef/5a4d3a] Submitted process > outputBams (umi5_umi3)
[36/0f3f4e] Submitted process > makeBedgraphs (umi5_umi3)
[9d/246c3a] Submitted process > bamToH5 (umi5_umi3)
[17/246cc7] Submitted process > generateStatsFigs (umi5_umi3)
Finished processing sample: umi5_umi3
[a2/95ecb9] Submitted process > renameTpms (umi5_umi3)
[59/beca3d] Submitted process > collateTpms (umi5_umi3)
[97/05f930] Submitted process > countReads
Workflow finished! (OK)

Check work directory for countReads:
$ ls work/97/05f93037c1b2293c37cccc7e846577/
read_counts.tsv  ribosome_fqs.yaml
$ cat work/97/05f93037c1b2293c37cccc7e846577/ribosome_fqs.yaml
fq_files: {umi5_umi3: umi5_umi3_umi_adaptor.fastq}
multiplex_fq_files: null

Check work directory for dashboard:
$ ls work/a3/2fb657645409e8c110cdb63673be78/
ribosome_fqs.yaml  tmp.txt
$ cat work/a3/2fb657645409e8c110cdb63673be78/ribosome_fqs.yaml
fq_files: {umi5_umi3: umi5_umi3_umi_adaptor.fastq}
multiplex_fq_files: null

The YAML file is present in both.
Note there is no need to use ribosome_fqs.yaml as a filename both countReads and dashboard. Again, I just do this as a convention and a reminder of the channel and value the file's contents come from.

Ah OK. I see what it is. I forgot that ribosome_fqs_yaml is not a channel, it is a value. Whereas channels need to be split (using into) to be used as inputs to multiple processes, values do not. Values can be used as-is as inputs to multiple processes.
Restore countReads to its old version:
    input:
        ...
        val ribosome_fqs_yaml from ribosome_fqs_yaml
        ...

Remove the ribosome_fqs_yaml.into block.
Update dashboard to have:
    input:
        val ribosome_fqs_yaml from ribosome_fqs_yaml

(Note: in Nextflow this essentially says create a channel within this process, ribosome_fqs_yaml, which provides the value ribosome_fqs_yaml. There is no need for the channel and value to have the same name - I do this to keep from having too many new names.)
dashboard is now:
process dashboard {
    input:
        val ribosome_fqs_yaml from ribosome_fqs_yaml
     output:
        file 'tmp.txt' into dashboard_output
    shell:
        """
	echo "${ribosome_fqs_yaml}" > ribosome_fqs.yaml
	touch tmp.txt
	"""
}

Update the dashboard_output.subscribe command as I noticed it wouldn't print the output file name. It does if changed to:
dashboard_output.subscribe { println "Received from 'dashboard': $it" }

I ran a workflow with this dashboard and dashboard_output.subscribe commands:
$ nextflow run prep_riboviz.nf -params-file vignette/simdata_umi_config.yaml -ansi-log false --publish_index_tmp
N E X T F L O W  ~  version 20.04.1
Launching `prep_riboviz.nf` [pedantic_mandelbrot] - revision: a4a10691e7
[86/17fdef] Submitted process > cutAdapters (umi5_umi3)
[94/d2b5c9] Submitted process > buildIndicesrRNA (yeast_rRNA)
[d1/e85699] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[a3/2fb657] Submitted process > dashboard
[49/78f476] Submitted process > extractUmis (umi5_umi3)
Received from 'dashboard': /home/ubuntu/riboviz/work/a3/2fb657645409e8c110cdb63673be78/tmp.txt
[a1/574afb] Submitted process > hisat2rRNA (umi5_umi3)
[ae/3f68e5] Submitted process > hisat2ORF (umi5_umi3)
[aa/b1a5ce] Submitted process > trim5pMismatches (umi5_umi3)
[6b/4b7e20] Submitted process > samViewSort (umi5_umi3)
[e4/cc3d39] Submitted process > dedupUmis (umi5_umi3)
[ef/e6e442] Submitted process > groupUmisPreDedup (umi5_umi3)
[75/5cb629] Submitted process > groupUmisPostDedup (umi5_umi3)
[ef/5a4d3a] Submitted process > outputBams (umi5_umi3)
[36/0f3f4e] Submitted process > makeBedgraphs (umi5_umi3)
[9d/246c3a] Submitted process > bamToH5 (umi5_umi3)
[17/246cc7] Submitted process > generateStatsFigs (umi5_umi3)
Finished processing sample: umi5_umi3
[a2/95ecb9] Submitted process > renameTpms (umi5_umi3)
[59/beca3d] Submitted process > collateTpms (umi5_umi3)
[97/05f930] Submitted process > countReads
Workflow finished! (OK)

Check work directory for countReads:
$ ls work/97/05f93037c1b2293c37cccc7e846577/
read_counts.tsv  ribosome_fqs.yaml
$ cat work/97/05f93037c1b2293c37cccc7e846577/ribosome_fqs.yaml
fq_files: {umi5_umi3: umi5_umi3_umi_adaptor.fastq}
multiplex_fq_files: null

Check work directory for dashboard:
$ ls work/a3/2fb657645409e8c110cdb63673be78/
ribosome_fqs.yaml  tmp.txt
$ cat work/a3/2fb657645409e8c110cdb63673be78/ribosome_fqs.yaml
fq_files: {umi5_umi3: umi5_umi3_umi_adaptor.fastq}
multiplex_fq_files: null

The YAML file is present in both.
Note there is no need to use ribosome_fqs.yaml as a filename both countReads and dashboard. Again, I just do this as a convention and a reminder of the channel and value the file's contents come from.

Thanks for that @mikej888; I made those changes, and commented out the #Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')" line & it ran fine (tested with the multiplex simdata yaml):
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [big_woese] - revision: 0f093300cb
[81/8e6356] Submitted process > buildIndicesrRNA (yeast_rRNA)
[96/7331c4] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[89/24fa5c] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[0a/15cd8e] Submitted process > dashboard
Received from 'dashboard': /home/fanders6/riboviz/riboviz/work/0a/15cd8e33dc2015dd09b7bb2d778629/tmp.txt
[88/67f877] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[75/e24096] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[5f/b5d384] Submitted process > hisat2rRNA (Tag0)
[4c/2b4fd3] Submitted process > hisat2rRNA (Tag1)
[ea/a8c61c] Submitted process > hisat2rRNA (Tag2)
[ca/00e5c0] Submitted process > hisat2ORF (Tag0)
[94/f8606e] Submitted process > hisat2ORF (Tag1)
[b0/78596e] Submitted process > hisat2ORF (Tag2)
[94/a6f21c] Submitted process > trim5pMismatches (Tag1)
[7d/a3e41d] Submitted process > trim5pMismatches (Tag0)
[13/2688fe] Submitted process > trim5pMismatches (Tag2)
[6c/5efcd7] Submitted process > samViewSort (Tag1)
[24/ddd805] Submitted process > samViewSort (Tag0)
[e5/dbb4c3] Submitted process > samViewSort (Tag2)
[8f/b68f01] Submitted process > groupUmisPreDedup (Tag1)
[b5/ba042f] Submitted process > groupUmisPreDedup (Tag0)
[ed/934c60] Submitted process > dedupUmis (Tag1)
[25/74696e] Submitted process > dedupUmis (Tag0)
[a7/85e37f] Submitted process > dedupUmis (Tag2)
[8b/0117d1] Submitted process > groupUmisPreDedup (Tag2)
[76/ac5899] Submitted process > groupUmisPostDedup (Tag0)
[a2/3d65dc] Submitted process > groupUmisPostDedup (Tag2)
[7c/090dfc] Submitted process > groupUmisPostDedup (Tag1)
[d8/825bcd] Submitted process > outputBams (Tag1)
[80/051be1] Submitted process > outputBams (Tag2)
[16/3f4cca] Submitted process > outputBams (Tag0)
[8a/22b866] Submitted process > makeBedgraphs (Tag1)
[03/fed178] Submitted process > makeBedgraphs (Tag2)
[cb/f438ea] Submitted process > bamToH5 (Tag1)
[f9/7de663] Submitted process > makeBedgraphs (Tag0)
[b0/fd3ea3] Submitted process > bamToH5 (Tag2)
[23/99c8e3] Submitted process > bamToH5 (Tag0)
[a3/a06780] Submitted process > generateStatsFigs (Tag1)
[be/70ac1f] Submitted process > generateStatsFigs (Tag0)
[e0/a93acb] Submitted process > generateStatsFigs (Tag2)
Finished processing sample: Tag1
[54/258a1f] Submitted process > renameTpms (Tag1)
Finished processing sample: Tag0
[ed/5561a1] Submitted process > renameTpms (Tag0)
Finished processing sample: Tag2
[c5/96bd80] Submitted process > renameTpms (Tag2)
[29/c3b52b] Submitted process > collateTpms (Tag1, Tag0, Tag2)
[55/1accbb] Submitted process > countReads
Workflow finished! (OK)

The expected ribosome fastq files value is also returned in the work folder for the process as expected, which is grand.
(riboviz) fanders6@ceres:~/riboviz/riboviz/work/0a/15cd8e33dc2015dd09b7bb2d778629$ cat ribosome_fqs.yaml
fq_files: null
multiplex_fq_files: [multiplex_umi_barcode_adaptor.fastq]

When I uncomment the rmarkdown render command to create the dashboard, I still get the same error, but this is an issue in the .Rmd code and something I hope @john-s-f and @p6t might be able to figure out while I work on getting the right parameters into the file?

Thanks for that @mikej888; I made those changes, and commented out the #Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')" line & it ran fine (tested with the multiplex simdata yaml):
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [big_woese] - revision: 0f093300cb
[81/8e6356] Submitted process > buildIndicesrRNA (yeast_rRNA)
[96/7331c4] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[89/24fa5c] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[0a/15cd8e] Submitted process > dashboard
Received from 'dashboard': /home/fanders6/riboviz/riboviz/work/0a/15cd8e33dc2015dd09b7bb2d778629/tmp.txt
[88/67f877] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[75/e24096] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[5f/b5d384] Submitted process > hisat2rRNA (Tag0)
[4c/2b4fd3] Submitted process > hisat2rRNA (Tag1)
[ea/a8c61c] Submitted process > hisat2rRNA (Tag2)
[ca/00e5c0] Submitted process > hisat2ORF (Tag0)
[94/f8606e] Submitted process > hisat2ORF (Tag1)
[b0/78596e] Submitted process > hisat2ORF (Tag2)
[94/a6f21c] Submitted process > trim5pMismatches (Tag1)
[7d/a3e41d] Submitted process > trim5pMismatches (Tag0)
[13/2688fe] Submitted process > trim5pMismatches (Tag2)
[6c/5efcd7] Submitted process > samViewSort (Tag1)
[24/ddd805] Submitted process > samViewSort (Tag0)
[e5/dbb4c3] Submitted process > samViewSort (Tag2)
[8f/b68f01] Submitted process > groupUmisPreDedup (Tag1)
[b5/ba042f] Submitted process > groupUmisPreDedup (Tag0)
[ed/934c60] Submitted process > dedupUmis (Tag1)
[25/74696e] Submitted process > dedupUmis (Tag0)
[a7/85e37f] Submitted process > dedupUmis (Tag2)
[8b/0117d1] Submitted process > groupUmisPreDedup (Tag2)
[76/ac5899] Submitted process > groupUmisPostDedup (Tag0)
[a2/3d65dc] Submitted process > groupUmisPostDedup (Tag2)
[7c/090dfc] Submitted process > groupUmisPostDedup (Tag1)
[d8/825bcd] Submitted process > outputBams (Tag1)
[80/051be1] Submitted process > outputBams (Tag2)
[16/3f4cca] Submitted process > outputBams (Tag0)
[8a/22b866] Submitted process > makeBedgraphs (Tag1)
[03/fed178] Submitted process > makeBedgraphs (Tag2)
[cb/f438ea] Submitted process > bamToH5 (Tag1)
[f9/7de663] Submitted process > makeBedgraphs (Tag0)
[b0/fd3ea3] Submitted process > bamToH5 (Tag2)
[23/99c8e3] Submitted process > bamToH5 (Tag0)
[a3/a06780] Submitted process > generateStatsFigs (Tag1)
[be/70ac1f] Submitted process > generateStatsFigs (Tag0)
[e0/a93acb] Submitted process > generateStatsFigs (Tag2)
Finished processing sample: Tag1
[54/258a1f] Submitted process > renameTpms (Tag1)
Finished processing sample: Tag0
[ed/5561a1] Submitted process > renameTpms (Tag0)
Finished processing sample: Tag2
[c5/96bd80] Submitted process > renameTpms (Tag2)
[29/c3b52b] Submitted process > collateTpms (Tag1, Tag0, Tag2)
[55/1accbb] Submitted process > countReads
Workflow finished! (OK)

The expected ribosome fastq files value is also returned in the work folder for the process as expected, which is grand.
(riboviz) fanders6@ceres:~/riboviz/riboviz/work/0a/15cd8e33dc2015dd09b7bb2d778629$ cat ribosome_fqs.yaml
fq_files: null
multiplex_fq_files: [multiplex_umi_barcode_adaptor.fastq]

When I uncomment the rmarkdown render command to create the dashboard, I still get the same error, but this is an issue in the .Rmd code and something I hope @john-s-f and @p6t might be able to figure out while I work on getting the right parameters into the file?

@p6t I have a question about what kind of output this .Rmd creates.  I've seen some different options for arguments to rmarkdown::render(), but is flexdashboard::flex_dashboard an output type separate from html_document? How does this work in terms of running from the console? Should I be specifying the flex_dashboard output type in the render() call do you think?  When we specify output in the nextflow process, would it be better to specify a .html output? Or is a flexdashboard output definitely NOT the same?

@p6t I have a question about what kind of output this .Rmd creates.  I've seen some different options for arguments to rmarkdown::render(), but is flexdashboard::flex_dashboard an output type separate from html_document? How does this work in terms of running from the console? Should I be specifying the flex_dashboard output type in the render() call do you think?  When we specify output in the nextflow process, would it be better to specify a .html output? Or is a flexdashboard output definitely NOT the same?

@FlicAnderson I'm not certain, every instance I've seen online has flexdashboard::flex_dashboard. I do believe flexdashboard is not an html output, as r code has to run whenever the user changes a selection.

@FlicAnderson I'm not certain, every instance I've seen online has flexdashboard::flex_dashboard. I do believe flexdashboard is not an html output, as r code has to run whenever the user changes a selection.

@p6t gotcha. I'm not sure how best to handle that 'opens browser' event within a Nextflow system (I guess wecould put it as the last process in the chain to cause less disruption if it suddenly starts a browser session?), but as a start we need to try getting it running from the commandline first I think.
Another thing to do is to make sure we add a new parameter so we can either run the dashboard process or not (thinking of running riboviz in HPC, where this wouldn't be handy).

@p6t gotcha. I'm not sure how best to handle that 'opens browser' event within a Nextflow system (I guess wecould put it as the last process in the chain to cause less disruption if it suddenly starts a browser session?), but as a start we need to try getting it running from the commandline first I think.
Another thing to do is to make sure we add a new parameter so we can either run the dashboard process or not (thinking of running riboviz in HPC, where this wouldn't be handy).

Example dumping whole configuration into a file:
config_yaml = new Yaml().dump(params)

process dashboard {
    input:
        val config_yaml from config_yaml
     output:
        file 'tmp.txt' into dashboard_output
    shell:
        """
        echo "${config_yaml}" > config.yaml
        touch tmp.txt
}

Example dumping whole configuration into a file:
config_yaml = new Yaml().dump(params)

process dashboard {
    input:
        val config_yaml from config_yaml
     output:
        file 'tmp.txt' into dashboard_output
    shell:
        """
        echo "${config_yaml}" > config.yaml
        touch tmp.txt
}

Commit 3f79efb moves dashboard process down to end of script (but does not yet have any dependencies which would ensure it runs as one of the last nextflow processes); adds when declaration (using new YAML parameter run_dashboard); and makes the configuration file changes Mike suggested earlier.
Commit f2116d9 adds new YAML config file parameter: run_dashboard which has default TRUE, and will run the dashboard process in nextflow if so.
This seems to work, as it didn't run dashboard process when I had set run_dashboard to false, and DID run it when this was set to true.  I still have the render() command commented out as this still produces the mysterious output object error detailed here.

Commit 3f79efb moves dashboard process down to end of script (but does not yet have any dependencies which would ensure it runs as one of the last nextflow processes); adds when declaration (using new YAML parameter run_dashboard); and makes the configuration file changes Mike suggested earlier.
Commit f2116d9 adds new YAML config file parameter: run_dashboard which has default TRUE, and will run the dashboard process in nextflow if so.
This seems to work, as it didn't run dashboard process when I had set run_dashboard to false, and DID run it when this was set to true.  I still have the render() command commented out as this still produces the mysterious output object error detailed here.

I've edited the new_visuazation.Rmd file {r input} block to eval=FALSE, which removes the "object 'output' not found" error (as well as breaking all the lovely dashboard stuff, but it does at least run create a new_visualization.html file).
I'm now trying to see if I can read the yaml file in as a parameter, and how to deal with the output new_visualization.html file.
This is how the nextflow process & related bits look:
// create 'new' yaml for use in dashboard process
config_yaml = new Yaml().dump(params)

process dashboard {
    publishDir "${params.dir_out}", mode: 'copy', overwrite: true

    input:
      val config_yaml from config_yaml

    // this is a temporary output, as @FlicAnderson still not clear on what output new_visualization.Rmd produces?
    output:
      file 'tmp.txt' into dashboard_output

    when:
      params.run_dashboard

    shell:
      """
      echo "${config_yaml}" > config.yaml
      #Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"
      touch tmp.txt
      """
}

dashboard_output.subscribe { println "Received from 'dashboard': $it" }

And I've added parameters to the .Rmd yaml section:
params:
  yamlfile: NA

and try and pull in this parameter to the code:
yamlfilepath <- here::here(params$yamlfile)
#...
yaml1 <- here("vignette", "simdata_multiplex_config.yaml")
yaml2 <- here("vignette", "vignette_config.yaml")
yaml3 <- yamlfilepath

print(params$yamlfile)
print(yaml1)
print(yaml2)
print(yaml3)

This should output into the .html the values input to the .Rmd of params$yamfile and a version of that (yaml3) with some here() filepath magic.
Running nextflow on multiplex yaml in vignette:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [grave_wilson] - revision: eb6082347d
[2e/74f5c1] Submitted process > buildIndicesrRNA (yeast_rRNA)
[60/10b375] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[56/5bb570] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[f6/a9ed77] Submitted process > dashboard
[96/5455af] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[fb/9d9b08] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[86/5c7e0c] Submitted process > hisat2rRNA (Tag2)
[b0/4f37c4] Submitted process > hisat2rRNA (Tag1)
[92/0f8ed5] Submitted process > hisat2rRNA (Tag0)
[75/afca61] Submitted process > hisat2ORF (Tag2)
[ad/4828b8] Submitted process > hisat2ORF (Tag1)
[8e/6dac77] Submitted process > hisat2ORF (Tag0)
[a5/127de0] Submitted process > trim5pMismatches (Tag0)
[af/c4f149] Submitted process > trim5pMismatches (Tag2)
[23/4c8259] Submitted process > trim5pMismatches (Tag1)
[c2/93505c] Submitted process > samViewSort (Tag1)
[66/dc6417] Submitted process > samViewSort (Tag2)
[2c/91e0ae] Submitted process > samViewSort (Tag0)
[e5/26dd42] Submitted process > dedupUmis (Tag2)
[46/380f0e] Submitted process > dedupUmis (Tag0)
[fe/1b7b3c] Submitted process > groupUmisPreDedup (Tag2)
[81/12f46d] Submitted process > dedupUmis (Tag1)
[8a/0f7fbf] Submitted process > groupUmisPreDedup (Tag0)
[3c/c61ccf] Submitted process > groupUmisPreDedup (Tag1)
[b1/423c8f] Submitted process > groupUmisPostDedup (Tag2)
[28/66f9c6] Submitted process > outputBams (Tag2)
[2f/090540] Submitted process > groupUmisPostDedup (Tag1)
[bb/f4c919] Submitted process > outputBams (Tag1)
[17/2bc81e] Submitted process > groupUmisPostDedup (Tag0)
[10/14d20c] Submitted process > outputBams (Tag0)
[8b/2238b1] Submitted process > bamToH5 (Tag1)
[2b/15beb9] Submitted process > makeBedgraphs (Tag1)
[8b/cbd2be] Submitted process > bamToH5 (Tag0)
Received from 'dashboard': /home/fanders6/riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c/tmp.txt
[37/063f8f] Submitted process > makeBedgraphs (Tag0)
[34/c508fa] Submitted process > makeBedgraphs (Tag2)
[50/6e4b6f] Submitted process > bamToH5 (Tag2)
[36/333709] Submitted process > generateStatsFigs (Tag1)
[9b/53594e] Submitted process > generateStatsFigs (Tag2)
[bf/cc0a28] Submitted process > generateStatsFigs (Tag0)
Finished processing sample: Tag2
[2c/f665c9] Submitted process > renameTpms (Tag2)
Finished processing sample: Tag1
[fd/72fe89] Submitted process > renameTpms (Tag1)
Finished processing sample: Tag0
[f9/673361] Submitted process > renameTpms (Tag0)
[12/07bd0e] Submitted process > collateTpms (Tag2, Tag1, Tag0)
[13/51c217] Submitted process > countReads
Workflow finished! (OK)

fanders6@ceres:~$ cd riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c/
fanders6@ceres:~/riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c$ cat .command.log
processing file: new_visualization.Rmd
  |...                                                                   |   4%
  ordinary text without R code

  |......                                                                |   8%
label: setup (with options)
List of 1
 $ include: logi FALSE

── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.3     ✔ dplyr   1.0.1
✔ tidyr   1.1.1     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.5.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

Attaching package: 'scales'

The following object is masked from 'package:purrr':

    discard

The following object is masked from 'package:readr':

    col_factor

here() starts at /home/fanders6/riboviz/riboviz
  |........                                                              |  12%
  ordinary text without R code

  |...........                                                           |  16%
label: unnamed-chunk-1
  |..............                                                        |  20%
  ordinary text without R code

  |.................                                                     |  24%
label: unnamed-chunk-2
  |....................                                                  |  28%
  ordinary text without R code

  |......................                                                |  32%
label: unnamed-chunk-3
  |.........................                                             |  36%
  ordinary text without R code

  |............................                                          |  40%
label: unnamed-chunk-4
Parsed with column specification:
cols(
  SampleID = col_character(),
  TagRead = col_character()
)
Parsed with column specification:
cols(
  SampleName = col_character(),
  Program = col_character(),
  File = col_character(),
  NumReads = col_double(),
  Description = col_character()
)
  |...............................                                       |  44%
  ordinary text without R code

  |..................................                                    |  48%
label: unnamed-chunk-5
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
  |....................................                                  |  52%
  ordinary text without R code

  |.......................................                               |  56%
label: unnamed-chunk-6
  |..........................................                            |  60%
  ordinary text without R code

  |.............................................                         |  64%
label: unnamed-chunk-7
  |................................................                      |  68%
  ordinary text without R code

  |..................................................                    |  72%
label: unnamed-chunk-8
  |.....................................................                 |  76%
label: unnamed-chunk-9
  |........................................................              |  80%
  ordinary text without R code

  |...........................................................           |  84%
label: reactive
  |..............................................................        |  88%
  ordinary text without R code

  |................................................................      |  92%
label: input (with options)
List of 1
 $ eval: logi FALSE

  |...................................................................   |  96%
  ordinary text without R code

  |......................................................................| 100%
label: output_bysample

output file: new_visualization.knit.md

/usr/bin/pandoc +RTS -K512m -RTS new_visualization.utf8.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output new_visualization.html --email-obfuscation none --self-contained --standalone --section-divs --template /home/fanders6/R/x86_64-pc-linux-gnu-library/4.0/flexdashboard/rmarkdown/templates/flex_dashboard/resources/default.html --id-prefix section- --variable 'theme:yeti' --include-in-header /tmp/RtmpLhu8AF/rmarkdown-str4e8c2dcb8220.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --include-in-header /tmp/RtmpLhu8AF/file4e8c2c74212fhtml --highlight-style pygments --include-before-body /tmp/RtmpLhu8AF/file4e8c3c3b3772.html --include-after-body /tmp/RtmpLhu8AF/file4e8c20c53a16.html

Output created: new_visualization.html

Lines of text in the .html output (found in /rmarkdown):

Based on this first line, we can see print(params$yamlfile) has the value fed in from the process dashboard Rscript render() parameter ('config.yaml'), but it's not doing it very cleverly, so I need to find out how we take the config.yaml file created in the echo "${config_yaml}" > config.yaml step into the next line as the input parameter for yamlfile (ie in .. params = list(yamlfile=<NEWYAMLGOESHERE).
@mikej888 I wasn't quite able to get my head around how to do this, given that config.yaml is created only in the process' work folder.  How can I reference that location in the Rscript render() call?
Also another file location question for @mikej888 (although I suspect that once we get the output object error solved it might not be necessary to keep this fix), how would I ensure that the output file new_visualization.html (which currently turns up in rmarkdown/) gets created in the correct directory? I can set output_dir in the R command like so:
Rscript -e "rmarkdown::render(
'$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd',
params = list(yamlfile='config.yaml'),
output_dir = ${params.dir_out})"  # this is the key bit in setting .Rmd output file location

#  which gets converted into this:

Rscript -e "rmarkdown::render(
'$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd',
params = list(yamlfile='config.yaml'),
output_dir = vignette/simdata_multiplex_output)"

but it doesn't include the riboviz directories part of the path.  Is something like workflow.projectDir what I need to tack onto the front somehow? Are there any gotchas to avoid when playing with this part, or do normal shell command formatting rules apply?

I've edited the new_visuazation.Rmd file {r input} block to eval=FALSE, which removes the "object 'output' not found" error (as well as breaking all the lovely dashboard stuff, but it does at least run create a new_visualization.html file).
I'm now trying to see if I can read the yaml file in as a parameter, and how to deal with the output new_visualization.html file.
This is how the nextflow process & related bits look:
// create 'new' yaml for use in dashboard process
config_yaml = new Yaml().dump(params)

process dashboard {
    publishDir "${params.dir_out}", mode: 'copy', overwrite: true

    input:
      val config_yaml from config_yaml

    // this is a temporary output, as @FlicAnderson still not clear on what output new_visualization.Rmd produces?
    output:
      file 'tmp.txt' into dashboard_output

    when:
      params.run_dashboard

    shell:
      """
      echo "${config_yaml}" > config.yaml
      #Rscript --vanilla -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd')"
      Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"
      touch tmp.txt
      """
}

dashboard_output.subscribe { println "Received from 'dashboard': $it" }

And I've added parameters to the .Rmd yaml section:
params:
  yamlfile: NA

and try and pull in this parameter to the code:
yamlfilepath <- here::here(params$yamlfile)
#...
yaml1 <- here("vignette", "simdata_multiplex_config.yaml")
yaml2 <- here("vignette", "vignette_config.yaml")
yaml3 <- yamlfilepath

print(params$yamlfile)
print(yaml1)
print(yaml2)
print(yaml3)

This should output into the .html the values input to the .Rmd of params$yamfile and a version of that (yaml3) with some here() filepath magic.
Running nextflow on multiplex yaml in vignette:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [grave_wilson] - revision: eb6082347d
[2e/74f5c1] Submitted process > buildIndicesrRNA (yeast_rRNA)
[60/10b375] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[56/5bb570] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[f6/a9ed77] Submitted process > dashboard
[96/5455af] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[fb/9d9b08] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[86/5c7e0c] Submitted process > hisat2rRNA (Tag2)
[b0/4f37c4] Submitted process > hisat2rRNA (Tag1)
[92/0f8ed5] Submitted process > hisat2rRNA (Tag0)
[75/afca61] Submitted process > hisat2ORF (Tag2)
[ad/4828b8] Submitted process > hisat2ORF (Tag1)
[8e/6dac77] Submitted process > hisat2ORF (Tag0)
[a5/127de0] Submitted process > trim5pMismatches (Tag0)
[af/c4f149] Submitted process > trim5pMismatches (Tag2)
[23/4c8259] Submitted process > trim5pMismatches (Tag1)
[c2/93505c] Submitted process > samViewSort (Tag1)
[66/dc6417] Submitted process > samViewSort (Tag2)
[2c/91e0ae] Submitted process > samViewSort (Tag0)
[e5/26dd42] Submitted process > dedupUmis (Tag2)
[46/380f0e] Submitted process > dedupUmis (Tag0)
[fe/1b7b3c] Submitted process > groupUmisPreDedup (Tag2)
[81/12f46d] Submitted process > dedupUmis (Tag1)
[8a/0f7fbf] Submitted process > groupUmisPreDedup (Tag0)
[3c/c61ccf] Submitted process > groupUmisPreDedup (Tag1)
[b1/423c8f] Submitted process > groupUmisPostDedup (Tag2)
[28/66f9c6] Submitted process > outputBams (Tag2)
[2f/090540] Submitted process > groupUmisPostDedup (Tag1)
[bb/f4c919] Submitted process > outputBams (Tag1)
[17/2bc81e] Submitted process > groupUmisPostDedup (Tag0)
[10/14d20c] Submitted process > outputBams (Tag0)
[8b/2238b1] Submitted process > bamToH5 (Tag1)
[2b/15beb9] Submitted process > makeBedgraphs (Tag1)
[8b/cbd2be] Submitted process > bamToH5 (Tag0)
Received from 'dashboard': /home/fanders6/riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c/tmp.txt
[37/063f8f] Submitted process > makeBedgraphs (Tag0)
[34/c508fa] Submitted process > makeBedgraphs (Tag2)
[50/6e4b6f] Submitted process > bamToH5 (Tag2)
[36/333709] Submitted process > generateStatsFigs (Tag1)
[9b/53594e] Submitted process > generateStatsFigs (Tag2)
[bf/cc0a28] Submitted process > generateStatsFigs (Tag0)
Finished processing sample: Tag2
[2c/f665c9] Submitted process > renameTpms (Tag2)
Finished processing sample: Tag1
[fd/72fe89] Submitted process > renameTpms (Tag1)
Finished processing sample: Tag0
[f9/673361] Submitted process > renameTpms (Tag0)
[12/07bd0e] Submitted process > collateTpms (Tag2, Tag1, Tag0)
[13/51c217] Submitted process > countReads
Workflow finished! (OK)

fanders6@ceres:~$ cd riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c/
fanders6@ceres:~/riboviz/riboviz/work/f6/a9ed77e18a04fab4e2b62bff3af47c$ cat .command.log
processing file: new_visualization.Rmd
  |...                                                                   |   4%
  ordinary text without R code

  |......                                                                |   8%
label: setup (with options)
List of 1
 $ include: logi FALSE

── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.3     ✔ dplyr   1.0.1
✔ tidyr   1.1.1     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.5.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

Attaching package: 'scales'

The following object is masked from 'package:purrr':

    discard

The following object is masked from 'package:readr':

    col_factor

here() starts at /home/fanders6/riboviz/riboviz
  |........                                                              |  12%
  ordinary text without R code

  |...........                                                           |  16%
label: unnamed-chunk-1
  |..............                                                        |  20%
  ordinary text without R code

  |.................                                                     |  24%
label: unnamed-chunk-2
  |....................                                                  |  28%
  ordinary text without R code

  |......................                                                |  32%
label: unnamed-chunk-3
  |.........................                                             |  36%
  ordinary text without R code

  |............................                                          |  40%
label: unnamed-chunk-4
Parsed with column specification:
cols(
  SampleID = col_character(),
  TagRead = col_character()
)
Parsed with column specification:
cols(
  SampleName = col_character(),
  Program = col_character(),
  File = col_character(),
  NumReads = col_double(),
  Description = col_character()
)
  |...............................                                       |  44%
  ordinary text without R code

  |..................................                                    |  48%
label: unnamed-chunk-5
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Length = col_double(),
  Counts = col_double()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  Pos = col_double(),
  Counts = col_double(),
  End = col_character()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
Parsed with column specification:
cols(
  gene = col_character(),
  Ct_fr0 = col_double(),
  Ct_fr1 = col_double(),
  Ct_fr2 = col_double(),
  pval_fr0vs1 = col_double(),
  pval_fr0vs2 = col_double(),
  pval_fr0vsboth = col_double()
)
  |....................................                                  |  52%
  ordinary text without R code

  |.......................................                               |  56%
label: unnamed-chunk-6
  |..........................................                            |  60%
  ordinary text without R code

  |.............................................                         |  64%
label: unnamed-chunk-7
  |................................................                      |  68%
  ordinary text without R code

  |..................................................                    |  72%
label: unnamed-chunk-8
  |.....................................................                 |  76%
label: unnamed-chunk-9
  |........................................................              |  80%
  ordinary text without R code

  |...........................................................           |  84%
label: reactive
  |..............................................................        |  88%
  ordinary text without R code

  |................................................................      |  92%
label: input (with options)
List of 1
 $ eval: logi FALSE

  |...................................................................   |  96%
  ordinary text without R code

  |......................................................................| 100%
label: output_bysample

output file: new_visualization.knit.md

/usr/bin/pandoc +RTS -K512m -RTS new_visualization.utf8.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output new_visualization.html --email-obfuscation none --self-contained --standalone --section-divs --template /home/fanders6/R/x86_64-pc-linux-gnu-library/4.0/flexdashboard/rmarkdown/templates/flex_dashboard/resources/default.html --id-prefix section- --variable 'theme:yeti' --include-in-header /tmp/RtmpLhu8AF/rmarkdown-str4e8c2dcb8220.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --include-in-header /tmp/RtmpLhu8AF/file4e8c2c74212fhtml --highlight-style pygments --include-before-body /tmp/RtmpLhu8AF/file4e8c3c3b3772.html --include-after-body /tmp/RtmpLhu8AF/file4e8c20c53a16.html

Output created: new_visualization.html

Lines of text in the .html output (found in /rmarkdown):

Based on this first line, we can see print(params$yamlfile) has the value fed in from the process dashboard Rscript render() parameter ('config.yaml'), but it's not doing it very cleverly, so I need to find out how we take the config.yaml file created in the echo "${config_yaml}" > config.yaml step into the next line as the input parameter for yamlfile (ie in .. params = list(yamlfile=<NEWYAMLGOESHERE).
@mikej888 I wasn't quite able to get my head around how to do this, given that config.yaml is created only in the process' work folder.  How can I reference that location in the Rscript render() call?
Also another file location question for @mikej888 (although I suspect that once we get the output object error solved it might not be necessary to keep this fix), how would I ensure that the output file new_visualization.html (which currently turns up in rmarkdown/) gets created in the correct directory? I can set output_dir in the R command like so:
Rscript -e "rmarkdown::render(
'$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd',
params = list(yamlfile='config.yaml'),
output_dir = ${params.dir_out})"  # this is the key bit in setting .Rmd output file location

#  which gets converted into this:

Rscript -e "rmarkdown::render(
'$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd',
params = list(yamlfile='config.yaml'),
output_dir = vignette/simdata_multiplex_output)"

but it doesn't include the riboviz directories part of the path.  Is something like workflow.projectDir what I need to tack onto the front somehow? Are there any gotchas to avoid when playing with this part, or do normal shell command formatting rules apply?

In
Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"

Replace 'config.yaml' with'${config.yaml}'. This should dereference the value of config_yaml and should insert the path to the YAML file in the work directory for the process.

In
Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"

Replace 'config.yaml' with'${config.yaml}'. This should dereference the value of config_yaml and should insert the path to the YAML file in the work directory for the process.

I'd hope/expect the Rscript command output to end up in the work directory.
publishDir tells a process where files declared in its output can be copied to. You could add to process dashboard:
publishDir "${params.dir_out}", mode: 'copy', overwrite: true

and this should publish the Rscript outputs to dir_out as defined in the YAML.

I'd hope/expect the Rscript command output to end up in the work directory.
publishDir tells a process where files declared in its output can be copied to. You could add to process dashboard:
publishDir "${params.dir_out}", mode: 'copy', overwrite: true

and this should publish the Rscript outputs to dir_out as defined in the YAML.

In
Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"

Replace 'config.yaml' with'${config.yaml}'. This should dereference the value of config_yaml and should insert the path to the YAML file in the work directory for the process.

I've tried that, and got an error:
Error executing process > 'dashboard'

Caused by:
  Unknown variable 'config' -- Make sure it is not misspelt and defined somewhere in the script before using it

Source block:
  """
        echo "${config_yaml}" > config.yaml
        Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config.yaml}'))"
        touch tmp.txt
        """

ConfigConfusion :scream_cat:

In
Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"

Replace 'config.yaml' with'${config.yaml}'. This should dereference the value of config_yaml and should insert the path to the YAML file in the work directory for the process.

I've tried that, and got an error:
Error executing process > 'dashboard'

Caused by:
  Unknown variable 'config' -- Make sure it is not misspelt and defined somewhere in the script before using it

Source block:
  """
        echo "${config_yaml}" > config.yaml
        Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config.yaml}'))"
        touch tmp.txt
        """

ConfigConfusion :scream_cat:

Sorry, ${config_yaml} - it should match the val defined in the input section.

Sorry, ${config_yaml} - it should match the val defined in the input section.

Sorry, ${config_yaml} - it should match the val defined in the input section.
Aha, gotcha!

Sorry, ${config_yaml} - it should match the val defined in the input section.
Aha, gotcha!

Once we get this to work with the vignette we can close it. We still need to figure out why RShiny does not work.
We need a new ticket to decide how we use this new functionality.

Once we get this to work with the vignette we can close it. We still need to figure out why RShiny does not work.
We need a new ticket to decide how we use this new functionality.

Sorry, ${config_yaml} - it should match the val defined in the input section.

Aha, gotcha!

Alas...
Using ${config_yaml} as suggested in Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config_yaml}'))"inprep_riboviz.nf` gives the WHOLE set of yaml parameters into the .Rmd render() command...

Can we define the name & location of config.yaml as a new value somehow then use that within render() @mikej888? Is this possibly a shell 101 thing?

Sorry, ${config_yaml} - it should match the val defined in the input section.

Aha, gotcha!

Alas...
Using ${config_yaml} as suggested in Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config_yaml}'))"inprep_riboviz.nf` gives the WHOLE set of yaml parameters into the .Rmd render() command...

Can we define the name & location of config.yaml as a new value somehow then use that within render() @mikej888? Is this possibly a shell 101 thing?

In relation to @FlicAnderson  and the actual YAML being put into the command line, sorry, I should have foreseen that.
The following adds a new process which dumps the YAML content into a config file. This will be created in a work directory...
process createConfigFile {
    input:
      val config_yaml from config_yaml
     output:
      file "config.yaml" into config_file_yaml
    when:
      params.run_dashboard
    shell:
      """
      echo "${config_yaml}" > ${config_file_yaml}
      """
}

Note that config_file_yaml - the config file (not the YAML contents of it) are declared as an output. This can then be provided as an input to dashboard, so, hopefully, a symbolic link to the config file created above will be created in the work directory for dashboard and so can also be dereferenced by ${config_file_yaml}...
process dashboard {
    ...
    input:
      file config_file_yaml from config_file_yaml
    ...

    shell:
      """
        Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config_file_yaml}'))"
      touch tmp.txt
      """
}

In relation to @FlicAnderson  and the actual YAML being put into the command line, sorry, I should have foreseen that.
The following adds a new process which dumps the YAML content into a config file. This will be created in a work directory...
process createConfigFile {
    input:
      val config_yaml from config_yaml
     output:
      file "config.yaml" into config_file_yaml
    when:
      params.run_dashboard
    shell:
      """
      echo "${config_yaml}" > ${config_file_yaml}
      """
}

Note that config_file_yaml - the config file (not the YAML contents of it) are declared as an output. This can then be provided as an input to dashboard, so, hopefully, a symbolic link to the config file created above will be created in the work directory for dashboard and so can also be dereferenced by ${config_file_yaml}...
process dashboard {
    ...
    input:
      file config_file_yaml from config_file_yaml
    ...

    shell:
      """
        Rscript -e "rmarkdown::render('$HOME/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='${config_file_yaml}'))"
      touch tmp.txt
      """
}

Now getting slightly further :D  We're getting R error messages.
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [sick_albattani] - revision: b81e8d31a9
[84/cf2f49] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[db/2117e0] Submitted process > buildIndicesrRNA (yeast_rRNA)
[16/51b8d3] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[72/76e927] Submitted process > createConfigFile
[a1/6a8a98] Submitted process > dashboard
[61/623a8d] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[ca/57e9f6] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[da/2c70d8] Submitted process > hisat2rRNA (Tag0)
[72/efabf9] Submitted process > hisat2rRNA (Tag2)
[a9/675ae2] Submitted process > hisat2rRNA (Tag1)
[4f/cdd615] Submitted process > hisat2ORF (Tag1)
[01/1cc8d7] Submitted process > hisat2ORF (Tag0)
[7b/1d7622] Submitted process > hisat2ORF (Tag2)
[08/d47c3b] Submitted process > trim5pMismatches (Tag1)
[36/c6d57d] Submitted process > trim5pMismatches (Tag2)
[6e/98f410] Submitted process > trim5pMismatches (Tag0)
Error executing process > 'dashboard'

Caused by:
  Process `dashboard` terminated with an error exit status (1)

Command executed:

  Rscript -e "rmarkdown::render('/home/fanders6/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"
  touch tmp.txt

Command exit status:
  1

Command output:
    |
    |                                                                      |   0%
    |
    |...                                                                   |   4%
    ordinary text without R code

    |
    |......                                                                |   8%
  label: setup (with options)
  List of 1
   $ include: logi FALSE

    |
    |........                                                              |  12%
    ordinary text without R code

    |
    |...........                                                           |  16%
  label: unnamed-chunk-1

    |
    |..............                                                        |  20%
    ordinary text without R code

    |
    |.................                                                     |  24%
  label: unnamed-chunk-2

    |
    |....................                                                  |  28%
    ordinary text without R code

    |
    |......................                                                |  32%
  label: unnamed-chunk-3

    |
    |.........................                                             |  36%
    ordinary text without R code

    |
    |............................                                          |  40%
  label: unnamed-chunk-4

Command error:

  processing file: new_visualization.Rmd
  ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
  ✔ ggplot2 3.3.2     ✔ purrr   0.3.4
  ✔ tibble  3.0.3     ✔ dplyr   1.0.1
  ✔ tidyr   1.1.1     ✔ stringr 1.4.0
  ✔ readr   1.3.1     ✔ forcats 0.5.0
  ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
  ✖ dplyr::filter() masks stats::filter()
  ✖ dplyr::lag()    masks stats::lag()

  Attaching package: 'scales'

  The following object is masked from 'package:purrr':

      discard

  The following object is masked from 'package:readr':

      col_factor

  here() starts at /home/fanders6/riboviz/riboviz
  Quitting from lines 259-294 (new_visualization.Rmd)
  Error in file(file, "rt", encoding = fileEncoding) :
    cannot open the connection
  Calls: <Anonymous> ... eval -> eval -> find_sample_names -> read_yaml -> file
  In addition: Warning message:
  In file(file, "rt", encoding = fileEncoding) :
    cannot open file 'config.yaml': No such file or directory
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/a1/6a8a98663d6418cf7dd0fdb57824bd

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Workflow finished! (failed)
WARN: Killing pending tasks (3)

This "cannot open file 'config.yaml': No such file or directory" error suggests that we're missing the rest of the path, perhaps?  I saw in countReads process there's some magic going on like "${file(params.dir_out).toAbsolutePath()}" - might something like this help get the absolute path of our 'config.yaml' file from the work directory to pass in as the parameter yamlfile?

Now getting slightly further :D  We're getting R error messages.
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [sick_albattani] - revision: b81e8d31a9
[84/cf2f49] Submitted process > cutAdaptersMultiplex (multiplex_umi_barcode_adaptor)
[db/2117e0] Submitted process > buildIndicesrRNA (yeast_rRNA)
[16/51b8d3] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[72/76e927] Submitted process > createConfigFile
[a1/6a8a98] Submitted process > dashboard
[61/623a8d] Submitted process > extractUmisMultiplex (multiplex_umi_barcode_adaptor)
[ca/57e9f6] Submitted process > demultiplex (multiplex_umi_barcode_adaptor)
Demultiplexed samples: [Tag0, Tag1, Tag2]
Non-demultiplexed samples: [Tag3]
[da/2c70d8] Submitted process > hisat2rRNA (Tag0)
[72/efabf9] Submitted process > hisat2rRNA (Tag2)
[a9/675ae2] Submitted process > hisat2rRNA (Tag1)
[4f/cdd615] Submitted process > hisat2ORF (Tag1)
[01/1cc8d7] Submitted process > hisat2ORF (Tag0)
[7b/1d7622] Submitted process > hisat2ORF (Tag2)
[08/d47c3b] Submitted process > trim5pMismatches (Tag1)
[36/c6d57d] Submitted process > trim5pMismatches (Tag2)
[6e/98f410] Submitted process > trim5pMismatches (Tag0)
Error executing process > 'dashboard'

Caused by:
  Process `dashboard` terminated with an error exit status (1)

Command executed:

  Rscript -e "rmarkdown::render('/home/fanders6/riboviz/riboviz/rmarkdown/new_visualization.Rmd', params = list(yamlfile='config.yaml'))"
  touch tmp.txt

Command exit status:
  1

Command output:
    |
    |                                                                      |   0%
    |
    |...                                                                   |   4%
    ordinary text without R code

    |
    |......                                                                |   8%
  label: setup (with options)
  List of 1
   $ include: logi FALSE

    |
    |........                                                              |  12%
    ordinary text without R code

    |
    |...........                                                           |  16%
  label: unnamed-chunk-1

    |
    |..............                                                        |  20%
    ordinary text without R code

    |
    |.................                                                     |  24%
  label: unnamed-chunk-2

    |
    |....................                                                  |  28%
    ordinary text without R code

    |
    |......................                                                |  32%
  label: unnamed-chunk-3

    |
    |.........................                                             |  36%
    ordinary text without R code

    |
    |............................                                          |  40%
  label: unnamed-chunk-4

Command error:

  processing file: new_visualization.Rmd
  ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
  ✔ ggplot2 3.3.2     ✔ purrr   0.3.4
  ✔ tibble  3.0.3     ✔ dplyr   1.0.1
  ✔ tidyr   1.1.1     ✔ stringr 1.4.0
  ✔ readr   1.3.1     ✔ forcats 0.5.0
  ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
  ✖ dplyr::filter() masks stats::filter()
  ✖ dplyr::lag()    masks stats::lag()

  Attaching package: 'scales'

  The following object is masked from 'package:purrr':

      discard

  The following object is masked from 'package:readr':

      col_factor

  here() starts at /home/fanders6/riboviz/riboviz
  Quitting from lines 259-294 (new_visualization.Rmd)
  Error in file(file, "rt", encoding = fileEncoding) :
    cannot open the connection
  Calls: <Anonymous> ... eval -> eval -> find_sample_names -> read_yaml -> file
  In addition: Warning message:
  In file(file, "rt", encoding = fileEncoding) :
    cannot open file 'config.yaml': No such file or directory
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/a1/6a8a98663d6418cf7dd0fdb57824bd

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Workflow finished! (failed)
WARN: Killing pending tasks (3)

This "cannot open file 'config.yaml': No such file or directory" error suggests that we're missing the rest of the path, perhaps?  I saw in countReads process there's some magic going on like "${file(params.dir_out).toAbsolutePath()}" - might something like this help get the absolute path of our 'config.yaml' file from the work directory to pass in as the parameter yamlfile?

Request that we give a more informative name than new_visualization.Rmd. Soon, it won't be new.
Is this going to be interactive_qc_visualization.Rmd - or will it do more than QC? Are there any goood examples of naming conventions for this kind of dashboard? Do we want riboviz in the name?

Request that we give a more informative name than new_visualization.Rmd. Soon, it won't be new.
Is this going to be interactive_qc_visualization.Rmd - or will it do more than QC? Are there any goood examples of naming conventions for this kind of dashboard? Do we want riboviz in the name?

... editing it to:
# process dashboard:
 # ...
    shell:
      """
      Rscript -e "rmarkdown::render( \
      '${workflow.projectDir}/rmarkdown/new_visualization.Rmd', \
      params = list(yamlfile='${workDir}/${config_file_yaml}'))"

      touch tmp.txt
      """

got me a bit closer:
# error from nextflow run:
# ...
  Quitting from lines 259-294 (new_visualization.Rmd)
  Error in file(file, "rt", encoding = fileEncoding) :
    cannot open the connection
  Calls: <Anonymous> ... eval -> eval -> find_sample_names -> read_yaml -> file
  In addition: Warning message:
  In file(file, "rt", encoding = fileEncoding) :
    cannot open file '/home/fanders6/riboviz/riboviz/work/config.yaml': No such file or directory
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/de/69179a94fecac0dbee93373a7788ad

I'm just not sure how to get the specific work directory hashes in there, though there must be a way...

... editing it to:
# process dashboard:
 # ...
    shell:
      """
      Rscript -e "rmarkdown::render( \
      '${workflow.projectDir}/rmarkdown/new_visualization.Rmd', \
      params = list(yamlfile='${workDir}/${config_file_yaml}'))"

      touch tmp.txt
      """

got me a bit closer:
# error from nextflow run:
# ...
  Quitting from lines 259-294 (new_visualization.Rmd)
  Error in file(file, "rt", encoding = fileEncoding) :
    cannot open the connection
  Calls: <Anonymous> ... eval -> eval -> find_sample_names -> read_yaml -> file
  In addition: Warning message:
  In file(file, "rt", encoding = fileEncoding) :
    cannot open file '/home/fanders6/riboviz/riboviz/work/config.yaml': No such file or directory
  Execution halted

Work dir:
  /home/fanders6/riboviz/riboviz/work/de/69179a94fecac0dbee93373a7788ad

I'm just not sure how to get the specific work directory hashes in there, though there must be a way...

Request that we give a more informative name than new_visualization.Rmd. Soon, it won't be new.
Is this going to be interactive_qc_visualization.Rmd - or will it do more than QC? Are there any goood examples of naming conventions for this kind of dashboard? Do we want riboviz in the name?

I agree with @ewallace that new_visualization.Rmd's name should be changed, but I suppose it's hard to know the future scope of what it would be used for just yet?  Perhaps something not involving QC, so it's a little more general and will cover us if it starts to include other cool plots/info?
(I think final naming rights should go to @p6t and @john-s-f 's team since they put it together :medal_sports:)

Request that we give a more informative name than new_visualization.Rmd. Soon, it won't be new.
Is this going to be interactive_qc_visualization.Rmd - or will it do more than QC? Are there any goood examples of naming conventions for this kind of dashboard? Do we want riboviz in the name?

I agree with @ewallace that new_visualization.Rmd's name should be changed, but I suppose it's hard to know the future scope of what it would be used for just yet?  Perhaps something not involving QC, so it's a little more general and will cover us if it starts to include other cool plots/info?
(I think final naming rights should go to @p6t and @john-s-f 's team since they put it together :medal_sports:)

Indeed, having riboviz in the name might be sensible seeing as how this is the visualization part. The name is certainly subject to change, I'm open to whatever names people think are appropriate.
@FlicAnderson We've managed to reproduce your error. We've attempted a solution which involved wrapping all of the code that does stuff (namely, create/reshape data frames, make plots) inside of
shinyServer(function(input, output,session){...}

Which gives us a different error of
Quitting from lines 433-434 (new_visualization.Rmd)
  Error in callModule(server, "RiboViz") :
    session must be a ShinySession or session_proxy object.
  Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> callModule
  Execution halted

I'll have to investigate this further. @p6t might have had some ideas how to resolve this but they're more involved than just wrapping the code in another function. I'll try some of them out.

Indeed, having riboviz in the name might be sensible seeing as how this is the visualization part. The name is certainly subject to change, I'm open to whatever names people think are appropriate.
@FlicAnderson We've managed to reproduce your error. We've attempted a solution which involved wrapping all of the code that does stuff (namely, create/reshape data frames, make plots) inside of
shinyServer(function(input, output,session){...}

Which gives us a different error of
Quitting from lines 433-434 (new_visualization.Rmd)
  Error in callModule(server, "RiboViz") :
    session must be a ShinySession or session_proxy object.
  Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> callModule
  Execution halted

I'll have to investigate this further. @p6t might have had some ideas how to resolve this but they're more involved than just wrapping the code in another function. I'll try some of them out.

Currently reading up on getParent as I was hoping it might work as something like: yamlfile='${config_file_yaml.getParent()}, since here in the Nextflow docs it suggests you can get file attributes such as the path of a file using it, but it just evaluated to 'null' somehow.
Oh well, dinnertime :grin:

Currently reading up on getParent as I was hoping it might work as something like: yamlfile='${config_file_yaml.getParent()}, since here in the Nextflow docs it suggests you can get file attributes such as the path of a file using it, but it just evaluated to 'null' somehow.
Oh well, dinnertime :grin:

Cracked it re: handing the nextflow process working directory path into rmarkdown!
Via: this discussion about passing current working directory of a process as input to a script, specifically to rmarkdown
Benjamin Wingfield @nebfield Jun 05 2017 10:56 UTC
is there a way to pass the current working directory of a process as an input to a script? I have an rmarkdown file that sets the working directory to its current location in $baseDir/reports/ - when I pass input files to rmarkdown it can't find them

Evan Floden @evanfloden Jun 05 2017 10:58 UTC
try $workDir https://www.nextflow.io/docs/latest/metadata.html

Benjamin Wingfield @nebfield Jun 05 2017 10:58 UTC
thanks, i'll have a look!

Benjamin Wingfield @nebfield Jun 05 2017 11:15 UTC
ah, $workDir gives top level access to the working directory only
I think I need the path printed by a submitted process

# ...

Benjamin Wingfield @nebfield Jun 05 2017 11:21 UTC
sorry I meant the output of nextflow, for example: [36/0140d4] Cached process > test

Evan Floden @evanfloden Jun 05 2017 11:25 UTC
I think I get what you mean now. So you need to access the dir from, for example, work/36/0140d4… in another downstream process?

Benjamin Wingfield @nebfield Jun 05 2017 11:25 UTC
yes, exactly
sorry, it would be the same process

Paolo Di Tommaso @pditommaso Jun 05 2017 11:25 UTC
use the bash implicit variable $PWD
taking care to escape it ie \$PWD

Benjamin Wingfield @nebfield Jun 05 2017 11:29 UTC
that makes sense, I'll try that thank you

Benjamin Wingfield @nebfield Jun 05 2017 11:45 UTC
that worked perfectly, thanks. after doing more reading rmarkdown files are a bit fragile and have some odd behaviour with working directories.

Paolo Di Tommaso @pditommaso Jun 05 2017 11:47 UTC
well NF can't fix rmarkdown fragility :)

Benjamin Wingfield @nebfield Jun 05 2017 11:48 UTC
haha unfortunately not :( thanks again!

Tried escaped $PWD with a slash, followed by our $config_file_yaml in dashboard process in prep_riboviz.nf:
      Rscript -e "rmarkdown::render( \
      '${workflow.projectDir}/rmarkdown/new_visualization.Rmd', \
      params = list(yamlfile='\$PWD/${config_file_yaml}'))"

& it worked!
Nextflow vignette run:
(riboviz) fanders6@ceres:~/riboviz/riboviz$ nextflow run prep_riboviz.nf -params-file vignette/simdata_multiplex_config.yaml -ansi-log false
N E X T F L O W  ~  version 20.07.1
Launching `prep_riboviz.nf` [furious_coulomb] - revision: 14ce0448bd