Discussion: Issues with large application development with Shiny

[Austin T]

I've been working on a large application for Shiny over the past few months, and I've found that Shiny's system doesn't seem to be ideal for this as it is right now. I'm going to try and list some of the problems I've had with trying to use Shiny in a larger application.

Modularity

Shiny, as it stands, is not very facilitating to encapsulation in development. This can pose a problem when I need to explicitly separate parts of an application. For example, suppose I want to make ModuleA and ModuleB and then make sure that each has its own reactive and regular environment, and can take in inputs from both the UI and other environments in the larger application, and also output from an application. Even with the recent Shiny modules proposal, it doesn't appear to be a clean fix: when I want to be able to create a function to be able to embed an arbitrary expression list into a module's server function, it doesn't evaluate correctly--with the ns(character()) function, it will not correctly capture the scope needed to embed something like observeEvent(input$button) under the real inputId: input$module.id-button and evaluate correctly.

I've been trying to fiddle with things like eval and substitute and I personally have a hard time trying to extend Shiny's functionality. I know a larger application might require more depth of knowledge, but as a user-friendly framework, the knowledge requirement of Shiny to create larger applications is particularly troublesome.

Reactivity

Reactivity is a great idea. However, having to introduce a second system of environments and dependencies turns out quickly to become a large headache in a bigger application. For simple relationships, such as choosing the range of a graph in a preview, this is fine. But, when I wanted to implement some sort of type checking and input curation from one module to another and have certain elements react to steps in this multi-step process, the logic can become quite large. I wouldn't actually mind choosing an environment for evaluation and then using callbacks. I quickly discarded the idea of reactive elements and isolate--the little differences seemed a bit esoteric, so I opted to simplify the design and use observeEvent() in many places. Also, when dealing with graphs that can take a long time to output, the reactivity can stall the application. Turning reactivity on and off might prove useful for those that want to disconnect parts of the application in the interest of saving time that would be eaten up by costly procedures hooked onto certain reactive variables. Man

About observeEvent()--the inputId can be a bit of problem due to the use of the additional evaluation and scoping problems as opposed to string manipulation of Id's. Again, when implementing a sort of tape-on namespacing system, eval/parse/substitute/quote got in the way and prevented me from doing so.

I hope I haven't offended anyone! I am clearly not a master in R, but I hope that I've put forth some issues that have some valid points and can drive discussion about the design of Shiny.

[Joe Cheng]

Hi Austin,

I'm not at all offended, but I'm also not even close to understanding what you're talking about. Can you give a small code example of how you're trying to communicate between module A and B? This is admittedly an under-documented area right now but I do have a clear idea in my head of how it should work, and "has its own reactive and regular environment" doesn't enter into it at all.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/65c7dac1-9af4-4813-8a7f-1fc457908d0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Austin T]

Hi Joe,

Here's an example of my code from one of my modules:

library(topGO)


#Create an environment under module_env.
module_env$topGO <- new.env()


#Set the current environment
env <- module_env$topGO
#Create a variable to hold reactive variables just for the module
env$rval <- reactiveValues()


env$rval$status_text <- 'Idle.'
env$rval$table_preview <- NULL


#Make sure that you evaluate all of your items in the new environment.
#Use evalq()
evalq(envir=env, {
  #UI object that will be rendered under ui_module
  ui_topGO <-
    tabsetPanel(
      tabPanel('topGO Enrichment Analysis',
        p('\n'),
        actionButton('topGO.start', "Start module (create topGO data object)"),
        actionButton('topGO.output', "Output data"),
        actionButton('topGO.clear', "Clear data"),
        HTML('<hr>'),
        column(4,
          h4('Analysis Settings'),
          textInput('topGO.mapping',
            value='org.Hs.eg.db',
            label='Package/Database For Terms (Mapping)'
          ),
          selectInput('topGO.select_ontology',
            selected = 'BP',
            label='Select Ontology',
            choices = c('BP', 'MF', 'CC'),
            multiple = FALSE
          ),
          numericInput('topGO.pvalue_threshold',
            label = 'P-value Threshold Of Interest',
            value = 0.05,
            min = 0,
            max = 1,
            step = 0.01
          ),
          selectInput('topGO.algorithm',
            label = 'Analysis Algorithm',
            selected = 'fisher',
            choices = c('fisher', 'ks', 't', 'globaltest', 'sum'),
            multiple = FALSE
          ),
          selectInput('topGO.statistic',
            label = 'Analysis Statistic',
            selected = 'classic',
            choices = c('classic', 'elim', 'weight01', 'lea'),
            multiple = FALSE
          ),
          actionButton('topGO.start_analysis', 'Start analysis'),
          HTML('<hr />'),
          h4('Format Output Table'),
          numericInput('topGO.numterms',
            label = 'Number Of Terms',
            value = 10,
            min = 1,
            step = 1
          ),
          actionButton('topGO.generate_table', 'Generate term table')
        ),
        column(8,
          tabsetPanel(
            tabPanel(title='Status',
              verbatimTextOutput('topGO.status_text')      
            ),
            tabPanel(title='Preview Term Table',
              tableOutput('topGO.term_table')
            )
          )
        )
      ),
      tabPanel('Help',
         p('\n'),
         shiny::includeMarkdown("modules/topGO/help.md")
      )
    )
 
  #Create your variables here, so that you can specify them as
  #input and output.
 
  #Input
  genes_and_pvalues <- NULL
 
  #Output
  terms_output <- NULL
 
  #Variables
  topGOdata <- NULL
  test_results <- NULL
 
 
  #Always remember to register your module at the end!
  register_module(
    name="Step [6] - Enrichment Analysis (topGO)",
    module=ui_topGO,
    envir=env,
    inputs=c("genes_and_pvalues"),
    outputs=c("terms_output")
  )
 
})

In this module's UI file, I load the required package and create an environment under a root-level environment called module_env to hold the environment for each module. Each environment in this module_env has its own reactive values environment that I use to serve all purposes where reactive values are needed to trigger the next step in any reactive process.

Notice how all of my inputId for the module are prefixed with topGO. This is due to the fact that namespaces did not exist at the time of writing, and are a recent proposal. I've instead opted to use this convention, which I do not like because it's purely by convention and any name collisions will be painful to debug.

With the new Shiny modules proposal and implementation, the way it works does not work correctly. This is due to the fact that the evaluation of the namespace ID's for the UI and the server are not the same due to the way environments are applied during the execution of the modules. In short, there exists a case where repeatedly applying ns() while modules are nested does not result in the correct ID's for the server-side, and the ID's are not statically placed in one entire UI object definition. The code above does not illustrate this. The code below is an example of an attempt I tried to implement my own modules in terms of Shiny modules. Note that I use shinyBS elements, which also have their own problems which make them incompatible with the namespacing scheme recently introduced with the Shiny modules:

moduleOptionUI <- function(id, title, ...) {
  # Create the namespace.
  ns <- shiny::NS(id)
  # Get the elements; THESE MUST BE bsCollapsePanel
  elements <- eval(substitute(...))
 
  return(
    bsCollapse(id=ns(id), multiple=TRUE, open='Hi',
      bsCollapsePanel(title=title, value=title,
        elements
      )
    )
  )
}


moduleOptionServer <- function(input, output, session, envir=parent.env(environment()), expr) {
  # Create a shorthand for the module environment.
  e <- envir
  # Preserve the executed expression.
  expr <- base::substitute(expr)
 
  # Execute the expression in a reactive context, but also encapsulated within the passed environment.
  shiny::observe({
    base::eval(expr, envir = e)
  })
}


ui <- navbarPage(title='test',
  fluidRow(
    column(3,
      p('asdfsadf')  
    ),
    column(9,
      column(4,
        moduleOptionUI(title = 'Module Options', id=ns('option1'),
          p('What is up.')                
        ),
        moduleOptionUI(title = 'Option 1', id=ns('option2'),
          p("Panel #2")  
        )
      ),
      column(8,
        p('preview part')  
      )
    )
  )
)


server <- function(input, output, session) {
 
}

My application requires inputs and outputs from not only the UI, but also other parts of the application as well, such as other modules and the global environment. To do this, I pair an environment per module and then specify the variable names for the inputs and outputs of the module. There exists a core framework for my application such that anytime I want to import and export variables in and out of modules, I can have the correct environment and variable names for the modules on hand in order to use the functions get() and assign() with the environments, thus allowing for I/O between modules and encapsulation between modules in an organized fashion.

Here is a short overview of how my application works:

Core Application Framework:

    - Handles I/O between modules

    - Requires modules to specify:

        - Input variable names

        - Output variable names

        - Module environments

    - Inputs to each module are invasive to the module environment.

    - Output to the core are invasive to the core environment.

    - Each module will have its UI hosted by the Core framework in its own panel.

    

What do I mean by invasive?

    - It means that an environment will be changed from the outside.

    - For inputs to a module, the core will insert the module inputs into the

      module environment.

    - For outputs from a module, the core will handle the output variables

      defined within the module environment and transfer it into the core environment.

      

Module Framework:

    - Each module has its own environment.

    - Within its own environment, a module has its own reactive variables.

    - Each module defines its own UI.

    - Each module has its own server file which will be run at the root level

      of the Shiny server function.

        - This is why I need to prefix all the UI names and encapsulate each

          module's functionality in its own environment.

    - On start up, each UI file is loaded once.

    - The source files for the server for each module is loaded as is into the

      server function using source(filename, local=TRUE).

      

Problems:

    - The UI id's are all global. There is no true encapsulation between module inputIds.

        - Shiny modules have solved this.

            - There is a problem, however, with the module server functions not

              getting the correct ID's. They seem to still be evaluated at the root

              server level and not within the module context.

    - Reactivity with the encapsulation system I've designed is a bit complex. I've

      opted to only use observeEvent()

        - observeEvent depends on a name, not a character type, for its bound

          reactive value or input ID. This means that namespacing cannot be applied

          without some work; I've tried to do a similar thing to the current namespacing

          scheme with paste, but something happened with that that prevented me from doing so.

          This is probably related to the quote/substitute/parse/eval issue again.

        - Reactivity also seems to change the way expressions are evaluated and

          this is not clear to the coder. If I want to create a module and then pass

          an observeEvent to a module's server function, I want it to react to the correct

          ID and inherit the correct environment for variables and reactive values.

    - Graphical output is clunky to handle--there is no way to arbitrarily pass around

      graphs independently of the data, which is unfortunate, as it can be expensive to

      host large data sets and regraph them if we are disatisfied with the resolution

      of the graphs or want to regenerate them.

        - Reactivity irritates this issue more because small changes can cause large

          delays. Having to explicitly code a guard to prevent this can take many lines of

          code.

        - Perhaps providing some sort of SVG file or resolution-independent format

          and then passing that around could allow for easier graphing in Shiny, and even

          R, in general. This would allow the storing of, for example, graphs as we

          dynamically adjust options and create more and more to be more trivial.

          Instead, we are currently forced to deal with files as the storage method, or 

          some sort of hand-rolled construct to handle the concept of passing a graph.

[Austin T]

Here is an example of the server file to be run as-is in the server function below:

env <- module_env$topGO


observeEvent(input$topGO.clear, {
  env <- module_env$topGO
 
  #Input
  env$gene_and_pvalues <- NULL
 
  #Output
  env$terms_output <- NULL
 
  #Variables
  env$rval$status_text <- "Cleared data."
  env$topGOdata <- NULL
 
  gc()
})


#Use observeEvent to trigger on the change of a reactive value
#or input
observeEvent(input$topGO.start, {
  #Unfortunately, we need to reset the environment
  #for every trigger we use.
  env <- module_env$topGO
 
  env$ready = FALSE
 
  withProgress(message='Loading gene/terms package/library...', {
    tryCatch({
      library(input$topGO.mapping, character.only = T)
      env$ready = TRUE
    }, warning = function(w) {
      suppressWarnings(input$topGO.mapping, character.only = T)
      env$ready = TRUE
      print(w)
      env$rval$status_text <- capture.output(print(w))
    }, error = function(e) {
      env$ready = FALSE
      print(e)
      env$rval$status_text <- capture.output(print(e))
    })
  })
 
  #Operates on values under a pvalue threshold
  if(env$ready) {
    shinyjs::js$playSound('core_sound_tick')
    withProgress(message='Creating topGO data object...', {
     
#       env$genelist <- env$genes_and_pvalues
#       env$temp_genelist <- env$genes_and_pvalues
#       env$interested_genes <- as.integer(env$genelist %in% env$genes)
#       names(env$interested_genes) <- env$genes_and_pvalues
     
      tryCatch({
        env$topGOdata <- invisible(
          new("topGOdata",
              ontology = "BP",
              allGenes = p.adjust(env$genes_and_pvalues, method='BH'),
              geneSel = function(x) {return(x < input$topGO.pvalue_threshold)},
              nodeSize = 5,
              annot = annFUN.org,
              mapping = input$topGO.mapping,
              ID = 'symbol'
          )
        )
        env$rval$status_text <- capture.output(print(env$topGOdata))
        shinyjs::js$playSound('core_sound_alert')
      }, warning = function(w) {
        env$topGOdata <- suppressWarnings(
          invisible(
            new("topGOdata",
                ontology = input$topGO.select_ontology,
                allGenes = p.adjust(env$genes_and_pvalues, method='BH'),
                geneSel = function(x) {return(x < input$topGO.pvalue_threshold)},
                nodeSize = 5,
                annot = annFUN.org,
                mapping = input$topGO.mapping,
                ID = 'symbol'
            )
          )
        )
        #Capture a short description after the data is read.
        env$rval$status_text <- capture.output(print(env$topGOdata))
        env$rval$status_text <- paste(env$rval$status_text, w)
        shinyjs::js$playSound('core_sound_alert')
      }, error = function(e) {
        print(e)
        env$rval$status_text <- e
      })
    })
  }
})


observeEvent(input$topGO.output, {
  #Unfortunately, we need to reset the environment
  #for every trigger we use.
  env <- module_env$topGO
 
  withProgress(message='Outputing term table...', {
    env$terms_output <- env$rval$table_preview
  })
 
  #Use module_output() to output your modules
  #disallow "Alice" from being output to the core
  module_output(disallow=is.null)
})


observeEvent(input$topGO.start_analysis, {
  env <- module_env$topGO
  if (!is.null(env$topGOdata)) {
    shinyjs::js$playSound('core_sound_tick')
    withProgress(message='Running analysis...', {
      env$rval$status_text <- capture.output(
        env$test_results <- runTest(
          object = env$topGOdata,
          algorithm = input$topGO.statistic,
          statistic = input$topGO.algorithm
        )
      )
    })
    shinyjs::js$playSound('core_sound_alert')
  }
})


observeEvent(input$topGO.generate_table, {
  env <- module_env$topGO
 
  if (!is.null(env$test_results)) {
    shinyjs::js$playSound('core_sound_tick')
    withProgress(message='Generating table...', {
      env$rval$table_preview <- GenTable(
        env$topGOdata,
        analysisScore = env$test_results,
        numChar=999,
        topNodes = input$topGO.numterms
      )
    })
  }
})


#status text
output$topGO.status_text <- renderPrint({
  env <- module_env$topGO
  print(env$rval$status_text)
})


output$topGO.term_table <- renderTable({
  env <- module_env$topGO
  return(env$rval$table_preview)
})

Notice the mandatory switching back to the correct module environment, which is held in the root scope.

[Joe Cheng]

Ack! I'm sorry that you have invested so much in this approach! It is disastrously incompatible with the way Shiny is designed to work. I hesitated to add the observeEvent feature for many releases despite its obvious utility, precisely because I thought some people might be misled into implementing their Shiny apps in this imperative style instead of reactively.

I can't back you out of all of this in a single email, but let's start with a single issue: you mentioned that when using the official Shiny modules, your server doesn't have access to the right (namespaced) inputs/outputs, but the top-level un-namespaced inputs/outputs. Are you using callModule() to invoke your module server function? And are you passing it the same id argument that you're passing to the corresponding module UI?

Once we determine what's going wrong with that, I'll explain the communication mechanism between modules. And once we are clear on that, we can talk about how to restructure your code around reactive() instead of observeEvent().

At the end of this, we'll have gotten rid of:

• Explicit creation of environments (new.env())
• Any mention of eval, evalq, substitute, parse, quote--these are very *very* rarely a good idea in Shiny apps
• Implicit knowledge of variables across module/core boundaries
• And likely, your performance problems as well!

(I don't suppose your app is on github? Or if not, would you be able and willing to share the code with me privately?)

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/54eab796-532d-49b0-870f-d39d93ea7752%40googlegroups.com.

[Austin T]

I don't think I can share the full code to the application directly since it's not only mine (it'll be released shortly), but I can share this short self-contained experiment that should work alone:

library(shinyBS)
library(shiny)

moduleOptionUI <- function(id, title, ...) {
  # Create the namespace.
  ns <- shiny::NS(id)
  # Get the elements; THESE MUST BE bsCollapsePanel
  elements <- eval(substitute(...))
 
  return(
    bsCollapse(id=ns(id), multiple=TRUE, open='Hi',
      bsCollapsePanel(title=title, value=title,
        elements
      )
    )
  )
}

moduleOption <- function(input, output, session, envir=parent.env(environment()), expr) {

  # Create a shorthand for the module environment.
  e <- envir
  # Preserve the executed expression.
  expr <- base::substitute(expr)

 
  # Execute the expression in a reactive context.

  shiny::observe({
    base::eval(expr, envir = e)
  })
}

ui <- navbarPage(title='Example',
  tabPanel('Workflow',
    fluidRow(
      column(4,
        actionButton('button', 'Stop'),
        uiOutput('ui_core')
      ),
      column(8,
        uiOutput('ui_module')
      )
    )
  )
)

core_env <- new.env()
core_env$rval <- reactiveValues()
core_env$rval$module_choices <- c('A', 'B', 'C')
core_env$rval$module_tags_category <- c('a', 'b', 'c')
core_env$rval$module_tags_workflow <- c('1', '2', '3')
core_env$rval$module_tags_function <- c('z', 'x', 'y')

core_env$rval$status_text <- 'Idle.'


coreStatusText <- function(string, core_env, session, style='warning') {
  core_env$rval$status_text <- string
  updateCollapse(session=session, id='Status', open='Status', style=list('Status'=style))
}


moduleCoreUI <- function(id, title, core_env, ...) {

  # Create the namespace.
  ns <- shiny::NS(id)
  # Get the elements; THESE MUST BE bsCollapsePanel
  elements <- eval(substitute(...))

 
  div(
    bsCollapse(
      bsCollapsePanel(title='Status', value='status_open', style='info',
        p('\n'),
        p(core_env$rval$status_text)
      )
    ),
    wellPanel(
      elements
    )
  )
}


moduleCoreServer <- function(input, output, session, core_env, expr) {
  ns <- session$ns
 
  # Create a shorthand for the module environment.
  e <- core_env
  # Preserve the executed expression.
  eval(expr)
}


moduleContentsUI <- function(id, title, core_env, ...) {

  # Create the namespace.
  ns <- shiny::NS(id)
  # Get the elements; THESE MUST BE bsCollapsePanel
  elements <- eval(substitute(...))

 
  div(
    fluidRow(
      column(6,
        selectInput(ns('module_select'),
          label='Select module',
          multiple = FALSE,
          choices = core_env$rval$module_choices,
          selectize = TRUE)
      ),
      column(2,
        selectInput(ns('module_select_category'),
          label='Category',
          multiple = FALSE,
          choices = core_env$rval$module_tags_category,
          selectize = TRUE)
      ),
      column(2,
        selectInput(ns('module_search_workflow'),
          label='Workflow',
          multiple = FALSE,
          choices = core_env$rval$module_tags_workflow,
          selectize = TRUE)
      ),
      column(2,
        selectInput(ns('module_select_function'),
          label='Function',
          multiple = FALSE,
          choices = core_env$rval$module_tags_function,
          selectize = TRUE)
      )
    ),
    fluidRow(
      column(4,
        h4('Options')
        #optionElements
      ),
      column(8,
        h4('Preview')
        #previewElements

      )
    )
  )
}


server <- function(input, output, session) {

  output$ui_core <- renderUI({
    ns <- session$ns
    moduleCoreUI('g_core', 'Core', core_env,
      actionButton(ns('button'), 'Change message')  
    )
  })
 
  output$ui_module <- renderUI({
    ns <- session$ns
    moduleContentsUI('g_module', 'Module', core_env)
  })
 
  observeEvent(input$button, {
    browser()
    updateCollapse(session=session, id='Status', open='Status', style=list('Status'='warning'))
  })
 
  callModule(moduleCoreServer, 'g_core', core_env=core_env, session=session, expr=
      observeEvent(input$button, quote({
        coreStatusText('Help!', core_env, session, 'warning')
        print('asdf')
      }))
  )
}


shinyApp(ui, server)

On Tuesday, January 19, 2016 at 1:40:13 PM UTC-10, Austin T wrote:

[Joe Cheng]

I couldn't figure out the overall intent of your example. The moduleOption looked like it maybe was factored out of moduleCore, but then wasn't actually called from anywhere. moduleContentsUI had no corresponding module server function. The various sets of module choices in the rval, I couldn't tell if those were to be dynamic or static, which would make a huge difference for how those were implemented either way. And the use of renderUI, while perfectly legal, would needlessly confuse the points I need to make here.

I decided to try to meet you halfway by creating an app with a single module that wraps bsCollapse and a status text for reuse like you were doing.

https://gist.github.com/jcheng5/a55890a84f90860befa6

There is so much that won't come across just from the code sample, though. I think maybe we should video chat about this, if you're up for that.

In the meantime, let me drop some random tips here:

1. ShinyBS bsCollapse works fine with modules, but note: updateCollapse's id parameter shouldn't be wrapped in session$ns(). You can see in my example that in my module UI I have bsCollapse(id = ns("collapse")) and in the module server function I have updateCollapse(session, "collapse"). This works because updateCollapse calls session$sendInputMessage, which automatically does the session$ns() wrapping for the id parameter. If you call updateCollapse(session, session$ns("collapse")) you'll end up double-namespacing it.
2. In my example, when I call statusBoxUI, I pass in several UI controls. Note that I do NOT attempt to wrap them in ns() (because it's at the top level), and I do NOT attempt to get them to use the namespace of the module I'm shoving them into. Rather, these controls are created at the top level, and are only displayed inside that module. If I want to handle/read those inputs, I need to do it at the top level.
3. So what happens if you want to coordinate between stuff that happens at the top level and in a module, or between module A and module B, or even between two instance of module A in the same page? You do this through input and output arguments to the module server function(s). You can see that my module lets the caller control the status text by passing in an rx_status reactive expression; in this case it's a complicated eventReactive, but in other cases you might do something as simple as rx_status=reactive(input$status). And the module returns a function that callers can use to open/close the bsCollapse.
4. All the places you had eval(substitute(...)) in the UI could be replaced with list(...) (or getting rid of the elements variable entirely and just passing a naked ... where you are passing elements).

An additional note on point #2. You were trying to do something akin to this in your example (you were doing it in renderUI, but the upshot is the same):

ui <- fluidPage(

  moduleA("one", actionButton(ns("button"), "Stop")),

  moduleA("two", actionButton(ns("button"), "Change message")),

)

So this definitely doesn't give those two action buttons ids that are namespaced in "one" and "two". But with that fact out of the way, even the attempt here to duplicate the "button" ID is misguided. Shiny Modules don't ever let you write code like this, where at the same "level" (i.e. in a given module UI function, or a top-level ui definition) you get to use the same ID twice to mean different things. It's still on you to ensure that the IDs are all unique within each level. What modules let you do is stop worrying once you've ensured that, whereas before you'd also have to worry if IDs were unique between different levels.

I'll have to stop there for now. Please do let me know whether you want to vchat.

--
You received this message because you are subscribed to the Google Groups "Shiny - Web Framework for R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shiny-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/d875ce7b-7a5b-477c-ac7d-2488a7363afe%40googlegroups.com.

[Austin T]

Hi Joe,

Sorry for the late reply; I had to contribute to some sort of project for a competition. If you'd like to video chat, I may be able to arrange that. It may or may not be difficult due to the time zone difference, though.

I'm going to make responses to the four points you brought up:

1. Ah, simple enough. It's nice to have implicit namespacing for the server.
2. Oh, but what about the case where I want to have the UI I give it with namespaced ID's? Otherwise, true encapsulation for those elements inside of the module namespace doesn't work. I think generic modules are quite important to prevent people from having to write a lot of the Shiny server boilerplate or other boilerplate over again if they're already decided on a general module format.
3. I was planning to pass the arguments through the module inputs, but not necessarily through the outputs. If the output of a module is expected to be some sort of data, what if I also want to provide graphical output in Shiny and "pass" it around? For example, say I want to keep track of all the graphs that I plot in a Shiny app, then I want to provide a list of those that they might want to preview or download later. As far as I know, R doesn't have an obvious, generic way of passing around graphs as data without the copying the data that the graph references (or wrapping it in a special reference object class). I suppose I could pass around closures with references to the data environment along with the data output, but then I mix the graphical data output and the data output from a module. I think, if I were to redo my module format, I'd use this method. I'm too far along in the development process to change it for my current application, though, which is sort of sad.
4. I was trying to do this in an effort to evaluate the ns() once, and then perhaps replace it with a changed ns() call such that: if I have a "button" inside of a module "A" inside of a module "B", I would want the eventual ID for the button to expand to B-A-button. However, if I simply pass the elements without any attempt to change the evaluation, I would think that they would either get immediately evaluated to the HTML, capturing only A as the namespace tag and ending up with A-button, or they would not call the ns() and instead end up with B-button somehow. 

As you can see, I really wanted to be able to pass in arbitrary elements like a button into a module, and then have the module system modify the namespacing until it reached the top level, cascading through the layers. My original idea was to implicitly derive the IDs from the environment hashes, but I wasn't able to successfully implement.

[Joe Cheng]

For #3, if you want your return value to be more than one thing, then return a list that contains more than one thing. And just to be clear, if you want a plot to appear you can do that by just having a plotOutput in that namespace; but if you want to do some further things with a plot (like you're saying, have a button that lets you download a whole bunch of different plots) then yes, you need to return "something" that represents a plot. You can use grDevices::recordPlot() to generate such an object, I believe; and then use grDevices::replayPlot() later.

Correct me if I'm wrong, but if you follow my advice for #3 then the other issues (#2 and #4) fall away?

The problem with inserting a button or whatever into a moduleUI, and wanting it then to be transformed into having the namespace of that module, and then inserting corresponding logic into a module server function, and similarly wanting it to inherit the input/output/session of that module... whew... is that this violates the entire purpose of modules, which is encapsulation.

As Shiny apps get large, they get hard to understand because the interaction between all the different parts becomes too much to hold in your head.

So we break each piece of functionality into a module. Within each module, you don't have to worry about anything other than what inputs/outputs, reactives, observers, variables, etc. exist just in that particular module. And it's absolutely critical that all of those interactions be treated as a "black box" to anyone on the outside. You have to be able to rely on the assumption that changes to your module won't affect any other module, as long as you don't change the function arguments you're expecting and the return value you're returning. And you have to similarly be able to assume that other modules can change whatever they like other than the function arguments and return values, and they won't affect your module either.

The function args and return value are the public contract that your module has with the rest of the world; everything else is an implementation detail. It's vital that the public contract be all anyone needs to know to successfully interact with your module.

Again, it's totally OK for a module to let a caller insert a widget into the middle of a module UI--as long as the interaction with that widget (i.e. the code that reads input$foo and does something with it) is performed by the caller.

If you follow all that logic, then there's no possible reason you could have for wanting to crack into the namespace of a module from the outside. It's simply not ever something you should want to do to crack that black box. If we allow that, it's game over for encapsulation, and anytime there's a problem with your module you have to look at your entire codebase, not just in your one module.

------

Everybody except for Austin should ignore the rest of this email!

Now, that's the theory. In the real world, you've written a (presumably) huge application that wasn't designed this way. And BTW we haven't even gotten to the proper use of reactivity, we're still just talking about namespaces! And you're releasing any moment or whatever. So I'll tell you how to crack the black box and wish you the best of luck--but I request that you talk to me before you start on your next large Shiny app so I can help you avoid this situation in the future.

Again, I do NOT find the following advisable unless you are truly stuck and can't go back and refactor things.

If you want to do this:

mymoduleUI("foo", actionButton(ns("stop"), "Stop"))

You can make it work like this:

mymoduleUI("foo", actionButton(NS("foo", "stop"), "Stop"))

Or if this line of code itself appears in another module UI, then:

mymoduleUI(ns("foo"), actionButton(NS(ns("foo"), "stop"), "Stop"))

Note that it's on you to make sure that mymoduleUI doesn't already have a widget with an ID of "stop". (Because we're breaking encapsulation, we can't rely on namespaces to protect us from collisions.)

Or if you're unsure whether mymoduleUI will directly embed the control(s) you're passing it, or whether it'll pass it further to another module it's calling, then instead of taking the controls as arguments, take a callback function as an argument. The callback function should have an id as argument, and return elements. Then the caller can write "ns <- NS(id)" at the top of the callback function, and use ns() as usual in the rest of the callback function code.

And for the server side, instead of this:

callModule(mymodule, "foo", observeEvent(input$stop, { ... }))

Then have your module take a function that it will pass input, output, and session to:

mymodule <- function(input, output, session, extraBehavior = NULL) {

  if (!is.null(extraBehavior)) {

    extraBehavior(input, output, session)

  }

}

callModule(mymodule, "foo", function(input, output, session) {

  observeEvent(input$stop, {

    # now you can access the module's private input and output

  })

})

It's awful, it breaks encapsulation to little pieces, but it should work.

To view this discussion on the web visit https://groups.google.com/d/msgid/shiny-discuss/d6852328-3799-49a6-b6a0-a54218b453d1%40googlegroups.com.