Off-list. R input> still readable
11 views
Skip to first unread message
Daniel Lee
Feb 3, 2017, 8:40:52 AM2/3/17
to stan-...@googlegroups.com
Hi Bob,
Just wanted to let you know that it's still readable. Different mailers might highlight it differently, but it's still getting through.
Daniel
> On Feb 3, 2017, at 2:56 AM, Bob Carpenter <ca...@alias-i.com> wrote:
>
> Ack. My mailer's treating the R input > as an embedded message.
> And this isn't Discourse, so those ticks didn't do anything.
> Hopefully it's still readable.
>
> - Bob
>
>> On Feb 3, 2017, at 2:54 AM, Bob Carpenter <ca...@alias-i.com> wrote:
>>
>> Part II is just the technical issue.
>>
>> But it's more confusing than I thought. The core concept here is
>> that R passes by value and that the value of (y ~ x) is a formula.
>> But it looks like it behaves differently than ("y ~ x") in a function
>> context.
>>
>> So this all looks OK:
>>
>> ``
>> $ foo <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm(y ~ x)); }
>> $ foo(10)
>>
>> Call:
>> lm(formula = y ~ x)
>>
>> Coefficients:
>> (Intercept) x
>> 199.6 100.4
>> ```
>>
>> But now what happens when I quote that?
>>
>> ```
>>> bar <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm("y ~ x")); }
>>> bar(10)
>> Error in eval(expr, envir, enclos) : object 'y' not found
>> ```
>>
>> Hmm. I honestly don't know what's going on there. But a little
>> spelunking shows me it's looking further up the environment chain:
>>
>> ```
>>> bar <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm("y ~ x")); }
>>> x <- rnorm(10); y <- rnorm(10)
>>> bar(10)
>>
>> Call:
>> lm(formula = "y ~ x")
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>> ```
>>
>> If you're in the top-level environment, now that x and y are defined,
>> quotes and no quotes appear to work the same way:
>>
>> ```
>>> lm(y ~ x)
>>
>> Call:
>> lm(formula = y ~ x)
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>>
>>> lm("y ~ x")
>>
>> Call:
>> lm(formula = "y ~ x")
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>> ```
>>
>>
>> I'm confused. And rather frightened by this behavior.
>>
>> - Bob
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On Feb 1, 2017, at 5:23 PM, Andrew Gelman <Gel...@stat.columbia.edu> wrote:
>>>
>>> Hi, Jonah. We can talk about it on Friday. Short answer is that when working with data and fitting models it's a real pain in the ass to keep working with a data frame. I actually don't like the whole "data frame" thing as it can act as a straitjacket, it's a Stata way of thinking.
>>>
>>> Here's an example. I start with this model (for simplicity I'm assuming we have a stan_logit() alias):
>>>
>>> stan_logit(y ~ dist + arsenic)
>>> or
>>> stan_logit(y ~ dist + arsenic, data="wells")
>>>
>>> Ok, fine so far. Now I decide I want to code as dist/100:
>>> dist100 <- dist/100
>>> stan_logit(y ~ dist100 + arsenic)
>>>
>>> That worked fine. Now the alternative:
>>> dist100 <- dist/100 INSIDE "wells"
>>> stan_logit(y ~ dist100 + arsenic, data="wells")
>>>
>>> So here's my question: can I do this:
>>> dist100 <- dist/100 INSIDE "wells"
>>> in a reasonable way?
>>>
>>> Right now I have to first pull dist out of wells, then compute dist100, then put it back into wells, something like this:
>>> dist <- wells$dist
>>> dist100 <- dist/100
>>> APPEND wells with dist100
>>> (I don't remember how to do that last command but I know it can be done in one line.)
>>>
>>> Anyway, that's the point. In the midst of running regressions I play with data, for example creating a new predictor:
>>> x_avg <- (x1 + x2 + x3)/3
>>> or whatever.
>>> If I have to keep putting things into the data frame just to run my regressions, that's a pain in the ass. It's _not_ just "a bit more typing," it's actual additional lines of code, it makes things harder to follow.
>>>
>>> In Stata world, you have a big data matrix and then pull out columns of it. I don't see this as being R world.
>>>
>>> A
>>>
>>>
>>>> On Feb 1, 2017, at 4:47 PM, Jonah Gabry <jga...@gmail.com> wrote:
>>>>
>>>> On Wednesday, February 1, 2017 at 4:06:31 PM UTC-5, Andrew Gelman wrote:
>>>> I'm only including code that runs!
>>>>
>>>> But, that's right, I don't want to include crap like "data=data" in all of my stan and stan_glm calls.
>>>> A
>>>>
>>>> Well, not "data=data" but maybe "data=wells" isn't so bad? Unlike the list you need to (or needed to) manually create for rstan, for rstanarm it would just be the name of the data frame where the variables in the formula can be found. I understand the appeal of not needing to do that, but I think that's outweighed by the mess it causes for software development. Here's a pros/cons list:
>>>>
>>>> Cons of searching for data in global (or local) environment if not specified:
>>>>
>>>> 1) Takes developers a lot more time to add anything new to rstanarm because it's not easy to ensure that the data is handled appropriately. This is not an exaggeration: at most 1/4 of the time I spent implementing kfold was getting the algorithm right and the rest was spent on avoiding errors when people specify data without a data argument. And even after all that time there are apparently still cases when it doesn't work!)
>>>> 2) Inevitably more bugs because the internal code needs to be much more complicated and is harder to test
>>>> 3) Easier for the user to make mistakes (need to be very vigilant about what's already in the global environment)
>>>> 4) I could probably come up with a bunch more!
>>>>
>>>> Pros of searching for data in global (or local) environment if not specified:
>>>>
>>>> 1) A little bit less typing for the user
>>>>
>>>> Am I missing any other advantages other than a bit less typing? I really don't want to sacrifice the quality of the software just to avoid typing something like data=wells, but if there are other advantages I'm missing then maybe those well tip the scale.
>>>>
>>>> Jonah
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wednesday, February 1, 2017 at 4:06:31 PM UTC-5, Andrew Gelman wrote:
>>>> I'm only including code that runs!
>>>>
>>>> But, that's right, I don't want to include crap like "data=data" in all of my stan and stan_glm calls.
>>>> A
>>>>
>>>>> On Feb 1, 2017, at 1:17 PM, Bob Carpenter <ca...@alias-i.com> wrote:
>>>>>
>>>>> I really don't like this "read out of the global environment"
>>>>> thing. I hope Andrew's not writing the new book to lean heavily
>>>>> on that.
>>>>>
>>>>> - Bob
>>>>>
>>>>>> On Feb 1, 2017, at 11:44 AM, Jonah Gabry <jga...@gmail.com> wrote:
>>>>>>
>>>>>> So... kfold attempts to account for the fact that sometimes data is specified using the data argument to stan_glm and other times using the global environment. When no data argument is specified it needs to construct a data frame that it can use to refit the model K times (each time modifying that data frame to drop the right set of observation). So it constructs a data frame consisting of the 2nd through 5th columns of X because that's what you implicitly told it was your data. But now when we need to go refit the model how is the formula including X[, 2:5] supposed to be interpreted relative to this data frame? There are ways around this but so far they seem to introduce other complications that are even more annoying.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>> To post to this group, send email to stan-...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>> To post to this group, send email to stan-...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Just wanted to let you know that it's still readable. Different mailers might highlight it differently, but it's still getting through.
Daniel
> On Feb 3, 2017, at 2:56 AM, Bob Carpenter <ca...@alias-i.com> wrote:
>
> Ack. My mailer's treating the R input > as an embedded message.
> And this isn't Discourse, so those ticks didn't do anything.
> Hopefully it's still readable.
>
> - Bob
>
>> On Feb 3, 2017, at 2:54 AM, Bob Carpenter <ca...@alias-i.com> wrote:
>>
>> Part II is just the technical issue.
>>
>> But it's more confusing than I thought. The core concept here is
>> that R passes by value and that the value of (y ~ x) is a formula.
>> But it looks like it behaves differently than ("y ~ x") in a function
>> context.
>>
>> So this all looks OK:
>>
>> ``
>> $ foo <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm(y ~ x)); }
>> $ foo(10)
>>
>> Call:
>> lm(formula = y ~ x)
>>
>> Coefficients:
>> (Intercept) x
>> 199.6 100.4
>> ```
>>
>> But now what happens when I quote that?
>>
>> ```
>>> bar <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm("y ~ x")); }
>>> bar(10)
>> Error in eval(expr, envir, enclos) : object 'y' not found
>> ```
>>
>> Hmm. I honestly don't know what's going on there. But a little
>> spelunking shows me it's looking further up the environment chain:
>>
>> ```
>>> bar <- function(n) { x <- rnorm(n); y <- 100 * x + 200 + rnorm(n); return(lm("y ~ x")); }
>>> x <- rnorm(10); y <- rnorm(10)
>>> bar(10)
>>
>> Call:
>> lm(formula = "y ~ x")
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>> ```
>>
>> If you're in the top-level environment, now that x and y are defined,
>> quotes and no quotes appear to work the same way:
>>
>> ```
>>> lm(y ~ x)
>>
>> Call:
>> lm(formula = y ~ x)
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>>
>>> lm("y ~ x")
>>
>> Call:
>> lm(formula = "y ~ x")
>>
>> Coefficients:
>> (Intercept) x
>> 0.02121 -0.20474
>> ```
>>
>>
>> I'm confused. And rather frightened by this behavior.
>>
>> - Bob
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On Feb 1, 2017, at 5:23 PM, Andrew Gelman <Gel...@stat.columbia.edu> wrote:
>>>
>>> Hi, Jonah. We can talk about it on Friday. Short answer is that when working with data and fitting models it's a real pain in the ass to keep working with a data frame. I actually don't like the whole "data frame" thing as it can act as a straitjacket, it's a Stata way of thinking.
>>>
>>> Here's an example. I start with this model (for simplicity I'm assuming we have a stan_logit() alias):
>>>
>>> stan_logit(y ~ dist + arsenic)
>>> or
>>> stan_logit(y ~ dist + arsenic, data="wells")
>>>
>>> Ok, fine so far. Now I decide I want to code as dist/100:
>>> dist100 <- dist/100
>>> stan_logit(y ~ dist100 + arsenic)
>>>
>>> That worked fine. Now the alternative:
>>> dist100 <- dist/100 INSIDE "wells"
>>> stan_logit(y ~ dist100 + arsenic, data="wells")
>>>
>>> So here's my question: can I do this:
>>> dist100 <- dist/100 INSIDE "wells"
>>> in a reasonable way?
>>>
>>> Right now I have to first pull dist out of wells, then compute dist100, then put it back into wells, something like this:
>>> dist <- wells$dist
>>> dist100 <- dist/100
>>> APPEND wells with dist100
>>> (I don't remember how to do that last command but I know it can be done in one line.)
>>>
>>> Anyway, that's the point. In the midst of running regressions I play with data, for example creating a new predictor:
>>> x_avg <- (x1 + x2 + x3)/3
>>> or whatever.
>>> If I have to keep putting things into the data frame just to run my regressions, that's a pain in the ass. It's _not_ just "a bit more typing," it's actual additional lines of code, it makes things harder to follow.
>>>
>>> In Stata world, you have a big data matrix and then pull out columns of it. I don't see this as being R world.
>>>
>>> A
>>>
>>>
>>>> On Feb 1, 2017, at 4:47 PM, Jonah Gabry <jga...@gmail.com> wrote:
>>>>
>>>> On Wednesday, February 1, 2017 at 4:06:31 PM UTC-5, Andrew Gelman wrote:
>>>> I'm only including code that runs!
>>>>
>>>> But, that's right, I don't want to include crap like "data=data" in all of my stan and stan_glm calls.
>>>> A
>>>>
>>>> Well, not "data=data" but maybe "data=wells" isn't so bad? Unlike the list you need to (or needed to) manually create for rstan, for rstanarm it would just be the name of the data frame where the variables in the formula can be found. I understand the appeal of not needing to do that, but I think that's outweighed by the mess it causes for software development. Here's a pros/cons list:
>>>>
>>>> Cons of searching for data in global (or local) environment if not specified:
>>>>
>>>> 1) Takes developers a lot more time to add anything new to rstanarm because it's not easy to ensure that the data is handled appropriately. This is not an exaggeration: at most 1/4 of the time I spent implementing kfold was getting the algorithm right and the rest was spent on avoiding errors when people specify data without a data argument. And even after all that time there are apparently still cases when it doesn't work!)
>>>> 2) Inevitably more bugs because the internal code needs to be much more complicated and is harder to test
>>>> 3) Easier for the user to make mistakes (need to be very vigilant about what's already in the global environment)
>>>> 4) I could probably come up with a bunch more!
>>>>
>>>> Pros of searching for data in global (or local) environment if not specified:
>>>>
>>>> 1) A little bit less typing for the user
>>>>
>>>> Am I missing any other advantages other than a bit less typing? I really don't want to sacrifice the quality of the software just to avoid typing something like data=wells, but if there are other advantages I'm missing then maybe those well tip the scale.
>>>>
>>>> Jonah
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wednesday, February 1, 2017 at 4:06:31 PM UTC-5, Andrew Gelman wrote:
>>>> I'm only including code that runs!
>>>>
>>>> But, that's right, I don't want to include crap like "data=data" in all of my stan and stan_glm calls.
>>>> A
>>>>
>>>>> On Feb 1, 2017, at 1:17 PM, Bob Carpenter <ca...@alias-i.com> wrote:
>>>>>
>>>>> I really don't like this "read out of the global environment"
>>>>> thing. I hope Andrew's not writing the new book to lean heavily
>>>>> on that.
>>>>>
>>>>> - Bob
>>>>>
>>>>>> On Feb 1, 2017, at 11:44 AM, Jonah Gabry <jga...@gmail.com> wrote:
>>>>>>
>>>>>> So... kfold attempts to account for the fact that sometimes data is specified using the data argument to stan_glm and other times using the global environment. When no data argument is specified it needs to construct a data frame that it can use to refit the model K times (each time modifying that data frame to drop the right set of observation). So it constructs a data frame consisting of the 2nd through 5th columns of X because that's what you implicitly told it was your data. But now when we need to go refit the model how is the formula including X[, 2:5] supposed to be interpreted relative to this data frame? There are ways around this but so far they seem to introduce other complications that are even more annoying.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>>> To post to this group, send email to stan-...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
>> To post to this group, send email to stan-...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Jonah Gabry
Feb 3, 2017, 10:33:58 AM2/3/17
to Stan users mailing list
Yeah still readable. That was still on list btw.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
>>>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to stan-...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to stan-...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to stan-...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
Daniel Lee
Feb 3, 2017, 12:08:47 PM2/3/17
to stan-...@googlegroups.com
Damn it. I'm definitely losing today.
Daniel
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages