R help - Aug 2010 - Question regarding significance of a covariate in a coxme survival

Using a p-value to make any kind of decision is questionable to begin
with, and especially unreliable in choosing covariates in regression.
Old studies, e.g. by Walls and Weeks and by Bendel and Afifi, have shown
that if predictive ability is the criterion of interest and one wishes
to use p-values for deciding whether to include a covariate, one should
set the p-value bar very large, at 0.25 and even 0.40.

By contrast, methods such as AIC are aimed at avoiding overfitting, by
penalizing models with large numbers of covariates.  Same for Mallows'
Cp, cross validation etc.

So, the p-value and AIC are answering quite different questions, and
thus should not be expected to give the same or even similar results.
But, worse than that, many point out that p-values tend not to be
answering ANY question of practical interest.

It's a shame that the use of p-values is so entrenched.  I can expand on
this, with references, if there is interest.

Norm Matloff
Professor of Computer Science (formerly Statistics)
University of California, Davis

What statistical measure(s) tend to be answering ALL(?) question of practical
interest? 


-- 
View this message in context:
http://r.789695.n4.nabble.com/Re-Question-regarding-significance-of-a-covariate-in-a-coxme-survival-tp2399386p2399524.html
Sent from the R help mailing list archive at Nabble.com.

What statistical measure(s) tend to be answering ALL(?) question of practical
interest? 

-- 
View this message in context:
http://r.789695.n4.nabble.com/Re-Question-regarding-significance-of-a-covariate-in-a-coxme-survival-tp2399386p2399577.html
Sent from the R help mailing list archive at Nabble.com.

In 2010-08-30, C. Peng <button_an at hotmail.com> wrote:
> What statistical measure(s) tend to be answering ALL(?) question of
> practical interest?
None.  All I had said was that significance testing doesn't really
answer any questions of practical interest.  Unfortunately, that doesn't
mean there's something to answer all such questions.

In the regression case brought up by the original poster (that was Cox
regression, but the principle is the same), prediction-oriented measures
such AIC or cross-validation directly address the question of interest,
if that question is predictive ability.

In general, one should at the very least form confidence intervals
instead of significance tests.  In the case in which tests are
demanded, e.g. medical journals, replace "instead of" by "in
addition
to."

Norm