R help - Nov 2001 - SQL implementations (was: Are you experienced in SAS and R ...)

An embedded message was scrubbed...
From: Douglas Bates <bates at stat.wisc.edu>
Subject: Re: [R] Are you experienced in SAS and R as well? Which of these would
you recommend me?
Date: 23 Nov 2001 15:16:59 -0600
Size: 4446
Url:
https://stat.ethz.ch/pipermail/r-help/attachments/20011123/8d0399f0/attachment.mht

Douglas Bates <bates at cs.wisc.edu> writes:
> My purpose is to give my opinions as a rather heave user of both MySQL
> and PostgreSQL.
That's a typo.  I meant to write "rather heavy user".

I was just creating a PostgreSQL database and realized another
advantage of PostgreSQL - in PostgreSQL tables you can use external
keys to enforce referential integrity.  In the example I was
describing where the records in the test score table include a student
identifier and a school identifier and a district identifier, I can
indicate in the definition of the table that the student id refers to
the primary key in the student table.  An attempt to insert a record
into the test score table with a student id that does not occur in the
student table will produce an error.
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Douglas Bates wrote:
> PostgreSQL is extensible.  You can define your own functions using SQL
> or PL/PGSQL or Perl or Python.  You can also define new data types.
And indeed, the REmbeddedPostgres package allows one to use R
functions directly in Postgres.  While we have also done this in MySQL
it is not in any way supported by MySQL and requires some
modifications to the system. For Postgres, this is a straightforward
extension that follows the Postgres documentation!  Similarly, we may
well be able to use data types in R as elements in Postgres tables in
the future.

Personally, I think that as this sort of embedding within
sophisticated and well-tuned database management systems becomes more
secure and stable, we will be able to handle significantly larger
datasets with R code.

The reason that I bring this to the attention of the r-help now is
that if one thinks this facility is promising, one doesn't want to
commit to MySQL now and then convert subsequently.

 Duncan

-- 
_______________________________________________________________

Duncan Temple Lang                duncan at research.bell-labs.com
Bell Labs, Lucent Technologies    office: (908)582-3217
700 Mountain Avenue, Room 2C-259  fax:    (908)582-3340
Murray Hill, NJ  07974-2070       
         http://cm.bell-labs.com/stat/duncan
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Dear R-users,

I am very much looking forward to the day when R will be able to handle very
large datasets. I however thought that getting there would be rather
difficult as most of the models are written assuming random access to the
data.

This makes me wonder how an embedded database mentioned by Duncan Temple
Lang (or R imbedded in the database) could help?

A related question. Prof. Ripley mentioned few times that one of his
approaches is to generate random samples from databases, which are then
analyzed in R (one doesn't need any sort of embedding for this). Are there
other techniques?

Thanks, Vadim

-----Original Message-----
From: Duncan Temple Lang [mailto:duncan at research.bell-labs.com]
Sent: Friday, November 23, 2001 3:19 PM
To: Douglas Bates
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] SQL implementations (was: Are you experienced in SAS
and R ...)


...

Personally, I think that as this sort of embedding within
sophisticated and well-tuned database management systems becomes more
secure and stable, we will be able to handle significantly larger
datasets with R code.

...

 Duncan

-- 
_______________________________________________________________

Duncan Temple Lang                duncan at research.bell-labs.com
Bell Labs, Lucent Technologies    office: (908)582-3217
700 Mountain Avenue, Room 2C-259  fax:    (908)582-3340
Murray Hill, NJ  07974-2070       
         http://cm.bell-labs.com/stat/duncan
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.
-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.
_._

-------------------------------------------------- 
DISCLAIMER 
This e-mail, and any attachments thereto, is intended only for use by the
addressee(s) named herein and may contain legally privileged and/or
confidential information.  If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or
copying of this e-mail, and any attachments thereto, is strictly prohibited.
If you have received this e-mail in error, please immediately notify me and
permanently delete the original and any copy of any e-mail and any printout
thereof. 

E-mail transmission cannot be guaranteed to be secure or error-free.  The
sender therefore does not accept liability for any errors or omissions in
the contents of this message which arise as a result of e-mail transmission.

NOTICE REGARDING PRIVACY AND CONFIDENTIALITY 

Knight Trading Group may, at its discretion, monitor and review the content
of all e-mail communications. 

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

On Fri, 23 Nov 2001, Dirk Eddelbuettel wrote:
>
>   "Vadim" == Vadim Ogranovich <vograno at arbitrade.com>
writes:
>   Vadim> A related question. Prof. Ripley mentioned few times that one
of
his
>   Vadim> approaches is to generate random samples from databases, which
are
>   Vadim> then analyzed in R (one doesn't need any sort of embedding
for
>   Vadim> this). Are there other techniques?
>
> You might be overlooking the fact that random subsampling from timeseries
can
> severely distory the dynamic structure of the series.
This is yet another reason I don't see the advantage of databases over
multi-pass algorithms. If you read a time series as a stream you can easily
stratify it, like Prof. Ripley mentioned in his other posting, and convert
it into another STREAM that looks like a DATASET of possibly large number of
columns. You can then feed this stream, which takes const memory, into the
multi-pass algorithm to model the temporal structure.
I don't know how one can practically do it with a database (certainly not
using SQL).

I believe I am missing the point behind the use of databases for modelling
very large data sets and would be grateful for any insight.

Thanks, Vadim

-------------------------------------------------- 
DISCLAIMER 
This e-mail, and any attachments thereto, is intended only for use by the
addressee(s) named herein and may contain legally privileged and/or
confidential information.  If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or
copying of this e-mail, and any attachments thereto, is strictly prohibited.
If you have received this e-mail in error, please immediately notify me and
permanently delete the original and any copy of any e-mail and any printout
thereof. 

E-mail transmission cannot be guaranteed to be secure or error-free.  The
sender therefore does not accept liability for any errors or omissions in
the contents of this message which arise as a result of e-mail transmission.

NOTICE REGARDING PRIVACY AND CONFIDENTIALITY 

Knight Trading Group may, at its discretion, monitor and review the content
of all e-mail communications. 

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._