R devel - Aug 2011 - getNativeSymbolInfo("user_unif_rand") returns different results on windows and linux

Hi,

When loading a package that provides the user-supplied RNG hook 
user_unif_rand, calling getNativeSymbolInfo("user_unif_rand") returns 
informations about the loaded symbol.
I am using this to identify which package currently provides the RNG hook.
The results are the same on windows and linux if only one library 
provides the hook.

If one loads a second package that provides this hook, then on linux 
(Ubuntu 10.10), calling again getNativeSymbolInfo("user_unif_rand") 
returns the latest symbol information (which I presume is the correct 
result).
On windows (XP and Vista) however the symbol information does not change 
(same pointer address) and the package data is NULL.
See results for both systems below. I tested this with R 2.12.1, 2.13.0 
and 2.13.1 on Windows.

Is this a normal behaviour for dll loaded on Windows?
Thank you.

Renaud


#####################
# LINUX
#####################
 > library(rlecuyer)
 > getNativeSymbolInfo("user_unif_rand")
$name
[1] "user_unif_rand"

$address
<pointer: 0x7ff55acffed0>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rlecuyer
Filename: 
/home/renaud/R/x86_64-pc-linux-gnu-library/2.12/rlecuyer/libs/rlecuyer.so
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
 > library(rstream)
 > getNativeSymbolInfo("user_unif_rand")
$name
[1] "user_unif_rand"

$address
<pointer: 0x7ff55aaf58c0>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rstream
Filename: 
/home/renaud/R/x86_64-pc-linux-gnu-library/2.12/rstream/libs/rstream.so
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"

#####################
# WINDOWS
#####################
 > library(rlecuyer)
 > getNativeSymbolInfo("user_unif_rand")
$name
[1] "user_unif_rand"

$address
<pointer: 0x6bb84fb8>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rlecuyer
Filename: C:/Program
         Files/R/R-2.13.1/library/rlecuyer/libs/i386/rlecuyer.dll
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
 > library(rstream)
 > getNativeSymbolInfo("user_unif_rand")
$name
[1] "user_unif_rand"

$address
<pointer: 0x6bb84fb8>
attr(,"class")
[1] "NativeSymbol"

$package
NULL

attr(,"class")
[1] "NativeSymbolInfo"






 

###
UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}

Hi Renaud

  I cannot presently step through the code on Windows to verify the cause of the
problem,
but looking at the code, I would _presume_ the reason is that symbol lookup is
cached on
Windows but not on Linux or OS X (at least by default).
Thus when we perform the second search for user_unif_rand, we find it in the
cache
rather than searching all the DLLs.

This happens because we did not specify a value for the PACKAGE parameter.

When we load a new DLL, the cache should probably be cleared or we
intelligently update it as necessary for the new DLLs on demand, i.e.
for a new search for a symbol we look in the new DLLs and then use the cache.
(This involves some book-keeping that could become convoluted.)

If we had specified a value for PACKAGE, e.g.

  getNativeSymbolInfo("user_unif_rand", "rstream")

we would get the version in that package's DLL.

So this gives a workaround that you can use with just R code

  findNativeSymbolInfo   function(sym,  dlls = rev(getLoadedDLLs()))  {

      for(d in dlls) {
         z = getNativeSymbolInfo(sym, d)
         if(!is.null(z))
          return(z)
      }

    NULL
  }

We'll think about whether to change the behaviour on Windows and how to do
it
without affecting performance excessively.

 Best,
   D.


On 8/17/11 9:12 AM, Renaud Gaujoux wrote:
> Hi,
> 
> When loading a package that provides the user-supplied RNG hook
user_unif_rand, calling
> getNativeSymbolInfo("user_unif_rand") returns informations about
the loaded symbol.
> I am using this to identify which package currently provides the RNG hook.
> The results are the same on windows and linux if only one library provides
the hook.
> 
> If one loads a second package that provides this hook, then on linux
(Ubuntu 10.10), calling again
> getNativeSymbolInfo("user_unif_rand") returns the latest symbol
information (which I presume is the correct result).
> On windows (XP and Vista) however the symbol information does not change
(same pointer address) and the package data is
> NULL.
> See results for both systems below. I tested this with R 2.12.1, 2.13.0 and
2.13.1 on Windows.
> 
> Is this a normal behaviour for dll loaded on Windows?
> Thank you.
> 
> Renaud
> 
> 
> #####################
> # LINUX
> #####################
>> library(rlecuyer)
>> getNativeSymbolInfo("user_unif_rand")
> $name
> [1] "user_unif_rand"
> 
> $address
> <pointer: 0x7ff55acffed0>
> attr(,"class")
> [1] "NativeSymbol"
> 
> $package
> DLL name: rlecuyer
> Filename:
/home/renaud/R/x86_64-pc-linux-gnu-library/2.12/rlecuyer/libs/rlecuyer.so
> Dynamic lookup: TRUE
> 
> attr(,"class")
> [1] "NativeSymbolInfo"
>> library(rstream)
>> getNativeSymbolInfo("user_unif_rand")
> $name
> [1] "user_unif_rand"
> 
> $address
> <pointer: 0x7ff55aaf58c0>
> attr(,"class")
> [1] "NativeSymbol"
> 
> $package
> DLL name: rstream
> Filename:
/home/renaud/R/x86_64-pc-linux-gnu-library/2.12/rstream/libs/rstream.so
> Dynamic lookup: TRUE
> 
> attr(,"class")
> [1] "NativeSymbolInfo"
> 
> #####################
> # WINDOWS
> #####################
>> library(rlecuyer)
>> getNativeSymbolInfo("user_unif_rand")
> $name
> [1] "user_unif_rand"
> 
> $address
> <pointer: 0x6bb84fb8>
> attr(,"class")
> [1] "NativeSymbol"
> 
> $package
> DLL name: rlecuyer
> Filename: C:/Program
>         Files/R/R-2.13.1/library/rlecuyer/libs/i386/rlecuyer.dll
> Dynamic lookup: TRUE
> 
> attr(,"class")
> [1] "NativeSymbolInfo"
>> library(rstream)
>> getNativeSymbolInfo("user_unif_rand")
> $name
> [1] "user_unif_rand"
> 
> $address
> <pointer: 0x6bb84fb8>
> attr(,"class")
> [1] "NativeSymbol"
> 
> $package
> NULL
> 
> attr(,"class")
> [1] "NativeSymbolInfo"
> 
> 
> 
> 
> 
> 
> 
> 
> ###
> UNIVERSITY OF CAPE TOWN
> This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Hi,

sorry to bump this post but I did not get any reply on this puzzling 
issue, which looks important though.

While investigating the issue it came out that the value returned by 
getNativeSymbolInfo('user_unif_rand') (on Windows XP) seems to depend on
whether it has already been called on the same symbol (see test code 
below. Each sequence needs to be run on on a fresh R session).

Things works perfectly under Linux.

Thank you for any explanation on the matter.

Renaud

########## SEQUENCE 1 ##################
library(rstream)
getNativeSymbolInfo('user_unif_rand')
# this returns complete info pointing to rstream's hook
#### RESULT
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rstream
Filename: C:/Program
         Files/R/R-2.13.1/library/rstream/libs/i386/rstream.dll
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
####

# call again getNativeSymbolInfo
getNativeSymbolInfo('user_unif_rand')
# this returns INcomplete info pointing to rstream's hook
### RESULT
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
NULL

attr(,"class")
[1] "NativeSymbolInfo"
############################


########## SEQUENCE 2 ##################
library(rstream)
library(rlecuyer)
getNativeSymbolInfo('user_unif_rand')
# this returns complete info pointing to relcuyer's hook
########## RESULT ##################
$name
[1] "user_unif_rand"

$address
<pointer: 0x6bb84fb8>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rlecuyer
Filename: C:/Program
         Files/R/R-2.13.1/library/rlecuyer/libs/i386/rlecuyer.dll
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
############################

########## SEQUENCE 3 ##################
####### Load library that provides a hook for user_unif_rand
library(rstream)
getNativeSymbolInfo('user_unif_rand')
# this returns complete info pointing to rstream's hook
## RESULT ##
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rstream
Filename: C:/Program
         Files/R/R-2.13.1/library/rstream/libs/i386/rstream.dll
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
##

####### Load other library that provides the hook
library(rlecuyer)
getNativeSymbolInfo('user_unif_rand')
# this returns INcomplete info pointing to rstream's hook
## RESULT ##
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
NULL

attr(,"class")
[1] "NativeSymbolInfo"
############################


 > sessionInfo()
R version 2.13.1 (2011-07-08)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_South Africa.1252  LC_CTYPE=English_South 
Africa.1252
[3] LC_MONETARY=English_South Africa.1252 LC_NUMERIC=C 

[5] LC_TIME=English_South Africa.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rlecuyer_0.3-1 rstream_1.3.1



###

UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}

Hi Martin,

oups... I indeed had not seen this reply (maybe I should drink less water).
Thank you and sorry for the gross bump.

Duncan, thank you for the work around. A cache issue makes sense as 
shown by the sample code in my bump-post.
The cache could be updated when loading a new DLL, by looking up it 
symbols against the cache.

Another issue is illustrated by the following calls:

library(rstream)
getNativeSymbolInfo("user_unif_rand", "rstream")
getNativeSymbolInfo("user_unif_rand", "rstream")

Although one specifies the package here, the results are different (see 
below: the first returns the complete information, the second has a NULL 
element package).

I noticed other issues which makes very difficult to predict which DLL 
will actually be resolved (e.g. when one plays with loading/unloading 
packages that provides the hook), but I guess caching is behind all these.

Renaud

################
 > library(rstream)
 > getNativeSymbolInfo("user_unif_rand", "rstream")
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
DLL name: rstream
Filename: C:/Program
         Files/R/R-2.13.1/library/rstream/libs/i386/rstream.dll
Dynamic lookup: TRUE

attr(,"class")
[1] "NativeSymbolInfo"
 > getNativeSymbolInfo("user_unif_rand", "rstream")
$name
[1] "user_unif_rand"

$address
<pointer: 0x6ee41280>
attr(,"class")
[1] "NativeSymbol"

$package
NULL

attr(,"class")
[1] "NativeSymbolInfo"



On 23/08/2011 17:58, Martin Morgan wrote:
> Hi Renaud -- did you miss this
>
> https://stat.ethz.ch/pipermail/r-devel/2011-August/061812.html
>
> ? Martin
>
> On 08/23/2011 07:40 AM, Renaud Gaujoux wrote:
>> Hi,
>>
>> sorry to bump this post but I did not get any reply on this puzzling
>> issue, which looks important though.
>>
>> While investigating the issue it came out that the value returned by
>> getNativeSymbolInfo('user_unif_rand') (on Windows XP) seems to
depend on
>> whether it has already been called on the same symbol (see test code
>> below. Each sequence needs to be run on on a fresh R session).
>>
>> Things works perfectly under Linux.
>>
>> Thank you for any explanation on the matter.
>>
>> Renaud
>>
>> ########## SEQUENCE 1 ##################
>> library(rstream)
>> getNativeSymbolInfo('user_unif_rand')
>> # this returns complete info pointing to rstream's hook
>> #### RESULT
>> $name
>> [1] "user_unif_rand"
>>
>> $address
>> <pointer: 0x6ee41280>
>> attr(,"class")
>> [1] "NativeSymbol"
>>
>> $package
>> DLL name: rstream
>> Filename: C:/Program
>> Files/R/R-2.13.1/library/rstream/libs/i386/rstream.dll
>> Dynamic lookup: TRUE
>>
>> attr(,"class")
>> [1] "NativeSymbolInfo"
>> ####
>>
>> # call again getNativeSymbolInfo
>> getNativeSymbolInfo('user_unif_rand')
>> # this returns INcomplete info pointing to rstream's hook
>> ### RESULT
>> $name
>> [1] "user_unif_rand"
>>
>> $address
>> <pointer: 0x6ee41280>
>> attr(,"class")
>> [1] "NativeSymbol"
>>
>> $package
>> NULL
>>
>> attr(,"class")
>> [1] "NativeSymbolInfo"
>> ############################
>>
>>
>> ########## SEQUENCE 2 ##################
>> library(rstream)
>> library(rlecuyer)
>> getNativeSymbolInfo('user_unif_rand')
>> # this returns complete info pointing to relcuyer's hook
>> ########## RESULT ##################
>> $name
>> [1] "user_unif_rand"
>>
>> $address
>> <pointer: 0x6bb84fb8>
>> attr(,"class")
>> [1] "NativeSymbol"
>>
>> $package
>> DLL name: rlecuyer
>> Filename: C:/Program
>> Files/R/R-2.13.1/library/rlecuyer/libs/i386/rlecuyer.dll
>> Dynamic lookup: TRUE
>>
>> attr(,"class")
>> [1] "NativeSymbolInfo"
>> ############################
>>
>> ########## SEQUENCE 3 ##################
>> ####### Load library that provides a hook for user_unif_rand
>> library(rstream)
>> getNativeSymbolInfo('user_unif_rand')
>> # this returns complete info pointing to rstream's hook
>> ## RESULT ##
>> $name
>> [1] "user_unif_rand"
>>
>> $address
>> <pointer: 0x6ee41280>
>> attr(,"class")
>> [1] "NativeSymbol"
>>
>> $package
>> DLL name: rstream
>> Filename: C:/Program
>> Files/R/R-2.13.1/library/rstream/libs/i386/rstream.dll
>> Dynamic lookup: TRUE
>>
>> attr(,"class")
>> [1] "NativeSymbolInfo"
>> ##
>>
>> ####### Load other library that provides the hook
>> library(rlecuyer)
>> getNativeSymbolInfo('user_unif_rand')
>> # this returns INcomplete info pointing to rstream's hook
>> ## RESULT ##
>> $name
>> [1] "user_unif_rand"
>>
>> $address
>> <pointer: 0x6ee41280>
>> attr(,"class")
>> [1] "NativeSymbol"
>>
>> $package
>> NULL
>>
>> attr(,"class")
>> [1] "NativeSymbolInfo"
>> ############################
>>
>>
>> > sessionInfo()
>> R version 2.13.1 (2011-07-08)
>> Platform: i386-pc-mingw32/i386 (32-bit)
>>
>> locale:
>> [1] LC_COLLATE=English_South Africa.1252 LC_CTYPE=English_South 
>> Africa.1252
>> [3] LC_MONETARY=English_South Africa.1252 LC_NUMERIC=C
>> [5] LC_TIME=English_South Africa.1252
>>
>> attached base packages:
>> [1] stats graphics grDevices utils datasets methods base
>>
>> other attached packages:
>> [1] rlecuyer_0.3-1 rstream_1.3.1
>>
>>
>>
>> ###
>>
>> UNIVERSITY OF CAPE TOWN
>> This e-mail is subject to the UCT ICT policies and
e-mai...{{dropped:5}}
>>
>> ______________________________________________
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>


###

UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:9}}