| Title: | Out of Memory Vectors |
|---|---|
| Description: | Core functionality for working with vectors (numeric, integer, logical and character) that are too large to keep in memory. The vectors are kept (partially) on disk using memory mapping. This package contains the basic functionality for working with these memory mapped vectors (e.g. creating, indexing, ordering and sorting) and provides C++ headers which can be used by other packages to extend the functionality provided in this package. Compatibility updates by Renova. |
| Authors: | Jan van der Laan [aut, cre] |
| Maintainer: | Jan van der Laan <[email protected]> |
| License: | GPL-3 |
| Version: | 0.2.5.9002 |
| Built: | 2026-11-16 22:19:30 UTC |
| Source: | https://github.com/renova-r/lvec |
Converts a primitive R-vector to lvec
as_lvec(x)as_lvec(x)
x |
the object to convert. This can be a vector of type
|
Returns an lvec of the same type as x. When x is
already and lvec, x is returned. For character vectors the
maximum length of the lvec is set to the maximum length found
in x.
# convert a character vector to lvec x <- as_lvec(letters) lget(x, 1:26)# convert a character vector to lvec x <- as_lvec(letters) lget(x, 1:26)
Convert complete lvec to R vector
as_rvec(x)as_rvec(x)
x |
|
Returns an R vector of type integer, numeric, character or logical.
The ranges have a maximum length.
chunk(x, ...) ## S3 method for class 'lvec' chunk(x, chunk_size = 1e+06, ...) ## Default S3 method: chunk(x, chunk_size = NULL, ...) ## S3 method for class 'data.frame' chunk(x, chunk_size = NULL, ...)chunk(x, ...) ## S3 method for class 'lvec' chunk(x, chunk_size = 1e+06, ...) ## Default S3 method: chunk(x, chunk_size = NULL, ...) ## S3 method for class 'data.frame' chunk(x, chunk_size = NULL, ...)
x |
an object for which the index ranges should be calculated. Should
support the |
... |
ignored; used to pass additional arguments to other methods. |
chunk_size |
a numeric vector of length 1 giving the maximum length of the chunks. |
The default chunk size can be changes by setting the option 'chunk_size', ('options(chunk_size = <new default chunk size>)').
Implementations of chunk for data frames and regular vectors are provided to make it easier to write code that works on both lvec objects and regular R objects.
Clone an lvec object
clone(x, ...) ## S3 method for class 'lvec' clone(x, ...)clone(x, ...) ## S3 method for class 'lvec' clone(x, ...)
x |
|
... |
ignored; used to pass additional arguments to other methods |
lvec objects are basically pointers to pieces of memory.
When copying an object only the pointer is copied and when modifying the copied
object also the original object is modified. The advantage of this is speed:
these is less copying of the complete vector. In order to obtain a true copy
of an lvec code can be used.
a <- as_lvec(1:3) # Copy b <- a # When modifying the copy also the original is modified lset(b, 1, 10) print(a) print(b) # Use clone to make a true copy b <- clone(a) lset(b, 1, 100) print(a) print(b)a <- as_lvec(1:3) # Copy b <- a # When modifying the copy also the original is modified lset(b, 1, 10) print(a) print(b) # Use clone to make a true copy b <- clone(a) lset(b, 1, 100) print(a) print(b)
Check if an object is of type lvec
is_lvec(x)is_lvec(x)
x |
the object to check |
Returns TRUE is the object is of type lvec and
FALSE otherwise.
Get and set the length of an lvec
## S3 method for class 'lvec' length(x) ## S3 replacement method for class 'lvec' length(x) <- value## S3 method for class 'lvec' length(x) ## S3 replacement method for class 'lvec' length(x) <- value
x |
the |
value |
the new length of the |
The length of the lvec.
Read elements from an lvec
lget(x, ...) ## S3 method for class 'lvec' lget(x, index = NULL, range = NULL, ...) ## Default S3 method: lget(x, index = NULL, range = NULL, ...) ## S3 method for class 'data.frame' lget(x, index = NULL, range = NULL, ...)lget(x, ...) ## S3 method for class 'lvec' lget(x, index = NULL, range = NULL, ...) ## Default S3 method: lget(x, index = NULL, range = NULL, ...) ## S3 method for class 'data.frame' lget(x, index = NULL, range = NULL, ...)
x |
the |
... |
used to pass on additional arguments to other methods. |
index |
a logical or numeric vector to index |
range |
a numeric vector of length 2 specifying a range of elements
to select. Specify either |
Indexing using index should follow the same rules as indexing a regular
R-vector using a logical or numeric index. The range given by range
includes both end elements. So, a range of c(1,3) selects the first
three elements.
Returns an lvec with the selected elements. In order to convert
the selection to an R-vector as_rvec can e used.
a <- as_lvec(letters[1:4]) # Select first two elements lget(a, 1:2) lget(a, c(TRUE, TRUE, FALSE, FALSE)) lget(a, range = c(1,2)) # Logical indices are recycled: select odd elements lget(a, c(TRUE, FALSE))a <- as_lvec(letters[1:4]) # Select first two elements lget(a, 1:2) lget(a, c(TRUE, TRUE, FALSE, FALSE)) lget(a, range = c(1,2)) # Logical indices are recycled: select odd elements lget(a, c(TRUE, FALSE))
Read and write lvec object to file
lsave(x, filename, overwrite = TRUE, compress = FALSE) lload(filename)lsave(x, filename, overwrite = TRUE, compress = FALSE) lload(filename)
x |
|
filename |
name of the file(s) to save the lvec to. See details. |
overwrite |
overwrite existing files or abort when files would be overwritten. |
compress |
a logical specifying if the data should be compressed.
(see |
The lvec is written in chunks to a number of RDS files using
saveRDS. When filename contains the extension 'RDS'
(capitalisation may differ), this extension is stripped from the filename.
After that the lvec is written in blocks (or chunks) to files having
names <filename>.00001.RDS, <filename>.00002.RDS etc. Some
additional data (data type, the number of blocks, the size of the lvec, etc)
is written to the file <filename>.RDS.
The size of the chunks can be controlled by the option 'chunk_size' (see
chunk).
lsave does not return anything. lload returns an
lvec.
Set values in an lvec
lset(x, ...) ## S3 method for class 'lvec' lset(x, index = NULL, values, range = NULL, ...) ## Default S3 method: lset(x, index = NULL, values, range = NULL, ...) ## S3 method for class 'data.frame' lset(x, index = NULL, values, range = NULL, ...)lset(x, ...) ## S3 method for class 'lvec' lset(x, index = NULL, values, range = NULL, ...) ## Default S3 method: lset(x, index = NULL, values, range = NULL, ...) ## S3 method for class 'data.frame' lset(x, index = NULL, values, range = NULL, ...)
x |
|
... |
used to pass additional arguments to other methods |
index |
a numeric or logical vector with indices at which the values should be set. |
values |
a vector with the new values. When shorter than the length of the indices the values are recycled. |
range |
a numeric vector of length 2 specifying a range of elements
to select. Specify either |
Should behave in the same way as assigning and indexing to a regular
R-vector. The range given by range includes both end elements. So, a
range of c(1,3) selects the first three elements.
When range is given, and values is not given it is assumed
index contains the values. Therefore, one can do
lset(x, range = c(1,4), NA), to set the first four elements of
x to missing.
a <- as_lvec(1:10) # set second element to 20 lset(a, 2, 20) print(a) # set odd elements to 20 lset(a, c(TRUE, FALSE), 20) print(a) # values are recycled lset(a, 1:4, 100:101) print(a) # range index; set first 3 elements to NA lset(a, range = c(1,3), NA) print(a)a <- as_lvec(1:10) # set second element to 20 lset(a, 2, 20) print(a) # set odd elements to 20 lset(a, c(TRUE, FALSE), 20) print(a) # values are recycled lset(a, 1:4, 100:101) print(a) # range index; set first 3 elements to NA lset(a, range = c(1,3), NA) print(a)
The data in these vectors are stored on disk (partially buffered for speed) allowing one to work with more data than fits into available memory.
lvec( size, type = c("numeric", "integer", "logical", "character"), strlen = NULL )lvec( size, type = c("numeric", "integer", "logical", "character"), strlen = NULL )
size |
the size of the vector |
type |
the type of the vector. Should be one of the following value:
|
strlen |
in case of a vector of type |
The minimum value of strlen is two. When a value smaller than that is
given it is automatically set to two. This is because a minimum of two bytes
is necessary to also store missing values correctly.
Returns an object of type lvec. Elements of this vector are stored on
file (partially buffered in memory for speed) allowing one to work with more
data than fits into memory.
# create an integer vector of length 100 x <- lvec(100, type = "integer") # Get the first 10 values; values are initialised to 0 by default lget(x, 1:10) # Set the first 10 values to 11:20 lset(x, 1:10, 11:20) # set maximum length of the string to 1, strings longer than that get # truncated. However, minimum value of strlen is 2. x <- lvec(10, type = "character", strlen = 1) lset(x, 1:3, c("a", "foo", NA)) lget(x, 1:3)# create an integer vector of length 100 x <- lvec(100, type = "integer") # Get the first 10 values; values are initialised to 0 by default lget(x, 1:10) # Set the first 10 values to 11:20 lset(x, 1:10, 11:20) # set maximum length of the string to 1, strings longer than that get # truncated. However, minimum value of strlen is 2. x <- lvec(10, type = "character", strlen = 1) lset(x, 1:3, c("a", "foo", NA)) lget(x, 1:3)
Get the type of the lvec
lvec_type(x)lvec_type(x)
x |
the |
Returns an character vector of length 1 with one of the following values:
"integer", "numeric", "logical" or "character".
Order a lvec
order(x, ...) ## Default S3 method: order(x, ...) ## S3 method for class 'lvec' order(x, ...)order(x, ...) ## Default S3 method: order(x, ...) ## S3 method for class 'lvec' order(x, ...)
x |
lvec to sort |
... |
unused. |
Returns the order of x. Unlike the default order
function in R, the sort used is not stable (e.g. in case there are multiple
records with the same value in x, there relative order after sorting
is not defined).
x <- as_lvec(rnorm(10)) order(x)x <- as_lvec(rnorm(10)) order(x)
Print an lvec
## S3 method for class 'lvec' print(x, ...)## S3 method for class 'lvec' print(x, ...)
x |
|
... |
unused |
Returns x invisibly.
Set and get attributes of the original R-vector stored in an lvec
rattr(x, which) rattr(x, which) <- valuerattr(x, which) rattr(x, which) <- value
x |
and object of type |
which |
a character vector of length one giving the name of the attribute. |
value |
the new value of the attribute |
The attributes of the lvec can be set and obtained using the
standard functions attr and attributes. However
when an lvec is converted to an R-vector using, for example,
as_rvec, the attributes of the resulting R-vector are set using
the result of rattr. This can be used to store vectors such as
factors and dates (POSIXct) in lvec objects, as these are basically
integer and numeric vectors with a number of additional attributes.
dates <- as_lvec(as.Date("2016-12-05", "2016-12-24")) # When printing and reading the result is converted back to a date object print(dates) as_rvec(dates) # make a factor of an integer lvec a <- as_lvec(1:3) rattr(a, "class") <- "factor" rattr(a, "levels") <- c("a", "b", "c") print(a)dates <- as_lvec(as.Date("2016-12-05", "2016-12-24")) # When printing and reading the result is converted back to a date object print(dates) as_rvec(dates) # make a factor of an integer lvec a <- as_lvec(1:3) rattr(a, "class") <- "factor" rattr(a, "levels") <- c("a", "b", "c") print(a)
Sort a lvec
## S3 method for class 'lvec' sort(x, decreasing = FALSE, clone = TRUE, ...)## S3 method for class 'lvec' sort(x, decreasing = FALSE, clone = TRUE, ...)
x |
lvec to sort |
decreasing |
unused (a value unequal to |
clone |
clone x before sorting |
... |
unused. |
Sorts x and returns a sorted copy of x. When clone is
FALSE the input vector is modified.
x <- as_lvec(rnorm(10)) sort(x) # Effect of clone a <- as_lvec(rnorm(10)) b <- sort(a, clone = FALSE) print(a)x <- as_lvec(rnorm(10)) sort(x) # Effect of clone a <- as_lvec(rnorm(10)) b <- sort(a, clone = FALSE) print(a)
Get and set the maximum string length of a character lvec
strlen(x) strlen(x) <- valuestrlen(x) strlen(x) <- value
x |
a lvec of type character. |
value |
the new value of the maximum string length. |
a <- as_lvec('123') strlen(a) # = 3 # Strings are truncated to strlen lset(a, 1, '123456') print(a) # '123' strlen(a) <- 5 lset(a, 1, '123456') print(a) # '12345'a <- as_lvec('123') strlen(a) # = 3 # Strings are truncated to strlen lset(a, 1, '123456') print(a) # '123' strlen(a) <- 5 lset(a, 1, '123456') print(a) # '12345'