| Title: | R Front End for 'Apache Spark' |
|---|---|
| Description: | Provides an R Front end for 'Apache Spark' <https://spark.apache.org>. Compatibility updates by Renova. |
| Authors: | Shivaram Venkataraman [aut], Xiangrui Meng [aut], Felix Cheung [aut, cre], The Apache Software Foundation [aut, cph] |
| Maintainer: | Felix Cheung <[email protected]> |
| License: | Apache License (== 2.0) |
| Version: | 3.1.2.9002 |
| Built: | 2026-11-19 14:09:06 UTC |
| Source: | https://github.com/renova-r/SparkR |
Equality test that is safe for null values.
x %<=>% value ## S4 method for signature 'Column' x %<=>% valuex %<=>% value ## S4 method for signature 'Column' x %<=>% value
x |
a Column |
value |
a value to compare |
Can be used, unlike standard equality operator, to perform null-safe joins.
Equivalent to Scala Column.<=> and Column.eqNullSafe.
%<=>% since 2.3.0
## Not run: df1 <- createDataFrame(data.frame( x = c(1, NA, 3, NA), y = c(2, 6, 3, NA) )) head(select(df1, df1$x == df1$y, df1$x %<=>% df1$y)) df2 <- createDataFrame(data.frame(y = c(3, NA))) count(join(df1, df2, df1$y == df2$y)) count(join(df1, df2, df1$y %<=>% df2$y)) ## End(Not run)## Not run: df1 <- createDataFrame(data.frame( x = c(1, NA, 3, NA), y = c(2, 6, 3, NA) )) head(select(df1, df1$x == df1$y, df1$x %<=>% df1$y)) df2 <- createDataFrame(data.frame(y = c(3, NA))) count(join(df1, df2, df1$y == df2$y)) count(join(df1, df2, df1$y %<=>% df2$y)) ## End(Not run)
Match a column with given values.
## S4 method for signature 'Column' x %in% table## S4 method for signature 'Column' x %in% table
x |
a Column. |
table |
a collection of values (coercible to list) to compare with. |
A matched values as a result of comparing with given values.
%in% since 1.5.0
## Not run: filter(df, "age in (10, 30)") where(df, df$age %in% c(10, 30)) ## End(Not run)## Not run: filter(df, "age in (10, 30)") where(df, df$age %in% c(10, 30)) ## End(Not run)
S4 class that represents a AFTSurvivalRegressionModel
jobj |
a Java object reference to the backing Scala AFTSurvivalRegressionWrapper |
AFTSurvivalRegressionModel since 2.0.0
Aggregates on the entire SparkDataFrame without groups. The resulting SparkDataFrame will also contain the grouping columns.
Compute aggregates by specifying a list of columns
agg(x, ...) summarize(x, ...) ## S4 method for signature 'GroupedData' agg(x, ...) ## S4 method for signature 'GroupedData' summarize(x, ...) ## S4 method for signature 'SparkDataFrame' agg(x, ...) ## S4 method for signature 'SparkDataFrame' summarize(x, ...)agg(x, ...) summarize(x, ...) ## S4 method for signature 'GroupedData' agg(x, ...) ## S4 method for signature 'GroupedData' summarize(x, ...) ## S4 method for signature 'SparkDataFrame' agg(x, ...) ## S4 method for signature 'SparkDataFrame' summarize(x, ...)
x |
a SparkDataFrame or GroupedData. |
... |
further arguments to be passed to or from other methods. |
df2 <- agg(df, <column> = <aggFunction>) df2 <- agg(df, newColName = aggFunction(column))
A SparkDataFrame.
agg since 1.4.0
summarize since 1.4.0
agg since 1.4.0
summarize since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df2 <- agg(df, age = "sum") # new column name will be created as 'SUM(age#0)' df3 <- agg(df, ageSum = sum(df$age)) # Creates a new column named ageSum df4 <- summarize(df, ageSum = max(df$age)) ## End(Not run)## Not run: df2 <- agg(df, age = "sum") # new column name will be created as 'SUM(age#0)' df3 <- agg(df, ageSum = sum(df$age)) # Creates a new column named ageSum df4 <- summarize(df, ageSum = max(df$age)) ## End(Not run)
Returns a new SparkDataFrame or a Column with an alias set. Equivalent to SQL "AS" keyword.
## S4 method for signature 'Column' alias(object, data) ## S4 method for signature 'SparkDataFrame' alias(object, data)## S4 method for signature 'Column' alias(object, data) ## S4 method for signature 'SparkDataFrame' alias(object, data)
object |
x a SparkDataFrame or a Column |
data |
new name to use |
a SparkDataFrame or a Column
alias(Column) since 1.4.0
alias(SparkDataFrame) since 2.3.0
Other column_func:
between(),
cast(),
endsWith(),
otherwise(),
over(),
startsWith(),
substr()
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(iris) head(select( df, alias(df$Sepal_Length, "slength"), alias(df$Petal_Length, "plength") )) ## End(Not run) ## Not run: df <- alias(createDataFrame(mtcars), "mtcars") avg_mpg <- alias(agg(groupBy(df, df$cyl), avg(df$mpg)), "avg_mpg") head(select(df, column("mtcars.mpg"))) head(join(df, avg_mpg, column("mtcars.cyl") == column("avg_mpg.cyl"))) ## End(Not run)## Not run: df <- createDataFrame(iris) head(select( df, alias(df$Sepal_Length, "slength"), alias(df$Petal_Length, "plength") )) ## End(Not run) ## Not run: df <- alias(createDataFrame(mtcars), "mtcars") avg_mpg <- alias(agg(groupBy(df, df$cyl), avg(df$mpg)), "avg_mpg") head(select(df, column("mtcars.mpg"))) head(join(df, avg_mpg, column("mtcars.cyl") == column("avg_mpg.cyl"))) ## End(Not run)
S4 class that represents an ALSModel
jobj |
a Java object reference to the backing Scala ALSWrapper |
ALSModel since 2.1.0
Calculates the approximate quantiles of numerical columns of a SparkDataFrame. The result of this algorithm has the following deterministic bound: If the SparkDataFrame has N elements and if we request the quantile at probability p up to error err, then the algorithm will return a sample x from the SparkDataFrame so that the *exact* rank of x is close to (p * N). More precisely, floor((p - err) * N) <= rank(x) <= ceil((p + err) * N). This method implements a variation of the Greenwald-Khanna algorithm (with some speed optimizations). The algorithm was first present in [[https://doi.org/10.1145/375663.375670 Space-efficient Online Computation of Quantile Summaries]] by Greenwald and Khanna. Note that NA values will be ignored in numerical columns before calculation. For columns only containing NA values, an empty list is returned.
## S4 method for signature 'SparkDataFrame,character,numeric,numeric' approxQuantile(x, cols, probabilities, relativeError)## S4 method for signature 'SparkDataFrame,character,numeric,numeric' approxQuantile(x, cols, probabilities, relativeError)
x |
A SparkDataFrame. |
cols |
A single column name, or a list of names for multiple columns. |
probabilities |
A list of quantile probabilities. Each number must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum. |
relativeError |
The relative target precision to achieve (>= 0). If set to zero, the exact quantiles are computed, which could be very expensive. Note that values greater than 1 are accepted but give the same result as 1. |
The approximate quantiles at the given probabilities. If the input is a single column name, the output is a list of approximate quantiles in that column; If the input is multiple column names, the output should be a list, and each element in it is a list of numeric values which represents the approximate quantiles in corresponding column.
approxQuantile since 2.0.0
Other stat functions:
corr(),
cov(),
crosstab(),
freqItems(),
sampleBy()
## Not run: df <- read.json("/path/to/file.json") quantiles <- approxQuantile(df, "key", c(0.5, 0.8), 0.0) ## End(Not run)## Not run: df <- read.json("/path/to/file.json") quantiles <- approxQuantile(df, "key", c(0.5, 0.8), 0.0) ## End(Not run)
Sort a SparkDataFrame by the specified column(s).
arrange(x, col, ...) ## S4 method for signature 'SparkDataFrame,Column' arrange(x, col, ..., withinPartitions = FALSE) ## S4 method for signature 'SparkDataFrame,character' arrange(x, col, ..., decreasing = FALSE, withinPartitions = FALSE) ## S4 method for signature 'SparkDataFrame,characterOrColumn' orderBy(x, col, ...)arrange(x, col, ...) ## S4 method for signature 'SparkDataFrame,Column' arrange(x, col, ..., withinPartitions = FALSE) ## S4 method for signature 'SparkDataFrame,character' arrange(x, col, ..., decreasing = FALSE, withinPartitions = FALSE) ## S4 method for signature 'SparkDataFrame,characterOrColumn' orderBy(x, col, ...)
x |
a SparkDataFrame to be sorted. |
col |
a character or Column object indicating the fields to sort on |
... |
additional sorting fields |
withinPartitions |
a logical argument indicating whether to sort only within each partition |
decreasing |
a logical argument indicating sorting order for columns when a character vector is specified for col |
A SparkDataFrame where all elements are sorted.
arrange(SparkDataFrame, Column) since 1.4.0
arrange(SparkDataFrame, character) since 1.4.0
orderBy(SparkDataFrame, characterOrColumn) since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) arrange(df, df$col1) arrange(df, asc(df$col1), desc(abs(df$col2))) arrange(df, "col1", decreasing = TRUE) arrange(df, "col1", "col2", decreasing = c(TRUE, FALSE)) arrange(df, "col1", "col2", withinPartitions = TRUE) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) arrange(df, df$col1) arrange(df, asc(df$col1), desc(abs(df$col2))) arrange(df, "col1", decreasing = TRUE) arrange(df, "col1", "col2", decreasing = c(TRUE, FALSE)) arrange(df, "col1", "col2", withinPartitions = TRUE) ## End(Not run)
This function downloads the contents of a SparkDataFrame into an R's data.frame. Since data.frames are held in memory, ensure that you have enough memory in your system to accommodate the contents.
as.data.frame(x, row.names = NULL, optional = FALSE, ...) ## S4 method for signature 'SparkDataFrame' as.data.frame(x, row.names = NULL, optional = FALSE, ...)as.data.frame(x, row.names = NULL, optional = FALSE, ...) ## S4 method for signature 'SparkDataFrame' as.data.frame(x, row.names = NULL, optional = FALSE, ...)
x |
a SparkDataFrame. |
row.names |
|
optional |
If |
... |
additional arguments to pass to base::as.data.frame. |
A data.frame.
as.data.frame since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: irisDF <- createDataFrame(iris) df <- as.data.frame(irisDF[irisDF$Species == "setosa", ]) ## End(Not run)## Not run: irisDF <- createDataFrame(iris) df <- as.data.frame(irisDF[irisDF$Species == "setosa", ]) ## End(Not run)
A set of operations working with SparkDataFrame columns
asc(x) asc_nulls_first(x) asc_nulls_last(x) contains(x, ...) desc(x) desc_nulls_first(x) desc_nulls_last(x) getField(x, ...) getItem(x, ...) isNaN(x) isNull(x) isNotNull(x) like(x, ...) rlike(x, ...)asc(x) asc_nulls_first(x) asc_nulls_last(x) contains(x, ...) desc(x) desc_nulls_first(x) desc_nulls_last(x) getField(x, ...) getItem(x, ...) isNaN(x) isNull(x) isNotNull(x) like(x, ...) rlike(x, ...)
x |
a Column object. |
... |
additional argument(s). |
The specified SparkDataFrame is attached to the R search path. This means that the SparkDataFrame is searched by R when evaluating a variable, so columns in the SparkDataFrame can be accessed by simply giving their names.
## S4 method for signature 'SparkDataFrame' attach( what, pos = 2L, name = paste(deparse(substitute(what), backtick = FALSE), collapse = " "), warn.conflicts = TRUE )## S4 method for signature 'SparkDataFrame' attach( what, pos = 2L, name = paste(deparse(substitute(what), backtick = FALSE), collapse = " "), warn.conflicts = TRUE )
what |
(SparkDataFrame) The SparkDataFrame to attach |
pos |
(integer) Specify position in search() where to attach. |
name |
(character) Name to use for the attached SparkDataFrame. Names starting with package: are reserved for library. |
warn.conflicts |
(logical) If TRUE, warnings are printed about conflicts from attaching the database, unless that SparkDataFrame contains an object |
attach since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: attach(irisDf) summary(Sepal_Width) ## End(Not run)## Not run: attach(irisDf) summary(Sepal_Width) ## End(Not run)
Aggregate function: returns the average of the values in a group.
avg(x, ...) ## S4 method for signature 'Column' avg(x)avg(x, ...) ## S4 method for signature 'Column' avg(x)
x |
Column to compute on or a GroupedData object. |
... |
additional argument(s) when |
avg since 1.4.0
Other aggregate functions:
column_aggregate_functions,
corr(),
count(),
cov(),
first(),
last()
## Not run: avg(df$c)## Not run: avg(df$c)
Waits for the termination of the query, either by stopQuery or by an error.
awaitTermination(x, timeout = NULL) ## S4 method for signature 'StreamingQuery' awaitTermination(x, timeout = NULL)awaitTermination(x, timeout = NULL) ## S4 method for signature 'StreamingQuery' awaitTermination(x, timeout = NULL)
x |
a StreamingQuery. |
timeout |
time to wait in milliseconds, if omitted, wait indefinitely until |
If the query has terminated, then all subsequent calls to this method will return TRUE immediately.
TRUE if query has terminated within the timeout period; nothing if timeout is not specified.
awaitTermination(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
explain(),
isActive(),
lastProgress(),
queryName(),
status(),
stopQuery()
## Not run: awaitTermination(sq, 10000)## Not run: awaitTermination(sq, 10000)
Test if the column is between the lower bound and upper bound, inclusive.
between(x, bounds) ## S4 method for signature 'Column' between(x, bounds)between(x, bounds) ## S4 method for signature 'Column' between(x, bounds)
x |
a Column |
bounds |
lower and upper bounds |
between since 1.5.0
Other column_func:
alias(),
cast(),
endsWith(),
otherwise(),
over(),
startsWith(),
substr()
S4 class that represents a BisectingKMeansModel
jobj |
a Java object reference to the backing Scala BisectingKMeansModel |
BisectingKMeansModel since 2.2.0
Return a new SparkDataFrame marked as small enough for use in broadcast joins.
broadcast(x) ## S4 method for signature 'SparkDataFrame' broadcast(x)broadcast(x) ## S4 method for signature 'SparkDataFrame' broadcast(x)
x |
a SparkDataFrame. |
Equivalent to hint(x, "broadcast").
a SparkDataFrame.
broadcast since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(mtcars) avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg") head(join(df, broadcast(avg_mpg), df$cyl == avg_mpg$cyl)) ## End(Not run)## Not run: df <- createDataFrame(mtcars) avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg") head(join(df, broadcast(avg_mpg), df$cyl == avg_mpg$cyl)) ## End(Not run)
Persist with the default storage level (MEMORY_ONLY).
cache(x) ## S4 method for signature 'SparkDataFrame' cache(x)cache(x) ## S4 method for signature 'SparkDataFrame' cache(x)
x |
A SparkDataFrame |
cache since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) cache(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) cache(df) ## End(Not run)
Caches the specified table in-memory.
cacheTable(tableName)cacheTable(tableName)
tableName |
the qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database. |
SparkDataFrame
cacheTable since 1.4.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") cacheTable("table") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") cacheTable("table") ## End(Not run)
Cancel active jobs for the specified group
cancelJobGroup(groupId)cancelJobGroup(groupId)
groupId |
the ID of job group to be cancelled |
cancelJobGroup since 1.5.0
## Not run: sparkR.session() cancelJobGroup("myJobGroup") ## End(Not run)## Not run: sparkR.session() cancelJobGroup("myJobGroup") ## End(Not run)
Casts the column to a different data type.
cast(x, dataType) ## S4 method for signature 'Column' cast(x, dataType)cast(x, dataType) ## S4 method for signature 'Column' cast(x, dataType)
x |
a Column. |
dataType |
a character object describing the target data type. See Spark Data Types for available data types. |
cast since 1.4.0
Other column_func:
alias(),
between(),
endsWith(),
otherwise(),
over(),
startsWith(),
substr()
## Not run: cast(df$age, "string") ## End(Not run)## Not run: cast(df$age, "string") ## End(Not run)
Returns a checkpointed version of this SparkDataFrame. Checkpointing can be used to truncate the
logical plan, which is especially useful in iterative algorithms where the plan may grow
exponentially. It will be saved to files inside the checkpoint directory set with
setCheckpointDir
checkpoint(x, eager = TRUE) ## S4 method for signature 'SparkDataFrame' checkpoint(x, eager = TRUE)checkpoint(x, eager = TRUE) ## S4 method for signature 'SparkDataFrame' checkpoint(x, eager = TRUE)
x |
A SparkDataFrame |
eager |
whether to checkpoint this SparkDataFrame immediately |
a new checkpointed SparkDataFrame
checkpoint since 2.2.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: setCheckpointDir("/checkpoint") df <- checkpoint(df) ## End(Not run)## Not run: setCheckpointDir("/checkpoint") df <- checkpoint(df) ## End(Not run)
Removes all cached tables from the in-memory cache.
clearCache()clearCache()
clearCache since 1.4.0
## Not run: clearCache() ## End(Not run)## Not run: clearCache() ## End(Not run)
Clear current job group ID and its description
clearJobGroup()clearJobGroup()
clearJobGroup since 1.5.0
## Not run: sparkR.session() clearJobGroup() ## End(Not run)## Not run: sparkR.session() clearJobGroup() ## End(Not run)
Returns a new SparkDataFrame that has exactly numPartitions partitions.
This operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100
partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of
the current partitions. If a larger number of partitions is requested, it will stay at the
current number of partitions.
coalesce(x, ...) ## S4 method for signature 'SparkDataFrame' coalesce(x, numPartitions)coalesce(x, ...) ## S4 method for signature 'SparkDataFrame' coalesce(x, numPartitions)
x |
a SparkDataFrame. |
... |
additional argument(s). |
numPartitions |
the number of partitions to use. |
However, if you're doing a drastic coalesce on a SparkDataFrame, e.g. to numPartitions = 1,
this may result in your computation taking place on fewer nodes than
you like (e.g. one node in the case of numPartitions = 1). To avoid this,
call repartition. This will add a shuffle step, but means the
current upstream partitions will be executed in parallel (per whatever
the current partitioning is).
coalesce(SparkDataFrame) since 2.1.1
repartition, repartitionByRange
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- coalesce(df, 1L) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- coalesce(df, 1L) ## End(Not run)
Collects all the elements of a SparkDataFrame and coerces them into an R data.frame.
collect(x, ...) ## S4 method for signature 'SparkDataFrame' collect(x, stringsAsFactors = FALSE)collect(x, ...) ## S4 method for signature 'SparkDataFrame' collect(x, stringsAsFactors = FALSE)
x |
a SparkDataFrame. |
... |
further arguments to be passed to or from other methods. |
stringsAsFactors |
(Optional) a logical indicating whether or not string columns should be converted to factors. FALSE by default. |
collect since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) collected <- collect(df) class(collected) firstName <- names(collected)[1] ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) collected <- collect(df) class(collected) firstName <- names(collected)[1] ## End(Not run)
Return a vector of column names.
colnames(x, do.NULL = TRUE, prefix = "col") colnames(x) <- value columns(x) ## S4 method for signature 'SparkDataFrame' columns(x) ## S4 method for signature 'SparkDataFrame' names(x) ## S4 replacement method for signature 'SparkDataFrame' names(x) <- value ## S4 method for signature 'SparkDataFrame' colnames(x) ## S4 replacement method for signature 'SparkDataFrame' colnames(x) <- valuecolnames(x, do.NULL = TRUE, prefix = "col") colnames(x) <- value columns(x) ## S4 method for signature 'SparkDataFrame' columns(x) ## S4 method for signature 'SparkDataFrame' names(x) ## S4 replacement method for signature 'SparkDataFrame' names(x) <- value ## S4 method for signature 'SparkDataFrame' colnames(x) ## S4 replacement method for signature 'SparkDataFrame' colnames(x) <- value
x |
a SparkDataFrame. |
do.NULL |
currently not used. |
prefix |
currently not used. |
value |
a character vector. Must have the same length as the number of columns to be renamed. |
columns since 1.4.0
names since 1.5.0
names<- since 1.5.0
colnames since 1.6.0
colnames<- since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) columns(df) colnames(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) columns(df) colnames(df) ## End(Not run)
Get column types of a SparkDataFrame
Set the column types of a SparkDataFrame.
coltypes(x) coltypes(x) <- value ## S4 method for signature 'SparkDataFrame' coltypes(x) ## S4 replacement method for signature 'SparkDataFrame,character' coltypes(x) <- valuecoltypes(x) coltypes(x) <- value ## S4 method for signature 'SparkDataFrame' coltypes(x) ## S4 replacement method for signature 'SparkDataFrame,character' coltypes(x) <- value
x |
A SparkDataFrame |
value |
A character vector with the target column types for the given SparkDataFrame. Column types can be one of integer, numeric/double, character, logical, or NA to keep that column as-is. |
value A character vector with the column types of the given SparkDataFrame
coltypes since 1.6.0
coltypes<- since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: irisDF <- createDataFrame(iris) coltypes(irisDF) # get column types ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) coltypes(df) <- c("character", "integer") # set column types coltypes(df) <- c(NA, "numeric") # set column types ## End(Not run)## Not run: irisDF <- createDataFrame(iris) coltypes(irisDF) # get column types ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) coltypes(df) <- c("character", "integer") # set column types coltypes(df) <- c(NA, "numeric") # set column types ## End(Not run)
The column class supports unary, binary operations on SparkDataFrame columns
Returns a Column based on the given column name.
column(x) ## S4 method for signature 'jobj' column(x) ## S4 method for signature 'character' column(x)column(x) ## S4 method for signature 'jobj' column(x) ## S4 method for signature 'character' column(x)
x |
Character column name. |
jcreference to JVM SparkDataFrame column
Column since 1.4.0
column since 1.6.0
Other non-aggregate functions:
column_nonaggregate_functions,
not()
## Not run: column("name")## Not run: column("name")
Aggregate functions defined for Column.
approx_count_distinct(x, ...) approxCountDistinct(x, ...) collect_list(x) collect_set(x) countDistinct(x, ...) grouping_bit(x) grouping_id(x, ...) kurtosis(x) n_distinct(x, ...) percentile_approx(x, percentage, ...) sd(x, na.rm = FALSE) skewness(x) stddev(x) stddev_pop(x) stddev_samp(x) sumDistinct(x) var(x, y = NULL, na.rm = FALSE, use) variance(x) var_pop(x) var_samp(x) ## S4 method for signature 'Column' approx_count_distinct(x, rsd = 0.05) ## S4 method for signature 'Column' approxCountDistinct(x, rsd = 0.05) ## S4 method for signature 'Column' kurtosis(x) ## S4 method for signature 'Column' max(x) ## S4 method for signature 'Column' mean(x) ## S4 method for signature 'Column' min(x) ## S4 method for signature 'characterOrColumn,numericOrColumn' percentile_approx(x, percentage, accuracy = 10000) ## S4 method for signature 'Column' sd(x) ## S4 method for signature 'Column' skewness(x) ## S4 method for signature 'Column' stddev(x) ## S4 method for signature 'Column' stddev_pop(x) ## S4 method for signature 'Column' stddev_samp(x) ## S4 method for signature 'Column' sum(x) ## S4 method for signature 'Column' sumDistinct(x) ## S4 method for signature 'Column' var(x) ## S4 method for signature 'Column' variance(x) ## S4 method for signature 'Column' var_pop(x) ## S4 method for signature 'Column' var_samp(x) ## S4 method for signature 'Column' approx_count_distinct(x, rsd = 0.05) ## S4 method for signature 'Column' approxCountDistinct(x, rsd = 0.05) ## S4 method for signature 'Column' countDistinct(x, ...) ## S4 method for signature 'Column' n_distinct(x, ...) ## S4 method for signature 'Column' collect_list(x) ## S4 method for signature 'Column' collect_set(x) ## S4 method for signature 'Column' grouping_bit(x) ## S4 method for signature 'Column' grouping_id(x, ...)approx_count_distinct(x, ...) approxCountDistinct(x, ...) collect_list(x) collect_set(x) countDistinct(x, ...) grouping_bit(x) grouping_id(x, ...) kurtosis(x) n_distinct(x, ...) percentile_approx(x, percentage, ...) sd(x, na.rm = FALSE) skewness(x) stddev(x) stddev_pop(x) stddev_samp(x) sumDistinct(x) var(x, y = NULL, na.rm = FALSE, use) variance(x) var_pop(x) var_samp(x) ## S4 method for signature 'Column' approx_count_distinct(x, rsd = 0.05) ## S4 method for signature 'Column' approxCountDistinct(x, rsd = 0.05) ## S4 method for signature 'Column' kurtosis(x) ## S4 method for signature 'Column' max(x) ## S4 method for signature 'Column' mean(x) ## S4 method for signature 'Column' min(x) ## S4 method for signature 'characterOrColumn,numericOrColumn' percentile_approx(x, percentage, accuracy = 10000) ## S4 method for signature 'Column' sd(x) ## S4 method for signature 'Column' skewness(x) ## S4 method for signature 'Column' stddev(x) ## S4 method for signature 'Column' stddev_pop(x) ## S4 method for signature 'Column' stddev_samp(x) ## S4 method for signature 'Column' sum(x) ## S4 method for signature 'Column' sumDistinct(x) ## S4 method for signature 'Column' var(x) ## S4 method for signature 'Column' variance(x) ## S4 method for signature 'Column' var_pop(x) ## S4 method for signature 'Column' var_samp(x) ## S4 method for signature 'Column' approx_count_distinct(x, rsd = 0.05) ## S4 method for signature 'Column' approxCountDistinct(x, rsd = 0.05) ## S4 method for signature 'Column' countDistinct(x, ...) ## S4 method for signature 'Column' n_distinct(x, ...) ## S4 method for signature 'Column' collect_list(x) ## S4 method for signature 'Column' collect_set(x) ## S4 method for signature 'Column' grouping_bit(x) ## S4 method for signature 'Column' grouping_id(x, ...)
x |
Column to compute on. |
... |
additional argument(s). For example, it could be used to pass additional Columns. |
percentage |
Numeric percentage at which percentile should be computed All values should be between 0 and 1. If length equals to 1 resulting column is of type double, otherwise, array type of double. |
y, na.rm, use
|
currently not used. |
rsd |
maximum relative standard deviation allowed (default = 0.05). |
accuracy |
A positive numeric literal (default: 10000) which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation. |
approx_count_distinct: Returns the approximate number of distinct items in a group.
approxCountDistinct: Returns the approximate number of distinct items in a group.
kurtosis: Returns the kurtosis of the values in a group.
max: Returns the maximum value of the expression in a group.
mean: Returns the average of the values in a group. Alias for avg.
min: Returns the minimum value of the expression in a group.
percentile_approx Returns the approximate percentile of the numeric column
col which is the smallest value in the ordered col values (sorted from least to
greatest) such that no more than percentage of col values is less than the value
or equal to that value.
sd: Alias for stddev_samp.
skewness: Returns the skewness of the values in a group.
stddev: Alias for std_dev.
stddev_pop: Returns the population standard deviation of the expression in a group.
stddev_samp: Returns the unbiased sample standard deviation of the expression in a group.
sum: Returns the sum of all values in the expression.
sumDistinct: Returns the sum of distinct values in the expression.
var: Alias for var_samp.
var_pop: Returns the population variance of the values in a group.
var_samp: Returns the unbiased variance of the values in a group.
countDistinct: Returns the number of distinct items in a group.
n_distinct: Returns the number of distinct items in a group.
collect_list: Creates a list of objects with duplicates.
Note: the function is non-deterministic because the order of collected results depends
on the order of the rows which may be non-deterministic after a shuffle.
collect_set: Creates a list of objects with duplicate elements eliminated.
Note: the function is non-deterministic because the order of collected results depends
on the order of the rows which may be non-deterministic after a shuffle.
grouping_bit: Indicates whether a specified column in a GROUP BY list is aggregated or
not, returns 1 for aggregated or 0 for not aggregated in the result set. Same as GROUPING
in SQL and grouping function in Scala.
grouping_id: Returns the level of grouping.
Equals to
grouping_bit(c1) * 2^(n - 1) + grouping_bit(c2) * 2^(n - 2) + ... + grouping_bit(cn)
.
approx_count_distinct(Column) since 3.0.0
approxCountDistinct(Column) since 1.4.0
kurtosis since 1.6.0
max since 1.5.0
mean since 1.5.0
min since 1.5.0
percentile_approx since 3.1.0
sd since 1.6.0
skewness since 1.6.0
stddev since 1.6.0
stddev_pop since 1.6.0
stddev_samp since 1.6.0
sum since 1.5.0
sumDistinct since 1.4.0
var since 1.6.0
variance since 1.6.0
var_pop since 1.5.0
var_samp since 1.6.0
approx_count_distinct(Column, numeric) since 3.0.0
approxCountDistinct(Column, numeric) since 1.4.0
countDistinct since 1.4.0
n_distinct since 1.4.0
collect_list since 2.3.0
collect_set since 2.3.0
grouping_bit since 2.3.0
grouping_id since 2.3.0
Other aggregate functions:
avg(),
corr(),
count(),
cov(),
first(),
last()
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ## End(Not run) ## Not run: head(select(df, approx_count_distinct(df$gear))) head(select(df, approx_count_distinct(df$gear, 0.02))) head(select(df, countDistinct(df$gear, df$cyl))) head(select(df, n_distinct(df$gear))) head(distinct(select(df, "gear"))) ## End(Not run) ## Not run: head(select(df, mean(df$mpg), sd(df$mpg), skewness(df$mpg), kurtosis(df$mpg))) ## End(Not run) ## Not run: head(select(df, avg(df$mpg), mean(df$mpg), sum(df$mpg), min(df$wt), max(df$qsec))) # metrics by num of cylinders tmp <- agg(groupBy(df, "cyl"), avg(df$mpg), avg(df$hp), avg(df$wt), avg(df$qsec)) head(orderBy(tmp, "cyl")) # car with the max mpg mpg_max <- as.numeric(collect(agg(df, max(df$mpg)))) head(where(df, df$mpg == mpg_max)) ## End(Not run) ## Not run: head(select(df, sd(df$mpg), stddev(df$mpg), stddev_pop(df$wt), stddev_samp(df$qsec))) ## End(Not run) ## Not run: head(select(df, sumDistinct(df$gear))) head(distinct(select(df, "gear"))) ## End(Not run) ## Not run: head(agg(df, var(df$mpg), variance(df$mpg), var_pop(df$mpg), var_samp(df$mpg))) ## End(Not run) ## Not run: df2 = df[df$mpg > 20, ] collect(select(df2, collect_list(df2$gear))) collect(select(df2, collect_set(df2$gear))) ## End(Not run) ## Not run: # With cube agg( cube(df, "cyl", "gear", "am"), mean(df$mpg), grouping_bit(df$cyl), grouping_bit(df$gear), grouping_bit(df$am) ) # With rollup agg( rollup(df, "cyl", "gear", "am"), mean(df$mpg), grouping_bit(df$cyl), grouping_bit(df$gear), grouping_bit(df$am) ) ## End(Not run) ## Not run: # With cube agg( cube(df, "cyl", "gear", "am"), mean(df$mpg), grouping_id(df$cyl, df$gear, df$am) ) # With rollup agg( rollup(df, "cyl", "gear", "am"), mean(df$mpg), grouping_id(df$cyl, df$gear, df$am) ) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ## End(Not run) ## Not run: head(select(df, approx_count_distinct(df$gear))) head(select(df, approx_count_distinct(df$gear, 0.02))) head(select(df, countDistinct(df$gear, df$cyl))) head(select(df, n_distinct(df$gear))) head(distinct(select(df, "gear"))) ## End(Not run) ## Not run: head(select(df, mean(df$mpg), sd(df$mpg), skewness(df$mpg), kurtosis(df$mpg))) ## End(Not run) ## Not run: head(select(df, avg(df$mpg), mean(df$mpg), sum(df$mpg), min(df$wt), max(df$qsec))) # metrics by num of cylinders tmp <- agg(groupBy(df, "cyl"), avg(df$mpg), avg(df$hp), avg(df$wt), avg(df$qsec)) head(orderBy(tmp, "cyl")) # car with the max mpg mpg_max <- as.numeric(collect(agg(df, max(df$mpg)))) head(where(df, df$mpg == mpg_max)) ## End(Not run) ## Not run: head(select(df, sd(df$mpg), stddev(df$mpg), stddev_pop(df$wt), stddev_samp(df$qsec))) ## End(Not run) ## Not run: head(select(df, sumDistinct(df$gear))) head(distinct(select(df, "gear"))) ## End(Not run) ## Not run: head(agg(df, var(df$mpg), variance(df$mpg), var_pop(df$mpg), var_samp(df$mpg))) ## End(Not run) ## Not run: df2 = df[df$mpg > 20, ] collect(select(df2, collect_list(df2$gear))) collect(select(df2, collect_set(df2$gear))) ## End(Not run) ## Not run: # With cube agg( cube(df, "cyl", "gear", "am"), mean(df$mpg), grouping_bit(df$cyl), grouping_bit(df$gear), grouping_bit(df$am) ) # With rollup agg( rollup(df, "cyl", "gear", "am"), mean(df$mpg), grouping_bit(df$cyl), grouping_bit(df$gear), grouping_bit(df$am) ) ## End(Not run) ## Not run: # With cube agg( cube(df, "cyl", "gear", "am"), mean(df$mpg), grouping_id(df$cyl, df$gear, df$am) ) # With rollup agg( rollup(df, "cyl", "gear", "am"), mean(df$mpg), grouping_id(df$cyl, df$gear, df$am) ) ## End(Not run)
Avro processing functions defined for Column.
from_avro(x, ...) to_avro(x, ...) ## S4 method for signature 'characterOrColumn' from_avro(x, jsonFormatSchema, ...) ## S4 method for signature 'characterOrColumn' to_avro(x, jsonFormatSchema = NULL)from_avro(x, ...) to_avro(x, ...) ## S4 method for signature 'characterOrColumn' from_avro(x, jsonFormatSchema, ...) ## S4 method for signature 'characterOrColumn' to_avro(x, jsonFormatSchema = NULL)
x |
Column to compute on. |
... |
additional argument(s) passed as parser options. |
jsonFormatSchema |
character Avro schema in JSON string format |
from_avro Converts a binary column of Avro format into its corresponding catalyst value.
The specified schema must match the read data, otherwise the behavior is undefined:
it may fail or return arbitrary result.
To deserialize the data with a compatible and evolved schema, the expected Avro schema can be
set via the option avroSchema.
to_avro Converts a column into binary of Avro format.
Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".
from_avro since 3.1.0
to_avro since 3.1.0
## Not run: df <- createDataFrame(iris) schema <- paste( c( '{"type": "record", "namespace": "example.avro", "name": "Iris", "fields": [', '{"type": ["double", "null"], "name": "Sepal_Length"},', '{"type": ["double", "null"], "name": "Sepal_Width"},', '{"type": ["double", "null"], "name": "Petal_Length"},', '{"type": ["double", "null"], "name": "Petal_Width"},', '{"type": ["string", "null"], "name": "Species"}]}' ), collapse="\\n" ) df_serialized <- select( df, alias(to_avro(alias(struct(column("*")), "fields")), "payload") ) df_deserialized <- select( df_serialized, from_avro(df_serialized$payload, schema) ) head(df_deserialized) ## End(Not run)## Not run: df <- createDataFrame(iris) schema <- paste( c( '{"type": "record", "namespace": "example.avro", "name": "Iris", "fields": [', '{"type": ["double", "null"], "name": "Sepal_Length"},', '{"type": ["double", "null"], "name": "Sepal_Width"},', '{"type": ["double", "null"], "name": "Petal_Length"},', '{"type": ["double", "null"], "name": "Petal_Width"},', '{"type": ["string", "null"], "name": "Species"}]}' ), collapse="\\n" ) df_serialized <- select( df, alias(to_avro(alias(struct(column("*")), "fields")), "payload") ) df_deserialized <- select( df_serialized, from_avro(df_serialized$payload, schema) ) head(df_deserialized) ## End(Not run)
Collection functions defined for Column.
array_aggregate(x, initialValue, merge, ...) array_contains(x, value) array_distinct(x) array_except(x, y) array_exists(x, f) array_forall(x, f) array_filter(x, f) array_intersect(x, y) array_join(x, delimiter, ...) array_max(x) array_min(x) array_position(x, value) array_remove(x, value) array_repeat(x, count) array_sort(x) array_transform(x, f) arrays_overlap(x, y) array_union(x, y) arrays_zip(x, ...) arrays_zip_with(x, y, f) concat(x, ...) element_at(x, extraction) explode(x) explode_outer(x) flatten(x) from_json(x, schema, ...) from_csv(x, schema, ...) map_concat(x, ...) map_entries(x) map_filter(x, f) map_from_arrays(x, y) map_from_entries(x) map_keys(x) map_values(x) map_zip_with(x, y, f) posexplode(x) posexplode_outer(x) reverse(x) schema_of_csv(x, ...) schema_of_json(x, ...) shuffle(x) size(x) slice(x, start, length) sort_array(x, asc = TRUE) transform_keys(x, f) transform_values(x, f) to_json(x, ...) to_csv(x, ...) ## S4 method for signature 'Column' reverse(x) ## S4 method for signature 'Column' to_json(x, ...) ## S4 method for signature 'Column' to_csv(x, ...) ## S4 method for signature 'Column' concat(x, ...) ## S4 method for signature 'Column,characterOrstructTypeOrColumn' from_json(x, schema, as.json.array = FALSE, ...) ## S4 method for signature 'characterOrColumn' schema_of_json(x, ...) ## S4 method for signature 'Column,characterOrstructTypeOrColumn' from_csv(x, schema, ...) ## S4 method for signature 'characterOrColumn' schema_of_csv(x, ...) ## S4 method for signature 'characterOrColumn,Column,'function'' array_aggregate(x, initialValue, merge, finish = NULL) ## S4 method for signature 'Column' array_contains(x, value) ## S4 method for signature 'Column' array_distinct(x) ## S4 method for signature 'Column,Column' array_except(x, y) ## S4 method for signature 'characterOrColumn,'function'' array_exists(x, f) ## S4 method for signature 'characterOrColumn,'function'' array_filter(x, f) ## S4 method for signature 'characterOrColumn,'function'' array_forall(x, f) ## S4 method for signature 'Column,Column' array_intersect(x, y) ## S4 method for signature 'Column,character' array_join(x, delimiter, nullReplacement = NULL) ## S4 method for signature 'Column' array_max(x) ## S4 method for signature 'Column' array_min(x) ## S4 method for signature 'Column' array_position(x, value) ## S4 method for signature 'Column' array_remove(x, value) ## S4 method for signature 'Column,numericOrColumn' array_repeat(x, count) ## S4 method for signature 'Column' array_sort(x) ## S4 method for signature 'characterOrColumn,'function'' array_transform(x, f) ## S4 method for signature 'Column,Column' arrays_overlap(x, y) ## S4 method for signature 'Column,Column' array_union(x, y) ## S4 method for signature 'Column' arrays_zip(x, ...) ## S4 method for signature 'characterOrColumn,characterOrColumn,'function'' arrays_zip_with(x, y, f) ## S4 method for signature 'Column' shuffle(x) ## S4 method for signature 'Column' flatten(x) ## S4 method for signature 'Column' map_concat(x, ...) ## S4 method for signature 'Column' map_entries(x) ## S4 method for signature 'characterOrColumn,'function'' map_filter(x, f) ## S4 method for signature 'Column,Column' map_from_arrays(x, y) ## S4 method for signature 'Column' map_from_entries(x) ## S4 method for signature 'Column' map_keys(x) ## S4 method for signature 'characterOrColumn,'function'' transform_keys(x, f) ## S4 method for signature 'characterOrColumn,'function'' transform_values(x, f) ## S4 method for signature 'Column' map_values(x) ## S4 method for signature 'characterOrColumn,characterOrColumn,'function'' map_zip_with(x, y, f) ## S4 method for signature 'Column' element_at(x, extraction) ## S4 method for signature 'Column' explode(x) ## S4 method for signature 'Column' size(x) ## S4 method for signature 'Column' slice(x, start, length) ## S4 method for signature 'Column' sort_array(x, asc = TRUE) ## S4 method for signature 'Column' posexplode(x) ## S4 method for signature 'Column' explode_outer(x) ## S4 method for signature 'Column' posexplode_outer(x)array_aggregate(x, initialValue, merge, ...) array_contains(x, value) array_distinct(x) array_except(x, y) array_exists(x, f) array_forall(x, f) array_filter(x, f) array_intersect(x, y) array_join(x, delimiter, ...) array_max(x) array_min(x) array_position(x, value) array_remove(x, value) array_repeat(x, count) array_sort(x) array_transform(x, f) arrays_overlap(x, y) array_union(x, y) arrays_zip(x, ...) arrays_zip_with(x, y, f) concat(x, ...) element_at(x, extraction) explode(x) explode_outer(x) flatten(x) from_json(x, schema, ...) from_csv(x, schema, ...) map_concat(x, ...) map_entries(x) map_filter(x, f) map_from_arrays(x, y) map_from_entries(x) map_keys(x) map_values(x) map_zip_with(x, y, f) posexplode(x) posexplode_outer(x) reverse(x) schema_of_csv(x, ...) schema_of_json(x, ...) shuffle(x) size(x) slice(x, start, length) sort_array(x, asc = TRUE) transform_keys(x, f) transform_values(x, f) to_json(x, ...) to_csv(x, ...) ## S4 method for signature 'Column' reverse(x) ## S4 method for signature 'Column' to_json(x, ...) ## S4 method for signature 'Column' to_csv(x, ...) ## S4 method for signature 'Column' concat(x, ...) ## S4 method for signature 'Column,characterOrstructTypeOrColumn' from_json(x, schema, as.json.array = FALSE, ...) ## S4 method for signature 'characterOrColumn' schema_of_json(x, ...) ## S4 method for signature 'Column,characterOrstructTypeOrColumn' from_csv(x, schema, ...) ## S4 method for signature 'characterOrColumn' schema_of_csv(x, ...) ## S4 method for signature 'characterOrColumn,Column,'function'' array_aggregate(x, initialValue, merge, finish = NULL) ## S4 method for signature 'Column' array_contains(x, value) ## S4 method for signature 'Column' array_distinct(x) ## S4 method for signature 'Column,Column' array_except(x, y) ## S4 method for signature 'characterOrColumn,'function'' array_exists(x, f) ## S4 method for signature 'characterOrColumn,'function'' array_filter(x, f) ## S4 method for signature 'characterOrColumn,'function'' array_forall(x, f) ## S4 method for signature 'Column,Column' array_intersect(x, y) ## S4 method for signature 'Column,character' array_join(x, delimiter, nullReplacement = NULL) ## S4 method for signature 'Column' array_max(x) ## S4 method for signature 'Column' array_min(x) ## S4 method for signature 'Column' array_position(x, value) ## S4 method for signature 'Column' array_remove(x, value) ## S4 method for signature 'Column,numericOrColumn' array_repeat(x, count) ## S4 method for signature 'Column' array_sort(x) ## S4 method for signature 'characterOrColumn,'function'' array_transform(x, f) ## S4 method for signature 'Column,Column' arrays_overlap(x, y) ## S4 method for signature 'Column,Column' array_union(x, y) ## S4 method for signature 'Column' arrays_zip(x, ...) ## S4 method for signature 'characterOrColumn,characterOrColumn,'function'' arrays_zip_with(x, y, f) ## S4 method for signature 'Column' shuffle(x) ## S4 method for signature 'Column' flatten(x) ## S4 method for signature 'Column' map_concat(x, ...) ## S4 method for signature 'Column' map_entries(x) ## S4 method for signature 'characterOrColumn,'function'' map_filter(x, f) ## S4 method for signature 'Column,Column' map_from_arrays(x, y) ## S4 method for signature 'Column' map_from_entries(x) ## S4 method for signature 'Column' map_keys(x) ## S4 method for signature 'characterOrColumn,'function'' transform_keys(x, f) ## S4 method for signature 'characterOrColumn,'function'' transform_values(x, f) ## S4 method for signature 'Column' map_values(x) ## S4 method for signature 'characterOrColumn,characterOrColumn,'function'' map_zip_with(x, y, f) ## S4 method for signature 'Column' element_at(x, extraction) ## S4 method for signature 'Column' explode(x) ## S4 method for signature 'Column' size(x) ## S4 method for signature 'Column' slice(x, start, length) ## S4 method for signature 'Column' sort_array(x, asc = TRUE) ## S4 method for signature 'Column' posexplode(x) ## S4 method for signature 'Column' explode_outer(x) ## S4 method for signature 'Column' posexplode_outer(x)
x |
Column to compute on. Note the difference in the following methods:
|
initialValue |
a |
merge |
a |
... |
additional argument(s).
|
value |
A value to compute on.
|
y |
Column to compute on. |
f |
a
|
delimiter |
a character string that is used to concatenate the elements of column. |
count |
a Column or constant determining the number of repetitions. |
extraction |
index to check for in array or key to check for in map |
schema |
|
start |
the starting index |
length |
the length of the slice |
asc |
a logical flag indicating the sorting order. TRUE, sorting is in ascending order. FALSE, sorting is in descending order. |
as.json.array |
indicating if input string is JSON array of objects or a single object. |
finish |
an unary |
nullReplacement |
an optional character string that is used to replace the Null values. |
reverse: Returns a reversed string or an array with reverse order of elements.
to_json: Converts a column containing a structType, a mapType
or an arrayType into a Column of JSON string.
Resolving the Column can fail if an unsupported type is encountered.
to_csv: Converts a column containing a structType into a Column of CSV string.
Resolving the Column can fail if an unsupported type is encountered.
concat: Concatenates multiple input columns together into a single column.
The function works with strings, binary and compatible array columns.
from_json: Parses a column containing a JSON string into a Column of structType
with the specified schema or array of structType if as.json.array is set
to TRUE. If the string is unparseable, the Column will contain the value NA.
schema_of_json: Parses a JSON string and infers its schema in DDL format.
from_csv: Parses a column containing a CSV string into a Column of structType
with the specified schema.
If the string is unparseable, the Column will contain the value NA.
schema_of_csv: Parses a CSV string and infers its schema in DDL format.
array_aggregate Applies a binary operator to an initial state
and all elements in the array, and reduces this to a single state.
The final state is converted into the final result by applying
a finish function.
array_contains: Returns null if the array is null, true if the array contains
the value, and false otherwise.
array_distinct: Removes duplicate values from the array.
array_except: Returns an array of the elements in the first array but not in the second
array, without duplicates. The order of elements in the result is not determined.
array_exists Returns whether a predicate holds for one or more elements in the array.
array_filter Returns an array of elements for which a predicate holds in a given array.
array_forall Returns whether a predicate holds for every element in the array.
array_intersect: Returns an array of the elements in the intersection of the given two
arrays, without duplicates.
array_join: Concatenates the elements of column using the delimiter.
Null values are replaced with nullReplacement if set, otherwise they are ignored.
array_max: Returns the maximum value of the array.
array_min: Returns the minimum value of the array.
array_position: Locates the position of the first occurrence of the given value
in the given array. Returns NA if either of the arguments are NA.
Note: The position is not zero based, but 1 based index. Returns 0 if the given
value could not be found in the array.
array_remove: Removes all elements that equal to element from the given array.
array_repeat: Creates an array containing x repeated the number of times
given by count.
array_sort: Sorts the input array in ascending order. The elements of the input array
must be orderable. NA elements will be placed at the end of the returned array.
array_transform Returns an array of elements after applying
a transformation to each element in the input array.
arrays_overlap: Returns true if the input arrays have at least one non-null element in
common. If not and both arrays are non-empty and any of them contains a null, it returns null.
It returns false otherwise.
array_union: Returns an array of the elements in the union of the given two arrays,
without duplicates.
arrays_zip: Returns a merged array of structs in which the N-th struct contains all N-th
values of input arrays.
arrays_zip_with Merge two given arrays, element-wise, into a single array
using a function. If one array is shorter, nulls are appended at the end
to match the length of the longer array, before applying the function.
shuffle: Returns a random permutation of the given array.
flatten: Creates a single array from an array of arrays.
If a structure of nested arrays is deeper than two levels, only one level of nesting is removed.
map_concat: Returns the union of all the given maps.
map_entries: Returns an unordered array of all entries in the given map.
map_filter Returns a map whose key-value pairs satisfy a predicate.
map_from_arrays: Creates a new map column. The array in the first column is used for
keys. The array in the second column is used for values. All elements in the array for key
should not be null.
map_from_entries: Returns a map created from the given array of entries.
map_keys: Returns an unordered array containing the keys of the map.
transform_keys Applies a function to every key-value pair in a map and returns
a map with the results of those applications as the new keys for the pairs.
transform_values Applies a function to every key-value pair in a map and returns
a map with the results of those applications as the new values for the pairs.
map_values: Returns an unordered array containing the values of the map.
map_zip Merge two given maps, key-wise into a single map using a function.
element_at: Returns element of array at given index in extraction if
x is array. Returns value for the given key in extraction if x is map.
Note: The position is not zero based, but 1 based index.
explode: Creates a new row for each element in the given array or map column.
Uses the default column name col for elements in the array and
key and value for elements in the map unless specified otherwise.
size: Returns length of array or map.
slice: Returns an array containing all the elements in x from the index start
(array indices start at 1, or from the end if start is negative) with the specified length.
sort_array: Sorts the input array in ascending or descending order according to
the natural ordering of the array elements. NA elements will be placed at the beginning of
the returned array in ascending order or at the end of the returned array in descending order.
posexplode: Creates a new row for each element with position in the given array
or map column. Uses the default column name pos for position, and col
for elements in the array and key and value for elements in the map
unless specified otherwise.
explode: Creates a new row for each element in the given array or map column.
Unlike explode, if the array/map is null or empty
then null is produced.
Uses the default column name col for elements in the array and
key and value for elements in the map unless specified otherwise.
posexplode_outer: Creates a new row for each element with position in the given
array or map column. Unlike posexplode, if the array/map is null or empty
then the row (null, null) is produced.
Uses the default column name pos for position, and col
for elements in the array and key and value for elements in the map
unless specified otherwise.
reverse since 1.5.0
to_json since 2.2.0
to_csv since 3.0.0
concat since 1.5.0
from_json since 2.2.0
schema_of_json since 3.0.0
from_csv since 3.0.0
schema_of_csv since 3.0.0
array_aggregate since 3.1.0
array_contains since 1.6.0
array_distinct since 2.4.0
array_except since 2.4.0
array_exists since 3.1.0
array_filter since 3.1.0
array_forall since 3.1.0
array_intersect since 2.4.0
array_join since 2.4.0
array_max since 2.4.0
array_min since 2.4.0
array_position since 2.4.0
array_remove since 2.4.0
array_repeat since 2.4.0
array_sort since 2.4.0
array_transform since 3.1.0
arrays_overlap since 2.4.0
array_union since 2.4.0
arrays_zip since 2.4.0
zip_with since 3.1.0
shuffle since 2.4.0
flatten since 2.4.0
map_concat since 3.0.0
map_entries since 3.0.0
map_filter since 3.1.0
map_from_arrays since 2.4.0
map_from_entries since 3.0.0
map_keys since 2.3.0
transform_keys since 3.1.0
transform_values since 3.1.0
map_values since 2.3.0
map_zip_with since 3.1.0
element_at since 2.4.0
explode since 1.5.0
size since 1.5.0
slice since 2.4.0
sort_array since 1.6.0
posexplode since 2.1.0
explode_outer since 2.3.0
posexplode_outer since 2.3.0
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) tmp <- mutate(df, v1 = create_array(df$mpg, df$cyl, df$hp)) head(select(tmp, array_contains(tmp$v1, 21), size(tmp$v1), shuffle(tmp$v1))) head(select(tmp, array_max(tmp$v1), array_min(tmp$v1), array_distinct(tmp$v1))) head(select(tmp, array_position(tmp$v1, 21), array_repeat(df$mpg, 3), array_sort(tmp$v1))) head(select(tmp, reverse(tmp$v1), array_remove(tmp$v1, 21))) head(select(tmp, array_transform("v1", function(x) x * 10))) head(select(tmp, array_exists("v1", function(x) x > 120))) head(select(tmp, array_forall("v1", function(x) x >= 8.0))) head(select(tmp, array_filter("v1", function(x) x < 10))) head(select(tmp, array_aggregate("v1", lit(0), function(acc, y) acc + y))) head(select( tmp, array_aggregate("v1", lit(0), function(acc, y) acc + y, function(acc) acc / 10))) tmp2 <- mutate(tmp, v2 = explode(tmp$v1)) head(tmp2) head(select(tmp, posexplode(tmp$v1))) head(select(tmp, slice(tmp$v1, 2L, 2L))) head(select(tmp, sort_array(tmp$v1))) head(select(tmp, sort_array(tmp$v1, asc = FALSE))) tmp3 <- mutate(df, v3 = create_map(df$model, df$cyl)) head(select(tmp3, map_entries(tmp3$v3), map_keys(tmp3$v3), map_values(tmp3$v3))) head(select(tmp3, element_at(tmp3$v3, "Valiant"), map_concat(tmp3$v3, tmp3$v3))) head(select(tmp3, transform_keys("v3", function(k, v) upper(k)))) head(select(tmp3, transform_values("v3", function(k, v) v * 10))) head(select(tmp3, map_filter("v3", function(k, v) v < 42))) tmp4 <- mutate(df, v4 = create_array(df$mpg, df$cyl), v5 = create_array(df$cyl, df$hp)) head(select(tmp4, concat(tmp4$v4, tmp4$v5), arrays_overlap(tmp4$v4, tmp4$v5))) head(select(tmp4, array_except(tmp4$v4, tmp4$v5), array_intersect(tmp4$v4, tmp4$v5))) head(select(tmp4, array_union(tmp4$v4, tmp4$v5))) head(select(tmp4, arrays_zip(tmp4$v4, tmp4$v5))) head(select(tmp, concat(df$mpg, df$cyl, df$hp))) head(select(tmp4, arrays_zip_with(tmp4$v4, tmp4$v5, function(x, y) x * y))) tmp5 <- mutate(df, v6 = create_array(df$model, df$model)) head(select(tmp5, array_join(tmp5$v6, "#"), array_join(tmp5$v6, "#", "NULL"))) tmp6 <- mutate(df, v7 = create_array(create_array(df$model, df$model))) head(select(tmp6, flatten(tmp6$v7))) tmp7 <- mutate(df, v8 = create_array(df$model, df$cyl), v9 = create_array(df$model, df$hp)) head(select(tmp7, arrays_zip_with("v8", "v9", function(x, y) (x * y) %% 3))) head(select(tmp7, map_from_arrays(tmp7$v8, tmp7$v9))) tmp8 <- mutate(df, v10 = create_array(struct(df$model, df$cyl))) head(select(tmp8, map_from_entries(tmp8$v10))) ## End(Not run) ## Not run: # Converts a struct into a JSON object df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") select(df2, to_json(df2$d, dateFormat = 'dd/MM/yyyy')) # Converts an array of structs into a JSON array df2 <- sql("SELECT array(named_struct('name', 'Bob'), named_struct('name', 'Alice')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts a map into a JSON object df2 <- sql("SELECT map('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts an array of maps into a JSON array df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts a map into a pretty JSON object df2 <- sql("SELECT map('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people, pretty = TRUE)) ## End(Not run) ## Not run: # Converts a struct into a CSV string df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") select(df2, to_csv(df2$d, dateFormat = 'dd/MM/yyyy')) ## End(Not run) ## Not run: df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") df2 <- mutate(df2, d2 = to_json(df2$d, dateFormat = 'dd/MM/yyyy')) schema <- structType(structField("date", "string")) head(select(df2, from_json(df2$d2, schema, dateFormat = 'dd/MM/yyyy'))) df2 <- sql("SELECT named_struct('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people)) schema <- structType(structField("name", "string")) head(select(df2, from_json(df2$people_json, schema))) head(select(df2, from_json(df2$people_json, "name STRING"))) head(select(df2, from_json(df2$people_json, schema_of_json(head(df2)$people_json)))) ## End(Not run) ## Not run: json <- "{\"name\":\"Bob\"}" df <- sql("SELECT * FROM range(1)") head(select(df, schema_of_json(json))) ## End(Not run) ## Not run: csv <- "Amsterdam,2018" df <- sql(paste0("SELECT '", csv, "' as csv")) schema <- "city STRING, year INT" head(select(df, from_csv(df$csv, schema))) head(select(df, from_csv(df$csv, structType(schema)))) head(select(df, from_csv(df$csv, schema_of_csv(csv)))) ## End(Not run) ## Not run: csv <- "Amsterdam,2018" df <- sql("SELECT * FROM range(1)") head(select(df, schema_of_csv(csv))) ## End(Not run) ## Not run: df2 <- createDataFrame(data.frame( id = c(1, 2, 3), text = c("a,b,c", NA, "d,e") )) head(select(df2, df2$id, explode_outer(split_string(df2$text, ",")))) head(select(df2, df2$id, posexplode_outer(split_string(df2$text, ",")))) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) tmp <- mutate(df, v1 = create_array(df$mpg, df$cyl, df$hp)) head(select(tmp, array_contains(tmp$v1, 21), size(tmp$v1), shuffle(tmp$v1))) head(select(tmp, array_max(tmp$v1), array_min(tmp$v1), array_distinct(tmp$v1))) head(select(tmp, array_position(tmp$v1, 21), array_repeat(df$mpg, 3), array_sort(tmp$v1))) head(select(tmp, reverse(tmp$v1), array_remove(tmp$v1, 21))) head(select(tmp, array_transform("v1", function(x) x * 10))) head(select(tmp, array_exists("v1", function(x) x > 120))) head(select(tmp, array_forall("v1", function(x) x >= 8.0))) head(select(tmp, array_filter("v1", function(x) x < 10))) head(select(tmp, array_aggregate("v1", lit(0), function(acc, y) acc + y))) head(select( tmp, array_aggregate("v1", lit(0), function(acc, y) acc + y, function(acc) acc / 10))) tmp2 <- mutate(tmp, v2 = explode(tmp$v1)) head(tmp2) head(select(tmp, posexplode(tmp$v1))) head(select(tmp, slice(tmp$v1, 2L, 2L))) head(select(tmp, sort_array(tmp$v1))) head(select(tmp, sort_array(tmp$v1, asc = FALSE))) tmp3 <- mutate(df, v3 = create_map(df$model, df$cyl)) head(select(tmp3, map_entries(tmp3$v3), map_keys(tmp3$v3), map_values(tmp3$v3))) head(select(tmp3, element_at(tmp3$v3, "Valiant"), map_concat(tmp3$v3, tmp3$v3))) head(select(tmp3, transform_keys("v3", function(k, v) upper(k)))) head(select(tmp3, transform_values("v3", function(k, v) v * 10))) head(select(tmp3, map_filter("v3", function(k, v) v < 42))) tmp4 <- mutate(df, v4 = create_array(df$mpg, df$cyl), v5 = create_array(df$cyl, df$hp)) head(select(tmp4, concat(tmp4$v4, tmp4$v5), arrays_overlap(tmp4$v4, tmp4$v5))) head(select(tmp4, array_except(tmp4$v4, tmp4$v5), array_intersect(tmp4$v4, tmp4$v5))) head(select(tmp4, array_union(tmp4$v4, tmp4$v5))) head(select(tmp4, arrays_zip(tmp4$v4, tmp4$v5))) head(select(tmp, concat(df$mpg, df$cyl, df$hp))) head(select(tmp4, arrays_zip_with(tmp4$v4, tmp4$v5, function(x, y) x * y))) tmp5 <- mutate(df, v6 = create_array(df$model, df$model)) head(select(tmp5, array_join(tmp5$v6, "#"), array_join(tmp5$v6, "#", "NULL"))) tmp6 <- mutate(df, v7 = create_array(create_array(df$model, df$model))) head(select(tmp6, flatten(tmp6$v7))) tmp7 <- mutate(df, v8 = create_array(df$model, df$cyl), v9 = create_array(df$model, df$hp)) head(select(tmp7, arrays_zip_with("v8", "v9", function(x, y) (x * y) %% 3))) head(select(tmp7, map_from_arrays(tmp7$v8, tmp7$v9))) tmp8 <- mutate(df, v10 = create_array(struct(df$model, df$cyl))) head(select(tmp8, map_from_entries(tmp8$v10))) ## End(Not run) ## Not run: # Converts a struct into a JSON object df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") select(df2, to_json(df2$d, dateFormat = 'dd/MM/yyyy')) # Converts an array of structs into a JSON array df2 <- sql("SELECT array(named_struct('name', 'Bob'), named_struct('name', 'Alice')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts a map into a JSON object df2 <- sql("SELECT map('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts an array of maps into a JSON array df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts a map into a pretty JSON object df2 <- sql("SELECT map('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people, pretty = TRUE)) ## End(Not run) ## Not run: # Converts a struct into a CSV string df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") select(df2, to_csv(df2$d, dateFormat = 'dd/MM/yyyy')) ## End(Not run) ## Not run: df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") df2 <- mutate(df2, d2 = to_json(df2$d, dateFormat = 'dd/MM/yyyy')) schema <- structType(structField("date", "string")) head(select(df2, from_json(df2$d2, schema, dateFormat = 'dd/MM/yyyy'))) df2 <- sql("SELECT named_struct('name', 'Bob') as people") df2 <- mutate(df2, people_json = to_json(df2$people)) schema <- structType(structField("name", "string")) head(select(df2, from_json(df2$people_json, schema))) head(select(df2, from_json(df2$people_json, "name STRING"))) head(select(df2, from_json(df2$people_json, schema_of_json(head(df2)$people_json)))) ## End(Not run) ## Not run: json <- "{\"name\":\"Bob\"}" df <- sql("SELECT * FROM range(1)") head(select(df, schema_of_json(json))) ## End(Not run) ## Not run: csv <- "Amsterdam,2018" df <- sql(paste0("SELECT '", csv, "' as csv")) schema <- "city STRING, year INT" head(select(df, from_csv(df$csv, schema))) head(select(df, from_csv(df$csv, structType(schema)))) head(select(df, from_csv(df$csv, schema_of_csv(csv)))) ## End(Not run) ## Not run: csv <- "Amsterdam,2018" df <- sql("SELECT * FROM range(1)") head(select(df, schema_of_csv(csv))) ## End(Not run) ## Not run: df2 <- createDataFrame(data.frame( id = c(1, 2, 3), text = c("a,b,c", NA, "d,e") )) head(select(df2, df2$id, explode_outer(split_string(df2$text, ",")))) head(select(df2, df2$id, posexplode_outer(split_string(df2$text, ",")))) ## End(Not run)
Date time arithmetic functions defined for Column.
add_months(y, x) datediff(y, x) date_add(y, x) date_format(y, x) date_sub(y, x) from_utc_timestamp(y, x) months_between(y, x, ...) next_day(y, x) to_utc_timestamp(y, x) ## S4 method for signature 'Column' datediff(y, x) ## S4 method for signature 'Column' months_between(y, x, roundOff = NULL) ## S4 method for signature 'Column,character' date_format(y, x) ## S4 method for signature 'Column,character' from_utc_timestamp(y, x) ## S4 method for signature 'Column,character' next_day(y, x) ## S4 method for signature 'Column,character' to_utc_timestamp(y, x) ## S4 method for signature 'Column,numeric' add_months(y, x) ## S4 method for signature 'Column,numeric' date_add(y, x) ## S4 method for signature 'Column,numeric' date_sub(y, x)add_months(y, x) datediff(y, x) date_add(y, x) date_format(y, x) date_sub(y, x) from_utc_timestamp(y, x) months_between(y, x, ...) next_day(y, x) to_utc_timestamp(y, x) ## S4 method for signature 'Column' datediff(y, x) ## S4 method for signature 'Column' months_between(y, x, roundOff = NULL) ## S4 method for signature 'Column,character' date_format(y, x) ## S4 method for signature 'Column,character' from_utc_timestamp(y, x) ## S4 method for signature 'Column,character' next_day(y, x) ## S4 method for signature 'Column,character' to_utc_timestamp(y, x) ## S4 method for signature 'Column,numeric' add_months(y, x) ## S4 method for signature 'Column,numeric' date_add(y, x) ## S4 method for signature 'Column,numeric' date_sub(y, x)
y |
Column to compute on. |
x |
For class
|
... |
additional argument(s).
|
roundOff |
an optional parameter to specify if the result is rounded off to 8 digits |
datediff: Returns the number of days from y to x.
If y is later than x then the result is positive.
months_between: Returns number of months between dates y and x.
If y is later than x, then the result is positive. If y and x
are on the same day of month, or both are the last day of month, time of day will be ignored.
Otherwise, the difference is calculated based on 31 days per month, and rounded to 8 digits.
date_format: Converts a date/timestamp/string to a value of string in the format
specified by the date format given by the second argument. A pattern could be for instance
dd.MM.yyyy and could return a string like '18.03.1993'. All
pattern letters of java.time.format.DateTimeFormatter can be used.
Note: Use when ever possible specialized functions like year. These benefit from a
specialized implementation.
from_utc_timestamp: This is a common function for databases supporting TIMESTAMP WITHOUT
TIMEZONE. This function takes a timestamp which is timezone-agnostic, and interprets it as a
timestamp in UTC, and renders that timestamp as a timestamp in the given time zone.
However, timestamp in Spark represents number of microseconds from the Unix epoch, which is not
timezone-agnostic. So in Spark this function just shift the timestamp value from UTC timezone to
the given timezone.
This function may return confusing result if the input is a string with timezone, e.g.
(2018-03-13T06:18:23+00:00). The reason is that, Spark firstly cast the string to
timestamp according to the timezone in the string, and finally display the result by converting
the timestamp to string according to the session local timezone.
next_day: Given a date column, returns the first date which is later than the value of
the date column that is on the specified day of the week. For example,
next_day("2015-07-27", "Sunday") returns 2015-08-02 because that is the first Sunday
after 2015-07-27. Day of the week parameter is case insensitive, and accepts first three or
two characters: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun".
to_utc_timestamp: This is a common function for databases supporting TIMESTAMP WITHOUT
TIMEZONE. This function takes a timestamp which is timezone-agnostic, and interprets it as a
timestamp in the given timezone, and renders that timestamp as a timestamp in UTC.
However, timestamp in Spark represents number of microseconds from the Unix epoch, which is not
timezone-agnostic. So in Spark this function just shift the timestamp value from the given
timezone to UTC timezone.
This function may return confusing result if the input is a string with timezone, e.g.
(2018-03-13T06:18:23+00:00). The reason is that, Spark firstly cast the string to
timestamp according to the timezone in the string, and finally display the result by converting
the timestamp to string according to the session local timezone.
add_months: Returns the date that is numMonths (x) after startDate (y).
date_add: Returns the date that is x days after.
date_sub: Returns the date that is x days before.
datediff since 1.5.0
months_between since 1.5.0
date_format since 1.5.0
from_utc_timestamp since 1.5.0
next_day since 1.5.0
to_utc_timestamp since 1.5.0
add_months since 1.5.0
date_add since 1.5.0
date_sub since 1.5.0
Other data time functions:
timestamp_seconds()
## Not run: dts <- c("2005-01-02 18:47:22", "2005-12-24 16:30:58", "2005-10-28 07:30:05", "2005-12-28 07:01:05", "2006-01-24 00:01:10") y <- c(2.0, 2.2, 3.4, 2.5, 1.8) df <- createDataFrame(data.frame(time = as.POSIXct(dts), y = y)) ## End(Not run) ## Not run: tmp <- createDataFrame(data.frame(time_string1 = as.POSIXct(dts), time_string2 = as.POSIXct(dts[order(runif(length(dts)))]))) tmp2 <- mutate(tmp, datediff = datediff(tmp$time_string1, tmp$time_string2), monthdiff = months_between(tmp$time_string1, tmp$time_string2)) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, from_utc = from_utc_timestamp(df$time, "PST"), to_utc = to_utc_timestamp(df$time, "PST")) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, t1 = add_months(df$time, 1), t2 = date_add(df$time, 2), t3 = date_sub(df$time, 3), t4 = next_day(df$time, "Sun")) head(tmp) ## End(Not run)## Not run: dts <- c("2005-01-02 18:47:22", "2005-12-24 16:30:58", "2005-10-28 07:30:05", "2005-12-28 07:01:05", "2006-01-24 00:01:10") y <- c(2.0, 2.2, 3.4, 2.5, 1.8) df <- createDataFrame(data.frame(time = as.POSIXct(dts), y = y)) ## End(Not run) ## Not run: tmp <- createDataFrame(data.frame(time_string1 = as.POSIXct(dts), time_string2 = as.POSIXct(dts[order(runif(length(dts)))]))) tmp2 <- mutate(tmp, datediff = datediff(tmp$time_string1, tmp$time_string2), monthdiff = months_between(tmp$time_string1, tmp$time_string2)) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, from_utc = from_utc_timestamp(df$time, "PST"), to_utc = to_utc_timestamp(df$time, "PST")) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, t1 = add_months(df$time, 1), t2 = date_add(df$time, 2), t3 = date_sub(df$time, 3), t4 = next_day(df$time, "Sun")) head(tmp) ## End(Not run)
Math functions defined for Column.
bin(x) bround(x, ...) cbrt(x) ceil(x) conv(x, fromBase, toBase) hex(x) hypot(y, x) pmod(y, x) rint(x) shiftLeft(y, x) shiftRight(y, x) shiftRightUnsigned(y, x) signum(x) degrees(x) toDegrees(x) radians(x) toRadians(x) unhex(x) ## S4 method for signature 'Column' abs(x) ## S4 method for signature 'Column' acos(x) ## S4 method for signature 'Column' acosh(x) ## S4 method for signature 'Column' asin(x) ## S4 method for signature 'Column' asinh(x) ## S4 method for signature 'Column' atan(x) ## S4 method for signature 'Column' atanh(x) ## S4 method for signature 'Column' bin(x) ## S4 method for signature 'Column' cbrt(x) ## S4 method for signature 'Column' ceil(x) ## S4 method for signature 'Column' ceiling(x) ## S4 method for signature 'Column' cos(x) ## S4 method for signature 'Column' cosh(x) ## S4 method for signature 'Column' exp(x) ## S4 method for signature 'Column' expm1(x) ## S4 method for signature 'Column' factorial(x) ## S4 method for signature 'Column' floor(x) ## S4 method for signature 'Column' hex(x) ## S4 method for signature 'Column' log(x) ## S4 method for signature 'Column' log10(x) ## S4 method for signature 'Column' log1p(x) ## S4 method for signature 'Column' log2(x) ## S4 method for signature 'Column' rint(x) ## S4 method for signature 'Column' round(x) ## S4 method for signature 'Column' bround(x, scale = 0) ## S4 method for signature 'Column' signum(x) ## S4 method for signature 'Column' sign(x) ## S4 method for signature 'Column' sin(x) ## S4 method for signature 'Column' sinh(x) ## S4 method for signature 'Column' sqrt(x) ## S4 method for signature 'Column' tan(x) ## S4 method for signature 'Column' tanh(x) ## S4 method for signature 'Column' toDegrees(x) ## S4 method for signature 'Column' degrees(x) ## S4 method for signature 'Column' toRadians(x) ## S4 method for signature 'Column' radians(x) ## S4 method for signature 'Column' unhex(x) ## S4 method for signature 'Column' atan2(y, x) ## S4 method for signature 'Column' hypot(y, x) ## S4 method for signature 'Column' pmod(y, x) ## S4 method for signature 'Column,numeric' shiftLeft(y, x) ## S4 method for signature 'Column,numeric' shiftRight(y, x) ## S4 method for signature 'Column,numeric' shiftRightUnsigned(y, x) ## S4 method for signature 'Column,numeric,numeric' conv(x, fromBase, toBase)bin(x) bround(x, ...) cbrt(x) ceil(x) conv(x, fromBase, toBase) hex(x) hypot(y, x) pmod(y, x) rint(x) shiftLeft(y, x) shiftRight(y, x) shiftRightUnsigned(y, x) signum(x) degrees(x) toDegrees(x) radians(x) toRadians(x) unhex(x) ## S4 method for signature 'Column' abs(x) ## S4 method for signature 'Column' acos(x) ## S4 method for signature 'Column' acosh(x) ## S4 method for signature 'Column' asin(x) ## S4 method for signature 'Column' asinh(x) ## S4 method for signature 'Column' atan(x) ## S4 method for signature 'Column' atanh(x) ## S4 method for signature 'Column' bin(x) ## S4 method for signature 'Column' cbrt(x) ## S4 method for signature 'Column' ceil(x) ## S4 method for signature 'Column' ceiling(x) ## S4 method for signature 'Column' cos(x) ## S4 method for signature 'Column' cosh(x) ## S4 method for signature 'Column' exp(x) ## S4 method for signature 'Column' expm1(x) ## S4 method for signature 'Column' factorial(x) ## S4 method for signature 'Column' floor(x) ## S4 method for signature 'Column' hex(x) ## S4 method for signature 'Column' log(x) ## S4 method for signature 'Column' log10(x) ## S4 method for signature 'Column' log1p(x) ## S4 method for signature 'Column' log2(x) ## S4 method for signature 'Column' rint(x) ## S4 method for signature 'Column' round(x) ## S4 method for signature 'Column' bround(x, scale = 0) ## S4 method for signature 'Column' signum(x) ## S4 method for signature 'Column' sign(x) ## S4 method for signature 'Column' sin(x) ## S4 method for signature 'Column' sinh(x) ## S4 method for signature 'Column' sqrt(x) ## S4 method for signature 'Column' tan(x) ## S4 method for signature 'Column' tanh(x) ## S4 method for signature 'Column' toDegrees(x) ## S4 method for signature 'Column' degrees(x) ## S4 method for signature 'Column' toRadians(x) ## S4 method for signature 'Column' radians(x) ## S4 method for signature 'Column' unhex(x) ## S4 method for signature 'Column' atan2(y, x) ## S4 method for signature 'Column' hypot(y, x) ## S4 method for signature 'Column' pmod(y, x) ## S4 method for signature 'Column,numeric' shiftLeft(y, x) ## S4 method for signature 'Column,numeric' shiftRight(y, x) ## S4 method for signature 'Column,numeric' shiftRightUnsigned(y, x) ## S4 method for signature 'Column,numeric,numeric' conv(x, fromBase, toBase)
x |
Column to compute on. In |
... |
additional argument(s). |
fromBase |
base to convert from. |
toBase |
base to convert to. |
y |
Column to compute on. |
scale |
round to |
abs: Computes the absolute value.
acos: Returns the inverse cosine of the given value,
as if computed by java.lang.Math.acos()
acosh: Computes inverse hyperbolic cosine of the input column.
asin: Returns the inverse sine of the given value,
as if computed by java.lang.Math.asin()
asinh: Computes inverse hyperbolic sine of the input column.
atan: Returns the inverse tangent of the given value,
as if computed by java.lang.Math.atan()
atanh: Computes inverse hyperbolic tangent of the input column.
bin: Returns the string representation of the binary value
of the given long column. For example, bin("12") returns "1100".
cbrt: Computes the cube-root of the given value.
ceil: Computes the ceiling of the given value.
ceiling: Alias for ceil.
cos: Returns the cosine of the given value,
as if computed by java.lang.Math.cos(). Units in radians.
cosh: Returns the hyperbolic cosine of the given value,
as if computed by java.lang.Math.cosh().
exp: Computes the exponential of the given value.
expm1: Computes the exponential of the given value minus one.
factorial: Computes the factorial of the given value.
floor: Computes the floor of the given value.
hex: Computes hex value of the given column.
log: Computes the natural logarithm of the given value.
log10: Computes the logarithm of the given value in base 10.
log1p: Computes the natural logarithm of the given value plus one.
log2: Computes the logarithm of the given column in base 2.
rint: Returns the double value that is closest in value to the argument and
is equal to a mathematical integer.
round: Returns the value of the column rounded to 0 decimal places
using HALF_UP rounding mode.
bround: Returns the value of the column e rounded to scale decimal places
using HALF_EVEN rounding mode if scale >= 0 or at integer part when scale < 0.
Also known as Gaussian rounding or bankers' rounding that rounds to the nearest even number.
bround(2.5, 0) = 2, bround(3.5, 0) = 4.
signum: Computes the signum of the given value.
sign: Alias for signum.
sin: Returns the sine of the given value,
as if computed by java.lang.Math.sin(). Units in radians.
sinh: Returns the hyperbolic sine of the given value,
as if computed by java.lang.Math.sinh().
sqrt: Computes the square root of the specified float value.
tan: Returns the tangent of the given value,
as if computed by java.lang.Math.tan().
Units in radians.
tanh: Returns the hyperbolic tangent of the given value,
as if computed by java.lang.Math.tanh().
toDegrees: Converts an angle measured in radians to an approximately equivalent angle
measured in degrees.
degrees: Converts an angle measured in radians to an approximately equivalent angle
measured in degrees.
toRadians: Converts an angle measured in degrees to an approximately equivalent angle
measured in radians.
radians: Converts an angle measured in degrees to an approximately equivalent angle
measured in radians.
unhex: Inverse of hex. Interprets each pair of characters as a hexadecimal number
and converts to the byte representation of number.
atan2: Returns the angle theta from the conversion of rectangular coordinates
(x, y) to polar coordinates (r, theta),
as if computed by java.lang.Math.atan2(). Units in radians.
hypot: Computes "sqrt(a^2 + b^2)" without intermediate overflow or underflow.
pmod: Returns the positive value of dividend mod divisor.
Column x is divisor column, and column y is the dividend column.
shiftLeft: Shifts the given value numBits left. If the given value is a long value,
this function will return a long value else it will return an integer value.
shiftRight: (Signed) shifts the given value numBits right. If the given value is a long
value, it will return a long value else it will return an integer value.
shiftRightUnsigned: (Unsigned) shifts the given value numBits right. If the given value is
a long value, it will return a long value else it will return an integer value.
conv: Converts a number in a string column from one base to another.
abs since 1.5.0
acos since 1.5.0
acosh since 3.1.0
asin since 1.5.0
asinh since 3.1.0
atan since 1.5.0
atanh since 3.1.0
bin since 1.5.0
cbrt since 1.4.0
ceil since 1.5.0
ceiling since 1.5.0
cos since 1.5.0
cosh since 1.5.0
exp since 1.5.0
expm1 since 1.5.0
factorial since 1.5.0
floor since 1.5.0
hex since 1.5.0
log since 1.5.0
log10 since 1.5.0
log1p since 1.5.0
log2 since 1.5.0
rint since 1.5.0
round since 1.5.0
bround since 2.0.0
signum since 1.5.0
sign since 1.5.0
sin since 1.5.0
sinh since 1.5.0
sqrt since 1.5.0
tan since 1.5.0
tanh since 1.5.0
toDegrees since 1.4.0
degrees since 3.0.0
toRadians since 1.4.0
radians since 3.0.0
unhex since 1.5.0
atan2 since 1.5.0
hypot since 1.4.0
pmod since 1.5.0
shiftLeft since 1.5.0
shiftRight since 1.5.0
shiftRightUnsigned since 1.5.0
conv since 1.5.0
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) tmp <- mutate(df, v1 = log(df$mpg), v2 = cbrt(df$disp), v3 = bround(df$wt, 1), v4 = bin(df$cyl), v5 = hex(df$wt), v6 = degrees(df$gear), v7 = atan2(df$cyl, df$am), v8 = hypot(df$cyl, df$am), v9 = pmod(df$hp, df$cyl), v10 = shiftLeft(df$disp, 1), v11 = conv(df$hp, 10, 16), v12 = sign(df$vs - 0.5), v13 = sqrt(df$disp), v14 = ceil(df$wt)) head(tmp) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) tmp <- mutate(df, v1 = log(df$mpg), v2 = cbrt(df$disp), v3 = bround(df$wt, 1), v4 = bin(df$cyl), v5 = hex(df$wt), v6 = degrees(df$gear), v7 = atan2(df$cyl, df$am), v8 = hypot(df$cyl, df$am), v9 = pmod(df$hp, df$cyl), v10 = shiftLeft(df$disp, 1), v11 = conv(df$hp, 10, 16), v12 = sign(df$vs - 0.5), v13 = sqrt(df$disp), v14 = ceil(df$wt)) head(tmp) ## End(Not run)
Miscellaneous functions defined for Column.
assert_true(x, errMsg = NULL) crc32(x) hash(x, ...) md5(x) raise_error(x) sha1(x) sha2(y, x) xxhash64(x, ...) ## S4 method for signature 'Column' crc32(x) ## S4 method for signature 'Column' hash(x, ...) ## S4 method for signature 'Column' xxhash64(x, ...) ## S4 method for signature 'Column' assert_true(x, errMsg = NULL) ## S4 method for signature 'characterOrColumn' raise_error(x) ## S4 method for signature 'Column' md5(x) ## S4 method for signature 'Column' sha1(x) ## S4 method for signature 'Column,numeric' sha2(y, x)assert_true(x, errMsg = NULL) crc32(x) hash(x, ...) md5(x) raise_error(x) sha1(x) sha2(y, x) xxhash64(x, ...) ## S4 method for signature 'Column' crc32(x) ## S4 method for signature 'Column' hash(x, ...) ## S4 method for signature 'Column' xxhash64(x, ...) ## S4 method for signature 'Column' assert_true(x, errMsg = NULL) ## S4 method for signature 'characterOrColumn' raise_error(x) ## S4 method for signature 'Column' md5(x) ## S4 method for signature 'Column' sha1(x) ## S4 method for signature 'Column,numeric' sha2(y, x)
x |
Column to compute on. In |
errMsg |
(optional) The error message to be thrown. |
... |
additional Columns. |
y |
Column to compute on. |
crc32: Calculates the cyclic redundancy check value (CRC32) of a binary column
and returns the value as a bigint.
hash: Calculates the hash code of given columns, and returns the result
as an int column.
xxhash64: Calculates the hash code of given columns using the 64-bit
variant of the xxHash algorithm, and returns the result as a long
column.
assert_true: Returns null if the input column is true; throws an exception
with the provided error message otherwise.
raise_error: Throws an exception with the provided error message.
md5: Calculates the MD5 digest of a binary column and returns the value
as a 32 character hex string.
sha1: Calculates the SHA-1 digest of a binary column and returns the value
as a 40 character hex string.
sha2: Calculates the SHA-2 family of hash functions of a binary column and
returns the value as a hex string. The second argument x specifies the number
of bits, and is one of 224, 256, 384, or 512.
crc32 since 1.5.0
hash since 2.0.0
xxhash64 since 3.0.0
assert_true since 3.1.0
raise_error since 3.1.0
md5 since 1.5.0
sha1 since 1.5.0
sha2 since 1.5.0
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)[, 1:2]) tmp <- mutate(df, v1 = crc32(df$model), v2 = hash(df$model), v3 = hash(df$model, df$mpg), v4 = md5(df$model), v5 = sha1(df$model), v6 = sha2(df$model, 256)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = assert_true(df$vs < 2), v2 = assert_true(df$vs < 2, "custom error message"), v3 = assert_true(df$vs < 2, df$vs)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = raise_error("error message")) head(tmp) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)[, 1:2]) tmp <- mutate(df, v1 = crc32(df$model), v2 = hash(df$model), v3 = hash(df$model, df$mpg), v4 = md5(df$model), v5 = sha1(df$model), v6 = sha2(df$model, 256)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = assert_true(df$vs < 2), v2 = assert_true(df$vs < 2, "custom error message"), v3 = assert_true(df$vs < 2, df$vs)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = raise_error("error message")) head(tmp) ## End(Not run)
ML functions defined for Column.
array_to_vector(x) vector_to_array(x, ...) ## S4 method for signature 'Column' array_to_vector(x) ## S4 method for signature 'Column' vector_to_array(x, dtype = c("float64", "float32"))array_to_vector(x) vector_to_array(x, ...) ## S4 method for signature 'Column' array_to_vector(x) ## S4 method for signature 'Column' vector_to_array(x, dtype = c("float64", "float32"))
x |
Column to compute on. |
... |
additional argument(s). |
dtype |
The data type of the output array. Valid values: "float64" or "float32". |
array_to_vector Converts a column of array of numeric type into
a column of dense vectors in MLlib
vector_to_array Converts a column of MLlib sparse/dense vectors into
a column of dense arrays.
array_to_vector since 3.1.0
vector_to_array since 3.1.0
## Not run: df <- read.df("data/mllib/sample_libsvm_data.txt", source = "libsvm") head( withColumn( withColumn(df, "array", vector_to_array(df$features)), "vector", array_to_vector(column("array")) ) ) ## End(Not run)## Not run: df <- read.df("data/mllib/sample_libsvm_data.txt", source = "libsvm") head( withColumn( withColumn(df, "array", vector_to_array(df$features)), "vector", array_to_vector(column("array")) ) ) ## End(Not run)
Non-aggregate functions defined for Column.
when(condition, value) bitwiseNOT(x) create_array(x, ...) create_map(x, ...) expr(x) greatest(x, ...) input_file_name(x = "missing") isnan(x) least(x, ...) lit(x) monotonically_increasing_id(x = "missing") nanvl(y, x) negate(x) rand(seed) randn(seed) spark_partition_id(x = "missing") struct(x, ...) ## S4 method for signature 'ANY' lit(x) ## S4 method for signature 'Column' bitwiseNOT(x) ## S4 method for signature 'Column' coalesce(x, ...) ## S4 method for signature 'Column' isnan(x) ## S4 method for signature 'Column' is.nan(x) ## S4 method for signature 'missing' monotonically_increasing_id() ## S4 method for signature 'Column' negate(x) ## S4 method for signature 'missing' spark_partition_id() ## S4 method for signature 'characterOrColumn' struct(x, ...) ## S4 method for signature 'Column' nanvl(y, x) ## S4 method for signature 'Column' greatest(x, ...) ## S4 method for signature 'Column' least(x, ...) ## S4 method for signature 'character' expr(x) ## S4 method for signature 'missing' rand(seed) ## S4 method for signature 'numeric' rand(seed) ## S4 method for signature 'missing' randn(seed) ## S4 method for signature 'numeric' randn(seed) ## S4 method for signature 'Column' when(condition, value) ## S4 method for signature 'Column' ifelse(test, yes, no) ## S4 method for signature 'Column' create_array(x, ...) ## S4 method for signature 'Column' create_map(x, ...) ## S4 method for signature 'missing' input_file_name()when(condition, value) bitwiseNOT(x) create_array(x, ...) create_map(x, ...) expr(x) greatest(x, ...) input_file_name(x = "missing") isnan(x) least(x, ...) lit(x) monotonically_increasing_id(x = "missing") nanvl(y, x) negate(x) rand(seed) randn(seed) spark_partition_id(x = "missing") struct(x, ...) ## S4 method for signature 'ANY' lit(x) ## S4 method for signature 'Column' bitwiseNOT(x) ## S4 method for signature 'Column' coalesce(x, ...) ## S4 method for signature 'Column' isnan(x) ## S4 method for signature 'Column' is.nan(x) ## S4 method for signature 'missing' monotonically_increasing_id() ## S4 method for signature 'Column' negate(x) ## S4 method for signature 'missing' spark_partition_id() ## S4 method for signature 'characterOrColumn' struct(x, ...) ## S4 method for signature 'Column' nanvl(y, x) ## S4 method for signature 'Column' greatest(x, ...) ## S4 method for signature 'Column' least(x, ...) ## S4 method for signature 'character' expr(x) ## S4 method for signature 'missing' rand(seed) ## S4 method for signature 'numeric' rand(seed) ## S4 method for signature 'missing' randn(seed) ## S4 method for signature 'numeric' randn(seed) ## S4 method for signature 'Column' when(condition, value) ## S4 method for signature 'Column' ifelse(test, yes, no) ## S4 method for signature 'Column' create_array(x, ...) ## S4 method for signature 'Column' create_map(x, ...) ## S4 method for signature 'missing' input_file_name()
condition |
the condition to test on. Must be a Column expression. |
value |
result expression. |
x |
Column to compute on. In |
... |
additional Columns. |
y |
Column to compute on. |
seed |
a random seed. Can be missing. |
test |
a Column expression that describes the condition. |
yes |
return values for |
no |
return values for |
lit: A new Column is created to represent the literal value.
If the parameter is a Column, it is returned unchanged.
bitwiseNOT: Computes bitwise NOT.
coalesce: Returns the first column that is not NA, or NA if all inputs are.
isnan: Returns true if the column is NaN.
is.nan: Alias for isnan.
monotonically_increasing_id: Returns a column that generates monotonically increasing
64-bit integers. The generated ID is guaranteed to be monotonically increasing and unique,
but not consecutive. The current implementation puts the partition ID in the upper 31 bits,
and the record number within each partition in the lower 33 bits. The assumption is that the
SparkDataFrame has less than 1 billion partitions, and each partition has less than 8 billion
records. As an example, consider a SparkDataFrame with two partitions, each with 3 records.
This expression would return the following IDs:
0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.
This is equivalent to the MONOTONICALLY_INCREASING_ID function in SQL.
The method should be used with no argument.
Note: the function is non-deterministic because its result depends on partition IDs.
negate: Unary minus, i.e. negate the expression.
spark_partition_id: Returns the partition ID as a SparkDataFrame column.
Note that this is nondeterministic because it depends on data partitioning and
task scheduling.
This is equivalent to the SPARK_PARTITION_ID function in SQL.
struct: Creates a new struct column that composes multiple input columns.
nanvl: Returns the first column (y) if it is not NaN, or the second column
(x) if the first column is NaN. Both inputs should be floating point columns
(DoubleType or FloatType).
greatest: Returns the greatest value of the list of column names, skipping null values.
This function takes at least 2 parameters. It will return null if all parameters are null.
least: Returns the least value of the list of column names, skipping null values.
This function takes at least 2 parameters. It will return null if all parameters are null.
expr: Parses the expression string into the column that it represents, similar to
SparkDataFrame.selectExpr
rand: Generates a random column with independent and identically distributed (i.i.d.)
samples uniformly distributed in [0.0, 1.0).
Note: the function is non-deterministic in general case.
randn: Generates a column with independent and identically distributed (i.i.d.) samples
from the standard normal distribution.
Note: the function is non-deterministic in general case.
when: Evaluates a list of conditions and returns one of multiple possible result
expressions. For unmatched expressions null is returned.
ifelse: Evaluates a list of conditions and returns yes if the conditions are
satisfied. Otherwise no is returned for unmatched conditions.
create_array: Creates a new array column. The input columns must all have the same data
type.
create_map: Creates a new map column. The input columns must be grouped as key-value
pairs, e.g. (key1, value1, key2, value2, ...).
The key columns must all have the same data type, and can't be null.
The value columns must all have the same data type.
input_file_name: Creates a string column with the input file name for a given row.
The method should be used with no argument.
lit since 1.5.0
bitwiseNOT since 1.5.0
coalesce(Column) since 2.1.1
isnan since 2.0.0
is.nan since 2.0.0
negate since 1.5.0
spark_partition_id since 2.0.0
struct since 1.6.0
nanvl since 1.5.0
greatest since 1.5.0
least since 1.5.0
expr since 1.5.0
rand since 1.5.0
rand(numeric) since 1.5.0
randn since 1.5.0
randn(numeric) since 1.5.0
when since 1.5.0
ifelse since 1.5.0
create_array since 2.3.0
create_map since 2.3.0
input_file_name since 2.3.0
coalesce,SparkDataFrame-method
Other non-aggregate functions:
column(),
not()
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = lit(df$mpg), v2 = lit("x"), v3 = lit("2015-01-01"), v4 = negate(df$mpg), v5 = expr('length(model)'), v6 = greatest(df$vs, df$am), v7 = least(df$vs, df$am), v8 = column("mpg")) head(tmp) ## End(Not run) ## Not run: head(select(df, bitwiseNOT(cast(df$vs, "int")))) ## End(Not run) ## Not run: head(select(df, monotonically_increasing_id())) ## Not run: head(select(df, spark_partition_id())) ## Not run: tmp <- mutate(df, v1 = struct(df$mpg, df$cyl), v2 = struct("hp", "wt", "vs"), v3 = create_array(df$mpg, df$cyl, df$hp), v4 = create_map(lit("x"), lit(1.0), lit("y"), lit(-1.0))) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, r1 = rand(), r2 = rand(10), r3 = randn(), r4 = randn(10)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, mpg_na = otherwise(when(df$mpg > 20, df$mpg), lit(NaN)), mpg2 = ifelse(df$mpg > 20 & df$am > 0, 0, 1), mpg3 = ifelse(df$mpg > 20, df$mpg, 20.0)) head(tmp) tmp <- mutate(tmp, ind_na1 = is.nan(tmp$mpg_na), ind_na2 = isnan(tmp$mpg_na)) head(select(tmp, coalesce(tmp$mpg_na, tmp$mpg))) head(select(tmp, nanvl(tmp$mpg_na, tmp$hp))) ## End(Not run) ## Not run: tmp <- read.text("README.md") head(select(tmp, input_file_name())) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = lit(df$mpg), v2 = lit("x"), v3 = lit("2015-01-01"), v4 = negate(df$mpg), v5 = expr('length(model)'), v6 = greatest(df$vs, df$am), v7 = least(df$vs, df$am), v8 = column("mpg")) head(tmp) ## End(Not run) ## Not run: head(select(df, bitwiseNOT(cast(df$vs, "int")))) ## End(Not run) ## Not run: head(select(df, monotonically_increasing_id())) ## Not run: head(select(df, spark_partition_id())) ## Not run: tmp <- mutate(df, v1 = struct(df$mpg, df$cyl), v2 = struct("hp", "wt", "vs"), v3 = create_array(df$mpg, df$cyl, df$hp), v4 = create_map(lit("x"), lit(1.0), lit("y"), lit(-1.0))) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, r1 = rand(), r2 = rand(10), r3 = randn(), r4 = randn(10)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, mpg_na = otherwise(when(df$mpg > 20, df$mpg), lit(NaN)), mpg2 = ifelse(df$mpg > 20 & df$am > 0, 0, 1), mpg3 = ifelse(df$mpg > 20, df$mpg, 20.0)) head(tmp) tmp <- mutate(tmp, ind_na1 = is.nan(tmp$mpg_na), ind_na2 = isnan(tmp$mpg_na)) head(select(tmp, coalesce(tmp$mpg_na, tmp$mpg))) head(select(tmp, nanvl(tmp$mpg_na, tmp$hp))) ## End(Not run) ## Not run: tmp <- read.text("README.md") head(select(tmp, input_file_name())) ## End(Not run)
String functions defined for Column.
ascii(x) base64(x) concat_ws(sep, x, ...) decode(x, charset) encode(x, charset) format_number(y, x) format_string(format, x, ...) initcap(x) instr(y, x) levenshtein(y, x) locate(substr, str, ...) lower(x) lpad(x, len, pad) ltrim(x, trimString) overlay(x, replace, pos, ...) regexp_extract(x, pattern, idx) regexp_replace(x, pattern, replacement) repeat_string(x, n) rpad(x, len, pad) rtrim(x, trimString) split_string(x, pattern, ...) soundex(x) substring_index(x, delim, count) translate(x, matchingString, replaceString) trim(x, trimString) unbase64(x) upper(x) ## S4 method for signature 'Column' ascii(x) ## S4 method for signature 'Column' base64(x) ## S4 method for signature 'Column,character' decode(x, charset) ## S4 method for signature 'Column,character' encode(x, charset) ## S4 method for signature 'Column' initcap(x) ## S4 method for signature 'Column' length(x) ## S4 method for signature 'Column' lower(x) ## S4 method for signature 'Column,missing' ltrim(x, trimString) ## S4 method for signature 'Column,character' ltrim(x, trimString) ## S4 method for signature 'Column,Column,numericOrColumn' overlay(x, replace, pos, len = -1) ## S4 method for signature 'Column,missing' rtrim(x, trimString) ## S4 method for signature 'Column,character' rtrim(x, trimString) ## S4 method for signature 'Column' soundex(x) ## S4 method for signature 'Column,missing' trim(x, trimString) ## S4 method for signature 'Column,character' trim(x, trimString) ## S4 method for signature 'Column' unbase64(x) ## S4 method for signature 'Column' upper(x) ## S4 method for signature 'Column' levenshtein(y, x) ## S4 method for signature 'Column,character' instr(y, x) ## S4 method for signature 'Column,numeric' format_number(y, x) ## S4 method for signature 'character,Column' concat_ws(sep, x, ...) ## S4 method for signature 'character,Column' format_string(format, x, ...) ## S4 method for signature 'character,Column' locate(substr, str, pos = 1) ## S4 method for signature 'Column,numeric,character' lpad(x, len, pad) ## S4 method for signature 'Column,character,numeric' regexp_extract(x, pattern, idx) ## S4 method for signature 'Column,character,character' regexp_replace(x, pattern, replacement) ## S4 method for signature 'Column,numeric,character' rpad(x, len, pad) ## S4 method for signature 'Column,character,numeric' substring_index(x, delim, count) ## S4 method for signature 'Column,character,character' translate(x, matchingString, replaceString) ## S4 method for signature 'Column,character' split_string(x, pattern, limit = -1) ## S4 method for signature 'Column,numeric' repeat_string(x, n)ascii(x) base64(x) concat_ws(sep, x, ...) decode(x, charset) encode(x, charset) format_number(y, x) format_string(format, x, ...) initcap(x) instr(y, x) levenshtein(y, x) locate(substr, str, ...) lower(x) lpad(x, len, pad) ltrim(x, trimString) overlay(x, replace, pos, ...) regexp_extract(x, pattern, idx) regexp_replace(x, pattern, replacement) repeat_string(x, n) rpad(x, len, pad) rtrim(x, trimString) split_string(x, pattern, ...) soundex(x) substring_index(x, delim, count) translate(x, matchingString, replaceString) trim(x, trimString) unbase64(x) upper(x) ## S4 method for signature 'Column' ascii(x) ## S4 method for signature 'Column' base64(x) ## S4 method for signature 'Column,character' decode(x, charset) ## S4 method for signature 'Column,character' encode(x, charset) ## S4 method for signature 'Column' initcap(x) ## S4 method for signature 'Column' length(x) ## S4 method for signature 'Column' lower(x) ## S4 method for signature 'Column,missing' ltrim(x, trimString) ## S4 method for signature 'Column,character' ltrim(x, trimString) ## S4 method for signature 'Column,Column,numericOrColumn' overlay(x, replace, pos, len = -1) ## S4 method for signature 'Column,missing' rtrim(x, trimString) ## S4 method for signature 'Column,character' rtrim(x, trimString) ## S4 method for signature 'Column' soundex(x) ## S4 method for signature 'Column,missing' trim(x, trimString) ## S4 method for signature 'Column,character' trim(x, trimString) ## S4 method for signature 'Column' unbase64(x) ## S4 method for signature 'Column' upper(x) ## S4 method for signature 'Column' levenshtein(y, x) ## S4 method for signature 'Column,character' instr(y, x) ## S4 method for signature 'Column,numeric' format_number(y, x) ## S4 method for signature 'character,Column' concat_ws(sep, x, ...) ## S4 method for signature 'character,Column' format_string(format, x, ...) ## S4 method for signature 'character,Column' locate(substr, str, pos = 1) ## S4 method for signature 'Column,numeric,character' lpad(x, len, pad) ## S4 method for signature 'Column,character,numeric' regexp_extract(x, pattern, idx) ## S4 method for signature 'Column,character,character' regexp_replace(x, pattern, replacement) ## S4 method for signature 'Column,numeric,character' rpad(x, len, pad) ## S4 method for signature 'Column,character,numeric' substring_index(x, delim, count) ## S4 method for signature 'Column,character,character' translate(x, matchingString, replaceString) ## S4 method for signature 'Column,character' split_string(x, pattern, limit = -1) ## S4 method for signature 'Column,numeric' repeat_string(x, n)
x |
Column to compute on except in the following methods:
|
sep |
separator to use. |
... |
additional Columns. |
charset |
character set to use (one of "US-ASCII", "ISO-8859-1", "UTF-8", "UTF-16BE", "UTF-16LE", "UTF-16"). |
y |
Column to compute on. |
format |
a character object of format strings. |
substr |
a character string to be matched. |
str |
a Column where matches are sought for each entry. |
len |
In
|
pad |
a character string to be padded with. |
trimString |
a character string to trim with |
replace |
a Column with replacement. |
pos |
In
|
pattern |
a regular expression. |
idx |
a group index. |
replacement |
a character string that a matched |
n |
number of repetitions. |
delim |
a delimiter string. |
count |
number of occurrences of |
matchingString |
a source string where each character will be translated. |
replaceString |
a target string where each |
limit |
determines the length of the returned array.
|
ascii: Computes the numeric value of the first character of the string column,
and returns the result as an int column.
base64: Computes the BASE64 encoding of a binary column and returns it as
a string column. This is the reverse of unbase64.
decode: Computes the first argument into a string from a binary using the provided
character set.
encode: Computes the first argument into a binary from a string using the provided
character set.
initcap: Returns a new string column by converting the first letter of
each word to uppercase. Words are delimited by whitespace. For example, "hello world"
will become "Hello World".
length: Computes the character length of a string data or number of bytes
of a binary data. The length of string data includes the trailing spaces.
The length of binary data includes binary zeros.
lower: Converts a string column to lower case.
ltrim: Trims the spaces from left end for the specified string value. Optionally a
trimString can be specified.
overlay: Overlay the specified portion of x with replace,
starting from byte position pos of src and proceeding for
len bytes.
rtrim: Trims the spaces from right end for the specified string value. Optionally a
trimString can be specified.
soundex: Returns the soundex code for the specified expression.
trim: Trims the spaces from both ends for the specified string column. Optionally a
trimString can be specified.
unbase64: Decodes a BASE64 encoded string column and returns it as a binary column.
This is the reverse of base64.
upper: Converts a string column to upper case.
levenshtein: Computes the Levenshtein distance of the two given string columns.
instr: Locates the position of the first occurrence of a substring (x)
in the given string column (y). Returns null if either of the arguments are null.
Note: The position is not zero based, but 1 based index. Returns 0 if the substring
could not be found in the string column.
format_number: Formats numeric column y to a format like '#,###,###.##',
rounded to x decimal places with HALF_EVEN round mode, and returns the result
as a string column.
If x is 0, the result has no decimal point or fractional part.
If x < 0, the result will be null.
concat_ws: Concatenates multiple input string columns together into a single
string column, using the given separator.
format_string: Formats the arguments in printf-style and returns the result
as a string column.
locate: Locates the position of the first occurrence of substr.
Note: The position is not zero based, but 1 based index. Returns 0 if substr
could not be found in str.
lpad: Left-padded with pad to a length of len.
regexp_extract: Extracts a specific idx group identified by a Java regex,
from the specified string column. If the regex did not match, or the specified group did
not match, an empty string is returned.
regexp_replace: Replaces all substrings of the specified string value that
match regexp with rep.
rpad: Right-padded with pad to a length of len.
substring_index: Returns the substring from string (x) before count
occurrences of the delimiter (delim). If count is positive, everything the left of
the final delimiter (counting from left) is returned. If count is negative, every to the
right of the final delimiter (counting from the right) is returned. substring_index
performs a case-sensitive match when searching for the delimiter.
translate: Translates any character in the src by a character in replaceString.
The characters in replaceString is corresponding to the characters in matchingString.
The translate will happen when any character in the string matching with the character
in the matchingString.
split_string: Splits string on regular expression.
Equivalent to split SQL function. Optionally a
limit can be specified
repeat_string: Repeats string n times.
Equivalent to repeat SQL function.
ascii since 1.5.0
base64 since 1.5.0
decode since 1.6.0
encode since 1.6.0
initcap since 1.5.0
length since 1.5.0
lower since 1.4.0
ltrim since 1.5.0
ltrim(Column, character) since 2.3.0
overlay since 3.0.0
rtrim since 1.5.0
rtrim(Column, character) since 2.3.0
soundex since 1.5.0
trim since 1.5.0
trim(Column, character) since 2.3.0
unbase64 since 1.5.0
upper since 1.4.0
levenshtein since 1.5.0
instr since 1.5.0
format_number since 1.5.0
concat_ws since 1.5.0
format_string since 1.5.0
locate since 1.5.0
lpad since 1.5.0
regexp_extract since 1.5.0
regexp_replace since 1.5.0
rpad since 1.5.0
substring_index since 1.5.0
translate since 1.5.0
split_string 2.3.0
repeat_string since 2.3.0
## Not run: # Dataframe used throughout this doc df <- createDataFrame(as.data.frame(Titanic, stringsAsFactors = FALSE)) ## End(Not run) ## Not run: head(select(df, ascii(df$Class), ascii(df$Sex))) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = encode(df$Class, "UTF-8")) str(tmp) tmp2 <- mutate(tmp, s2 = base64(tmp$s1), s3 = decode(tmp$s1, "UTF-8"), s4 = soundex(tmp$Sex)) head(tmp2) head(select(tmp2, unbase64(tmp2$s2))) ## End(Not run) ## Not run: tmp <- mutate(df, sex_lower = lower(df$Sex), age_upper = upper(df$age), sex_age = concat_ws(" ", lower(df$sex), lower(df$age))) head(tmp) tmp2 <- mutate(tmp, s1 = initcap(tmp$sex_lower), s2 = initcap(tmp$sex_age), s3 = reverse(df$Sex)) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, SexLpad = lpad(df$Sex, 6, " "), SexRpad = rpad(df$Sex, 7, " ")) head(select(tmp, length(tmp$Sex), length(tmp$SexLpad), length(tmp$SexRpad))) tmp2 <- mutate(tmp, SexLtrim = ltrim(tmp$SexLpad), SexRtrim = rtrim(tmp$SexRpad), SexTrim = trim(tmp$SexLpad)) head(select(tmp2, length(tmp2$Sex), length(tmp2$SexLtrim), length(tmp2$SexRtrim), length(tmp2$SexTrim))) tmp <- mutate(df, SexLpad = lpad(df$Sex, 6, "xx"), SexRpad = rpad(df$Sex, 7, "xx")) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, d1 = levenshtein(df$Class, df$Sex), d2 = levenshtein(df$Age, df$Sex), d3 = levenshtein(df$Age, df$Age)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = instr(df$Sex, "m"), s2 = instr(df$Sex, "M"), s3 = locate("m", df$Sex), s4 = locate("m", df$Sex, pos = 4)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = df$Freq/3) head(select(tmp, format_number(tmp$v1, 0), format_number(tmp$v1, 2), format_string("%4.2f %s", tmp$v1, tmp$Sex)), 10) ## End(Not run) ## Not run: # concatenate strings tmp <- mutate(df, s1 = concat_ws("_", df$Class, df$Sex), s2 = concat_ws("+", df$Class, df$Sex, df$Age, df$Survived)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = regexp_extract(df$Class, "(\\d+)\\w+", 1), s2 = regexp_extract(df$Sex, "^(\\w)\\w+", 1), s3 = regexp_replace(df$Class, "\\D+", ""), s4 = substring_index(df$Sex, "a", 1), s5 = substring_index(df$Sex, "a", -1), s6 = translate(df$Sex, "ale", ""), s7 = translate(df$Sex, "a", "-")) head(tmp) ## End(Not run) ## Not run: head(select(df, split_string(df$Class, "\\d", 2))) head(select(df, split_string(df$Sex, "a"))) head(select(df, split_string(df$Class, "\\d"))) # This is equivalent to the following SQL expression head(selectExpr(df, "split(Class, '\\\\d')")) ## End(Not run) ## Not run: head(select(df, repeat_string(df$Class, 3))) # This is equivalent to the following SQL expression head(selectExpr(df, "repeat(Class, 3)")) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(as.data.frame(Titanic, stringsAsFactors = FALSE)) ## End(Not run) ## Not run: head(select(df, ascii(df$Class), ascii(df$Sex))) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = encode(df$Class, "UTF-8")) str(tmp) tmp2 <- mutate(tmp, s2 = base64(tmp$s1), s3 = decode(tmp$s1, "UTF-8"), s4 = soundex(tmp$Sex)) head(tmp2) head(select(tmp2, unbase64(tmp2$s2))) ## End(Not run) ## Not run: tmp <- mutate(df, sex_lower = lower(df$Sex), age_upper = upper(df$age), sex_age = concat_ws(" ", lower(df$sex), lower(df$age))) head(tmp) tmp2 <- mutate(tmp, s1 = initcap(tmp$sex_lower), s2 = initcap(tmp$sex_age), s3 = reverse(df$Sex)) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, SexLpad = lpad(df$Sex, 6, " "), SexRpad = rpad(df$Sex, 7, " ")) head(select(tmp, length(tmp$Sex), length(tmp$SexLpad), length(tmp$SexRpad))) tmp2 <- mutate(tmp, SexLtrim = ltrim(tmp$SexLpad), SexRtrim = rtrim(tmp$SexRpad), SexTrim = trim(tmp$SexLpad)) head(select(tmp2, length(tmp2$Sex), length(tmp2$SexLtrim), length(tmp2$SexRtrim), length(tmp2$SexTrim))) tmp <- mutate(df, SexLpad = lpad(df$Sex, 6, "xx"), SexRpad = rpad(df$Sex, 7, "xx")) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, d1 = levenshtein(df$Class, df$Sex), d2 = levenshtein(df$Age, df$Sex), d3 = levenshtein(df$Age, df$Age)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = instr(df$Sex, "m"), s2 = instr(df$Sex, "M"), s3 = locate("m", df$Sex), s4 = locate("m", df$Sex, pos = 4)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, v1 = df$Freq/3) head(select(tmp, format_number(tmp$v1, 0), format_number(tmp$v1, 2), format_string("%4.2f %s", tmp$v1, tmp$Sex)), 10) ## End(Not run) ## Not run: # concatenate strings tmp <- mutate(df, s1 = concat_ws("_", df$Class, df$Sex), s2 = concat_ws("+", df$Class, df$Sex, df$Age, df$Survived)) head(tmp) ## End(Not run) ## Not run: tmp <- mutate(df, s1 = regexp_extract(df$Class, "(\\d+)\\w+", 1), s2 = regexp_extract(df$Sex, "^(\\w)\\w+", 1), s3 = regexp_replace(df$Class, "\\D+", ""), s4 = substring_index(df$Sex, "a", 1), s5 = substring_index(df$Sex, "a", -1), s6 = translate(df$Sex, "ale", ""), s7 = translate(df$Sex, "a", "-")) head(tmp) ## End(Not run) ## Not run: head(select(df, split_string(df$Class, "\\d", 2))) head(select(df, split_string(df$Sex, "a"))) head(select(df, split_string(df$Class, "\\d"))) # This is equivalent to the following SQL expression head(selectExpr(df, "split(Class, '\\\\d')")) ## End(Not run) ## Not run: head(select(df, repeat_string(df$Class, 3))) # This is equivalent to the following SQL expression head(selectExpr(df, "repeat(Class, 3)")) ## End(Not run)
Window functions defined for Column.
cume_dist(x = "missing") dense_rank(x = "missing") lag(x, ...) lead(x, offset, defaultValue = NULL) nth_value(x, offset, ...) ntile(x) percent_rank(x = "missing") rank(x, ...) row_number(x = "missing") ## S4 method for signature 'missing' cume_dist() ## S4 method for signature 'missing' dense_rank() ## S4 method for signature 'characterOrColumn' lag(x, offset = 1, defaultValue = NULL) ## S4 method for signature 'characterOrColumn,numeric' lead(x, offset = 1, defaultValue = NULL) ## S4 method for signature 'characterOrColumn,numeric' nth_value(x, offset, na.rm = FALSE) ## S4 method for signature 'numeric' ntile(x) ## S4 method for signature 'missing' percent_rank() ## S4 method for signature 'missing' rank() ## S4 method for signature 'ANY' rank(x, ...) ## S4 method for signature 'missing' row_number()cume_dist(x = "missing") dense_rank(x = "missing") lag(x, ...) lead(x, offset, defaultValue = NULL) nth_value(x, offset, ...) ntile(x) percent_rank(x = "missing") rank(x, ...) row_number(x = "missing") ## S4 method for signature 'missing' cume_dist() ## S4 method for signature 'missing' dense_rank() ## S4 method for signature 'characterOrColumn' lag(x, offset = 1, defaultValue = NULL) ## S4 method for signature 'characterOrColumn,numeric' lead(x, offset = 1, defaultValue = NULL) ## S4 method for signature 'characterOrColumn,numeric' nth_value(x, offset, na.rm = FALSE) ## S4 method for signature 'numeric' ntile(x) ## S4 method for signature 'missing' percent_rank() ## S4 method for signature 'missing' rank() ## S4 method for signature 'ANY' rank(x, ...) ## S4 method for signature 'missing' row_number()
x |
In |
... |
additional argument(s). |
offset |
a numeric indicating number of row to use as the value |
defaultValue |
(optional) default to use when the offset row does not exist. |
na.rm |
a logical which indicates that the Nth value should skip null in the determination of which row to use |
cume_dist: Returns the cumulative distribution of values within a window partition,
i.e. the fraction of rows that are below the current row:
(number of values before and including x) / (total number of rows in the partition).
This is equivalent to the CUME_DIST function in SQL.
The method should be used with no argument.
dense_rank: Returns the rank of rows within a window partition, without any gaps.
The difference between rank and dense_rank is that dense_rank leaves no gaps in ranking
sequence when there are ties. That is, if you were ranking a competition using dense_rank
and had three people tie for second place, you would say that all three were in second
place and that the next person came in third. Rank would give me sequential numbers, making
the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the DENSE_RANK function in SQL.
The method should be used with no argument.
lag: Returns the value that is offset rows before the current row, and
defaultValue if there is less than offset rows before the current row. For example,
an offset of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
lead: Returns the value that is offset rows after the current row, and
defaultValue if there is less than offset rows after the current row.
For example, an offset of one will return the next row at any given point
in the window partition.
This is equivalent to the LEAD function in SQL.
nth_value: Window function: returns the value that is the offsetth
row of the window frame# (counting from 1), and null if the size of window
frame is less than offset rows.
ntile: Returns the ntile group id (from 1 to n inclusive) in an ordered window
partition. For example, if n is 4, the first quarter of the rows will get value 1, the second
quarter will get 2, the third quarter will get 3, and the last quarter will get 4.
This is equivalent to the NTILE function in SQL.
percent_rank: Returns the relative rank (i.e. percentile) of rows within a window
partition.
This is computed by: (rank of row in its partition - 1) / (number of rows in the partition - 1).
This is equivalent to the PERCENT_RANK function in SQL.
The method should be used with no argument.
rank: Returns the rank of rows within a window partition.
The difference between rank and dense_rank is that dense_rank leaves no gaps in ranking
sequence when there are ties. That is, if you were ranking a competition using dense_rank
and had three people tie for second place, you would say that all three were in second
place and that the next person came in third. Rank would give me sequential numbers, making
the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the RANK function in SQL.
The method should be used with no argument.
row_number: Returns a sequential number starting at 1 within a window partition.
This is equivalent to the ROW_NUMBER function in SQL.
The method should be used with no argument.
cume_dist since 1.6.0
dense_rank since 1.6.0
lag since 1.6.0
lead since 1.6.0
nth_value since 3.1.0
ntile since 1.6.0
percent_rank since 1.6.0
rank since 1.6.0
row_number since 1.6.0
## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ws <- orderBy(windowPartitionBy("am"), "hp") tmp <- mutate(df, dist = over(cume_dist(), ws), dense_rank = over(dense_rank(), ws), lag = over(lag(df$mpg), ws), lead = over(lead(df$mpg, 1), ws), percent_rank = over(percent_rank(), ws), rank = over(rank(), ws), row_number = over(row_number(), ws), nth_value = over(nth_value(df$mpg, 3), ws)) # Get ntile group id (1-4) for hp tmp <- mutate(tmp, ntile = over(ntile(4), ws)) head(tmp) ## End(Not run)## Not run: # Dataframe used throughout this doc df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) ws <- orderBy(windowPartitionBy("am"), "hp") tmp <- mutate(df, dist = over(cume_dist(), ws), dense_rank = over(dense_rank(), ws), lag = over(lag(df$mpg), ws), lead = over(lead(df$mpg, 1), ws), percent_rank = over(percent_rank(), ws), rank = over(rank(), ws), row_number = over(row_number(), ws), nth_value = over(nth_value(df$mpg, 3), ws)) # Get ntile group id (1-4) for hp tmp <- mutate(tmp, ntile = over(ntile(4), ws)) head(tmp) ## End(Not run)
Computes the Pearson Correlation Coefficient for two Columns.
corr(x, ...) ## S4 method for signature 'Column' corr(x, col2) ## S4 method for signature 'SparkDataFrame' corr(x, colName1, colName2, method = "pearson")corr(x, ...) ## S4 method for signature 'Column' corr(x, col2) ## S4 method for signature 'SparkDataFrame' corr(x, colName1, colName2, method = "pearson")
x |
a Column or a SparkDataFrame. |
... |
additional argument(s). If |
col2 |
a (second) Column. |
colName1 |
the name of the first column |
colName2 |
the name of the second column |
method |
Optional. A character specifying the method for calculating the correlation. only "pearson" is allowed now. |
The Pearson Correlation Coefficient as a Double.
corr since 1.6.0
corr since 1.6.0
Other aggregate functions:
avg(),
column_aggregate_functions,
count(),
cov(),
first(),
last()
Other stat functions:
approxQuantile(),
cov(),
crosstab(),
freqItems(),
sampleBy()
## Not run: df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) head(select(df, corr(df$mpg, df$hp))) ## End(Not run) ## Not run: corr(df, "mpg", "hp") corr(df, "mpg", "hp", method = "pearson") ## End(Not run)## Not run: df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) head(select(df, corr(df$mpg, df$hp))) ## End(Not run) ## Not run: corr(df, "mpg", "hp") corr(df, "mpg", "hp", method = "pearson") ## End(Not run)
Count the number of rows for each group when we have GroupedData input.
The resulting SparkDataFrame will also contain the grouping columns.
This can be used as a column aggregate function with Column as input,
and returns the number of items in a group.
count(x) n(x) ## S4 method for signature 'GroupedData' count(x) ## S4 method for signature 'Column' count(x) ## S4 method for signature 'Column' n(x)count(x) n(x) ## S4 method for signature 'GroupedData' count(x) ## S4 method for signature 'Column' count(x) ## S4 method for signature 'Column' n(x)
x |
a GroupedData or Column. |
A SparkDataFrame.
count since 1.4.0
count since 1.4.0
n since 1.4.0
Other aggregate functions:
avg(),
column_aggregate_functions,
corr(),
cov(),
first(),
last()
## Not run: count(groupBy(df, "name")) ## End(Not run) ## Not run: count(df$c) ## Not run: n(df$c)## Not run: count(groupBy(df, "name")) ## End(Not run) ## Not run: count(df$c) ## Not run: n(df$c)
Compute the covariance between two expressions.
cov(x, ...) covar_samp(col1, col2) covar_pop(col1, col2) ## S4 method for signature 'characterOrColumn' cov(x, col2) ## S4 method for signature 'characterOrColumn,characterOrColumn' covar_samp(col1, col2) ## S4 method for signature 'characterOrColumn,characterOrColumn' covar_pop(col1, col2) ## S4 method for signature 'SparkDataFrame' cov(x, colName1, colName2)cov(x, ...) covar_samp(col1, col2) covar_pop(col1, col2) ## S4 method for signature 'characterOrColumn' cov(x, col2) ## S4 method for signature 'characterOrColumn,characterOrColumn' covar_samp(col1, col2) ## S4 method for signature 'characterOrColumn,characterOrColumn' covar_pop(col1, col2) ## S4 method for signature 'SparkDataFrame' cov(x, colName1, colName2)
x |
a Column or a SparkDataFrame. |
... |
additional argument(s). If |
col1 |
the first Column. |
col2 |
the second Column. |
colName1 |
the name of the first column |
colName2 |
the name of the second column |
cov: Compute the sample covariance between two expressions.
covar_sample: Alias for cov.
covar_pop: Computes the population covariance between two expressions.
cov: When applied to SparkDataFrame, this calculates the sample covariance of two
numerical columns of one SparkDataFrame.
The covariance of the two columns.
cov since 1.6.0
covar_samp since 2.0.0
covar_pop since 2.0.0
cov since 1.6.0
Other aggregate functions:
avg(),
column_aggregate_functions,
corr(),
count(),
first(),
last()
Other stat functions:
approxQuantile(),
corr(),
crosstab(),
freqItems(),
sampleBy()
## Not run: df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) head(select(df, cov(df$mpg, df$hp), cov("mpg", "hp"), covar_samp(df$mpg, df$hp), covar_samp("mpg", "hp"), covar_pop(df$mpg, df$hp), covar_pop("mpg", "hp"))) ## End(Not run) ## Not run: cov(df, "mpg", "hp") cov(df, df$mpg, df$hp) ## End(Not run)## Not run: df <- createDataFrame(cbind(model = rownames(mtcars), mtcars)) head(select(df, cov(df$mpg, df$hp), cov("mpg", "hp"), covar_samp(df$mpg, df$hp), covar_samp("mpg", "hp"), covar_pop(df$mpg, df$hp), covar_pop("mpg", "hp"))) ## End(Not run) ## Not run: cov(df, "mpg", "hp") cov(df, df$mpg, df$hp) ## End(Not run)
Create o.a.s.sql.expressions.LambdaFunction corresponding to transformation described by func. Used by higher order functions.
create_lambda(fun)create_lambda(fun)
fun |
R |
JVM LambdaFunction object
Converts R data.frame or list into SparkDataFrame.
createDataFrame(data, schema = NULL, samplingRatio = 1, numPartitions = NULL) as.DataFrame(data, schema = NULL, samplingRatio = 1, numPartitions = NULL)createDataFrame(data, schema = NULL, samplingRatio = 1, numPartitions = NULL) as.DataFrame(data, schema = NULL, samplingRatio = 1, numPartitions = NULL)
data |
a list or data.frame. |
schema |
a list of column names or named list (StructType), optional. |
samplingRatio |
Currently not used. |
numPartitions |
the number of partitions of the SparkDataFrame. Defaults to 1, this is limited by length of the list or number of rows of the data.frame |
A SparkDataFrame.
createDataFrame since 1.4.0
as.DataFrame since 1.6.0
## Not run: sparkR.session() df1 <- as.DataFrame(iris) df2 <- as.DataFrame(list(3,4,5,6)) df3 <- createDataFrame(iris) df4 <- createDataFrame(cars, numPartitions = 2) ## End(Not run)## Not run: sparkR.session() df1 <- as.DataFrame(iris) df2 <- as.DataFrame(list(3,4,5,6)) df3 <- createDataFrame(iris) df4 <- createDataFrame(cars, numPartitions = 2) ## End(Not run)
Creates an external table based on the dataset in a data source, Returns a SparkDataFrame associated with the external table.
createExternalTable(tableName, path = NULL, source = NULL, schema = NULL, ...)createExternalTable(tableName, path = NULL, source = NULL, schema = NULL, ...)
tableName |
a name of the table. |
path |
the path of files to load. |
source |
the name of external data source. |
schema |
the schema of the data required for some data sources. |
... |
additional argument(s) passed to the method. |
The data source is specified by the source and a set of options(...).
If source is not specified, the default data source configured by
"spark.sql.sources.default" will be used.
A SparkDataFrame.
createExternalTable since 1.4.0
## Not run: sparkR.session() df <- createExternalTable("myjson", path="path/to/json", source="json", schema) ## End(Not run)## Not run: sparkR.session() df <- createExternalTable("myjson", path="path/to/json", source="json", schema) ## End(Not run)
Creates a new temporary view using a SparkDataFrame in the Spark Session. If a temporary view with the same name already exists, replaces it.
createOrReplaceTempView(x, viewName) ## S4 method for signature 'SparkDataFrame,character' createOrReplaceTempView(x, viewName)createOrReplaceTempView(x, viewName) ## S4 method for signature 'SparkDataFrame,character' createOrReplaceTempView(x, viewName)
x |
A SparkDataFrame |
viewName |
A character vector containing the name of the table |
createOrReplaceTempView since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "json_df") new_df <- sql("SELECT * FROM json_df") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "json_df") new_df <- sql("SELECT * FROM json_df") ## End(Not run)
Creates a table based on the dataset in a data source. Returns a SparkDataFrame associated with the table.
createTable(tableName, path = NULL, source = NULL, schema = NULL, ...)createTable(tableName, path = NULL, source = NULL, schema = NULL, ...)
tableName |
the qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database. |
path |
(optional) the path of files to load. |
source |
(optional) the name of the data source. |
schema |
(optional) the schema of the data required for some data sources. |
... |
additional named parameters as options for the data source. |
The data source is specified by the source and a set of options(...).
If source is not specified, the default data source configured by
"spark.sql.sources.default" will be used. When a path is specified, an external table is
created from the data at the given path. Otherwise a managed table is created.
A SparkDataFrame.
createTable since 2.2.0
## Not run: sparkR.session() df <- createTable("myjson", path="path/to/json", source="json", schema) createTable("people", source = "json", schema = schema) insertInto(df, "people") ## End(Not run)## Not run: sparkR.session() df <- createTable("myjson", path="path/to/json", source="json", schema) createTable("people", source = "json", schema = schema) insertInto(df, "people") ## End(Not run)
Returns Cartesian Product on two SparkDataFrames.
## S4 method for signature 'SparkDataFrame,SparkDataFrame' crossJoin(x, y)## S4 method for signature 'SparkDataFrame,SparkDataFrame' crossJoin(x, y)
x |
A SparkDataFrame |
y |
A SparkDataFrame |
A SparkDataFrame containing the result of the join operation.
crossJoin since 2.1.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) crossJoin(df1, df2) # Performs a Cartesian ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) crossJoin(df1, df2) # Performs a Cartesian ## End(Not run)
Computes a pair-wise frequency table of the given columns. Also known as a contingency table. The number of distinct values for each column should be less than 1e4. At most 1e6 non-zero pair frequencies will be returned.
## S4 method for signature 'SparkDataFrame,character,character' crosstab(x, col1, col2)## S4 method for signature 'SparkDataFrame,character,character' crosstab(x, col1, col2)
x |
a SparkDataFrame |
col1 |
name of the first column. Distinct items will make the first item of each row. |
col2 |
name of the second column. Distinct items will make the column names of the output. |
a local R data.frame representing the contingency table. The first column of each row
will be the distinct values of col1 and the column names will be the distinct
values of col2. The name of the first column will be "col1_col2".
Pairs that have no occurrences will have zero as their counts.
crosstab since 1.5.0
Other stat functions:
approxQuantile(),
corr(),
cov(),
freqItems(),
sampleBy()
## Not run: df <- read.json("/path/to/file.json") ct <- crosstab(df, "title", "gender") ## End(Not run)## Not run: df <- read.json("/path/to/file.json") ct <- crosstab(df, "title", "gender") ## End(Not run)
Create a multi-dimensional cube for the SparkDataFrame using the specified columns.
cube(x, ...) ## S4 method for signature 'SparkDataFrame' cube(x, ...)cube(x, ...) ## S4 method for signature 'SparkDataFrame' cube(x, ...)
x |
a SparkDataFrame. |
... |
character name(s) or Column(s) to group on. |
If grouping expression is missing cube creates a single global aggregate and is
equivalent to direct application of agg.
A GroupedData.
cube since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(mtcars) mean(cube(df, "cyl", "gear", "am"), "mpg") # Following calls are equivalent agg(cube(df), mean(df$mpg)) agg(df, mean(df$mpg)) ## End(Not run)## Not run: df <- createDataFrame(mtcars) mean(cube(df, "cyl", "gear", "am"), "mpg") # Following calls are equivalent agg(cube(df), mean(df$mpg)) agg(df, mean(df$mpg)) ## End(Not run)
Returns the current default database.
currentDatabase()currentDatabase()
name of the current default database.
since 2.2.0
## Not run: sparkR.session() currentDatabase() ## End(Not run)## Not run: sparkR.session() currentDatabase() ## End(Not run)
Apply a function to each partition of a SparkDataFrame.
dapply(x, func, schema) ## S4 method for signature 'SparkDataFrame,'function',characterOrstructType' dapply(x, func, schema)dapply(x, func, schema) ## S4 method for signature 'SparkDataFrame,'function',characterOrstructType' dapply(x, func, schema)
x |
A SparkDataFrame |
func |
A function to be applied to each partition of the SparkDataFrame. func should have only one parameter, to which a R data.frame corresponds to each partition will be passed. The output of func should be a R data.frame. |
schema |
The schema of the resulting SparkDataFrame after the function is applied. It must match the output of func. Since Spark 2.3, the DDL-formatted string is also supported for the schema. |
dapply since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(iris) df1 <- dapply(df, function(x) { x }, schema(df)) collect(df1) # filter and add a column df <- createDataFrame( list(list(1L, 1, "1"), list(2L, 2, "2"), list(3L, 3, "3")), c("a", "b", "c")) schema <- structType(structField("a", "integer"), structField("b", "double"), structField("c", "string"), structField("d", "integer")) df1 <- dapply( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }, schema) # The schema also can be specified in a DDL-formatted string. schema <- "a INT, d DOUBLE, c STRING, d INT" df1 <- dapply( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }, schema) collect(df1) # the result # a b c d # 1 2 2 2 3 # 2 3 3 3 4 ## End(Not run)## Not run: df <- createDataFrame(iris) df1 <- dapply(df, function(x) { x }, schema(df)) collect(df1) # filter and add a column df <- createDataFrame( list(list(1L, 1, "1"), list(2L, 2, "2"), list(3L, 3, "3")), c("a", "b", "c")) schema <- structType(structField("a", "integer"), structField("b", "double"), structField("c", "string"), structField("d", "integer")) df1 <- dapply( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }, schema) # The schema also can be specified in a DDL-formatted string. schema <- "a INT, d DOUBLE, c STRING, d INT" df1 <- dapply( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }, schema) collect(df1) # the result # a b c d # 1 2 2 2 3 # 2 3 3 3 4 ## End(Not run)
Apply a function to each partition of a SparkDataFrame and collect the result back to R as a data.frame.
dapplyCollect(x, func) ## S4 method for signature 'SparkDataFrame,'function'' dapplyCollect(x, func)dapplyCollect(x, func) ## S4 method for signature 'SparkDataFrame,'function'' dapplyCollect(x, func)
x |
A SparkDataFrame |
func |
A function to be applied to each partition of the SparkDataFrame. func should have only one parameter, to which a R data.frame corresponds to each partition will be passed. The output of func should be a R data.frame. |
dapplyCollect since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(iris) ldf <- dapplyCollect(df, function(x) { x }) # filter and add a column df <- createDataFrame( list(list(1L, 1, "1"), list(2L, 2, "2"), list(3L, 3, "3")), c("a", "b", "c")) ldf <- dapplyCollect( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }) # the result # a b c d # 2 2 2 3 # 3 3 3 4 ## End(Not run)## Not run: df <- createDataFrame(iris) ldf <- dapplyCollect(df, function(x) { x }) # filter and add a column df <- createDataFrame( list(list(1L, 1, "1"), list(2L, 2, "2"), list(3L, 3, "3")), c("a", "b", "c")) ldf <- dapplyCollect( df, function(x) { y <- x[x[1] > 1, ] y <- cbind(y, y[1] + 1L) }) # the result # a b c d # 2 2 2 3 # 3 3 3 4 ## End(Not run)
S4 class that represents a DecisionTreeClassificationModel
jobj |
a Java object reference to the backing Scala DecisionTreeClassificationModel |
DecisionTreeClassificationModel since 2.3.0
S4 class that represents a DecisionTreeRegressionModel
jobj |
a Java object reference to the backing Scala DecisionTreeRegressionModel |
DecisionTreeRegressionModel since 2.3.0
Computes statistics for numeric and string columns. If no columns are given, this function computes statistics for all numerical or string columns.
describe(x, col, ...) ## S4 method for signature 'SparkDataFrame,character' describe(x, col, ...) ## S4 method for signature 'SparkDataFrame,ANY' describe(x)describe(x, col, ...) ## S4 method for signature 'SparkDataFrame,character' describe(x, col, ...) ## S4 method for signature 'SparkDataFrame,ANY' describe(x)
x |
a SparkDataFrame to be computed. |
col |
a string of name. |
... |
additional expressions. |
A SparkDataFrame.
describe(SparkDataFrame, character) since 1.4.0
describe(SparkDataFrame) since 1.4.0
See summary for expanded statistics and control over which statistics to compute.
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) describe(df) describe(df, "col1") describe(df, "col1", "col2") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) describe(df) describe(df, "col1") describe(df, "col1", "col2") ## End(Not run)
Returns the dimensions (number of rows and columns) of a SparkDataFrame
## S4 method for signature 'SparkDataFrame' dim(x)## S4 method for signature 'SparkDataFrame' dim(x)
x |
a SparkDataFrame |
dim since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dim(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dim(df) ## End(Not run)
Return a new SparkDataFrame containing the distinct rows in this SparkDataFrame.
distinct(x) ## S4 method for signature 'SparkDataFrame' distinct(x) ## S4 method for signature 'SparkDataFrame' unique(x)distinct(x) ## S4 method for signature 'SparkDataFrame' distinct(x) ## S4 method for signature 'SparkDataFrame' unique(x)
x |
A SparkDataFrame |
distinct since 1.4.0
unique since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) distinctDF <- distinct(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) distinctDF <- distinct(df) ## End(Not run)
Returns a new SparkDataFrame with columns dropped. This is a no-op if schema doesn't contain column name(s).
drop(x, ...) ## S4 method for signature 'SparkDataFrame' drop(x, col) ## S4 method for signature 'ANY' drop(x)drop(x, ...) ## S4 method for signature 'SparkDataFrame' drop(x, col) ## S4 method for signature 'ANY' drop(x)
x |
a SparkDataFrame. |
... |
further arguments to be passed to or from other methods. |
col |
a character vector of column names or a Column. |
A SparkDataFrame.
drop since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) drop(df, "col1") drop(df, c("col1", "col2")) drop(df, df$col1) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) drop(df, "col1") drop(df, c("col1", "col2")) drop(df, df$col1) ## End(Not run)
Returns a new SparkDataFrame with duplicate rows removed, considering only the subset of columns.
dropDuplicates(x, ...) ## S4 method for signature 'SparkDataFrame' dropDuplicates(x, ...)dropDuplicates(x, ...) ## S4 method for signature 'SparkDataFrame' dropDuplicates(x, ...)
x |
A SparkDataFrame. |
... |
A character vector of column names or string column names. If the first argument contains a character vector, the followings are ignored. |
A SparkDataFrame with duplicate rows removed.
dropDuplicates since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dropDuplicates(df) dropDuplicates(df, "col1", "col2") dropDuplicates(df, c("col1", "col2")) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dropDuplicates(df) dropDuplicates(df, "col1", "col2") dropDuplicates(df, c("col1", "col2")) ## End(Not run)
Drops fields in a struct Column by name.
dropFields(x, ...) ## S4 method for signature 'Column' dropFields(x, ...)dropFields(x, ...) ## S4 method for signature 'Column' dropFields(x, ...)
x |
a Column |
... |
names of the fields to be dropped. |
dropFields since 3.1.0
## Not run: df <- select( createDataFrame(iris), alias( struct( column("Sepal_Width"), column("Sepal_Length"), alias( struct( column("Petal_Width"), column("Petal_Length"), alias( column("Petal_Width") * column("Petal_Length"), "Petal_Product" ) ), "Petal" ) ), "dimensions" ) ) head(withColumn(df, "dimensions", dropFields(df$dimensions, "Petal"))) head( withColumn( df, "dimensions", dropFields(df$dimensions, "Sepal_Width", "Sepal_Length") ) ) # This method supports dropping multiple nested fields directly e.g. head( withColumn( df, "dimensions", dropFields(df$dimensions, "Petal.Petal_Width", "Petal.Petal_Length") ) ) # However, if you are going to add/replace multiple nested fields, # it is preferred to extract out the nested struct before # adding/replacing multiple fields e.g. head( withColumn( df, "dimensions", withField( column("dimensions"), "Petal", dropFields(column("dimensions.Petal"), "Petal_Width", "Petal_Length") ) ) ) ## End(Not run)## Not run: df <- select( createDataFrame(iris), alias( struct( column("Sepal_Width"), column("Sepal_Length"), alias( struct( column("Petal_Width"), column("Petal_Length"), alias( column("Petal_Width") * column("Petal_Length"), "Petal_Product" ) ), "Petal" ) ), "dimensions" ) ) head(withColumn(df, "dimensions", dropFields(df$dimensions, "Petal"))) head( withColumn( df, "dimensions", dropFields(df$dimensions, "Sepal_Width", "Sepal_Length") ) ) # This method supports dropping multiple nested fields directly e.g. head( withColumn( df, "dimensions", dropFields(df$dimensions, "Petal.Petal_Width", "Petal.Petal_Length") ) ) # However, if you are going to add/replace multiple nested fields, # it is preferred to extract out the nested struct before # adding/replacing multiple fields e.g. head( withColumn( df, "dimensions", withField( column("dimensions"), "Petal", dropFields(column("dimensions.Petal"), "Petal_Width", "Petal_Length") ) ) ) ## End(Not run)
dropna, na.omit - Returns a new SparkDataFrame omitting rows with null values.
dropna(x, how = c("any", "all"), minNonNulls = NULL, cols = NULL) na.omit(object, ...) fillna(x, value, cols = NULL) ## S4 method for signature 'SparkDataFrame' dropna(x, how = c("any", "all"), minNonNulls = NULL, cols = NULL) ## S4 method for signature 'SparkDataFrame' na.omit(object, how = c("any", "all"), minNonNulls = NULL, cols = NULL) ## S4 method for signature 'SparkDataFrame' fillna(x, value, cols = NULL)dropna(x, how = c("any", "all"), minNonNulls = NULL, cols = NULL) na.omit(object, ...) fillna(x, value, cols = NULL) ## S4 method for signature 'SparkDataFrame' dropna(x, how = c("any", "all"), minNonNulls = NULL, cols = NULL) ## S4 method for signature 'SparkDataFrame' na.omit(object, how = c("any", "all"), minNonNulls = NULL, cols = NULL) ## S4 method for signature 'SparkDataFrame' fillna(x, value, cols = NULL)
x |
a SparkDataFrame. |
how |
"any" or "all".
if "any", drop a row if it contains any nulls.
if "all", drop a row only if all its values are null.
if |
minNonNulls |
if specified, drop rows that have less than
|
cols |
optional list of column names to consider. In |
object |
a SparkDataFrame. |
... |
further arguments to be passed to or from other methods. |
value |
value to replace null values with. Should be an integer, numeric, character or named list. If the value is a named list, then cols is ignored and value must be a mapping from column name (character) to replacement value. The replacement value must be an integer, numeric or character. |
A SparkDataFrame.
dropna since 1.4.0
na.omit since 1.5.0
fillna since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dropna(df) ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) fillna(df, 1) fillna(df, list("age" = 20, "name" = "unknown")) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dropna(df) ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) fillna(df, 1) fillna(df, list("age" = 20, "name" = "unknown")) ## End(Not run)
Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.
dropTempTable(tableName)dropTempTable(tableName)
tableName |
The name of the SparkSQL table to be dropped. |
dropTempTable since 1.4.0
## Not run: sparkR.session() df <- read.df(path, "parquet") createOrReplaceTempView(df, "table") dropTempTable("table") ## End(Not run)## Not run: sparkR.session() df <- read.df(path, "parquet") createOrReplaceTempView(df, "table") dropTempTable("table") ## End(Not run)
Drops the temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.
dropTempView(viewName)dropTempView(viewName)
viewName |
the name of the temporary view to be dropped. |
TRUE if the view is dropped successfully, FALSE otherwise.
since 2.0.0
## Not run: sparkR.session() df <- read.df(path, "parquet") createOrReplaceTempView(df, "table") dropTempView("table") ## End(Not run)## Not run: sparkR.session() df <- read.df(path, "parquet") createOrReplaceTempView(df, "table") dropTempView("table") ## End(Not run)
Return all column names and their data types as a list
dtypes(x) ## S4 method for signature 'SparkDataFrame' dtypes(x)dtypes(x) ## S4 method for signature 'SparkDataFrame' dtypes(x)
x |
A SparkDataFrame |
dtypes since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dtypes(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dtypes(df) ## End(Not run)
Determines if entries of x end with string (entries of) suffix respectively, where strings are recycled to common lengths.
endsWith(x, suffix) ## S4 method for signature 'Column' endsWith(x, suffix)endsWith(x, suffix) ## S4 method for signature 'Column' endsWith(x, suffix)
x |
vector of character string whose "ends" are considered |
suffix |
character vector (often of length one) |
endsWith since 1.4.0
Other column_func:
alias(),
between(),
cast(),
otherwise(),
over(),
startsWith(),
substr()
Return a new SparkDataFrame containing rows in this SparkDataFrame
but not in another SparkDataFrame. This is equivalent to EXCEPT DISTINCT in SQL.
except(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' except(x, y)except(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' except(x, y)
x |
a SparkDataFrame. |
y |
a SparkDataFrame. |
A SparkDataFrame containing the result of the except operation.
except since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) exceptDF <- except(df, df2) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) exceptDF <- except(df, df2) ## End(Not run)
Return a new SparkDataFrame containing rows in this SparkDataFrame
but not in another SparkDataFrame while preserving the duplicates.
This is equivalent to EXCEPT ALL in SQL. Also as standard in
SQL, this function resolves columns by position (not by name).
exceptAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' exceptAll(x, y)exceptAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' exceptAll(x, y)
x |
a SparkDataFrame. |
y |
a SparkDataFrame. |
A SparkDataFrame containing the result of the except all operation.
exceptAll since 2.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) exceptAllDF <- exceptAll(df1, df2) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) exceptAllDF <- exceptAll(df1, df2) ## End(Not run)
Print the logical and physical Catalyst plans to the console for debugging.
explain(x, ...) ## S4 method for signature 'SparkDataFrame' explain(x, extended = FALSE) ## S4 method for signature 'StreamingQuery' explain(x, extended = FALSE)explain(x, ...) ## S4 method for signature 'SparkDataFrame' explain(x, extended = FALSE) ## S4 method for signature 'StreamingQuery' explain(x, extended = FALSE)
x |
a SparkDataFrame or a StreamingQuery. |
... |
further arguments to be passed to or from other methods. |
extended |
Logical. If extended is FALSE, prints only the physical plan. |
explain since 1.4.0
explain(StreamingQuery) since 2.2.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
Other StreamingQuery methods:
awaitTermination(),
isActive(),
lastProgress(),
queryName(),
status(),
stopQuery()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) explain(df, TRUE) ## End(Not run) ## Not run: explain(sq)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) explain(df, TRUE) ## End(Not run) ## Not run: explain(sq)
Filter the rows of a SparkDataFrame according to a given condition.
filter(x, condition) where(x, condition) ## S4 method for signature 'SparkDataFrame,characterOrColumn' filter(x, condition) ## S4 method for signature 'SparkDataFrame,characterOrColumn' where(x, condition)filter(x, condition) where(x, condition) ## S4 method for signature 'SparkDataFrame,characterOrColumn' filter(x, condition) ## S4 method for signature 'SparkDataFrame,characterOrColumn' where(x, condition)
x |
A SparkDataFrame to be sorted. |
condition |
The condition to filter on. This may either be a Column expression or a string containing a SQL statement |
A SparkDataFrame containing only the rows that meet the condition.
filter since 1.4.0
where since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
Other subsetting functions:
select(),
subset()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) filter(df, "col1 > 0") filter(df, df$col2 != "abcdefg") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) filter(df, "col1 > 0") filter(df, df$col2 != "abcdefg") ## End(Not run)
Aggregate function: returns the first value in a group.
first(x, ...) ## S4 method for signature 'SparkDataFrame' first(x) ## S4 method for signature 'characterOrColumn' first(x, na.rm = FALSE)first(x, ...) ## S4 method for signature 'SparkDataFrame' first(x) ## S4 method for signature 'characterOrColumn' first(x, na.rm = FALSE)
x |
a SparkDataFrame or a column used in aggregation function. |
... |
further arguments to be passed to or from other methods. |
na.rm |
a logical value indicating whether NA values should be stripped before the computation proceeds. |
The function by default returns the first values it sees. It will return the first non-missing value it sees when na.rm is set to true. If all values are missing, then NA is returned. Note: the function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
first(SparkDataFrame) since 1.4.0
first(characterOrColumn) since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
Other aggregate functions:
avg(),
column_aggregate_functions,
corr(),
count(),
cov(),
last()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) first(df) ## End(Not run) ## Not run: first(df$c) first(df$c, TRUE) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) first(df) ## End(Not run) ## Not run: first(df$c) first(df$c, TRUE) ## End(Not run)
Get fitted result from a k-means model, similarly to R's fitted(). Note: A saved-loaded model does not support this method.
fitted(object, ...) ## S4 method for signature 'KMeansModel' fitted(object, method = c("centers", "classes"))fitted(object, ...) ## S4 method for signature 'KMeansModel' fitted(object, method = c("centers", "classes"))
object |
a fitted k-means model. |
... |
additional argument(s) passed to the method. |
method |
type of fitted results, |
fitted returns a SparkDataFrame containing fitted values.
fitted since 2.0.0
## Not run: model <- spark.kmeans(trainingData, ~ ., 2) fitted.model <- fitted(model) showDF(fitted.model) ## End(Not run)## Not run: model <- spark.kmeans(trainingData, ~ ., 2) fitted.model <- fitted(model) showDF(fitted.model) ## End(Not run)
S4 class that represents a FMClassificationModel
jobj |
a Java object reference to the backing Scala FMClassifierWrapper |
FMClassificationModel since 3.1.0
S4 class that represents a FMRegressionModel
jobj |
a Java object reference to the backing Scala FMRegressorWrapper |
FMRegressionModel since 3.1.0
S4 class that represents a FPGrowthModel
jobj |
a Java object reference to the backing Scala FPGrowthModel |
FPGrowthModel since 2.2.0
Finding frequent items for columns, possibly with false positives. Using the frequent element count algorithm described in https://dl.acm.org/doi/10.1145/762471.762473, proposed by Karp, Schenker, and Papadimitriou.
## S4 method for signature 'SparkDataFrame,character' freqItems(x, cols, support = 0.01)## S4 method for signature 'SparkDataFrame,character' freqItems(x, cols, support = 0.01)
x |
A SparkDataFrame. |
cols |
A vector column names to search frequent items in. |
support |
(Optional) The minimum frequency for an item to be considered |
a local R data.frame with the frequent items in each column
freqItems since 1.6.0
Other stat functions:
approxQuantile(),
corr(),
cov(),
crosstab(),
sampleBy()
## Not run: df <- read.json("/path/to/file.json") fi = freqItems(df, c("title", "gender")) ## End(Not run)## Not run: df <- read.json("/path/to/file.json") fi = freqItems(df, c("title", "gender")) ## End(Not run)
Groups the SparkDataFrame using the specified columns and applies the R function to each group.
gapply(x, ...) ## S4 method for signature 'GroupedData' gapply(x, func, schema) ## S4 method for signature 'SparkDataFrame' gapply(x, cols, func, schema)gapply(x, ...) ## S4 method for signature 'GroupedData' gapply(x, func, schema) ## S4 method for signature 'SparkDataFrame' gapply(x, cols, func, schema)
x |
a SparkDataFrame or GroupedData. |
... |
additional argument(s) passed to the method. |
func |
a function to be applied to each group partition specified by grouping column of the SparkDataFrame. See Details. |
schema |
the schema of the resulting SparkDataFrame after the function is applied.
The schema must match to output of |
cols |
grouping columns. |
func is a function of two arguments. The first, usually named key
(though this is not enforced) corresponds to the grouping key, will be an
unnamed list of length(cols) length-one objects corresponding
to the grouping columns' values for the current group.
The second, herein x, will be a local data.frame with the
columns of the input not in cols for the rows corresponding to key.
The output of func must be a data.frame matching schema –
in particular this means the names of the output data.frame are irrelevant
A SparkDataFrame.
gapply(GroupedData) since 2.0.0
gapply(SparkDataFrame) since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: # Computes the arithmetic mean of the second column by grouping # on the first and third columns. Output the grouping values and the average. df <- createDataFrame ( list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), c("a", "b", "c", "d")) # Here our output contains three columns, the key which is a combination of two # columns with data types integer and string and the mean which is a double. schema <- structType(structField("a", "integer"), structField("c", "string"), structField("avg", "double")) result <- gapply( df, c("a", "c"), function(key, x) { # key will either be list(1L, '1') (for the group where a=1L,c='1') or # list(3L, '3') (for the group where a=3L,c='3') y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) # The schema also can be specified in a DDL-formatted string. schema <- "a INT, c STRING, avg DOUBLE" result <- gapply( df, c("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) # We can also group the data and afterwards call gapply on GroupedData. # For example: gdf <- group_by(df, "a", "c") result <- gapply( gdf, function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) collect(result) # Result # ------ # a c avg # 3 3 3.0 # 1 1 1.5 # Fits linear models on iris dataset by grouping on the 'Species' column and # using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' # and 'Petal_Width' as training features. df <- createDataFrame (iris) schema <- structType(structField("(Intercept)", "double"), structField("Sepal_Width", "double"),structField("Petal_Length", "double"), structField("Petal_Width", "double")) df1 <- gapply( df, df$"Species", function(key, x) { m <- suppressWarnings(lm(Sepal_Length ~ Sepal_Width + Petal_Length + Petal_Width, x)) data.frame(t(coef(m))) }, schema) collect(df1) # Result # --------- # Model (Intercept) Sepal_Width Petal_Length Petal_Width # 1 0.699883 0.3303370 0.9455356 -0.1697527 # 2 1.895540 0.3868576 0.9083370 -0.6792238 # 3 2.351890 0.6548350 0.2375602 0.2521257 ## End(Not run)## Not run: # Computes the arithmetic mean of the second column by grouping # on the first and third columns. Output the grouping values and the average. df <- createDataFrame ( list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), c("a", "b", "c", "d")) # Here our output contains three columns, the key which is a combination of two # columns with data types integer and string and the mean which is a double. schema <- structType(structField("a", "integer"), structField("c", "string"), structField("avg", "double")) result <- gapply( df, c("a", "c"), function(key, x) { # key will either be list(1L, '1') (for the group where a=1L,c='1') or # list(3L, '3') (for the group where a=3L,c='3') y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) # The schema also can be specified in a DDL-formatted string. schema <- "a INT, c STRING, avg DOUBLE" result <- gapply( df, c("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) # We can also group the data and afterwards call gapply on GroupedData. # For example: gdf <- group_by(df, "a", "c") result <- gapply( gdf, function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) collect(result) # Result # ------ # a c avg # 3 3 3.0 # 1 1 1.5 # Fits linear models on iris dataset by grouping on the 'Species' column and # using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' # and 'Petal_Width' as training features. df <- createDataFrame (iris) schema <- structType(structField("(Intercept)", "double"), structField("Sepal_Width", "double"),structField("Petal_Length", "double"), structField("Petal_Width", "double")) df1 <- gapply( df, df$"Species", function(key, x) { m <- suppressWarnings(lm(Sepal_Length ~ Sepal_Width + Petal_Length + Petal_Width, x)) data.frame(t(coef(m))) }, schema) collect(df1) # Result # --------- # Model (Intercept) Sepal_Width Petal_Length Petal_Width # 1 0.699883 0.3303370 0.9455356 -0.1697527 # 2 1.895540 0.3868576 0.9083370 -0.6792238 # 3 2.351890 0.6548350 0.2375602 0.2521257 ## End(Not run)
Groups the SparkDataFrame using the specified columns, applies the R function to each group and collects the result back to R as data.frame.
gapplyCollect(x, ...) ## S4 method for signature 'GroupedData' gapplyCollect(x, func) ## S4 method for signature 'SparkDataFrame' gapplyCollect(x, cols, func)gapplyCollect(x, ...) ## S4 method for signature 'GroupedData' gapplyCollect(x, func) ## S4 method for signature 'SparkDataFrame' gapplyCollect(x, cols, func)
x |
a SparkDataFrame or GroupedData. |
... |
additional argument(s) passed to the method. |
func |
a function to be applied to each group partition specified by grouping column of the SparkDataFrame. See Details. |
cols |
grouping columns. |
func is a function of two arguments. The first, usually named key
(though this is not enforced) corresponds to the grouping key, will be an
unnamed list of length(cols) length-one objects corresponding
to the grouping columns' values for the current group.
The second, herein x, will be a local data.frame with the
columns of the input not in cols for the rows corresponding to key.
The output of func must be a data.frame matching schema –
in particular this means the names of the output data.frame are irrelevant
A data.frame.
gapplyCollect(GroupedData) since 2.0.0
gapplyCollect(SparkDataFrame) since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: # Computes the arithmetic mean of the second column by grouping # on the first and third columns. Output the grouping values and the average. df <- createDataFrame ( list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), c("a", "b", "c", "d")) result <- gapplyCollect( df, c("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) colnames(y) <- c("key_a", "key_c", "mean_b") y }) # We can also group the data and afterwards call gapply on GroupedData. # For example: gdf <- group_by(df, "a", "c") result <- gapplyCollect( gdf, function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) colnames(y) <- c("key_a", "key_c", "mean_b") y }) # Result # ------ # key_a key_c mean_b # 3 3 3.0 # 1 1 1.5 # Fits linear models on iris dataset by grouping on the 'Species' column and # using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' # and 'Petal_Width' as training features. df <- createDataFrame (iris) result <- gapplyCollect( df, df$"Species", function(key, x) { m <- suppressWarnings(lm(Sepal_Length ~ Sepal_Width + Petal_Length + Petal_Width, x)) data.frame(t(coef(m))) }) # Result # --------- # Model X.Intercept. Sepal_Width Petal_Length Petal_Width # 1 0.699883 0.3303370 0.9455356 -0.1697527 # 2 1.895540 0.3868576 0.9083370 -0.6792238 # 3 2.351890 0.6548350 0.2375602 0.2521257 ## End(Not run)## Not run: # Computes the arithmetic mean of the second column by grouping # on the first and third columns. Output the grouping values and the average. df <- createDataFrame ( list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), c("a", "b", "c", "d")) result <- gapplyCollect( df, c("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) colnames(y) <- c("key_a", "key_c", "mean_b") y }) # We can also group the data and afterwards call gapply on GroupedData. # For example: gdf <- group_by(df, "a", "c") result <- gapplyCollect( gdf, function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) colnames(y) <- c("key_a", "key_c", "mean_b") y }) # Result # ------ # key_a key_c mean_b # 3 3 3.0 # 1 1 1.5 # Fits linear models on iris dataset by grouping on the 'Species' column and # using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' # and 'Petal_Width' as training features. df <- createDataFrame (iris) result <- gapplyCollect( df, df$"Species", function(key, x) { m <- suppressWarnings(lm(Sepal_Length ~ Sepal_Width + Petal_Length + Petal_Width, x)) data.frame(t(coef(m))) }) # Result # --------- # Model X.Intercept. Sepal_Width Petal_Length Petal_Width # 1 0.699883 0.3303370 0.9455356 -0.1697527 # 2 1.895540 0.3868576 0.9083370 -0.6792238 # 3 2.351890 0.6548350 0.2375602 0.2521257 ## End(Not run)
S4 class that represents a GaussianMixtureModel
jobj |
a Java object reference to the backing Scala GaussianMixtureModel |
GaussianMixtureModel since 2.1.0
S4 class that represents a GBTClassificationModel
jobj |
a Java object reference to the backing Scala GBTClassificationModel |
GBTClassificationModel since 2.1.0
S4 class that represents a GBTRegressionModel
jobj |
a Java object reference to the backing Scala GBTRegressionModel |
GBTRegressionModel since 2.1.0
S4 class that represents a generalized linear model
jobj |
a Java object reference to the backing Scala GeneralizedLinearRegressionWrapper |
GeneralizedLinearRegressionModel since 2.0.0
NULL if it is missing. See
setLocalProperty.Get a local property set in this thread, or NULL if it is missing. See
setLocalProperty.
getLocalProperty(key)getLocalProperty(key)
key |
The key for a local property. |
getLocalProperty since 2.3.0
## Not run: getLocalProperty("spark.scheduler.pool") ## End(Not run)## Not run: getLocalProperty("spark.scheduler.pool") ## End(Not run)
Return the number of partitions
## S4 method for signature 'SparkDataFrame' getNumPartitions(x)## S4 method for signature 'SparkDataFrame' getNumPartitions(x)
x |
A SparkDataFrame |
getNumPartitions since 2.1.1
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df <- createDataFrame(cars, numPartitions = 2) getNumPartitions(df) ## End(Not run)## Not run: sparkR.session() df <- createDataFrame(cars, numPartitions = 2) getNumPartitions(df) ## End(Not run)
Fits a generalized linear model, similarly to R's glm().
## S4 method for signature 'formula,ANY,SparkDataFrame' glm( formula, family = gaussian, data, epsilon = 1e-06, maxit = 25, weightCol = NULL, var.power = 0, link.power = 1 - var.power, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc"), offsetCol = NULL )## S4 method for signature 'formula,ANY,SparkDataFrame' glm( formula, family = gaussian, data, epsilon = 1e-06, maxit = 25, weightCol = NULL, var.power = 0, link.power = 1 - var.power, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc"), offsetCol = NULL )
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
family |
a description of the error distribution and link function to be used in the model.
This can be a character string naming a family function, a family function or
the result of a call to a family function. Refer R family at
https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html.
Currently these families are supported: |
data |
a SparkDataFrame or R's glm data for training. |
epsilon |
positive convergence tolerance of iterations. |
maxit |
integer giving the maximal number of IRLS iterations. |
weightCol |
the weight column name. If this is not set or |
var.power |
the index of the power variance function in the Tweedie family. |
link.power |
the index of the power link function in the Tweedie family. |
stringIndexerOrderType |
how to order categories of a string feature column. This is used to decide the base level of a string feature as the last category after ordering is dropped when encoding strings. Supported options are "frequencyDesc", "frequencyAsc", "alphabetDesc", and "alphabetAsc". The default value is "frequencyDesc". When the ordering is set to "alphabetDesc", this drops the same category as R when encoding strings. |
offsetCol |
the offset column name. If this is not set or empty, we treat all instance offsets as 0.0. The feature specified as offset has a constant coefficient of 1.0. |
glm returns a fitted generalized linear model.
glm since 1.5.0
## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- glm(Freq ~ Sex + Age, df, family = "gaussian") summary(model) ## End(Not run)## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- glm(Freq ~ Sex + Age, df, family = "gaussian") summary(model) ## End(Not run)
Groups the SparkDataFrame using the specified columns, so we can run aggregation on them.
group_by(x, ...) groupBy(x, ...) ## S4 method for signature 'SparkDataFrame' groupBy(x, ...) ## S4 method for signature 'SparkDataFrame' group_by(x, ...)group_by(x, ...) groupBy(x, ...) ## S4 method for signature 'SparkDataFrame' groupBy(x, ...) ## S4 method for signature 'SparkDataFrame' group_by(x, ...)
x |
a SparkDataFrame. |
... |
character name(s) or Column(s) to group on. |
A GroupedData.
groupBy since 1.4.0
group_by since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: # Compute the average for all numeric columns grouped by department. avg(groupBy(df, "department")) # Compute the max age and average salary, grouped by department and gender. agg(groupBy(df, "department", "gender"), salary="avg", "age" -> "max") ## End(Not run)## Not run: # Compute the average for all numeric columns grouped by department. avg(groupBy(df, "department")) # Compute the max age and average salary, grouped by department and gender. agg(groupBy(df, "department", "gender"), salary="avg", "age" -> "max") ## End(Not run)
GroupedDatas can be created using groupBy() on a SparkDataFrame
groupedData(sgd)groupedData(sgd)
sgd |
A Java object reference to the backing Scala GroupedData |
GroupedData since 1.4.0
groupBy
Java-style function to compute the hashCode for the given object. Returns an integer value.
hashCode(key)hashCode(key)
key |
the object to be hashed |
This only works for integer, numeric and character types right now.
the hash code as an integer
hashCode since 1.4.0
## Not run: hashCode(1L) # 1 hashCode(1.0) # 1072693248 hashCode("1") # 49 ## End(Not run)## Not run: hashCode(1L) # 1 hashCode(1.0) # 1072693248 hashCode("1") # 49 ## End(Not run)
Return the first num rows of a SparkDataFrame as a R data.frame. If num is not
specified, then head() returns the first 6 rows as with R data.frame.
## S4 method for signature 'SparkDataFrame' head(x, num = 6L)## S4 method for signature 'SparkDataFrame' head(x, num = 6L)
x |
a SparkDataFrame. |
num |
the number of rows to return. Default is 6. |
A data.frame.
head since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) head(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) head(df) ## End(Not run)
Specifies execution plan hint and return a new SparkDataFrame.
hint(x, name, ...) ## S4 method for signature 'SparkDataFrame,character' hint(x, name, ...)hint(x, name, ...) ## S4 method for signature 'SparkDataFrame,character' hint(x, name, ...)
x |
a SparkDataFrame. |
name |
a name of the hint. |
... |
optional parameters for the hint. |
A SparkDataFrame.
hint since 2.2.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(mtcars) avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg") head(join(df, hint(avg_mpg, "broadcast"), df$cyl == avg_mpg$cyl)) ## End(Not run)## Not run: df <- createDataFrame(mtcars) avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg") head(join(df, hint(avg_mpg, "broadcast"), df$cyl == avg_mpg$cyl)) ## End(Not run)
This function computes a histogram for a given SparkR Column.
## S4 method for signature 'SparkDataFrame,characterOrColumn' histogram(df, col, nbins = 10)## S4 method for signature 'SparkDataFrame,characterOrColumn' histogram(df, col, nbins = 10)
df |
the SparkDataFrame containing the Column to build the histogram from. |
col |
the column as Character string or a Column to build the histogram from. |
nbins |
the number of bins (optional). Default value is 10. |
a data.frame with the histogram statistics, i.e., counts and centroids.
histogram since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: # Create a SparkDataFrame from the Iris dataset irisDF <- createDataFrame(iris) # Compute histogram statistics histStats <- histogram(irisDF, irisDF$Sepal_Length, nbins = 12) # Once SparkR has computed the histogram statistics, the histogram can be # rendered using the ggplot2 library: require(ggplot2) plot <- ggplot(histStats, aes(x = centroids, y = counts)) + geom_bar(stat = "identity") + xlab("Sepal_Length") + ylab("Frequency") ## End(Not run)## Not run: # Create a SparkDataFrame from the Iris dataset irisDF <- createDataFrame(iris) # Compute histogram statistics histStats <- histogram(irisDF, irisDF$Sepal_Length, nbins = 12) # Once SparkR has computed the histogram statistics, the histogram can be # rendered using the ggplot2 library: require(ggplot2) plot <- ggplot(histStats, aes(x = centroids, y = counts)) + geom_bar(stat = "identity") + xlab("Sepal_Length") + ylab("Frequency") ## End(Not run)
Insert the contents of a SparkDataFrame into a table registered in the current SparkSession.
insertInto(x, tableName, ...) ## S4 method for signature 'SparkDataFrame,character' insertInto(x, tableName, overwrite = FALSE)insertInto(x, tableName, ...) ## S4 method for signature 'SparkDataFrame,character' insertInto(x, tableName, overwrite = FALSE)
x |
a SparkDataFrame. |
tableName |
a character vector containing the name of the table. |
... |
further arguments to be passed to or from other methods. the existing rows in the table. |
overwrite |
a logical argument indicating whether or not to overwrite. |
insertInto since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df <- read.df(path, "parquet") df2 <- read.df(path2, "parquet") saveAsTable(df, "table1") insertInto(df2, "table1", overwrite = TRUE) ## End(Not run)## Not run: sparkR.session() df <- read.df(path, "parquet") df2 <- read.df(path2, "parquet") saveAsTable(df, "table1") insertInto(df2, "table1", overwrite = TRUE) ## End(Not run)
install.spark downloads and installs Spark to a local directory if
it is not found. If SPARK_HOME is set in the environment, and that directory is found, that is
returned. The Spark version we use is the same as the SparkR version. Users can specify a desired
Hadoop version, the remote mirror site, and the directory where the package is installed locally.
install.spark( hadoopVersion = "2.7", mirrorUrl = NULL, localDir = NULL, overwrite = FALSE )install.spark( hadoopVersion = "2.7", mirrorUrl = NULL, localDir = NULL, overwrite = FALSE )
hadoopVersion |
Version of Hadoop to install. Default is |
mirrorUrl |
base URL of the repositories to use. The directory layout should follow Apache mirrors. |
localDir |
a local directory where Spark is installed. The directory contains version-specific folders of Spark packages. Default is path to the cache directory:
|
overwrite |
If |
The full url of remote file is inferred from mirrorUrl and hadoopVersion.
mirrorUrl specifies the remote path to a Spark folder. It is followed by a subfolder
named after the Spark version (that corresponds to SparkR), and then the tar filename.
The filename is composed of four parts, i.e. [Spark version]-bin-[Hadoop version].tgz.
For example, the full path for a Spark 2.0.0 package for Hadoop 2.7 from
http://apache.osuosl.org has path:
http://apache.osuosl.org/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz.
For hadoopVersion = "without", [Hadoop version] in the filename is then
without-hadoop.
the (invisible) local directory where Spark is found or installed
install.spark since 2.1.0
See available Hadoop versions: Apache Spark
## Not run: install.spark() ## End(Not run)## Not run: install.spark() ## End(Not run)
Return a new SparkDataFrame containing rows only in both this SparkDataFrame
and another SparkDataFrame. This is equivalent to INTERSECT in SQL.
intersect(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' intersect(x, y)intersect(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' intersect(x, y)
x |
A SparkDataFrame |
y |
A SparkDataFrame |
A SparkDataFrame containing the result of the intersect.
intersect since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) intersectDF <- intersect(df, df2) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) intersectDF <- intersect(df, df2) ## End(Not run)
Return a new SparkDataFrame containing rows in both this SparkDataFrame
and another SparkDataFrame while preserving the duplicates.
This is equivalent to INTERSECT ALL in SQL. Also as standard in
SQL, this function resolves columns by position (not by name).
intersectAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' intersectAll(x, y)intersectAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' intersectAll(x, y)
x |
a SparkDataFrame. |
y |
a SparkDataFrame. |
A SparkDataFrame containing the result of the intersect all operation.
intersectAll since 2.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) intersectAllDF <- intersectAll(df1, df2) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) intersectAllDF <- intersectAll(df1, df2) ## End(Not run)
Invokes higher order function expression identified by name, (relative to o.a.s.sql.catalyst.expressions)
invoke_higher_order_function(name, cols, funs)invoke_higher_order_function(name, cols, funs)
name |
character |
cols |
list of character or Column objects |
funs |
list of named list(fun = ..., expected_narg = ...) |
a Column representing name applied to cols with funs
Returns TRUE if this query is actively running.
isActive(x) ## S4 method for signature 'StreamingQuery' isActive(x)isActive(x) ## S4 method for signature 'StreamingQuery' isActive(x)
x |
a StreamingQuery. |
TRUE if query is actively running, FALSE if stopped.
isActive(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
awaitTermination(),
explain(),
lastProgress(),
queryName(),
status(),
stopQuery()
## Not run: isActive(sq)## Not run: isActive(sq)
Returns True if the collect and take methods can be run locally
(without any Spark executors).
isLocal(x) ## S4 method for signature 'SparkDataFrame' isLocal(x)isLocal(x) ## S4 method for signature 'SparkDataFrame' isLocal(x)
x |
A SparkDataFrame |
isLocal since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) isLocal(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) isLocal(df) ## End(Not run)
S4 class that represents an IsotonicRegressionModel
jobj |
a Java object reference to the backing Scala IsotonicRegressionModel |
IsotonicRegressionModel since 2.1.0
Returns TRUE if this SparkDataFrame contains one or more sources that continuously return data
as it arrives. A dataset that reads data from a streaming source must be executed as a
StreamingQuery using write.stream.
isStreaming(x) ## S4 method for signature 'SparkDataFrame' isStreaming(x)isStreaming(x) ## S4 method for signature 'SparkDataFrame' isStreaming(x)
x |
A SparkDataFrame |
TRUE if this SparkDataFrame is from a streaming source
isStreaming since 2.2.0
experimental
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) isStreaming(df) ## End(Not run)## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) isStreaming(df) ## End(Not run)
Joins two SparkDataFrames based on the given join expression.
## S4 method for signature 'SparkDataFrame,SparkDataFrame' join(x, y, joinExpr = NULL, joinType = NULL)## S4 method for signature 'SparkDataFrame,SparkDataFrame' join(x, y, joinExpr = NULL, joinType = NULL)
x |
A SparkDataFrame |
y |
A SparkDataFrame |
joinExpr |
(Optional) The expression used to perform the join. joinExpr must be a Column expression. If joinExpr is omitted, the default, inner join is attempted and an error is thrown if it would be a Cartesian Product. For Cartesian join, use crossJoin instead. |
joinType |
The type of join to perform, default 'inner'. Must be one of: 'inner', 'cross', 'outer', 'full', 'fullouter', 'full_outer', 'left', 'leftouter', 'left_outer', 'right', 'rightouter', 'right_outer', 'semi', 'leftsemi', 'left_semi', 'anti', 'leftanti', 'left_anti'. |
A SparkDataFrame containing the result of the join operation.
join since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) join(df1, df2, df1$col1 == df2$col2) # Performs an inner join based on expression join(df1, df2, df1$col1 == df2$col2, "right_outer") join(df1, df2) # Attempts an inner join ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) join(df1, df2, df1$col1 == df2$col2) # Performs an inner join based on expression join(df1, df2, df1$col1 == df2$col2, "right_outer") join(df1, df2) # Attempts an inner join ## End(Not run)
S4 class that represents a KMeansModel
jobj |
a Java object reference to the backing Scala KMeansModel |
KMeansModel since 2.0.0
S4 class that represents an KSTest
jobj |
a Java object reference to the backing Scala KSTestWrapper |
KSTest since 2.1.0
Aggregate function: returns the last value in a group.
last(x, ...) ## S4 method for signature 'characterOrColumn' last(x, na.rm = FALSE)last(x, ...) ## S4 method for signature 'characterOrColumn' last(x, na.rm = FALSE)
x |
column to compute on. |
... |
further arguments to be passed to or from other methods. |
na.rm |
a logical value indicating whether NA values should be stripped before the computation proceeds. |
The function by default returns the last values it sees. It will return the last non-missing value it sees when na.rm is set to true. If all values are missing, then NA is returned. Note: the function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
last since 1.4.0
Other aggregate functions:
avg(),
column_aggregate_functions,
corr(),
count(),
cov(),
first()
## Not run: last(df$c) last(df$c, TRUE) ## End(Not run)## Not run: last(df$c) last(df$c, TRUE) ## End(Not run)
Prints the most recent progress update of this streaming query in JSON format.
lastProgress(x) ## S4 method for signature 'StreamingQuery' lastProgress(x)lastProgress(x) ## S4 method for signature 'StreamingQuery' lastProgress(x)
x |
a StreamingQuery. |
lastProgress(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
awaitTermination(),
explain(),
isActive(),
queryName(),
status(),
stopQuery()
## Not run: lastProgress(sq)## Not run: lastProgress(sq)
S4 class that represents an LDAModel
jobj |
a Java object reference to the backing Scala LDAWrapper |
LDAModel since 2.1.0
Limit the resulting SparkDataFrame to the number of rows specified.
limit(x, num) ## S4 method for signature 'SparkDataFrame,numeric' limit(x, num)limit(x, num) ## S4 method for signature 'SparkDataFrame,numeric' limit(x, num)
x |
A SparkDataFrame |
num |
The number of rows to return |
A new SparkDataFrame containing the number of rows specified.
limit since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) limitedDF <- limit(df, 10) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) limitedDF <- limit(df, 10) ## End(Not run)
S4 class that represents a LinearRegressionModel
jobj |
a Java object reference to the backing Scala LinearRegressionWrapper |
LinearRegressionModel since 3.1.0
S4 class that represents an LinearSVCModel
jobj |
a Java object reference to the backing Scala LinearSVCModel |
LinearSVCModel since 2.2.0
Returns a list of columns for the given table/view in the specified database.
listColumns(tableName, databaseName = NULL)listColumns(tableName, databaseName = NULL)
tableName |
the qualified or unqualified name that designates a table/view. If no database
identifier is provided, it refers to a table/view in the current database.
If |
databaseName |
(optional) name of the database |
a SparkDataFrame of the list of column descriptions.
since 2.2.0
## Not run: sparkR.session() listColumns("mytable") ## End(Not run)## Not run: sparkR.session() listColumns("mytable") ## End(Not run)
Returns a list of databases available.
listDatabases()listDatabases()
a SparkDataFrame of the list of databases.
since 2.2.0
## Not run: sparkR.session() listDatabases() ## End(Not run)## Not run: sparkR.session() listDatabases() ## End(Not run)
Returns a list of functions registered in the specified database. This includes all temporary functions.
listFunctions(databaseName = NULL)listFunctions(databaseName = NULL)
databaseName |
(optional) name of the database |
a SparkDataFrame of the list of function descriptions.
since 2.2.0
## Not run: sparkR.session() listFunctions() ## End(Not run)## Not run: sparkR.session() listFunctions() ## End(Not run)
Returns a list of tables or views in the specified database. This includes all temporary views.
listTables(databaseName = NULL)listTables(databaseName = NULL)
databaseName |
(optional) name of the database |
a SparkDataFrame of the list of tables.
since 2.2.0
## Not run: sparkR.session() listTables() listTables("default") ## End(Not run)## Not run: sparkR.session() listTables() listTables("default") ## End(Not run)
Returns a locally checkpointed version of this SparkDataFrame. Checkpointing can be used to truncate the logical plan, which is especially useful in iterative algorithms where the plan may grow exponentially. Local checkpoints are stored in the executors using the caching subsystem and therefore they are not reliable.
localCheckpoint(x, eager = TRUE) ## S4 method for signature 'SparkDataFrame' localCheckpoint(x, eager = TRUE)localCheckpoint(x, eager = TRUE) ## S4 method for signature 'SparkDataFrame' localCheckpoint(x, eager = TRUE)
x |
A SparkDataFrame |
eager |
whether to locally checkpoint this SparkDataFrame immediately |
a new locally checkpointed SparkDataFrame
localCheckpoint since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- localCheckpoint(df) ## End(Not run)## Not run: df <- localCheckpoint(df) ## End(Not run)
S4 class that represents an LogisticRegressionModel
jobj |
a Java object reference to the backing Scala LogisticRegressionModel |
LogisticRegressionModel since 2.1.0
Merges two data frames
merge(x, y, ...) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' merge( x, y, by = intersect(names(x), names(y)), by.x = by, by.y = by, all = FALSE, all.x = all, all.y = all, sort = TRUE, suffixes = c("_x", "_y"), ... )merge(x, y, ...) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' merge( x, y, by = intersect(names(x), names(y)), by.x = by, by.y = by, all = FALSE, all.x = all, all.y = all, sort = TRUE, suffixes = c("_x", "_y"), ... )
x |
the first data frame to be joined. |
y |
the second data frame to be joined. |
... |
additional argument(s) passed to the method. |
by |
a character vector specifying the join columns. If by is not
specified, the common column names in |
by.x |
a character vector specifying the joining columns for x. |
by.y |
a character vector specifying the joining columns for y. |
all |
a boolean value setting |
all.x |
a boolean value indicating whether all the rows in x should be including in the join. |
all.y |
a boolean value indicating whether all the rows in y should be including in the join. |
sort |
a logical argument indicating whether the resulting columns should be sorted. |
suffixes |
a string vector of length 2 used to make colnames of
|
If all.x and all.y are set to FALSE, a natural join will be returned. If all.x is set to TRUE and all.y is set to FALSE, a left outer join will be returned. If all.x is set to FALSE and all.y is set to TRUE, a right outer join will be returned. If all.x and all.y are set to TRUE, a full outer join will be returned.
merge since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) merge(df1, df2) # Performs an inner join by common columns merge(df1, df2, by = "col1") # Performs an inner join based on expression merge(df1, df2, by.x = "col1", by.y = "col2", all.y = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all.x = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all.x = TRUE, all.y = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all = TRUE, sort = FALSE) merge(df1, df2, by = "col1", all = TRUE, suffixes = c("-X", "-Y")) merge(df1, df2, by = NULL) # Performs a Cartesian join ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) merge(df1, df2) # Performs an inner join by common columns merge(df1, df2, by = "col1") # Performs an inner join based on expression merge(df1, df2, by.x = "col1", by.y = "col2", all.y = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all.x = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all.x = TRUE, all.y = TRUE) merge(df1, df2, by.x = "col1", by.y = "col2", all = TRUE, sort = FALSE) merge(df1, df2, by = "col1", all = TRUE, suffixes = c("-X", "-Y")) merge(df1, df2, by = NULL) # Performs a Cartesian join ## End(Not run)
S4 class that represents a MultilayerPerceptronClassificationModel
jobj |
a Java object reference to the backing Scala MultilayerPerceptronClassifierWrapper |
MultilayerPerceptronClassificationModel since 2.1.0
Return a new SparkDataFrame with the specified columns added or replaced.
mutate(.data, ...) transform(`_data`, ...) ## S4 method for signature 'SparkDataFrame' mutate(.data, ...) ## S4 method for signature 'SparkDataFrame' transform(`_data`, ...)mutate(.data, ...) transform(`_data`, ...) ## S4 method for signature 'SparkDataFrame' mutate(.data, ...) ## S4 method for signature 'SparkDataFrame' transform(`_data`, ...)
.data |
a SparkDataFrame. |
... |
additional column argument(s) each in the form name = col. |
_data |
a SparkDataFrame. |
A new SparkDataFrame with the new columns added or replaced.
mutate since 1.4.0
transform since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- mutate(df, newCol = df$col1 * 5, newCol2 = df$col1 * 2) names(newDF) # Will contain newCol, newCol2 newDF2 <- transform(df, newCol = df$col1 / 5, newCol2 = df$col1 * 2) df <- createDataFrame(list(list("Andy", 30L), list("Justin", 19L)), c("name", "age")) # Replace the "age" column df1 <- mutate(df, age = df$age + 1L) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- mutate(df, newCol = df$col1 * 5, newCol2 = df$col1 * 2) names(newDF) # Will contain newCol, newCol2 newDF2 <- transform(df, newCol = df$col1 / 5, newCol2 = df$col1 * 2) df <- createDataFrame(list(list("Andy", 30L), list("Justin", 19L)), c("name", "age")) # Replace the "age" column df1 <- mutate(df, age = df$age + 1L) ## End(Not run)
S4 class that represents a NaiveBayesModel
jobj |
a Java object reference to the backing Scala NaiveBayesWrapper |
NaiveBayesModel since 2.0.0
Returns the number of columns in a SparkDataFrame
## S4 method for signature 'SparkDataFrame' ncol(x)## S4 method for signature 'SparkDataFrame' ncol(x)
x |
a SparkDataFrame |
ncol since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) ncol(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) ncol(df) ## End(Not run)
Inversion of boolean expression.
Inversion of boolean expression.
not(x) ## S4 method for signature 'Column' !x ## S4 method for signature 'Column' not(x)not(x) ## S4 method for signature 'Column' !x ## S4 method for signature 'Column' not(x)
x |
Column to compute on |
not and ! cannot be applied directly to numerical column.
To achieve R-like truthiness column has to be casted to BooleanType.
! since 2.3.0
not since 2.3.0
Other non-aggregate functions:
column_nonaggregate_functions,
column()
## Not run: df <- createDataFrame(data.frame(x = c(-1, 0, 1))) head(select(df, !column("x") > 0)) ## End(Not run) ## Not run: df <- createDataFrame(data.frame( is_true = c(TRUE, FALSE, NA), flag = c(1, 0, 1) )) head(select(df, not(df$is_true))) # Explicit cast is required when working with numeric column head(select(df, not(cast(df$flag, "boolean")))) ## End(Not run)## Not run: df <- createDataFrame(data.frame(x = c(-1, 0, 1))) head(select(df, !column("x") > 0)) ## End(Not run) ## Not run: df <- createDataFrame(data.frame( is_true = c(TRUE, FALSE, NA), flag = c(1, 0, 1) )) head(select(df, not(df$is_true))) # Explicit cast is required when working with numeric column head(select(df, not(cast(df$flag, "boolean")))) ## End(Not run)
Returns the number of rows in a SparkDataFrame
## S4 method for signature 'SparkDataFrame' count(x) ## S4 method for signature 'SparkDataFrame' nrow(x)## S4 method for signature 'SparkDataFrame' count(x) ## S4 method for signature 'SparkDataFrame' nrow(x)
x |
a SparkDataFrame. |
count since 1.4.0
nrow since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) count(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) count(df) ## End(Not run)
Defines the ordering columns in a WindowSpec.
orderBy(x, col, ...) ## S4 method for signature 'WindowSpec,character' orderBy(x, col, ...) ## S4 method for signature 'WindowSpec,Column' orderBy(x, col, ...)orderBy(x, col, ...) ## S4 method for signature 'WindowSpec,character' orderBy(x, col, ...) ## S4 method for signature 'WindowSpec,Column' orderBy(x, col, ...)
x |
a WindowSpec |
col |
a character or Column indicating an ordering column |
... |
additional sorting fields |
A WindowSpec.
orderBy(WindowSpec, character) since 2.0.0
orderBy(WindowSpec, Column) since 2.0.0
See arrange for use in sorting a SparkDataFrame
Other windowspec_method:
partitionBy(),
rangeBetween(),
rowsBetween()
## Not run: orderBy(ws, "col1", "col2") orderBy(ws, df$col1, df$col2) ## End(Not run)## Not run: orderBy(ws, "col1", "col2") orderBy(ws, df$col1, df$col2) ## End(Not run)
If values in the specified column are null, returns the value.
Can be used in conjunction with when to specify a default value for expressions.
otherwise(x, value) ## S4 method for signature 'Column' otherwise(x, value)otherwise(x, value) ## S4 method for signature 'Column' otherwise(x, value)
x |
a Column. |
value |
value to replace when the corresponding entry in |
otherwise since 1.5.0
Other column_func:
alias(),
between(),
cast(),
endsWith(),
over(),
startsWith(),
substr()
Define a windowing column.
over(x, window) ## S4 method for signature 'Column,WindowSpec' over(x, window)over(x, window) ## S4 method for signature 'Column,WindowSpec' over(x, window)
x |
a Column, usually one returned by window function(s). |
window |
a WindowSpec object. Can be created by |
over since 2.0.0
Other column_func:
alias(),
between(),
cast(),
endsWith(),
otherwise(),
startsWith(),
substr()
## Not run: df <- createDataFrame(mtcars) # Partition by am (transmission) and order by hp (horsepower) ws <- orderBy(windowPartitionBy("am"), "hp") # Rank on hp within each partition out <- select(df, over(rank(), ws), df$hp, df$am) # Lag mpg values by 1 row on the partition-and-ordered table out <- select(df, over(lead(df$mpg), ws), df$mpg, df$hp, df$am) ## End(Not run)## Not run: df <- createDataFrame(mtcars) # Partition by am (transmission) and order by hp (horsepower) ws <- orderBy(windowPartitionBy("am"), "hp") # Rank on hp within each partition out <- select(df, over(rank(), ws), df$hp, df$am) # Lag mpg values by 1 row on the partition-and-ordered table out <- select(df, over(lead(df$mpg), ws), df$mpg, df$hp, df$am) ## End(Not run)
Defines the partitioning columns in a WindowSpec.
partitionBy(x, ...) ## S4 method for signature 'WindowSpec' partitionBy(x, col, ...)partitionBy(x, ...) ## S4 method for signature 'WindowSpec' partitionBy(x, col, ...)
x |
a WindowSpec. |
... |
additional column(s) to partition on. |
col |
a column to partition on (described by the name or Column). |
A WindowSpec.
partitionBy(WindowSpec) since 2.0.0
Other windowspec_method:
orderBy(),
rangeBetween(),
rowsBetween()
## Not run: partitionBy(ws, "col1", "col2") partitionBy(ws, df$col1, df$col2) ## End(Not run)## Not run: partitionBy(ws, "col1", "col2") partitionBy(ws, df$col1, df$col2) ## End(Not run)
Persist this SparkDataFrame with the specified storage level. For details of the supported storage levels, refer to http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence.
persist(x, newLevel) ## S4 method for signature 'SparkDataFrame,character' persist(x, newLevel)persist(x, newLevel) ## S4 method for signature 'SparkDataFrame,character' persist(x, newLevel)
x |
the SparkDataFrame to persist. |
newLevel |
storage level chosen for the persistence. See available options in the description. |
persist since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") ## End(Not run)
Pivot a column of the GroupedData and perform the specified aggregation. There are two versions of pivot function: one that requires the caller to specify the list of distinct values to pivot on, and one that does not. The latter is more concise but less efficient, because Spark needs to first compute the list of distinct values internally.
## S4 method for signature 'GroupedData,character' pivot(x, colname, values = list())## S4 method for signature 'GroupedData,character' pivot(x, colname, values = list())
x |
a GroupedData object |
colname |
A column name |
values |
A value or a list/vector of distinct values for the output columns. |
GroupedData object
pivot since 2.0.0
## Not run: df <- createDataFrame(data.frame( earnings = c(10000, 10000, 11000, 15000, 12000, 20000, 21000, 22000), course = c("R", "Python", "R", "Python", "R", "Python", "R", "Python"), period = c("1H", "1H", "2H", "2H", "1H", "1H", "2H", "2H"), year = c(2015, 2015, 2015, 2015, 2016, 2016, 2016, 2016) )) group_sum <- sum(pivot(groupBy(df, "year"), "course"), "earnings") group_min <- min(pivot(groupBy(df, "year"), "course", "R"), "earnings") group_max <- max(pivot(groupBy(df, "year"), "course", c("Python", "R")), "earnings") group_mean <- mean(pivot(groupBy(df, "year"), "course", list("Python", "R")), "earnings") ## End(Not run)## Not run: df <- createDataFrame(data.frame( earnings = c(10000, 10000, 11000, 15000, 12000, 20000, 21000, 22000), course = c("R", "Python", "R", "Python", "R", "Python", "R", "Python"), period = c("1H", "1H", "2H", "2H", "1H", "1H", "2H", "2H"), year = c(2015, 2015, 2015, 2015, 2016, 2016, 2016, 2016) )) group_sum <- sum(pivot(groupBy(df, "year"), "course"), "earnings") group_min <- min(pivot(groupBy(df, "year"), "course", "R"), "earnings") group_max <- max(pivot(groupBy(df, "year"), "course", c("Python", "R")), "earnings") group_mean <- mean(pivot(groupBy(df, "year"), "course", list("Python", "R")), "earnings") ## End(Not run)
S4 class that represents a PowerIterationClustering
jobj |
a Java object reference to the backing Scala PowerIterationClustering |
PowerIterationClustering since 3.0.0
Makes predictions from a MLlib model. For more information, see the specific MLlib model below.
predict(object, ...)predict(object, ...)
object |
a fitted ML model object. |
... |
additional argument(s) passed to the method. |
spark.als, spark.bisectingKmeans, spark.decisionTree,
spark.gaussianMixture, spark.gbt,
spark.logit, spark.mlp, spark.naiveBayes,
spark.randomForest, spark.survreg, spark.svmLinear
S4 class that represents a PrefixSpan
jobj |
a Java object reference to the backing Scala PrefixSpan |
PrefixSpan since 3.0.0
This function prints the type and id for an object stored in the SparkR JVM backend.
## S3 method for class 'jobj' print(x, ...)## S3 method for class 'jobj' print(x, ...)
x |
The JVM object reference |
... |
further arguments passed to or from other methods |
print.jobj since 1.4.0
This function prints the contents of a StructField returned from the SparkR JVM backend.
## S3 method for class 'structField' print(x, ...)## S3 method for class 'structField' print(x, ...)
x |
A StructField object |
... |
further arguments passed to or from other methods |
print.structField since 1.4.0
This function prints the contents of a StructType returned from the SparkR JVM backend.
## S3 method for class 'structType' print(x, ...)## S3 method for class 'structType' print(x, ...)
x |
A StructType object |
... |
further arguments passed to or from other methods |
print.structType since 1.4.0
Prints out the schema in tree format
printSchema(x) ## S4 method for signature 'SparkDataFrame' printSchema(x)printSchema(x) ## S4 method for signature 'SparkDataFrame' printSchema(x)
x |
A SparkDataFrame |
printSchema since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) printSchema(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) printSchema(df) ## End(Not run)
Returns the user-specified name of the query. This is specified in
write.stream(df, queryName = "query"). This name, if set, must be unique across all active
queries.
queryName(x) ## S4 method for signature 'StreamingQuery' queryName(x)queryName(x) ## S4 method for signature 'StreamingQuery' queryName(x)
x |
a StreamingQuery. |
The name of the query, or NULL if not specified.
queryName(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
awaitTermination(),
explain(),
isActive(),
lastProgress(),
status(),
stopQuery()
## Not run: queryName(sq)## Not run: queryName(sq)
S4 class that represents a RandomForestClassificationModel
jobj |
a Java object reference to the backing Scala RandomForestClassificationModel |
RandomForestClassificationModel since 2.1.0
S4 class that represents a RandomForestRegressionModel
jobj |
a Java object reference to the backing Scala RandomForestRegressionModel |
RandomForestRegressionModel since 2.1.0
Return a list of randomly split dataframes with the provided weights.
randomSplit(x, weights, seed) ## S4 method for signature 'SparkDataFrame,numeric' randomSplit(x, weights, seed)randomSplit(x, weights, seed) ## S4 method for signature 'SparkDataFrame,numeric' randomSplit(x, weights, seed)
x |
A SparkDataFrame |
weights |
A vector of weights for splits, will be normalized if they don't sum to 1 |
seed |
A seed to use for random split |
randomSplit since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df <- createDataFrame(data.frame(id = 1:1000)) df_list <- randomSplit(df, c(2, 3, 5), 0) # df_list contains 3 SparkDataFrames with each having about 200, 300 and 500 rows respectively sapply(df_list, count) ## End(Not run)## Not run: sparkR.session() df <- createDataFrame(data.frame(id = 1:1000)) df_list <- randomSplit(df, c(2, 3, 5), 0) # df_list contains 3 SparkDataFrames with each having about 200, 300 and 500 rows respectively sapply(df_list, count) ## End(Not run)
Defines the frame boundaries, from start (inclusive) to end (inclusive).
rangeBetween(x, start, end) ## S4 method for signature 'WindowSpec,numeric,numeric' rangeBetween(x, start, end)rangeBetween(x, start, end) ## S4 method for signature 'WindowSpec,numeric,numeric' rangeBetween(x, start, end)
x |
a WindowSpec |
start |
boundary start, inclusive. The frame is unbounded if this is the minimum long value. |
end |
boundary end, inclusive. The frame is unbounded if this is the maximum long value. |
Both start and end are relative from the current row. For example, "0" means
"current row", while "-1" means one off before the current row, and "5" means the five off
after the current row.
We recommend users use Window.unboundedPreceding, Window.unboundedFollowing,
and Window.currentRow to specify special boundary values, rather than using long values
directly.
A range-based boundary is based on the actual value of the ORDER BY expression(s). An offset is used to alter the value of the ORDER BY expression, for instance if the current ORDER BY expression has a value of 10 and the lower bound offset is -3, the resulting lower bound for the current row will be 10 - 3 = 7. This however puts a number of constraints on the ORDER BY expressions: there can be only one expression and this expression must have a numerical data type. An exception can be made when the offset is unbounded, because no value modification is needed, in this case multiple and non-numeric ORDER BY expression are allowed.
a WindowSpec
rangeBetween since 2.0.0
Other windowspec_method:
orderBy(),
partitionBy(),
rowsBetween()
## Not run: id <- c(rep(1, 3), rep(2, 3), 3) desc <- c('New', 'New', 'Good', 'New', 'Good', 'Good', 'New') df <- data.frame(id, desc) df <- createDataFrame(df) w1 <- orderBy(windowPartitionBy('desc'), df$id) w2 <- rangeBetween(w1, 0, 3) df1 <- withColumn(df, "sum", over(sum(df$id), w2)) head(df1) ## End(Not run)## Not run: id <- c(rep(1, 3), rep(2, 3), 3) desc <- c('New', 'New', 'Good', 'New', 'Good', 'Good', 'New') df <- data.frame(id, desc) df <- createDataFrame(df) w1 <- orderBy(windowPartitionBy('desc'), df$id) w2 <- rangeBetween(w1, 0, 3) df1 <- withColumn(df, "sum", over(sum(df$id), w2)) head(df1) ## End(Not run)
Union two or more SparkDataFrames by row. As in R's rbind, this method
requires that the input SparkDataFrames have the same column names.
rbind(..., deparse.level = 1) ## S4 method for signature 'SparkDataFrame' rbind(x, ..., deparse.level = 1)rbind(..., deparse.level = 1) ## S4 method for signature 'SparkDataFrame' rbind(x, ..., deparse.level = 1)
... |
additional SparkDataFrame(s). |
deparse.level |
currently not used (put here to match the signature of the base implementation). |
x |
a SparkDataFrame. |
Note: This does not remove duplicate rows across the two SparkDataFrames.
A SparkDataFrame containing the result of the union.
rbind since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() unions <- rbind(df, df2, df3, df4) ## End(Not run)## Not run: sparkR.session() unions <- rbind(df, df2, df3, df4) ## End(Not run)
Returns the dataset in a data source as a SparkDataFrame
read.df(path = NULL, source = NULL, schema = NULL, na.strings = "NA", ...) loadDF(path = NULL, source = NULL, schema = NULL, ...)read.df(path = NULL, source = NULL, schema = NULL, na.strings = "NA", ...) loadDF(path = NULL, source = NULL, schema = NULL, ...)
path |
The path of files to load |
source |
The name of external data source |
schema |
The data schema defined in structType or a DDL-formatted string. |
na.strings |
Default string value for NA when source is "csv" |
... |
additional external data source specific named properties. |
The data source is specified by the source and a set of options(...).
If source is not specified, the default data source configured by
"spark.sql.sources.default" will be used.
Similar to R read.csv, when source is "csv", by default, a value of "NA" will be
interpreted as NA.
SparkDataFrame
read.df since 1.4.0
loadDF since 1.6.0
## Not run: sparkR.session() df1 <- read.df("path/to/file.json", source = "json") schema <- structType(structField("name", "string"), structField("info", "map<string,double>")) df2 <- read.df(mapTypeJsonPath, "json", schema, multiLine = TRUE) df3 <- loadDF("data/test_table", "parquet", mergeSchema = "true") stringSchema <- "name STRING, info MAP<STRING, DOUBLE>" df4 <- read.df(mapTypeJsonPath, "json", stringSchema, multiLine = TRUE) ## End(Not run)## Not run: sparkR.session() df1 <- read.df("path/to/file.json", source = "json") schema <- structType(structField("name", "string"), structField("info", "map<string,double>")) df2 <- read.df(mapTypeJsonPath, "json", schema, multiLine = TRUE) df3 <- loadDF("data/test_table", "parquet", mergeSchema = "true") stringSchema <- "name STRING, info MAP<STRING, DOUBLE>" df4 <- read.df(mapTypeJsonPath, "json", stringSchema, multiLine = TRUE) ## End(Not run)
Additional JDBC database connection properties can be set (...)
read.jdbc( url, tableName, partitionColumn = NULL, lowerBound = NULL, upperBound = NULL, numPartitions = 0L, predicates = list(), ... )read.jdbc( url, tableName, partitionColumn = NULL, lowerBound = NULL, upperBound = NULL, numPartitions = 0L, predicates = list(), ... )
url |
JDBC database url of the form |
tableName |
the name of the table in the external database |
partitionColumn |
the name of a column of numeric, date, or timestamp type that will be used for partitioning. |
lowerBound |
the minimum value of |
upperBound |
the maximum value of |
numPartitions |
the number of partitions, This, along with |
predicates |
a list of conditions in the where clause; each one defines one partition |
... |
additional JDBC database connection named properties. |
Only one of partitionColumn or predicates should be set. Partitions of the table will be
retrieved in parallel based on the numPartitions or by the predicates.
Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems.
SparkDataFrame
read.jdbc since 2.0.0
## Not run: sparkR.session() jdbcUrl <- "jdbc:mysql://localhost:3306/databasename" df <- read.jdbc(jdbcUrl, "table", predicates = list("field<=123"), user = "username") df2 <- read.jdbc(jdbcUrl, "table2", partitionColumn = "index", lowerBound = 0, upperBound = 10000, user = "username", password = "password") ## End(Not run)## Not run: sparkR.session() jdbcUrl <- "jdbc:mysql://localhost:3306/databasename" df <- read.jdbc(jdbcUrl, "table", predicates = list("field<=123"), user = "username") df2 <- read.jdbc(jdbcUrl, "table2", partitionColumn = "index", lowerBound = 0, upperBound = 10000, user = "username", password = "password") ## End(Not run)
Loads a JSON file, returning the result as a SparkDataFrame
By default, (JSON Lines text format or newline-delimited JSON
) is supported. For JSON (one record per file), set a named property multiLine to
TRUE.
It goes through the entire dataset once to determine the schema.
read.json(path, ...)read.json(path, ...)
path |
Path of file to read. A vector of multiple paths is allowed. |
... |
additional external data source specific named properties. |
SparkDataFrame
read.json since 1.6.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) df <- read.json(path, multiLine = TRUE) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) df <- read.json(path, multiLine = TRUE) ## End(Not run)
Load a fitted MLlib model from the input path.
read.ml(path)read.ml(path)
path |
path of the model to read. |
A fitted MLlib model.
read.ml since 2.0.0
## Not run: path <- "path/to/model" model <- read.ml(path) ## End(Not run)## Not run: path <- "path/to/model" model <- read.ml(path) ## End(Not run)
Loads an ORC file, returning the result as a SparkDataFrame.
read.orc(path, ...)read.orc(path, ...)
path |
Path of file to read. |
... |
additional external data source specific named properties. |
SparkDataFrame
read.orc since 2.0.0
Loads a Parquet file, returning the result as a SparkDataFrame.
read.parquet(path, ...)read.parquet(path, ...)
path |
path of file to read. A vector of multiple paths is allowed. |
... |
additional data source specific named properties. |
SparkDataFrame
read.parquet since 1.6.0
Returns the dataset in a data source as a SparkDataFrame
read.stream(source = NULL, schema = NULL, ...)read.stream(source = NULL, schema = NULL, ...)
source |
The name of external data source |
schema |
The data schema defined in structType or a DDL-formatted string, this is required for file-based streaming data source |
... |
additional external data source specific named options, for instance |
The data source is specified by the source and a set of options(...).
If source is not specified, the default data source configured by
"spark.sql.sources.default" will be used.
SparkDataFrame
read.stream since 2.2.0
experimental
## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) q <- write.stream(df, "text", path = "/home/user/out", checkpointLocation = "/home/user/cp") df <- read.stream("json", path = jsonDir, schema = schema, maxFilesPerTrigger = 1) stringSchema <- "name STRING, info MAP<STRING, DOUBLE>" df1 <- read.stream("json", path = jsonDir, schema = stringSchema, maxFilesPerTrigger = 1) ## End(Not run)## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) q <- write.stream(df, "text", path = "/home/user/out", checkpointLocation = "/home/user/cp") df <- read.stream("json", path = jsonDir, schema = schema, maxFilesPerTrigger = 1) stringSchema <- "name STRING, info MAP<STRING, DOUBLE>" df1 <- read.stream("json", path = jsonDir, schema = stringSchema, maxFilesPerTrigger = 1) ## End(Not run)
Loads text files and returns a SparkDataFrame whose schema starts with a string column named "value", and followed by partitioned columns if there are any. The text files must be encoded as UTF-8.
read.text(path, ...)read.text(path, ...)
path |
Path of file to read. A vector of multiple paths is allowed. |
... |
additional external data source specific named properties. |
Each line in the text file is a new row in the resulting SparkDataFrame.
SparkDataFrame
read.text since 1.6.1
## Not run: sparkR.session() path <- "path/to/file.txt" df <- read.text(path) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.txt" df <- read.text(path) ## End(Not run)
Recovers all the partitions in the directory of a table and update the catalog. The name should reference a partitioned table, and not a view.
recoverPartitions(tableName)recoverPartitions(tableName)
tableName |
the qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database. |
since 2.2.0
## Not run: sparkR.session() recoverPartitions("myTable") ## End(Not run)## Not run: sparkR.session() recoverPartitions("myTable") ## End(Not run)
Invalidates and refreshes all the cached data (and the associated metadata) for any SparkDataFrame that contains the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.
refreshByPath(path)refreshByPath(path)
path |
the path of the data source. |
since 2.2.0
## Not run: sparkR.session() refreshByPath("/path") ## End(Not run)## Not run: sparkR.session() refreshByPath("/path") ## End(Not run)
Invalidates and refreshes all the cached data and metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.
refreshTable(tableName)refreshTable(tableName)
tableName |
the qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database. |
If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.
since 2.2.0
## Not run: sparkR.session() refreshTable("myTable") ## End(Not run)## Not run: sparkR.session() refreshTable("myTable") ## End(Not run)
Registers a SparkDataFrame as a Temporary Table in the SparkSession
registerTempTable(x, tableName) ## S4 method for signature 'SparkDataFrame,character' registerTempTable(x, tableName)registerTempTable(x, tableName) ## S4 method for signature 'SparkDataFrame,character' registerTempTable(x, tableName)
x |
A SparkDataFrame |
tableName |
A character vector containing the name of the table |
registerTempTable since 1.4.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) registerTempTable(df, "json_df") new_df <- sql("SELECT * FROM json_df") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) registerTempTable(df, "json_df") new_df <- sql("SELECT * FROM json_df") ## End(Not run)
Rename an existing column in a SparkDataFrame.
rename(x, ...) withColumnRenamed(x, existingCol, newCol) ## S4 method for signature 'SparkDataFrame,character,character' withColumnRenamed(x, existingCol, newCol) ## S4 method for signature 'SparkDataFrame' rename(x, ...)rename(x, ...) withColumnRenamed(x, existingCol, newCol) ## S4 method for signature 'SparkDataFrame,character,character' withColumnRenamed(x, existingCol, newCol) ## S4 method for signature 'SparkDataFrame' rename(x, ...)
x |
A SparkDataFrame |
... |
A named pair of the form new_column_name = existing_column |
existingCol |
The name of the column you want to change. |
newCol |
The new column name. |
A SparkDataFrame with the column name changed.
withColumnRenamed since 1.4.0
rename since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- withColumnRenamed(df, "col1", "newCol1") ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- rename(df, col1 = df$newCol1) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- withColumnRenamed(df, "col1", "newCol1") ## End(Not run) ## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- rename(df, col1 = df$newCol1) ## End(Not run)
The following options for repartition are possible:
Return a new SparkDataFrame that has exactly numPartitions.
Return a new SparkDataFrame hash partitioned by
the given columns into numPartitions.
Return a new SparkDataFrame hash partitioned by the given column(s),
using spark.sql.shuffle.partitions as number of partitions.
repartition(x, ...) ## S4 method for signature 'SparkDataFrame' repartition(x, numPartitions = NULL, col = NULL, ...)repartition(x, ...) ## S4 method for signature 'SparkDataFrame' repartition(x, numPartitions = NULL, col = NULL, ...)
x |
a SparkDataFrame. |
... |
additional column(s) to be used in the partitioning. |
numPartitions |
the number of partitions to use. |
col |
the column by which the partitioning will be performed. |
repartition since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- repartition(df, 2L) newDF <- repartition(df, numPartitions = 2L) newDF <- repartition(df, col = df$"col1", df$"col2") newDF <- repartition(df, 3L, col = df$"col1", df$"col2") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- repartition(df, 2L) newDF <- repartition(df, numPartitions = 2L) newDF <- repartition(df, col = df$"col1", df$"col2") newDF <- repartition(df, 3L, col = df$"col1", df$"col2") ## End(Not run)
The following options for repartition by range are possible:
Return a new SparkDataFrame range partitioned by
the given columns into numPartitions.
Return a new SparkDataFrame range partitioned by the given column(s),
using spark.sql.shuffle.partitions as number of partitions.
At least one partition-by expression must be specified. When no explicit sort order is specified, "ascending nulls first" is assumed.
repartitionByRange(x, ...) ## S4 method for signature 'SparkDataFrame' repartitionByRange(x, numPartitions = NULL, col = NULL, ...)repartitionByRange(x, ...) ## S4 method for signature 'SparkDataFrame' repartitionByRange(x, numPartitions = NULL, col = NULL, ...)
x |
a SparkDataFrame. |
... |
additional column(s) to be used in the range partitioning. |
numPartitions |
the number of partitions to use. |
col |
the column by which the range partitioning will be performed. |
Note that due to performance reasons this method uses sampling to estimate the ranges.
Hence, the output may not be consistent, since sampling can return different values.
The sample size can be controlled by the config
spark.sql.execution.rangeExchange.sampleSizePerPartition.
repartitionByRange since 2.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- repartitionByRange(df, col = df$col1, df$col2) newDF <- repartitionByRange(df, 3L, col = df$col1, df$col2) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- repartitionByRange(df, col = df$col1, df$col2) newDF <- repartitionByRange(df, 3L, col = df$col1, df$col2) ## End(Not run)
Create a multi-dimensional rollup for the SparkDataFrame using the specified columns.
rollup(x, ...) ## S4 method for signature 'SparkDataFrame' rollup(x, ...)rollup(x, ...) ## S4 method for signature 'SparkDataFrame' rollup(x, ...)
x |
a SparkDataFrame. |
... |
character name(s) or Column(s) to group on. |
If grouping expression is missing rollup creates a single global aggregate and is
equivalent to direct application of agg.
A GroupedData.
rollup since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: df <- createDataFrame(mtcars) mean(rollup(df, "cyl", "gear", "am"), "mpg") # Following calls are equivalent agg(rollup(df), mean(df$mpg)) agg(df, mean(df$mpg)) ## End(Not run)## Not run: df <- createDataFrame(mtcars) mean(rollup(df, "cyl", "gear", "am"), "mpg") # Following calls are equivalent agg(rollup(df), mean(df$mpg)) agg(df, mean(df$mpg)) ## End(Not run)
Defines the frame boundaries, from start (inclusive) to end (inclusive).
rowsBetween(x, start, end) ## S4 method for signature 'WindowSpec,numeric,numeric' rowsBetween(x, start, end)rowsBetween(x, start, end) ## S4 method for signature 'WindowSpec,numeric,numeric' rowsBetween(x, start, end)
x |
a WindowSpec |
start |
boundary start, inclusive. The frame is unbounded if this is the minimum long value. |
end |
boundary end, inclusive. The frame is unbounded if this is the maximum long value. |
Both start and end are relative positions from the current row. For example,
"0" means "current row", while "-1" means the row before the current row, and "5" means the
fifth row after the current row.
We recommend users use Window.unboundedPreceding, Window.unboundedFollowing,
and Window.currentRow to specify special boundary values, rather than using long values
directly.
A row based boundary is based on the position of the row within the partition. An offset indicates the number of rows above or below the current row, the frame for the current row starts or ends. For instance, given a row based sliding frame with a lower bound offset of -1 and a upper bound offset of +2. The frame for row with index 5 would range from index 4 to index 6.
a WindowSpec
rowsBetween since 2.0.0
Other windowspec_method:
orderBy(),
partitionBy(),
rangeBetween()
## Not run: id <- c(rep(1, 3), rep(2, 3), 3) desc <- c('New', 'New', 'Good', 'New', 'Good', 'Good', 'New') df <- data.frame(id, desc) df <- createDataFrame(df) w1 <- orderBy(windowPartitionBy('desc'), df$id) w2 <- rowsBetween(w1, 0, 3) df1 <- withColumn(df, "sum", over(sum(df$id), w2)) head(df1) ## End(Not run)## Not run: id <- c(rep(1, 3), rep(2, 3), 3) desc <- c('New', 'New', 'Good', 'New', 'Good', 'Good', 'New') df <- data.frame(id, desc) df <- createDataFrame(df) w1 <- orderBy(windowPartitionBy('desc'), df$id) w2 <- rowsBetween(w1, 0, 3) df1 <- withColumn(df, "sum", over(sum(df$id), w2)) head(df1) ## End(Not run)
Return a sampled subset of this SparkDataFrame using a random seed. Note: this is not guaranteed to provide exactly the fraction specified of the total count of of the given SparkDataFrame.
sample(x, withReplacement = FALSE, fraction, seed) sample_frac(x, withReplacement = FALSE, fraction, seed) ## S4 method for signature 'SparkDataFrame' sample(x, withReplacement = FALSE, fraction, seed) ## S4 method for signature 'SparkDataFrame' sample_frac(x, withReplacement = FALSE, fraction, seed)sample(x, withReplacement = FALSE, fraction, seed) sample_frac(x, withReplacement = FALSE, fraction, seed) ## S4 method for signature 'SparkDataFrame' sample(x, withReplacement = FALSE, fraction, seed) ## S4 method for signature 'SparkDataFrame' sample_frac(x, withReplacement = FALSE, fraction, seed)
x |
A SparkDataFrame |
withReplacement |
Sampling with replacement or not |
fraction |
The (rough) sample target fraction |
seed |
Randomness seed value. Default is a random seed. |
sample since 1.4.0
sample_frac since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) collect(sample(df, fraction = 0.5)) collect(sample(df, FALSE, 0.5)) collect(sample(df, TRUE, 0.5, seed = 3)) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) collect(sample(df, fraction = 0.5)) collect(sample(df, FALSE, 0.5)) collect(sample(df, TRUE, 0.5, seed = 3)) ## End(Not run)
Returns a stratified sample without replacement based on the fraction given on each stratum.
sampleBy(x, col, fractions, seed) ## S4 method for signature 'SparkDataFrame,character,list,numeric' sampleBy(x, col, fractions, seed)sampleBy(x, col, fractions, seed) ## S4 method for signature 'SparkDataFrame,character,list,numeric' sampleBy(x, col, fractions, seed)
x |
A SparkDataFrame |
col |
column that defines strata |
fractions |
A named list giving sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero. |
seed |
random seed |
A new SparkDataFrame that represents the stratified sample
sampleBy since 1.6.0
Other stat functions:
approxQuantile(),
corr(),
cov(),
crosstab(),
freqItems()
## Not run: df <- read.json("/path/to/file.json") sample <- sampleBy(df, "key", fractions, 36) ## End(Not run)## Not run: df <- read.json("/path/to/file.json") sample <- sampleBy(df, "key", fractions, 36) ## End(Not run)
The data source is specified by the source and a set of options (...).
If source is not specified, the default data source configured by
spark.sql.sources.default will be used.
saveAsTable(df, tableName, source = NULL, mode = "error", ...) ## S4 method for signature 'SparkDataFrame,character' saveAsTable(df, tableName, source = NULL, mode = "error", ...)saveAsTable(df, tableName, source = NULL, mode = "error", ...) ## S4 method for signature 'SparkDataFrame,character' saveAsTable(df, tableName, source = NULL, mode = "error", ...)
df |
a SparkDataFrame. |
tableName |
a name for the table. |
source |
a name for external data source. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
... |
additional option(s) passed to the method. |
Additionally, mode is used to specify the behavior of the save operation when
data already exists in the data source. There are four modes:
'append': Contents of this SparkDataFrame are expected to be appended to existing data.
'overwrite': Existing data is expected to be overwritten by the contents of this
SparkDataFrame.
'error' or 'errorifexists': An exception is expected to be thrown.
'ignore': The save operation is expected to not save the contents of the SparkDataFrame
and to not change the existing data.
saveAsTable since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) saveAsTable(df, "myfile") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) saveAsTable(df, "myfile") ## End(Not run)
Returns the schema of this SparkDataFrame as a structType object.
schema(x) ## S4 method for signature 'SparkDataFrame' schema(x)schema(x) ## S4 method for signature 'SparkDataFrame' schema(x)
x |
A SparkDataFrame |
schema since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dfSchema <- schema(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) dfSchema <- schema(df) ## End(Not run)
Selects a set of columns with names or Column expressions.
select(x, col, ...) ## S4 method for signature 'SparkDataFrame' x$name ## S4 replacement method for signature 'SparkDataFrame' x$name <- value ## S4 method for signature 'SparkDataFrame,character' select(x, col, ...) ## S4 method for signature 'SparkDataFrame,Column' select(x, col, ...) ## S4 method for signature 'SparkDataFrame,list' select(x, col)select(x, col, ...) ## S4 method for signature 'SparkDataFrame' x$name ## S4 replacement method for signature 'SparkDataFrame' x$name <- value ## S4 method for signature 'SparkDataFrame,character' select(x, col, ...) ## S4 method for signature 'SparkDataFrame,Column' select(x, col, ...) ## S4 method for signature 'SparkDataFrame,list' select(x, col)
x |
a SparkDataFrame. |
col |
a list of columns or single Column or name. |
... |
additional column(s) if only one column is specified in |
name |
name of a Column (without being wrapped by |
value |
a Column or an atomic vector in the length of 1 as literal value, or |
A new SparkDataFrame with selected columns.
$ since 1.4.0
$<- since 1.4.0
select(SparkDataFrame, character) since 1.4.0
select(SparkDataFrame, Column) since 1.4.0
select(SparkDataFrame, list) since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
Other subsetting functions:
filter(),
subset()
## Not run: select(df, "*") select(df, "col1", "col2") select(df, df$name, df$age + 1) select(df, c("col1", "col2")) select(df, list(df$name, df$age + 1)) # Similar to R data frames columns can also be selected using $ df[,df$age] ## End(Not run)## Not run: select(df, "*") select(df, "col1", "col2") select(df, df$name, df$age + 1) select(df, c("col1", "col2")) select(df, list(df$name, df$age + 1)) # Similar to R data frames columns can also be selected using $ df[,df$age] ## End(Not run)
Select from a SparkDataFrame using a set of SQL expressions.
selectExpr(x, expr, ...) ## S4 method for signature 'SparkDataFrame,character' selectExpr(x, expr, ...)selectExpr(x, expr, ...) ## S4 method for signature 'SparkDataFrame,character' selectExpr(x, expr, ...)
x |
A SparkDataFrame to be selected from. |
expr |
A string containing a SQL expression |
... |
Additional expressions |
A SparkDataFrame
selectExpr since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) selectExpr(df, "col1", "(col2 * 5) as newCol") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) selectExpr(df, "col1", "(col2 * 5) as newCol") ## End(Not run)
Set the directory under which SparkDataFrame are going to be checkpointed. The directory must be an HDFS path if running on a cluster.
setCheckpointDir(directory)setCheckpointDir(directory)
directory |
Directory path to checkpoint to |
setCheckpointDir since 2.2.0
## Not run: setCheckpointDir("/checkpoint") ## End(Not run)## Not run: setCheckpointDir("/checkpoint") ## End(Not run)
Sets the current default database.
setCurrentDatabase(databaseName)setCurrentDatabase(databaseName)
databaseName |
name of the database |
since 2.2.0
## Not run: sparkR.session() setCurrentDatabase("default") ## End(Not run)## Not run: sparkR.session() setCurrentDatabase("default") ## End(Not run)
Set a description that is shown as a job description in UI.
setJobDescription(value)setJobDescription(value)
value |
The job description of the current job. |
setJobDescription since 2.3.0
## Not run: setJobDescription("This is an example job.") ## End(Not run)## Not run: setJobDescription("This is an example job.") ## End(Not run)
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
setJobGroup(groupId, description, interruptOnCancel)setJobGroup(groupId, description, interruptOnCancel)
groupId |
the ID to be assigned to job groups. |
description |
description for the job group ID. |
interruptOnCancel |
flag to indicate if the job is interrupted on job cancellation. |
setJobGroup since 1.5.0
## Not run: sparkR.session() setJobGroup("myJobGroup", "My job group description", TRUE) ## End(Not run)## Not run: sparkR.session() setJobGroup("myJobGroup", "My job group description", TRUE) ## End(Not run)
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.
setLocalProperty(key, value)setLocalProperty(key, value)
key |
The key for a local property. |
value |
The value for a local property. |
setLocalProperty since 2.3.0
## Not run: setLocalProperty("spark.scheduler.pool", "poolA") ## End(Not run)## Not run: setLocalProperty("spark.scheduler.pool", "poolA") ## End(Not run)
Set new log level: "ALL", "DEBUG", "ERROR", "FATAL", "INFO", "OFF", "TRACE", "WARN"
setLogLevel(level)setLogLevel(level)
level |
New log level |
setLogLevel since 2.0.0
## Not run: setLogLevel("ERROR") ## End(Not run)## Not run: setLogLevel("ERROR") ## End(Not run)
If eager evaluation is enabled and the Spark object is a SparkDataFrame, evaluate the SparkDataFrame and print top rows of the SparkDataFrame, otherwise, print the class and type information of the Spark object.
## S4 method for signature 'Column' show(object) ## S4 method for signature 'GroupedData' show(object) ## S4 method for signature 'SparkDataFrame' show(object) ## S4 method for signature 'WindowSpec' show(object) ## S4 method for signature 'StreamingQuery' show(object)## S4 method for signature 'Column' show(object) ## S4 method for signature 'GroupedData' show(object) ## S4 method for signature 'SparkDataFrame' show(object) ## S4 method for signature 'WindowSpec' show(object) ## S4 method for signature 'StreamingQuery' show(object)
object |
a Spark object. Can be a SparkDataFrame, Column, GroupedData, WindowSpec. |
show(Column) since 1.4.0
show(GroupedData) since 1.4.0
show(SparkDataFrame) since 1.4.0
show(WindowSpec) since 2.0.0
show(StreamingQuery) since 2.2.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) show(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) show(df) ## End(Not run)
Print the first numRows rows of a SparkDataFrame
showDF(x, ...) ## S4 method for signature 'SparkDataFrame' showDF(x, numRows = 20, truncate = TRUE, vertical = FALSE)showDF(x, ...) ## S4 method for signature 'SparkDataFrame' showDF(x, numRows = 20, truncate = TRUE, vertical = FALSE)
x |
a SparkDataFrame. |
... |
further arguments to be passed to or from other methods. |
numRows |
the number of rows to print. Defaults to 20. |
truncate |
whether truncate long strings. If |
vertical |
whether print output rows vertically (one line per column value). |
showDF since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) showDF(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) showDF(df) ## End(Not run)
The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, use spark.getSparkFiles(fileName) to find its download location.
spark.addFile(path, recursive = FALSE)spark.addFile(path, recursive = FALSE)
path |
The path of the file to be added |
recursive |
Whether to add files recursively from the path. Default is FALSE. |
A directory can be given if the recursive option is set to true. Currently directories are only supported for Hadoop-supported filesystems. Refer Hadoop-supported filesystems at https://cwiki.apache.org/confluence/display/HADOOP2/HCFS.
Note: A path can be added only once. Subsequent additions of the same path are ignored.
spark.addFile since 2.1.0
## Not run: spark.addFile("~/myfile") ## End(Not run)## Not run: spark.addFile("~/myfile") ## End(Not run)
spark.als learns latent factors in collaborative filtering via alternating least
squares. Users can call summary to obtain fitted latent factors, predict
to make predictions on new data, and write.ml/read.ml to save/load fitted models.
spark.als(data, ...) ## S4 method for signature 'SparkDataFrame' spark.als( data, ratingCol = "rating", userCol = "user", itemCol = "item", rank = 10, regParam = 0.1, maxIter = 10, nonnegative = FALSE, implicitPrefs = FALSE, alpha = 1, numUserBlocks = 10, numItemBlocks = 10, checkpointInterval = 10, seed = 0 ) ## S4 method for signature 'ALSModel' summary(object) ## S4 method for signature 'ALSModel' predict(object, newData) ## S4 method for signature 'ALSModel,character' write.ml(object, path, overwrite = FALSE)spark.als(data, ...) ## S4 method for signature 'SparkDataFrame' spark.als( data, ratingCol = "rating", userCol = "user", itemCol = "item", rank = 10, regParam = 0.1, maxIter = 10, nonnegative = FALSE, implicitPrefs = FALSE, alpha = 1, numUserBlocks = 10, numItemBlocks = 10, checkpointInterval = 10, seed = 0 ) ## S4 method for signature 'ALSModel' summary(object) ## S4 method for signature 'ALSModel' predict(object, newData) ## S4 method for signature 'ALSModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
... |
additional argument(s) passed to the method. |
ratingCol |
column name for ratings. |
userCol |
column name for user ids. Ids must be (or can be coerced into) integers. |
itemCol |
column name for item ids. Ids must be (or can be coerced into) integers. |
rank |
rank of the matrix factorization (> 0). |
regParam |
regularization parameter (>= 0). |
maxIter |
maximum number of iterations (>= 0). |
nonnegative |
logical value indicating whether to apply nonnegativity constraints. |
implicitPrefs |
logical value indicating whether to use implicit preference. |
alpha |
alpha parameter in the implicit preference formulation (>= 0). |
numUserBlocks |
number of user blocks used to parallelize computation (> 0). |
numItemBlocks |
number of item blocks used to parallelize computation (> 0). |
checkpointInterval |
number of checkpoint intervals (>= 1) or disable checkpoint (-1). Note: this setting will be ignored if the checkpoint directory is not set. |
seed |
integer seed for random number generation. |
object |
a fitted ALS model. |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
logical value indicating whether to overwrite if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
For more details, see MLlib: Collaborative Filtering.
spark.als returns a fitted ALS model.
summary returns summary information of the fitted model, which is a list.
The list includes user (the names of the user column),
item (the item column), rating (the rating column), userFactors
(the estimated user factors), itemFactors (the estimated item factors),
and rank (rank of the matrix factorization model).
predict returns a SparkDataFrame containing predicted values.
spark.als since 2.1.0
the input rating dataframe to the ALS implementation should be deterministic.
Nondeterministic data can cause failure during fitting ALS model. For example,
an order-sensitive operation like sampling after a repartition makes dataframe output
nondeterministic, like sample(repartition(df, 2L), FALSE, 0.5, 1618L).
Checkpointing sampled dataframe or adding a sort before sampling can help make the
dataframe deterministic.
summary(ALSModel) since 2.1.0
predict(ALSModel) since 2.1.0
write.ml(ALSModel, character) since 2.1.0
## Not run: ratings <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0), list(2, 1, 1.0), list(2, 2, 5.0)) df <- createDataFrame(ratings, c("user", "item", "rating")) model <- spark.als(df, "rating", "user", "item") # extract latent factors stats <- summary(model) userFactors <- stats$userFactors itemFactors <- stats$itemFactors # make predictions predicted <- predict(model, df) showDF(predicted) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # set other arguments modelS <- spark.als(df, "rating", "user", "item", rank = 20, regParam = 0.1, nonnegative = TRUE) statsS <- summary(modelS) ## End(Not run)## Not run: ratings <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0), list(2, 1, 1.0), list(2, 2, 5.0)) df <- createDataFrame(ratings, c("user", "item", "rating")) model <- spark.als(df, "rating", "user", "item") # extract latent factors stats <- summary(model) userFactors <- stats$userFactors itemFactors <- stats$itemFactors # make predictions predicted <- predict(model, df) showDF(predicted) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # set other arguments modelS <- spark.als(df, "rating", "user", "item", rank = 20, regParam = 0.1, nonnegative = TRUE) statsS <- summary(modelS) ## End(Not run)
A scalable graph clustering algorithm. Users can call spark.assignClusters to
return a cluster assignment for each input vertex.
Run the PIC algorithm and returns a cluster assignment for each input vertex.
spark.assignClusters(data, ...) ## S4 method for signature 'SparkDataFrame' spark.assignClusters( data, k = 2L, initMode = c("random", "degree"), maxIter = 20L, sourceCol = "src", destinationCol = "dst", weightCol = NULL )spark.assignClusters(data, ...) ## S4 method for signature 'SparkDataFrame' spark.assignClusters( data, k = 2L, initMode = c("random", "degree"), maxIter = 20L, sourceCol = "src", destinationCol = "dst", weightCol = NULL )
data |
a SparkDataFrame. |
... |
additional argument(s) passed to the method. |
k |
the number of clusters to create. |
initMode |
the initialization algorithm; "random" or "degree" |
maxIter |
the maximum number of iterations. |
sourceCol |
the name of the input column for source vertex IDs. |
destinationCol |
the name of the input column for destination vertex IDs |
weightCol |
weight column name. If this is not set or |
A dataset that contains columns of vertex id and the corresponding cluster for the id.
The schema of it will be: id: integer, cluster: integer
spark.assignClusters(SparkDataFrame) since 3.0.0
## Not run: df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0), list(1L, 2L, 1.0), list(3L, 4L, 1.0), list(4L, 0L, 0.1)), schema = c("src", "dst", "weight")) clusters <- spark.assignClusters(df, initMode = "degree", weightCol = "weight") showDF(clusters) ## End(Not run)## Not run: df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0), list(1L, 2L, 1.0), list(3L, 4L, 1.0), list(4L, 0L, 0.1)), schema = c("src", "dst", "weight")) clusters <- spark.assignClusters(df, initMode = "degree", weightCol = "weight") showDF(clusters) ## End(Not run)
Fits a bisecting k-means clustering model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
Get fitted result from a bisecting k-means model. Note: A saved-loaded model does not support this method.
spark.bisectingKmeans(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.bisectingKmeans( data, formula, k = 4, maxIter = 20, seed = NULL, minDivisibleClusterSize = 1 ) ## S4 method for signature 'BisectingKMeansModel' summary(object) ## S4 method for signature 'BisectingKMeansModel' predict(object, newData) ## S4 method for signature 'BisectingKMeansModel' fitted(object, method = c("centers", "classes")) ## S4 method for signature 'BisectingKMeansModel,character' write.ml(object, path, overwrite = FALSE)spark.bisectingKmeans(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.bisectingKmeans( data, formula, k = 4, maxIter = 20, seed = NULL, minDivisibleClusterSize = 1 ) ## S4 method for signature 'BisectingKMeansModel' summary(object) ## S4 method for signature 'BisectingKMeansModel' predict(object, newData) ## S4 method for signature 'BisectingKMeansModel' fitted(object, method = c("centers", "classes")) ## S4 method for signature 'BisectingKMeansModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', '-', '*', and '^'. Note that the response variable of formula is empty in spark.bisectingKmeans. |
... |
additional argument(s) passed to the method. |
k |
the desired number of leaf clusters. Must be > 1. The actual number could be smaller if there are no divisible leaf clusters. |
maxIter |
maximum iteration number. |
seed |
the random seed. |
minDivisibleClusterSize |
The minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster. Note that it is an expert parameter. The default value should be good enough for most cases. |
object |
a fitted bisecting k-means model. |
newData |
a SparkDataFrame for testing. |
method |
type of fitted results, |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.bisectingKmeans returns a fitted bisecting k-means model.
summary returns summary information of the fitted model, which is a list.
The list includes the model's k (number of cluster centers),
coefficients (model cluster centers),
size (number of data points in each cluster), cluster
(cluster centers of the transformed data; cluster is NULL if is.loaded is TRUE),
and is.loaded (whether the model is loaded from a saved file).
predict returns the predicted values based on a bisecting k-means model.
fitted returns a SparkDataFrame containing fitted values.
spark.bisectingKmeans since 2.2.0
summary(BisectingKMeansModel) since 2.2.0
predict(BisectingKMeansModel) since 2.2.0
fitted since 2.2.0
write.ml(BisectingKMeansModel, character) since 2.2.0
## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.bisectingKmeans(df, Class ~ Survived, k = 4) summary(model) # get fitted result from a bisecting k-means model fitted.model <- fitted(model, "centers") showDF(fitted.model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Class", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.bisectingKmeans(df, Class ~ Survived, k = 4) summary(model) # get fitted result from a bisecting k-means model fitted.model <- fitted(model, "centers") showDF(fitted.model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Class", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.decisionTree fits a Decision Tree Regression model or Classification model on
a SparkDataFrame. Users can call summary to get a summary of the fitted Decision Tree
model, predict to make predictions on new data, and write.ml/read.ml to
save/load fitted models.
For more details, see
Decision Tree Regression and
Decision Tree Classification
spark.decisionTree(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.decisionTree( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, impurity = NULL, seed = NULL, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'DecisionTreeRegressionModel' summary(object) ## S3 method for class 'summary.DecisionTreeRegressionModel' print(x, ...) ## S4 method for signature 'DecisionTreeClassificationModel' summary(object) ## S3 method for class 'summary.DecisionTreeClassificationModel' print(x, ...) ## S4 method for signature 'DecisionTreeRegressionModel' predict(object, newData) ## S4 method for signature 'DecisionTreeClassificationModel' predict(object, newData) ## S4 method for signature 'DecisionTreeRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'DecisionTreeClassificationModel,character' write.ml(object, path, overwrite = FALSE)spark.decisionTree(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.decisionTree( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, impurity = NULL, seed = NULL, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'DecisionTreeRegressionModel' summary(object) ## S3 method for class 'summary.DecisionTreeRegressionModel' print(x, ...) ## S4 method for signature 'DecisionTreeClassificationModel' summary(object) ## S3 method for class 'summary.DecisionTreeClassificationModel' print(x, ...) ## S4 method for signature 'DecisionTreeRegressionModel' predict(object, newData) ## S4 method for signature 'DecisionTreeClassificationModel' predict(object, newData) ## S4 method for signature 'DecisionTreeRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'DecisionTreeClassificationModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
type |
type of model, one of "regression" or "classification", to fit |
maxDepth |
Maximum depth of the tree (>= 0). |
maxBins |
Maximum number of bins used for discretizing continuous features and for choosing how to split on features at each node. More bins give higher granularity. Must be >= 2 and >= number of categories in any categorical feature. |
impurity |
Criterion used for information gain calculation. For regression, must be "variance". For classification, must be one of "entropy" and "gini", default is "gini". |
seed |
integer seed for random number generation. |
minInstancesPerNode |
Minimum number of instances each child must have after split. |
minInfoGain |
Minimum information gain for a split to be considered at a tree node. |
checkpointInterval |
Param for set checkpoint interval (>= 1) or disable checkpoint (-1). Note: this setting will be ignored if the checkpoint directory is not set. |
maxMemoryInMB |
Maximum memory in MiB allocated to histogram aggregation. |
cacheNodeIds |
If FALSE, the algorithm will pass trees to executors to match instances with nodes. If TRUE, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type in classification model. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
A fitted Decision Tree regression model or classification model. |
x |
summary object of Decision Tree regression model or classification model
returned by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.decisionTree returns a fitted Decision Tree model.
summary returns summary information of the fitted model, which is a list.
The list of components includes formula (formula),
numFeatures (number of features), features (list of features),
featureImportances (feature importances), and maxDepth (max depth of
trees).
predict returns a SparkDataFrame containing predicted labeled in a column named
"prediction".
spark.decisionTree since 2.3.0
summary(DecisionTreeRegressionModel) since 2.3.0
print.summary.DecisionTreeRegressionModel since 2.3.0
summary(DecisionTreeClassificationModel) since 2.3.0
print.summary.DecisionTreeClassificationModel since 2.3.0
predict(DecisionTreeRegressionModel) since 2.3.0
predict(DecisionTreeClassificationModel) since 2.3.0
write.ml(DecisionTreeRegressionModel, character) since 2.3.0
write.ml(DecisionTreeClassificationModel, character) since 2.3.0
## Not run: # fit a Decision Tree Regression Model df <- createDataFrame(longley) model <- spark.decisionTree(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Decision Tree Classification Model t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.decisionTree(df, Survived ~ Freq + Age, "classification") ## End(Not run)## Not run: # fit a Decision Tree Regression Model df <- createDataFrame(longley) model <- spark.decisionTree(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Decision Tree Classification Model t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.decisionTree(df, Survived ~ Freq + Age, "classification") ## End(Not run)
A parallel PrefixSpan algorithm to mine frequent sequential patterns.
spark.findFrequentSequentialPatterns returns a complete set of frequent sequential
patterns.
For more details, see
PrefixSpan.
spark.findFrequentSequentialPatterns(data, ...) ## S4 method for signature 'SparkDataFrame' spark.findFrequentSequentialPatterns( data, minSupport = 0.1, maxPatternLength = 10L, maxLocalProjDBSize = 32000000L, sequenceCol = "sequence" )spark.findFrequentSequentialPatterns(data, ...) ## S4 method for signature 'SparkDataFrame' spark.findFrequentSequentialPatterns( data, minSupport = 0.1, maxPatternLength = 10L, maxLocalProjDBSize = 32000000L, sequenceCol = "sequence" )
data |
A SparkDataFrame. |
... |
additional argument(s) passed to the method. |
minSupport |
Minimal support level. |
maxPatternLength |
Maximal pattern length. |
maxLocalProjDBSize |
Maximum number of items (including delimiters used in the internal storage format) allowed in a projected database before local processing. |
sequenceCol |
name of the sequence column in dataset. |
A complete set of frequent sequential patterns in the input sequences of itemsets.
The returned SparkDataFrame contains columns of sequence and corresponding
frequency. The schema of it will be:
sequence: ArrayType(ArrayType(T)), freq: integer
where T is the item type
spark.findFrequentSequentialPatterns(SparkDataFrame) since 3.0.0
## Not run: df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))), list(list(list(1L), list(3L, 2L), list(1L, 2L))), list(list(list(1L, 2L), list(5L))), list(list(list(6L)))), schema = c("sequence")) frequency <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5, maxPatternLength = 5L, maxLocalProjDBSize = 32000000L) showDF(frequency) ## End(Not run)## Not run: df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))), list(list(list(1L), list(3L, 2L), list(1L, 2L))), list(list(list(1L, 2L), list(5L))), list(list(list(6L)))), schema = c("sequence")) frequency <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5, maxPatternLength = 5L, maxLocalProjDBSize = 32000000L) showDF(frequency) ## End(Not run)
spark.fmClassifier fits a factorization classification model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
Only categorical data is supported.
spark.fmClassifier(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.fmClassifier( data, formula, factorSize = 8, fitLinear = TRUE, regParam = 0, miniBatchFraction = 1, initStd = 0.01, maxIter = 100, stepSize = 1, tol = 1e-06, solver = c("adamW", "gd"), thresholds = NULL, seed = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'FMClassificationModel' summary(object) ## S4 method for signature 'FMClassificationModel' predict(object, newData) ## S4 method for signature 'FMClassificationModel,character' write.ml(object, path, overwrite = FALSE)spark.fmClassifier(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.fmClassifier( data, formula, factorSize = 8, fitLinear = TRUE, regParam = 0, miniBatchFraction = 1, initStd = 0.01, maxIter = 100, stepSize = 1, tol = 1e-06, solver = c("adamW", "gd"), thresholds = NULL, seed = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'FMClassificationModel' summary(object) ## S4 method for signature 'FMClassificationModel' predict(object, newData) ## S4 method for signature 'FMClassificationModel,character' write.ml(object, path, overwrite = FALSE)
data |
a |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
factorSize |
dimensionality of the factors. |
fitLinear |
whether to fit linear term. # TODO Can we express this with formula? |
regParam |
the regularization parameter. |
miniBatchFraction |
the mini-batch fraction parameter. |
initStd |
the standard deviation of initial coefficients. |
maxIter |
maximum iteration number. |
stepSize |
stepSize parameter. |
tol |
convergence tolerance of iterations. |
solver |
solver parameter, supported options: "gd" (minibatch gradient descent) or "adamW". |
thresholds |
in binary classification, in range [0, 1]. If the estimated probability of class label 1 is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0 more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with threshold p is equivalent to setting thresholds c(1-p, p). |
seed |
seed parameter for weights initialization. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
a FM Classification model fitted by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.fmClassifier returns a fitted Factorization Machines Classification Model.
summary returns summary information of the fitted model, which is a list.
predict returns the predicted values based on a FM Classification model.
spark.fmClassifier since 3.1.0
summary(FMClassificationModel) since 3.1.0
predict(FMClassificationModel) since 3.1.0
write.ml(FMClassificationModel, character) since 3.1.0
## Not run: df <- read.df("data/mllib/sample_binary_classification_data.txt", source = "libsvm") # fit Factorization Machines Classification Model model <- spark.fmClassifier( df, label ~ features, regParam = 0.01, maxIter = 10, fitLinear = TRUE ) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: df <- read.df("data/mllib/sample_binary_classification_data.txt", source = "libsvm") # fit Factorization Machines Classification Model model <- spark.fmClassifier( df, label ~ features, regParam = 0.01, maxIter = 10, fitLinear = TRUE ) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.fmRegressor fits a factorization regression model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
spark.fmRegressor(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.fmRegressor( data, formula, factorSize = 8, fitLinear = TRUE, regParam = 0, miniBatchFraction = 1, initStd = 0.01, maxIter = 100, stepSize = 1, tol = 1e-06, solver = c("adamW", "gd"), seed = NULL, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'FMRegressionModel' summary(object) ## S4 method for signature 'FMRegressionModel' predict(object, newData) ## S4 method for signature 'FMRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.fmRegressor(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.fmRegressor( data, formula, factorSize = 8, fitLinear = TRUE, regParam = 0, miniBatchFraction = 1, initStd = 0.01, maxIter = 100, stepSize = 1, tol = 1e-06, solver = c("adamW", "gd"), seed = NULL, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'FMRegressionModel' summary(object) ## S4 method for signature 'FMRegressionModel' predict(object, newData) ## S4 method for signature 'FMRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
a |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
factorSize |
dimensionality of the factors. |
fitLinear |
whether to fit linear term. # TODO Can we express this with formula? |
regParam |
the regularization parameter. |
miniBatchFraction |
the mini-batch fraction parameter. |
initStd |
the standard deviation of initial coefficients. |
maxIter |
maximum iteration number. |
stepSize |
stepSize parameter. |
tol |
convergence tolerance of iterations. |
solver |
solver parameter, supported options: "gd" (minibatch gradient descent) or "adamW". |
seed |
seed parameter for weights initialization. |
stringIndexerOrderType |
how to order categories of a string feature column. This is used to decide the base level of a string feature as the last category after ordering is dropped when encoding strings. Supported options are "frequencyDesc", "frequencyAsc", "alphabetDesc", and "alphabetAsc". The default value is "frequencyDesc". When the ordering is set to "alphabetDesc", this drops the same category as R when encoding strings. |
object |
a FM Regression Model model fitted by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.fmRegressor returns a fitted Factorization Machines Regression Model.
summary returns summary information of the fitted model, which is a list.
predict returns the predicted values based on an FMRegressionModel.
spark.fmRegressor since 3.1.0
summary(FMRegressionModel) since 3.1.0
predict(FMRegressionModel) since 3.1.0
write.ml(FMRegressionModel, character) since 3.1.0
## Not run: df <- read.df("data/mllib/sample_linear_regression_data.txt", source = "libsvm") # fit Factorization Machines Regression Model model <- spark.fmRegressor( df, label ~ features, regParam = 0.01, maxIter = 10, fitLinear = TRUE ) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: df <- read.df("data/mllib/sample_linear_regression_data.txt", source = "libsvm") # fit Factorization Machines Regression Model model <- spark.fmRegressor( df, label ~ features, regParam = 0.01, maxIter = 10, fitLinear = TRUE ) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
A parallel FP-growth algorithm to mine frequent itemsets.
spark.fpGrowth fits a FP-growth model on a SparkDataFrame. Users can
spark.freqItemsets to get frequent itemsets, spark.associationRules to get
association rules, predict to make predictions on new data based on generated association
rules, and write.ml/read.ml to save/load fitted models.
For more details, see
FP-growth.
spark.fpGrowth(data, ...) spark.freqItemsets(object) spark.associationRules(object) ## S4 method for signature 'SparkDataFrame' spark.fpGrowth( data, minSupport = 0.3, minConfidence = 0.8, itemsCol = "items", numPartitions = NULL ) ## S4 method for signature 'FPGrowthModel' spark.freqItemsets(object) ## S4 method for signature 'FPGrowthModel' spark.associationRules(object) ## S4 method for signature 'FPGrowthModel' predict(object, newData) ## S4 method for signature 'FPGrowthModel,character' write.ml(object, path, overwrite = FALSE)spark.fpGrowth(data, ...) spark.freqItemsets(object) spark.associationRules(object) ## S4 method for signature 'SparkDataFrame' spark.fpGrowth( data, minSupport = 0.3, minConfidence = 0.8, itemsCol = "items", numPartitions = NULL ) ## S4 method for signature 'FPGrowthModel' spark.freqItemsets(object) ## S4 method for signature 'FPGrowthModel' spark.associationRules(object) ## S4 method for signature 'FPGrowthModel' predict(object, newData) ## S4 method for signature 'FPGrowthModel,character' write.ml(object, path, overwrite = FALSE)
data |
A SparkDataFrame for training. |
... |
additional argument(s) passed to the method. |
object |
a fitted FPGrowth model. |
minSupport |
Minimal support level. |
minConfidence |
Minimal confidence level. |
itemsCol |
Features column name. |
numPartitions |
Number of partitions used for fitting. |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
logical value indicating whether to overwrite if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.fpGrowth returns a fitted FPGrowth model.
A SparkDataFrame with frequent itemsets.
The SparkDataFrame contains two columns:
items (an array of the same type as the input column)
and freq (frequency of the itemset).
A SparkDataFrame with association rules.
The SparkDataFrame contains five columns:
antecedent (an array of the same type as the input column),
consequent (an array of the same type as the input column),
confidence (confidence for the rule)
lift (lift for the rule)
and support (support for the rule)
predict returns a SparkDataFrame containing predicted values.
spark.fpGrowth since 2.2.0
spark.freqItemsets(FPGrowthModel) since 2.2.0
spark.associationRules(FPGrowthModel) since 2.2.0
predict(FPGrowthModel) since 2.2.0
write.ml(FPGrowthModel, character) since 2.2.0
## Not run: raw_data <- read.df( "data/mllib/sample_fpgrowth.txt", source = "csv", schema = structType(structField("raw_items", "string"))) data <- selectExpr(raw_data, "split(raw_items, ' ') as items") model <- spark.fpGrowth(data) # Show frequent itemsets frequent_itemsets <- spark.freqItemsets(model) showDF(frequent_itemsets) # Show association rules association_rules <- spark.associationRules(model) showDF(association_rules) # Predict on new data new_itemsets <- data.frame(items = c("t", "t,s")) new_data <- selectExpr(createDataFrame(new_itemsets), "split(items, ',') as items") predict(model, new_data) # Save and load model path <- "/path/to/model" write.ml(model, path) read.ml(path) # Optional arguments baskets_data <- selectExpr(createDataFrame(itemsets), "split(items, ',') as baskets") another_model <- spark.fpGrowth(data, minSupport = 0.1, minConfidence = 0.5, itemsCol = "baskets", numPartitions = 10) ## End(Not run)## Not run: raw_data <- read.df( "data/mllib/sample_fpgrowth.txt", source = "csv", schema = structType(structField("raw_items", "string"))) data <- selectExpr(raw_data, "split(raw_items, ' ') as items") model <- spark.fpGrowth(data) # Show frequent itemsets frequent_itemsets <- spark.freqItemsets(model) showDF(frequent_itemsets) # Show association rules association_rules <- spark.associationRules(model) showDF(association_rules) # Predict on new data new_itemsets <- data.frame(items = c("t", "t,s")) new_data <- selectExpr(createDataFrame(new_itemsets), "split(items, ',') as items") predict(model, new_data) # Save and load model path <- "/path/to/model" write.ml(model, path) read.ml(path) # Optional arguments baskets_data <- selectExpr(createDataFrame(itemsets), "split(items, ',') as baskets") another_model <- spark.fpGrowth(data, minSupport = 0.1, minConfidence = 0.5, itemsCol = "baskets", numPartitions = 10) ## End(Not run)
Fits multivariate gaussian mixture model against a SparkDataFrame, similarly to R's
mvnormalmixEM(). Users can call summary to print a summary of the fitted model,
predict to make predictions on new data, and write.ml/read.ml
to save/load fitted models.
spark.gaussianMixture(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.gaussianMixture(data, formula, k = 2, maxIter = 100, tol = 0.01) ## S4 method for signature 'GaussianMixtureModel' summary(object) ## S4 method for signature 'GaussianMixtureModel' predict(object, newData) ## S4 method for signature 'GaussianMixtureModel,character' write.ml(object, path, overwrite = FALSE)spark.gaussianMixture(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.gaussianMixture(data, formula, k = 2, maxIter = 100, tol = 0.01) ## S4 method for signature 'GaussianMixtureModel' summary(object) ## S4 method for signature 'GaussianMixtureModel' predict(object, newData) ## S4 method for signature 'GaussianMixtureModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. Note that the response variable of formula is empty in spark.gaussianMixture. |
... |
additional arguments passed to the method. |
k |
number of independent Gaussians in the mixture model. |
maxIter |
maximum iteration number. |
tol |
the convergence tolerance. |
object |
a fitted gaussian mixture model. |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.gaussianMixture returns a fitted multivariate gaussian mixture model.
summary returns summary of the fitted model, which is a list.
The list includes the model's lambda (lambda), mu (mu),
sigma (sigma), loglik (loglik), and posterior (posterior).
predict returns a SparkDataFrame containing predicted labels in a column named
"prediction".
spark.gaussianMixture since 2.1.0
summary(GaussianMixtureModel) since 2.1.0
predict(GaussianMixtureModel) since 2.1.0
write.ml(GaussianMixtureModel, character) since 2.1.0
mixtools: https://cran.r-project.org/package=mixtools
## Not run: sparkR.session() library(mvtnorm) set.seed(100) a <- rmvnorm(4, c(0, 0)) b <- rmvnorm(6, c(3, 4)) data <- rbind(a, b) df <- createDataFrame(as.data.frame(data)) model <- spark.gaussianMixture(df, ~ V1 + V2, k = 2) summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "V1", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: sparkR.session() library(mvtnorm) set.seed(100) a <- rmvnorm(4, c(0, 0)) b <- rmvnorm(6, c(3, 4)) data <- rbind(a, b) df <- createDataFrame(as.data.frame(data)) model <- spark.gaussianMixture(df, ~ V1 + V2, k = 2) summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "V1", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.gbt fits a Gradient Boosted Tree Regression model or Classification model on a
SparkDataFrame. Users can call summary to get a summary of the fitted
Gradient Boosted Tree model, predict to make predictions on new data, and
write.ml/read.ml to save/load fitted models.
For more details, see
GBT Regression and
GBT Classification
spark.gbt(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.gbt( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, maxIter = 20, stepSize = 0.1, lossType = NULL, seed = NULL, subsamplingRate = 1, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'GBTRegressionModel' summary(object) ## S3 method for class 'summary.GBTRegressionModel' print(x, ...) ## S4 method for signature 'GBTClassificationModel' summary(object) ## S3 method for class 'summary.GBTClassificationModel' print(x, ...) ## S4 method for signature 'GBTRegressionModel' predict(object, newData) ## S4 method for signature 'GBTClassificationModel' predict(object, newData) ## S4 method for signature 'GBTRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'GBTClassificationModel,character' write.ml(object, path, overwrite = FALSE)spark.gbt(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.gbt( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, maxIter = 20, stepSize = 0.1, lossType = NULL, seed = NULL, subsamplingRate = 1, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'GBTRegressionModel' summary(object) ## S3 method for class 'summary.GBTRegressionModel' print(x, ...) ## S4 method for signature 'GBTClassificationModel' summary(object) ## S3 method for class 'summary.GBTClassificationModel' print(x, ...) ## S4 method for signature 'GBTRegressionModel' predict(object, newData) ## S4 method for signature 'GBTClassificationModel' predict(object, newData) ## S4 method for signature 'GBTRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'GBTClassificationModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', ':', '+', '-', '*', and '^'. |
... |
additional arguments passed to the method. |
type |
type of model, one of "regression" or "classification", to fit |
maxDepth |
Maximum depth of the tree (>= 0). |
maxBins |
Maximum number of bins used for discretizing continuous features and for choosing how to split on features at each node. More bins give higher granularity. Must be >= 2 and >= number of categories in any categorical feature. |
maxIter |
Param for maximum number of iterations (>= 0). |
stepSize |
Param for Step size to be used for each iteration of optimization. |
lossType |
Loss function which GBT tries to minimize. For classification, must be "logistic". For regression, must be one of "squared" (L2) and "absolute" (L1), default is "squared". |
seed |
integer seed for random number generation. |
subsamplingRate |
Fraction of the training data used for learning each decision tree, in range (0, 1]. |
minInstancesPerNode |
Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. |
minInfoGain |
Minimum information gain for a split to be considered at a tree node. |
checkpointInterval |
Param for set checkpoint interval (>= 1) or disable checkpoint (-1). Note: this setting will be ignored if the checkpoint directory is not set. |
maxMemoryInMB |
Maximum memory in MiB allocated to histogram aggregation. |
cacheNodeIds |
If FALSE, the algorithm will pass trees to executors to match instances with nodes. If TRUE, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type in classification model. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
A fitted Gradient Boosted Tree regression model or classification model. |
x |
summary object of Gradient Boosted Tree regression model or classification model
returned by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.gbt returns a fitted Gradient Boosted Tree model.
summary returns summary information of the fitted model, which is a list.
The list of components includes formula (formula),
numFeatures (number of features), features (list of features),
featureImportances (feature importances), maxDepth (max depth of trees),
numTrees (number of trees), and treeWeights (tree weights).
predict returns a SparkDataFrame containing predicted labeled in a column named
"prediction".
spark.gbt since 2.1.0
summary(GBTRegressionModel) since 2.1.0
print.summary.GBTRegressionModel since 2.1.0
summary(GBTClassificationModel) since 2.1.0
print.summary.GBTClassificationModel since 2.1.0
predict(GBTRegressionModel) since 2.1.0
predict(GBTClassificationModel) since 2.1.0
write.ml(GBTRegressionModel, character) since 2.1.0
write.ml(GBTClassificationModel, character) since 2.1.0
## Not run: # fit a Gradient Boosted Tree Regression Model df <- createDataFrame(longley) model <- spark.gbt(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Gradient Boosted Tree Classification Model # label must be binary - Only binary classification is supported for GBT. t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.gbt(df, Survived ~ Age + Freq, "classification") # numeric label is also supported t2 <- as.data.frame(Titanic) t2$NumericGender <- ifelse(t2$Sex == "Male", 0, 1) df <- createDataFrame(t2) model <- spark.gbt(df, NumericGender ~ ., type = "classification") ## End(Not run)## Not run: # fit a Gradient Boosted Tree Regression Model df <- createDataFrame(longley) model <- spark.gbt(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Gradient Boosted Tree Classification Model # label must be binary - Only binary classification is supported for GBT. t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.gbt(df, Survived ~ Age + Freq, "classification") # numeric label is also supported t2 <- as.data.frame(Titanic) t2$NumericGender <- ifelse(t2$Sex == "Male", 0, 1) df <- createDataFrame(t2) model <- spark.gbt(df, NumericGender ~ ., type = "classification") ## End(Not run)
Get the absolute path of a file added through spark.addFile.
spark.getSparkFiles(fileName)spark.getSparkFiles(fileName)
fileName |
The name of the file added through spark.addFile |
the absolute path of a file added through spark.addFile.
spark.getSparkFiles since 2.1.0
## Not run: spark.getSparkFiles("myfile") ## End(Not run)## Not run: spark.getSparkFiles("myfile") ## End(Not run)
Get the root directory that contains files added through spark.addFile.
spark.getSparkFilesRootDirectory()spark.getSparkFilesRootDirectory()
the root directory that contains files added through spark.addFile
spark.getSparkFilesRootDirectory since 2.1.0
## Not run: spark.getSparkFilesRootDirectory() ## End(Not run)## Not run: spark.getSparkFilesRootDirectory() ## End(Not run)
Fits generalized linear model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
spark.glm(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.glm( data, formula, family = gaussian, tol = 1e-06, maxIter = 25, weightCol = NULL, regParam = 0, var.power = 0, link.power = 1 - var.power, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc"), offsetCol = NULL ) ## S4 method for signature 'GeneralizedLinearRegressionModel' summary(object) ## S3 method for class 'summary.GeneralizedLinearRegressionModel' print(x, ...) ## S4 method for signature 'GeneralizedLinearRegressionModel' predict(object, newData) ## S4 method for signature 'GeneralizedLinearRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.glm(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.glm( data, formula, family = gaussian, tol = 1e-06, maxIter = 25, weightCol = NULL, regParam = 0, var.power = 0, link.power = 1 - var.power, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc"), offsetCol = NULL ) ## S4 method for signature 'GeneralizedLinearRegressionModel' summary(object) ## S3 method for class 'summary.GeneralizedLinearRegressionModel' print(x, ...) ## S4 method for signature 'GeneralizedLinearRegressionModel' predict(object, newData) ## S4 method for signature 'GeneralizedLinearRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', '-', '*', and '^'. |
... |
additional arguments passed to the method. |
family |
a description of the error distribution and link function to be used in the model.
This can be a character string naming a family function, a family function or
the result of a call to a family function. Refer R family at
https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html.
Currently these families are supported: Note that there are two ways to specify the tweedie family.
|
tol |
positive convergence tolerance of iterations. |
maxIter |
integer giving the maximal number of IRLS iterations. |
weightCol |
the weight column name. If this is not set or |
regParam |
regularization parameter for L2 regularization. |
var.power |
the power in the variance function of the Tweedie distribution which provides the relationship between the variance and mean of the distribution. Only applicable to the Tweedie family. |
link.power |
the index in the power link function. Only applicable to the Tweedie family. |
stringIndexerOrderType |
how to order categories of a string feature column. This is used to decide the base level of a string feature as the last category after ordering is dropped when encoding strings. Supported options are "frequencyDesc", "frequencyAsc", "alphabetDesc", and "alphabetAsc". The default value is "frequencyDesc". When the ordering is set to "alphabetDesc", this drops the same category as R when encoding strings. |
offsetCol |
the offset column name. If this is not set or empty, we treat all instance offsets as 0.0. The feature specified as offset has a constant coefficient of 1.0. |
object |
a fitted generalized linear model. |
x |
summary object of fitted generalized linear model returned by |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.glm returns a fitted generalized linear model.
summary returns summary information of the fitted model, which is a list.
The list of components includes at least the coefficients (coefficients matrix,
which includes coefficients, standard error of coefficients, t value and p value),
null.deviance (null/residual degrees of freedom), aic (AIC)
and iter (number of iterations IRLS takes). If there are collinear columns in
the data, the coefficients matrix only provides coefficients.
predict returns a SparkDataFrame containing predicted labels in a column named
"prediction".
spark.glm since 2.0.0
summary(GeneralizedLinearRegressionModel) since 2.0.0
print.summary.GeneralizedLinearRegressionModel since 2.0.0
predict(GeneralizedLinearRegressionModel) since 1.5.0
write.ml(GeneralizedLinearRegressionModel, character) since 2.0.0
## Not run: sparkR.session() t <- as.data.frame(Titanic, stringsAsFactors = FALSE) df <- createDataFrame(t) model <- spark.glm(df, Freq ~ Sex + Age, family = "gaussian") summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Freq", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) # note that the default string encoding is different from R's glm model2 <- glm(Freq ~ Sex + Age, family = "gaussian", data = t) summary(model2) # use stringIndexerOrderType = "alphabetDesc" to force string encoding # to be consistent with R model3 <- spark.glm(df, Freq ~ Sex + Age, family = "gaussian", stringIndexerOrderType = "alphabetDesc") summary(model3) # fit tweedie model model <- spark.glm(df, Freq ~ Sex + Age, family = "tweedie", var.power = 1.2, link.power = 0) summary(model) # use the tweedie family from statmod library(statmod) model <- spark.glm(df, Freq ~ Sex + Age, family = tweedie(1.2, 0)) summary(model) ## End(Not run)## Not run: sparkR.session() t <- as.data.frame(Titanic, stringsAsFactors = FALSE) df <- createDataFrame(t) model <- spark.glm(df, Freq ~ Sex + Age, family = "gaussian") summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Freq", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) # note that the default string encoding is different from R's glm model2 <- glm(Freq ~ Sex + Age, family = "gaussian", data = t) summary(model2) # use stringIndexerOrderType = "alphabetDesc" to force string encoding # to be consistent with R model3 <- spark.glm(df, Freq ~ Sex + Age, family = "gaussian", stringIndexerOrderType = "alphabetDesc") summary(model3) # fit tweedie model model <- spark.glm(df, Freq ~ Sex + Age, family = "tweedie", var.power = 1.2, link.power = 0) summary(model) # use the tweedie family from statmod library(statmod) model <- spark.glm(df, Freq ~ Sex + Age, family = tweedie(1.2, 0)) summary(model) ## End(Not run)
Fits an Isotonic Regression model against a SparkDataFrame, similarly to R's isoreg(). Users can print, make predictions on the produced model and save the model to the input path.
spark.isoreg(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.isoreg( data, formula, isotonic = TRUE, featureIndex = 0, weightCol = NULL ) ## S4 method for signature 'IsotonicRegressionModel' summary(object) ## S4 method for signature 'IsotonicRegressionModel' predict(object, newData) ## S4 method for signature 'IsotonicRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.isoreg(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.isoreg( data, formula, isotonic = TRUE, featureIndex = 0, weightCol = NULL ) ## S4 method for signature 'IsotonicRegressionModel' summary(object) ## S4 method for signature 'IsotonicRegressionModel' predict(object, newData) ## S4 method for signature 'IsotonicRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
SparkDataFrame for training. |
formula |
A symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
isotonic |
Whether the output sequence should be isotonic/increasing (TRUE) or antitonic/decreasing (FALSE). |
featureIndex |
The index of the feature if |
weightCol |
The weight column name. |
object |
a fitted IsotonicRegressionModel. |
newData |
SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.isoreg returns a fitted Isotonic Regression model.
summary returns summary information of the fitted model, which is a list.
The list includes model's boundaries (boundaries in increasing order)
and predictions (predictions associated with the boundaries at the same index).
predict returns a SparkDataFrame containing predicted values.
spark.isoreg since 2.1.0
summary(IsotonicRegressionModel) since 2.1.0
predict(IsotonicRegressionModel) since 2.1.0
write.ml(IsotonicRegression, character) since 2.1.0
## Not run: sparkR.session() data <- list(list(7.0, 0.0), list(5.0, 1.0), list(3.0, 2.0), list(5.0, 3.0), list(1.0, 4.0)) df <- createDataFrame(data, c("label", "feature")) model <- spark.isoreg(df, label ~ feature, isotonic = FALSE) # return model boundaries and prediction as lists result <- summary(model, df) # prediction based on fitted model predict_data <- list(list(-2.0), list(-1.0), list(0.5), list(0.75), list(1.0), list(2.0), list(9.0)) predict_df <- createDataFrame(predict_data, c("feature")) # get prediction column predict_result <- collect(select(predict(model, predict_df), "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: sparkR.session() data <- list(list(7.0, 0.0), list(5.0, 1.0), list(3.0, 2.0), list(5.0, 3.0), list(1.0, 4.0)) df <- createDataFrame(data, c("label", "feature")) model <- spark.isoreg(df, label ~ feature, isotonic = FALSE) # return model boundaries and prediction as lists result <- summary(model, df) # prediction based on fitted model predict_data <- list(list(-2.0), list(-1.0), list(0.5), list(0.75), list(1.0), list(2.0), list(9.0)) predict_df <- createDataFrame(predict_data, c("feature")) # get prediction column predict_result <- collect(select(predict(model, predict_df), "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
Fits a k-means clustering model against a SparkDataFrame, similarly to R's kmeans().
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
spark.kmeans(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.kmeans( data, formula, k = 2, maxIter = 20, initMode = c("k-means||", "random"), seed = NULL, initSteps = 2, tol = 1e-04 ) ## S4 method for signature 'KMeansModel' summary(object) ## S4 method for signature 'KMeansModel' predict(object, newData) ## S4 method for signature 'KMeansModel,character' write.ml(object, path, overwrite = FALSE)spark.kmeans(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.kmeans( data, formula, k = 2, maxIter = 20, initMode = c("k-means||", "random"), seed = NULL, initSteps = 2, tol = 1e-04 ) ## S4 method for signature 'KMeansModel' summary(object) ## S4 method for signature 'KMeansModel' predict(object, newData) ## S4 method for signature 'KMeansModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. Note that the response variable of formula is empty in spark.kmeans. |
... |
additional argument(s) passed to the method. |
k |
number of centers. |
maxIter |
maximum iteration number. |
initMode |
the initialization algorithm chosen to fit the model. |
seed |
the random seed for cluster initialization. |
initSteps |
the number of steps for the k-means|| initialization mode. This is an advanced setting, the default of 2 is almost always enough. Must be > 0. |
tol |
convergence tolerance of iterations. |
object |
a fitted k-means model. |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.kmeans returns a fitted k-means model.
summary returns summary information of the fitted model, which is a list.
The list includes the model's k (the configured number of cluster centers),
coefficients (model cluster centers),
size (number of data points in each cluster), cluster
(cluster centers of the transformed data), is.loaded (whether the model is loaded
from a saved file), and clusterSize
(the actual number of cluster centers. When using initMode = "random",
clusterSize may not equal to k).
predict returns the predicted values based on a k-means model.
spark.kmeans since 2.0.0
summary(KMeansModel) since 2.0.0
predict(KMeansModel) since 2.0.0
write.ml(KMeansModel, character) since 2.0.0
## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.kmeans(df, Class ~ Survived, k = 4, initMode = "random") summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Class", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: sparkR.session() t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.kmeans(df, Class ~ Survived, k = 4, initMode = "random") summary(model) # fitted values on training data fitted <- predict(model, df) head(select(fitted, "Class", "prediction")) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and print savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.kstest Conduct the two-sided Kolmogorov-Smirnov (KS) test for data sampled from a
continuous distribution.
By comparing the largest difference between the empirical cumulative distribution of the sample data and the theoretical distribution we can provide a test for the the null hypothesis that the sample data comes from that theoretical distribution.
Users can call summary to obtain a summary of the test, and print.summary.KSTest
to print out a summary result.
spark.kstest(data, ...) ## S4 method for signature 'SparkDataFrame' spark.kstest( data, testCol = "test", nullHypothesis = c("norm"), distParams = c(0, 1) ) ## S4 method for signature 'KSTest' summary(object) ## S3 method for class 'summary.KSTest' print(x, ...)spark.kstest(data, ...) ## S4 method for signature 'SparkDataFrame' spark.kstest( data, testCol = "test", nullHypothesis = c("norm"), distParams = c(0, 1) ) ## S4 method for signature 'KSTest' summary(object) ## S3 method for class 'summary.KSTest' print(x, ...)
data |
a SparkDataFrame of user data. |
... |
additional argument(s) passed to the method. |
testCol |
column name where the test data is from. It should be a column of double type. |
nullHypothesis |
name of the theoretical distribution tested against. Currently only
|
distParams |
parameters(s) of the distribution. For |
object |
test result object of KSTest by |
x |
summary object of KSTest returned by |
spark.kstest returns a test result object.
summary returns summary information of KSTest object, which is a list.
The list includes the p.value (p-value), statistic (test statistic
computed for the test), nullHypothesis (the null hypothesis with its
parameters tested against) and degreesOfFreedom (degrees of freedom of the test).
spark.kstest since 2.1.0
summary(KSTest) since 2.1.0
print.summary.KSTest since 2.1.0
## Not run: data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25)) df <- createDataFrame(data) test <- spark.kstest(df, "test", "norm", c(0, 1)) # get a summary of the test result testSummary <- summary(test) testSummary # print out the summary in an organized way print.summary.KSTest(testSummary) ## End(Not run)## Not run: data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25)) df <- createDataFrame(data) test <- spark.kstest(df, "test", "norm", c(0, 1)) # get a summary of the test result testSummary <- summary(test) testSummary # print out the summary in an organized way print.summary.KSTest(testSummary) ## End(Not run)
Run a function over a list of elements, distributing the computations with Spark. Applies a function in a manner that is similar to doParallel or lapply to elements of a list. The computations are distributed using Spark. It is conceptually the same as the following code: lapply(list, func)
spark.lapply(list, func)spark.lapply(list, func)
list |
the list of elements |
func |
a function that takes one argument. |
Known limitations:
variable scoping and capture: compared to R's rich support for variable resolutions, the distributed nature of SparkR limits how variables are resolved at runtime. All the variables that are available through lexical scoping are embedded in the closure of the function and available as read-only variables within the function. The environment variables should be stored into temporary variables outside the function, and not directly accessed within the function.
loading external packages: In order to use a package, you need to load it inside the closure. For example, if you rely on the MASS module, here is how you would use it:
train <- function(hyperparam) {
library(MASS)
lm.ridge("y ~ x+z", data, lambda=hyperparam)
model
}
a list of results (the exact type being determined by the function)
spark.lapply since 2.0.0
## Not run: sparkR.session() doubled <- spark.lapply(1:10, function(x) {2 * x}) ## End(Not run)## Not run: sparkR.session() doubled <- spark.lapply(1:10, function(x) {2 * x}) ## End(Not run)
spark.lda fits a Latent Dirichlet Allocation model on a SparkDataFrame. Users can call
summary to get a summary of the fitted LDA model, spark.posterior to compute
posterior probabilities on new data, spark.perplexity to compute log perplexity on new
data and write.ml/read.ml to save/load fitted models.
spark.lda(data, ...) spark.posterior(object, newData) spark.perplexity(object, data) ## S4 method for signature 'SparkDataFrame' spark.lda( data, features = "features", k = 10, maxIter = 20, optimizer = c("online", "em"), subsamplingRate = 0.05, topicConcentration = -1, docConcentration = -1, customizedStopWords = "", maxVocabSize = bitwShiftL(1, 18) ) ## S4 method for signature 'LDAModel' summary(object, maxTermsPerTopic) ## S4 method for signature 'LDAModel,SparkDataFrame' spark.perplexity(object, data) ## S4 method for signature 'LDAModel,SparkDataFrame' spark.posterior(object, newData) ## S4 method for signature 'LDAModel,character' write.ml(object, path, overwrite = FALSE)spark.lda(data, ...) spark.posterior(object, newData) spark.perplexity(object, data) ## S4 method for signature 'SparkDataFrame' spark.lda( data, features = "features", k = 10, maxIter = 20, optimizer = c("online", "em"), subsamplingRate = 0.05, topicConcentration = -1, docConcentration = -1, customizedStopWords = "", maxVocabSize = bitwShiftL(1, 18) ) ## S4 method for signature 'LDAModel' summary(object, maxTermsPerTopic) ## S4 method for signature 'LDAModel,SparkDataFrame' spark.perplexity(object, data) ## S4 method for signature 'LDAModel,SparkDataFrame' spark.posterior(object, newData) ## S4 method for signature 'LDAModel,character' write.ml(object, path, overwrite = FALSE)
data |
A SparkDataFrame for training. |
... |
additional argument(s) passed to the method. |
object |
A Latent Dirichlet Allocation model fitted by |
newData |
A SparkDataFrame for testing. |
features |
Features column name. Either libSVM-format column or character-format column is valid. |
k |
Number of topics. |
maxIter |
Maximum iterations. |
optimizer |
Optimizer to train an LDA model, "online" or "em", default is "online". |
subsamplingRate |
(For online optimizer) Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1]. |
topicConcentration |
concentration parameter (commonly named |
docConcentration |
concentration parameter (commonly named |
customizedStopWords |
stopwords that need to be removed from the given corpus. Ignore the parameter if libSVM-format column is used as the features column. |
maxVocabSize |
maximum vocabulary size, default 1 << 18 |
maxTermsPerTopic |
Maximum number of terms to collect for each topic. Default value of 10. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.lda returns a fitted Latent Dirichlet Allocation model.
summary returns summary information of the fitted model, which is a list.
The list includes
docConcentration |
concentration parameter commonly named |
topicConcentration |
concentration parameter commonly named |
logLikelihood |
log likelihood of the entire corpus |
logPerplexity |
log perplexity |
isDistributed |
TRUE for distributed model while FALSE for local model |
vocabSize |
number of terms in the corpus |
topics |
top 10 terms and their weights of all topics |
vocabulary |
whole terms of the training corpus, NULL if libsvm format file used as training set |
trainingLogLikelihood |
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters) It is only for distributed LDA model (i.e., optimizer = "em") |
logPrior |
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | Dirichlet hyperparameters) It is only for distributed LDA model (i.e., optimizer = "em") |
spark.perplexity returns the log perplexity of given SparkDataFrame, or the log
perplexity of the training data if missing argument "data".
spark.posterior returns a SparkDataFrame containing posterior probabilities
vectors named "topicDistribution".
spark.lda since 2.1.0
summary(LDAModel) since 2.1.0
spark.perplexity(LDAModel) since 2.1.0
spark.posterior(LDAModel) since 2.1.0
write.ml(LDAModel, character) since 2.1.0
topicmodels: https://cran.r-project.org/package=topicmodels
## Not run: text <- read.df("data/mllib/sample_lda_libsvm_data.txt", source = "libsvm") model <- spark.lda(data = text, optimizer = "em") # get a summary of the model summary(model) # compute posterior probabilities posterior <- spark.posterior(model, text) showDF(posterior) # compute perplexity perplexity <- spark.perplexity(model, text) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: text <- read.df("data/mllib/sample_lda_libsvm_data.txt", source = "libsvm") model <- spark.lda(data = text, optimizer = "em") # get a summary of the model summary(model) # compute posterior probabilities posterior <- spark.posterior(model, text) showDF(posterior) # compute perplexity perplexity <- spark.perplexity(model, text) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.lm fits a linear regression model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model,
predict to make predictions on new data,
and write.ml/read.ml to save/load fitted models.
spark.lm(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.lm( data, formula, maxIter = 100L, regParam = 0, elasticNetParam = 0, tol = 1e-06, standardization = TRUE, solver = c("auto", "l-bfgs", "normal"), weightCol = NULL, aggregationDepth = 2L, loss = c("squaredError", "huber"), epsilon = 1.35, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'LinearRegressionModel' summary(object) ## S4 method for signature 'LinearRegressionModel' predict(object, newData) ## S4 method for signature 'LinearRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.lm(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.lm( data, formula, maxIter = 100L, regParam = 0, elasticNetParam = 0, tol = 1e-06, standardization = TRUE, solver = c("auto", "l-bfgs", "normal"), weightCol = NULL, aggregationDepth = 2L, loss = c("squaredError", "huber"), epsilon = 1.35, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'LinearRegressionModel' summary(object) ## S4 method for signature 'LinearRegressionModel' predict(object, newData) ## S4 method for signature 'LinearRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
a |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
maxIter |
maximum iteration number. |
regParam |
the regularization parameter. |
elasticNetParam |
the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty. |
tol |
convergence tolerance of iterations. |
standardization |
whether to standardize the training features before fitting the model. |
solver |
The solver algorithm for optimization. Supported options: "l-bfgs", "normal" and "auto". |
weightCol |
weight column name. |
aggregationDepth |
suggested depth for treeAggregate (>= 2). |
loss |
the loss function to be optimized. Supported options: "squaredError" and "huber". |
epsilon |
the shape parameter to control the amount of robustness. |
stringIndexerOrderType |
how to order categories of a string feature column. This is used to decide the base level of a string feature as the last category after ordering is dropped when encoding strings. Supported options are "frequencyDesc", "frequencyAsc", "alphabetDesc", and "alphabetAsc". The default value is "frequencyDesc". When the ordering is set to "alphabetDesc", this drops the same category as R when encoding strings. |
object |
a Linear Regression Model model fitted by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.lm returns a fitted Linear Regression Model.
summary returns summary information of the fitted model, which is a list.
predict returns the predicted values based on a LinearRegressionModel.
spark.lm since 3.1.0
summary(LinearRegressionModel) since 3.1.0
predict(LinearRegressionModel) since 3.1.0
write.ml(LinearRegressionModel, character) since 3.1.0
## Not run: df <- read.df("data/mllib/sample_linear_regression_data.txt", source = "libsvm") # fit Linear Regression Model model <- spark.lm(df, label ~ features, regParam = 0.01, maxIter = 1) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: df <- read.df("data/mllib/sample_linear_regression_data.txt", source = "libsvm") # fit Linear Regression Model model <- spark.lm(df, label ~ features, regParam = 0.01, maxIter = 1) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
Fits an logistic regression model against a SparkDataFrame. It supports "binomial": Binary logistic regression with pivoting; "multinomial": Multinomial logistic (softmax) regression without pivoting, similar to glmnet. Users can print, make predictions on the produced model and save the model to the input path.
spark.logit(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.logit( data, formula, regParam = 0, elasticNetParam = 0, maxIter = 100, tol = 1e-06, family = "auto", standardization = TRUE, thresholds = 0.5, weightCol = NULL, aggregationDepth = 2, lowerBoundsOnCoefficients = NULL, upperBoundsOnCoefficients = NULL, lowerBoundsOnIntercepts = NULL, upperBoundsOnIntercepts = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'LogisticRegressionModel' summary(object) ## S4 method for signature 'LogisticRegressionModel' predict(object, newData) ## S4 method for signature 'LogisticRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.logit(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.logit( data, formula, regParam = 0, elasticNetParam = 0, maxIter = 100, tol = 1e-06, family = "auto", standardization = TRUE, thresholds = 0.5, weightCol = NULL, aggregationDepth = 2, lowerBoundsOnCoefficients = NULL, upperBoundsOnCoefficients = NULL, lowerBoundsOnIntercepts = NULL, upperBoundsOnIntercepts = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'LogisticRegressionModel' summary(object) ## S4 method for signature 'LogisticRegressionModel' predict(object, newData) ## S4 method for signature 'LogisticRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
SparkDataFrame for training. |
formula |
A symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
regParam |
the regularization parameter. |
elasticNetParam |
the ElasticNet mixing parameter. For alpha = 0.0, the penalty is an L2 penalty. For alpha = 1.0, it is an L1 penalty. For 0.0 < alpha < 1.0, the penalty is a combination of L1 and L2. Default is 0.0 which is an L2 penalty. |
maxIter |
maximum iteration number. |
tol |
convergence tolerance of iterations. |
family |
the name of family which is a description of the label distribution to be used in the model. Supported options:
|
standardization |
whether to standardize the training features before fitting the model. The coefficients of models will be always returned on the original scale, so it will be transparent for users. Note that with/without standardization, the models should be always converged to the same solution when no regularization is applied. Default is TRUE, same as glmnet. |
thresholds |
in binary classification, in range [0, 1]. If the estimated probability of class label 1 is > threshold, then predict 1, else 0. A high threshold encourages the model to predict 0 more often; a low threshold encourages the model to predict 1 more often. Note: Setting this with threshold p is equivalent to setting thresholds c(1-p, p). In multiclass (or binary) classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold. |
weightCol |
The weight column name. |
aggregationDepth |
The depth for treeAggregate (greater than or equal to 2). If the dimensions of features or the number of partitions are large, this param could be adjusted to a larger size. This is an expert parameter. Default value should be good for most cases. |
lowerBoundsOnCoefficients |
The lower bounds on coefficients if fitting under bound constrained optimization. The bound matrix must be compatible with the shape (1, number of features) for binomial regression, or (number of classes, number of features) for multinomial regression. It is a R matrix. |
upperBoundsOnCoefficients |
The upper bounds on coefficients if fitting under bound constrained optimization. The bound matrix must be compatible with the shape (1, number of features) for binomial regression, or (number of classes, number of features) for multinomial regression. It is a R matrix. |
lowerBoundsOnIntercepts |
The lower bounds on intercepts if fitting under bound constrained optimization. The bounds vector size must be equal to 1 for binomial regression, or the number of classes for multinomial regression. |
upperBoundsOnIntercepts |
The upper bounds on intercepts if fitting under bound constrained optimization. The bound vector size must be equal to 1 for binomial regression, or the number of classes for multinomial regression. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
an LogisticRegressionModel fitted by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.logit returns a fitted logistic regression model.
summary returns summary information of the fitted model, which is a list.
The list includes coefficients (coefficients matrix of the fitted model).
predict returns the predicted values based on an LogisticRegressionModel.
spark.logit since 2.1.0
summary(LogisticRegressionModel) since 2.1.0
predict(LogisticRegressionModel) since 2.1.0
write.ml(LogisticRegression, character) since 2.1.0
## Not run: sparkR.session() # binary logistic regression t <- as.data.frame(Titanic) training <- createDataFrame(t) model <- spark.logit(training, Survived ~ ., regParam = 0.5) summary <- summary(model) # fitted values on training data fitted <- predict(model, training) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and predict # Note that summary deos not work on loaded model savedModel <- read.ml(path) summary(savedModel) # binary logistic regression against two classes with # upperBoundsOnCoefficients and upperBoundsOnIntercepts ubc <- matrix(c(1.0, 0.0, 1.0, 0.0), nrow = 1, ncol = 4) model <- spark.logit(training, Species ~ ., upperBoundsOnCoefficients = ubc, upperBoundsOnIntercepts = 1.0) # multinomial logistic regression model <- spark.logit(training, Class ~ ., regParam = 0.5) summary <- summary(model) # multinomial logistic regression with # lowerBoundsOnCoefficients and lowerBoundsOnIntercepts lbc <- matrix(c(0.0, -1.0, 0.0, -1.0, 0.0, -1.0, 0.0, -1.0), nrow = 2, ncol = 4) lbi <- as.array(c(0.0, 0.0)) model <- spark.logit(training, Species ~ ., family = "multinomial", lowerBoundsOnCoefficients = lbc, lowerBoundsOnIntercepts = lbi) ## End(Not run)## Not run: sparkR.session() # binary logistic regression t <- as.data.frame(Titanic) training <- createDataFrame(t) model <- spark.logit(training, Survived ~ ., regParam = 0.5) summary <- summary(model) # fitted values on training data fitted <- predict(model, training) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and predict # Note that summary deos not work on loaded model savedModel <- read.ml(path) summary(savedModel) # binary logistic regression against two classes with # upperBoundsOnCoefficients and upperBoundsOnIntercepts ubc <- matrix(c(1.0, 0.0, 1.0, 0.0), nrow = 1, ncol = 4) model <- spark.logit(training, Species ~ ., upperBoundsOnCoefficients = ubc, upperBoundsOnIntercepts = 1.0) # multinomial logistic regression model <- spark.logit(training, Class ~ ., regParam = 0.5) summary <- summary(model) # multinomial logistic regression with # lowerBoundsOnCoefficients and lowerBoundsOnIntercepts lbc <- matrix(c(0.0, -1.0, 0.0, -1.0, 0.0, -1.0, 0.0, -1.0), nrow = 2, ncol = 4) lbi <- as.array(c(0.0, 0.0)) model <- spark.logit(training, Species ~ ., family = "multinomial", lowerBoundsOnCoefficients = lbc, lowerBoundsOnIntercepts = lbi) ## End(Not run)
spark.mlp fits a multi-layer perceptron neural network model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
Only categorical data is supported.
For more details, see
Multilayer Perceptron
spark.mlp(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.mlp( data, formula, layers, blockSize = 128, solver = "l-bfgs", maxIter = 100, tol = 1e-06, stepSize = 0.03, seed = NULL, initialWeights = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'MultilayerPerceptronClassificationModel' summary(object) ## S4 method for signature 'MultilayerPerceptronClassificationModel' predict(object, newData) ## S4 method for signature 'MultilayerPerceptronClassificationModel,character' write.ml(object, path, overwrite = FALSE)spark.mlp(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.mlp( data, formula, layers, blockSize = 128, solver = "l-bfgs", maxIter = 100, tol = 1e-06, stepSize = 0.03, seed = NULL, initialWeights = NULL, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'MultilayerPerceptronClassificationModel' summary(object) ## S4 method for signature 'MultilayerPerceptronClassificationModel' predict(object, newData) ## S4 method for signature 'MultilayerPerceptronClassificationModel,character' write.ml(object, path, overwrite = FALSE)
data |
a |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
layers |
integer vector containing the number of nodes for each layer. |
blockSize |
blockSize parameter. |
solver |
solver parameter, supported options: "gd" (minibatch gradient descent) or "l-bfgs". |
maxIter |
maximum iteration number. |
tol |
convergence tolerance of iterations. |
stepSize |
stepSize parameter. |
seed |
seed parameter for weights initialization. |
initialWeights |
initialWeights parameter for weights initialization, it should be a numeric vector. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
a Multilayer Perceptron Classification Model fitted by |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.mlp returns a fitted Multilayer Perceptron Classification Model.
summary returns summary information of the fitted model, which is a list.
The list includes numOfInputs (number of inputs), numOfOutputs
(number of outputs), layers (array of layer sizes including input
and output layers), and weights (the weights of layers).
For weights, it is a numeric vector with length equal to the expected
given the architecture (i.e., for 8-10-2 network, 112 connection weights).
predict returns a SparkDataFrame containing predicted labeled in a column named
"prediction".
spark.mlp since 2.1.0
summary(MultilayerPerceptronClassificationModel) since 2.1.0
predict(MultilayerPerceptronClassificationModel) since 2.1.0
write.ml(MultilayerPerceptronClassificationModel, character) since 2.1.0
## Not run: df <- read.df("data/mllib/sample_multiclass_classification_data.txt", source = "libsvm") # fit a Multilayer Perceptron Classification Model model <- spark.mlp(df, label ~ features, blockSize = 128, layers = c(4, 3), solver = "l-bfgs", maxIter = 100, tol = 0.5, stepSize = 1, seed = 1, initialWeights = c(0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 9, 9, 9, 9, 9)) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: df <- read.df("data/mllib/sample_multiclass_classification_data.txt", source = "libsvm") # fit a Multilayer Perceptron Classification Model model <- spark.mlp(df, label ~ features, blockSize = 128, layers = c(4, 3), solver = "l-bfgs", maxIter = 100, tol = 0.5, stepSize = 1, seed = 1, initialWeights = c(0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 9, 9, 9, 9, 9)) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.naiveBayes fits a Bernoulli naive Bayes model against a SparkDataFrame.
Users can call summary to print a summary of the fitted model, predict to make
predictions on new data, and write.ml/read.ml to save/load fitted models.
Only categorical data is supported.
spark.naiveBayes(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.naiveBayes( data, formula, smoothing = 1, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'NaiveBayesModel' summary(object) ## S4 method for signature 'NaiveBayesModel' predict(object, newData) ## S4 method for signature 'NaiveBayesModel,character' write.ml(object, path, overwrite = FALSE)spark.naiveBayes(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.naiveBayes( data, formula, smoothing = 1, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'NaiveBayesModel' summary(object) ## S4 method for signature 'NaiveBayesModel' predict(object, newData) ## S4 method for signature 'NaiveBayesModel,character' write.ml(object, path, overwrite = FALSE)
data |
a |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', and '-'. |
... |
additional argument(s) passed to the method. Currently only |
smoothing |
smoothing parameter. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
a naive Bayes model fitted by |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.naiveBayes returns a fitted naive Bayes model.
summary returns summary information of the fitted model, which is a list.
The list includes apriori (the label distribution) and
tables (conditional probabilities given the target label).
predict returns a SparkDataFrame containing predicted labeled in a column named
"prediction".
spark.naiveBayes since 2.0.0
summary(NaiveBayesModel) since 2.0.0
predict(NaiveBayesModel) since 2.0.0
write.ml(NaiveBayesModel, character) since 2.0.0
e1071: https://cran.r-project.org/package=e1071
## Not run: data <- as.data.frame(UCBAdmissions) df <- createDataFrame(data) # fit a Bernoulli naive Bayes model model <- spark.naiveBayes(df, Admit ~ Gender + Dept, smoothing = 0) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: data <- as.data.frame(UCBAdmissions) df <- createDataFrame(data) # fit a Bernoulli naive Bayes model model <- spark.naiveBayes(df, Admit ~ Gender + Dept, smoothing = 0) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
spark.randomForest fits a Random Forest Regression model or Classification model on
a SparkDataFrame. Users can call summary to get a summary of the fitted Random Forest
model, predict to make predictions on new data, and write.ml/read.ml to
save/load fitted models.
For more details, see
Random Forest Regression and
Random Forest Classification
spark.randomForest(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.randomForest( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, numTrees = 20, impurity = NULL, featureSubsetStrategy = "auto", seed = NULL, subsamplingRate = 1, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip"), bootstrap = TRUE ) ## S4 method for signature 'RandomForestRegressionModel' summary(object) ## S3 method for class 'summary.RandomForestRegressionModel' print(x, ...) ## S4 method for signature 'RandomForestClassificationModel' summary(object) ## S3 method for class 'summary.RandomForestClassificationModel' print(x, ...) ## S4 method for signature 'RandomForestRegressionModel' predict(object, newData) ## S4 method for signature 'RandomForestClassificationModel' predict(object, newData) ## S4 method for signature 'RandomForestRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'RandomForestClassificationModel,character' write.ml(object, path, overwrite = FALSE)spark.randomForest(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.randomForest( data, formula, type = c("regression", "classification"), maxDepth = 5, maxBins = 32, numTrees = 20, impurity = NULL, featureSubsetStrategy = "auto", seed = NULL, subsamplingRate = 1, minInstancesPerNode = 1, minInfoGain = 0, checkpointInterval = 10, maxMemoryInMB = 256, cacheNodeIds = FALSE, handleInvalid = c("error", "keep", "skip"), bootstrap = TRUE ) ## S4 method for signature 'RandomForestRegressionModel' summary(object) ## S3 method for class 'summary.RandomForestRegressionModel' print(x, ...) ## S4 method for signature 'RandomForestClassificationModel' summary(object) ## S3 method for class 'summary.RandomForestClassificationModel' print(x, ...) ## S4 method for signature 'RandomForestRegressionModel' predict(object, newData) ## S4 method for signature 'RandomForestClassificationModel' predict(object, newData) ## S4 method for signature 'RandomForestRegressionModel,character' write.ml(object, path, overwrite = FALSE) ## S4 method for signature 'RandomForestClassificationModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', ':', '+', and '-'. |
... |
additional arguments passed to the method. |
type |
type of model, one of "regression" or "classification", to fit |
maxDepth |
Maximum depth of the tree (>= 0). |
maxBins |
Maximum number of bins used for discretizing continuous features and for choosing how to split on features at each node. More bins give higher granularity. Must be >= 2 and >= number of categories in any categorical feature. |
numTrees |
Number of trees to train (>= 1). |
impurity |
Criterion used for information gain calculation. For regression, must be "variance". For classification, must be one of "entropy" and "gini", default is "gini". |
featureSubsetStrategy |
The number of features to consider for splits at each tree node. Supported options: "auto" (choose automatically for task: If numTrees == 1, set to "all." If numTrees > 1 (forest), set to "sqrt" for classification and to "onethird" for regression), "all" (use all features), "onethird" (use 1/3 of the features), "sqrt" (use sqrt(number of features)), "log2" (use log2(number of features)), "n": (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). Default is "auto". |
seed |
integer seed for random number generation. |
subsamplingRate |
Fraction of the training data used for learning each decision tree, in range (0, 1]. |
minInstancesPerNode |
Minimum number of instances each child must have after split. |
minInfoGain |
Minimum information gain for a split to be considered at a tree node. |
checkpointInterval |
Param for set checkpoint interval (>= 1) or disable checkpoint (-1). Note: this setting will be ignored if the checkpoint directory is not set. |
maxMemoryInMB |
Maximum memory in MiB allocated to histogram aggregation. |
cacheNodeIds |
If FALSE, the algorithm will pass trees to executors to match instances with nodes. If TRUE, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type in classification model. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
bootstrap |
Whether bootstrap samples are used when building trees. |
object |
A fitted Random Forest regression model or classification model. |
x |
summary object of Random Forest regression model or classification model
returned by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.randomForest returns a fitted Random Forest model.
summary returns summary information of the fitted model, which is a list.
The list of components includes formula (formula),
numFeatures (number of features), features (list of features),
featureImportances (feature importances), maxDepth (max depth of trees),
numTrees (number of trees), and treeWeights (tree weights).
predict returns a SparkDataFrame containing predicted labeled in a column named
"prediction".
spark.randomForest since 2.1.0
summary(RandomForestRegressionModel) since 2.1.0
print.summary.RandomForestRegressionModel since 2.1.0
summary(RandomForestClassificationModel) since 2.1.0
print.summary.RandomForestClassificationModel since 2.1.0
predict(RandomForestRegressionModel) since 2.1.0
predict(RandomForestClassificationModel) since 2.1.0
write.ml(RandomForestRegressionModel, character) since 2.1.0
write.ml(RandomForestClassificationModel, character) since 2.1.0
## Not run: # fit a Random Forest Regression Model df <- createDataFrame(longley) model <- spark.randomForest(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Random Forest Classification Model t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.randomForest(df, Survived ~ Freq + Age, "classification") ## End(Not run)## Not run: # fit a Random Forest Regression Model df <- createDataFrame(longley) model <- spark.randomForest(df, Employed ~ ., type = "regression", maxDepth = 5, maxBins = 16) # get the summary of the model summary(model) # make predictions predictions <- predict(model, df) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) # fit a Random Forest Classification Model t <- as.data.frame(Titanic) df <- createDataFrame(t) model <- spark.randomForest(df, Survived ~ Freq + Age, "classification") ## End(Not run)
spark.survreg fits an accelerated failure time (AFT) survival regression model on
a SparkDataFrame. Users can call summary to get a summary of the fitted AFT model,
predict to make predictions on new data, and write.ml/read.ml to
save/load fitted models.
spark.survreg(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.survreg( data, formula, aggregationDepth = 2, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'AFTSurvivalRegressionModel' summary(object) ## S4 method for signature 'AFTSurvivalRegressionModel' predict(object, newData) ## S4 method for signature 'AFTSurvivalRegressionModel,character' write.ml(object, path, overwrite = FALSE)spark.survreg(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.survreg( data, formula, aggregationDepth = 2, stringIndexerOrderType = c("frequencyDesc", "frequencyAsc", "alphabetDesc", "alphabetAsc") ) ## S4 method for signature 'AFTSurvivalRegressionModel' summary(object) ## S4 method for signature 'AFTSurvivalRegressionModel' predict(object, newData) ## S4 method for signature 'AFTSurvivalRegressionModel,character' write.ml(object, path, overwrite = FALSE)
data |
a SparkDataFrame for training. |
formula |
a symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', ':', '+', and '-'. Note that operator '.' is not supported currently. |
... |
additional arguments passed to the method. |
aggregationDepth |
The depth for treeAggregate (greater than or equal to 2). If the dimensions of features or the number of partitions are large, this param could be adjusted to a larger size. This is an expert parameter. Default value should be good for most cases. |
stringIndexerOrderType |
how to order categories of a string feature column. This is used to decide the base level of a string feature as the last category after ordering is dropped when encoding strings. Supported options are "frequencyDesc", "frequencyAsc", "alphabetDesc", and "alphabetAsc". The default value is "frequencyDesc". When the ordering is set to "alphabetDesc", this drops the same category as R when encoding strings. |
object |
a fitted AFT survival regression model. |
newData |
a SparkDataFrame for testing. |
path |
the directory where the model is saved. |
overwrite |
overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.survreg returns a fitted AFT survival regression model.
summary returns summary information of the fitted model, which is a list.
The list includes the model's coefficients (features, coefficients,
intercept and log(scale)).
predict returns a SparkDataFrame containing predicted values
on the original scale of the data (mean predicted value at scale = 1.0).
spark.survreg since 2.0.0
summary(AFTSurvivalRegressionModel) since 2.0.0
predict(AFTSurvivalRegressionModel) since 2.0.0
write.ml(AFTSurvivalRegressionModel, character) since 2.0.0
survival: https://cran.r-project.org/package=survival
## Not run: df <- createDataFrame(ovarian) model <- spark.survreg(df, Surv(futime, fustat) ~ ecog_ps + rx) # get a summary of the model summary(model) # make predictions predicted <- predict(model, df) showDF(predicted) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: df <- createDataFrame(ovarian) model <- spark.survreg(df, Surv(futime, fustat) ~ ecog_ps + rx) # get a summary of the model summary(model) # make predictions predicted <- predict(model, df) showDF(predicted) # save and load the model path <- "path/to/model" write.ml(model, path) savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
Fits a linear SVM model against a SparkDataFrame, similar to svm in e1071 package. Currently only supports binary classification model with linear kernel. Users can print, make predictions on the produced model and save the model to the input path.
spark.svmLinear(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.svmLinear( data, formula, regParam = 0, maxIter = 100, tol = 1e-06, standardization = TRUE, threshold = 0, weightCol = NULL, aggregationDepth = 2, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'LinearSVCModel' predict(object, newData) ## S4 method for signature 'LinearSVCModel' summary(object) ## S4 method for signature 'LinearSVCModel,character' write.ml(object, path, overwrite = FALSE)spark.svmLinear(data, formula, ...) ## S4 method for signature 'SparkDataFrame,formula' spark.svmLinear( data, formula, regParam = 0, maxIter = 100, tol = 1e-06, standardization = TRUE, threshold = 0, weightCol = NULL, aggregationDepth = 2, handleInvalid = c("error", "keep", "skip") ) ## S4 method for signature 'LinearSVCModel' predict(object, newData) ## S4 method for signature 'LinearSVCModel' summary(object) ## S4 method for signature 'LinearSVCModel,character' write.ml(object, path, overwrite = FALSE)
data |
SparkDataFrame for training. |
formula |
A symbolic description of the model to be fitted. Currently only a few formula operators are supported, including '~', '.', ':', '+', '-', '*', and '^'. |
... |
additional arguments passed to the method. |
regParam |
The regularization parameter. Only supports L2 regularization currently. |
maxIter |
Maximum iteration number. |
tol |
Convergence tolerance of iterations. |
standardization |
Whether to standardize the training features before fitting the model. The coefficients of models will be always returned on the original scale, so it will be transparent for users. Note that with/without standardization, the models should be always converged to the same solution when no regularization is applied. |
threshold |
The threshold in binary classification applied to the linear model prediction. This threshold can be any real number, where Inf will make all predictions 0.0 and -Inf will make all predictions 1.0. |
weightCol |
The weight column name. |
aggregationDepth |
The depth for treeAggregate (greater than or equal to 2). If the dimensions of features or the number of partitions are large, this param could be adjusted to a larger size. This is an expert parameter. Default value should be good for most cases. |
handleInvalid |
How to handle invalid data (unseen labels or NULL values) in features and label column of string type. Supported options: "skip" (filter out rows with invalid data), "error" (throw an error), "keep" (put invalid data in a special additional bucket, at index numLabels). Default is "error". |
object |
a LinearSVCModel fitted by |
newData |
a SparkDataFrame for testing. |
path |
The directory where the model is saved. |
overwrite |
Overwrites or not if the output path already exists. Default is FALSE which means throw exception if the output path exists. |
spark.svmLinear returns a fitted linear SVM model.
predict returns the predicted values based on a LinearSVCModel.
summary returns summary information of the fitted model, which is a list.
The list includes coefficients (coefficients of the fitted model),
numClasses (number of classes), numFeatures (number of features).
spark.svmLinear since 2.2.0
predict(LinearSVCModel) since 2.2.0
summary(LinearSVCModel) since 2.2.0
write.ml(LogisticRegression, character) since 2.2.0
## Not run: sparkR.session() t <- as.data.frame(Titanic) training <- createDataFrame(t) model <- spark.svmLinear(training, Survived ~ ., regParam = 0.5) summary <- summary(model) # fitted values on training data fitted <- predict(model, training) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and predict # Note that summary deos not work on loaded model savedModel <- read.ml(path) summary(savedModel) ## End(Not run)## Not run: sparkR.session() t <- as.data.frame(Titanic) training <- createDataFrame(t) model <- spark.svmLinear(training, Survived ~ ., regParam = 0.5) summary <- summary(model) # fitted values on training data fitted <- predict(model, training) # save fitted model to input path path <- "path/to/model" write.ml(model, path) # can also read back the saved model and predict # Note that summary deos not work on loaded model savedModel <- read.ml(path) summary(savedModel) ## End(Not run)
SparkDataFrames can be created using functions like createDataFrame, read.json, table etc.
envAn R environment that stores bookkeeping states of the SparkDataFrame
sdfA Java object reference to the backing Scala DataFrame
SparkDataFrame since 2.0.0
createDataFrame, read.json, table
https://spark.apache.org/docs/latest/sparkr.html#sparkr-dataframes
Other SparkDataFrame functions:
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df <- createDataFrame(faithful) ## End(Not run)## Not run: sparkR.session() df <- createDataFrame(faithful) ## End(Not run)
Call a Java method in the JVM running the Spark driver. The return values are automatically converted to R objects for simple objects. Other values are returned as "jobj" which are references to objects on JVM.
sparkR.callJMethod(x, methodName, ...)sparkR.callJMethod(x, methodName, ...)
x |
object to invoke the method on. Should be a "jobj" created by newJObject. |
methodName |
method name to call. |
... |
parameters to pass to the Java method. |
This is a low level function to access the JVM directly and should only be used for advanced use cases. The arguments and return values that are primitive R types (like integer, numeric, character, lists) are automatically translated to/from Java types (like Integer, Double, String, Array). A full list can be found in serialize.R and deserialize.R in the Apache Spark code base.
the return value of the Java method. Either returned as a R object if it can be deserialized or returned as a "jobj". See details section for more.
sparkR.callJMethod since 2.0.1
sparkR.callJStatic, sparkR.newJObject
## Not run: sparkR.session() # Need to have a Spark JVM running before calling newJObject # Create a Java ArrayList and populate it jarray <- sparkR.newJObject("java.util.ArrayList") sparkR.callJMethod(jarray, "add", 42L) sparkR.callJMethod(jarray, "get", 0L) # Will print 42 ## End(Not run)## Not run: sparkR.session() # Need to have a Spark JVM running before calling newJObject # Create a Java ArrayList and populate it jarray <- sparkR.newJObject("java.util.ArrayList") sparkR.callJMethod(jarray, "add", 42L) sparkR.callJMethod(jarray, "get", 0L) # Will print 42 ## End(Not run)
Call a static method in the JVM running the Spark driver. The return value is automatically converted to R objects for simple objects. Other values are returned as "jobj" which are references to objects on JVM.
sparkR.callJStatic(x, methodName, ...)sparkR.callJStatic(x, methodName, ...)
x |
fully qualified Java class name that contains the static method to invoke. |
methodName |
name of static method to invoke. |
... |
parameters to pass to the Java method. |
This is a low level function to access the JVM directly and should only be used for advanced use cases. The arguments and return values that are primitive R types (like integer, numeric, character, lists) are automatically translated to/from Java types (like Integer, Double, String, Array). A full list can be found in serialize.R and deserialize.R in the Apache Spark code base.
the return value of the Java method. Either returned as a R object if it can be deserialized or returned as a "jobj". See details section for more.
sparkR.callJStatic since 2.0.1
sparkR.callJMethod, sparkR.newJObject
## Not run: sparkR.session() # Need to have a Spark JVM running before calling callJStatic sparkR.callJStatic("java.lang.System", "currentTimeMillis") sparkR.callJStatic("java.lang.System", "getProperty", "java.home") ## End(Not run)## Not run: sparkR.session() # Need to have a Spark JVM running before calling callJStatic sparkR.callJStatic("java.lang.System", "currentTimeMillis") sparkR.callJStatic("java.lang.System", "getProperty", "java.home") ## End(Not run)
Get Runtime Config from the current active SparkSession.
To change SparkSession Runtime Config, please see sparkR.session().
sparkR.conf(key, defaultValue)sparkR.conf(key, defaultValue)
key |
(optional) The key of the config to get, if omitted, all config is returned |
defaultValue |
(optional) The default value of the config to return if they config is not set, if omitted, the call fails if the config key is not set |
a list of config values with keys as their names
sparkR.conf since 2.0.0
## Not run: sparkR.session() allConfigs <- sparkR.conf() masterValue <- unlist(sparkR.conf("spark.master")) namedConfig <- sparkR.conf("spark.executor.memory", "0g") ## End(Not run)## Not run: sparkR.session() allConfigs <- sparkR.conf() masterValue <- unlist(sparkR.conf("spark.master")) namedConfig <- sparkR.conf("spark.executor.memory", "0g") ## End(Not run)
This function initializes a new SparkContext.
sparkR.init( master = "", appName = "SparkR", sparkHome = Sys.getenv("SPARK_HOME"), sparkEnvir = list(), sparkExecutorEnv = list(), sparkJars = "", sparkPackages = "" )sparkR.init( master = "", appName = "SparkR", sparkHome = Sys.getenv("SPARK_HOME"), sparkEnvir = list(), sparkExecutorEnv = list(), sparkJars = "", sparkPackages = "" )
master |
The Spark master URL |
appName |
Application name to register with cluster manager |
sparkHome |
Spark Home directory |
sparkEnvir |
Named list of environment variables to set on worker nodes |
sparkExecutorEnv |
Named list of environment variables to be used when launching executors |
sparkJars |
Character vector of jar files to pass to the worker nodes |
sparkPackages |
Character vector of package coordinates |
sparkR.init since 1.4.0
## Not run: sc <- sparkR.init("local[2]", "SparkR", "/home/spark") sc <- sparkR.init("local[2]", "SparkR", "/home/spark", list(spark.executor.memory="1g")) sc <- sparkR.init("yarn-client", "SparkR", "/home/spark", list(spark.executor.memory="4g"), list(LD_LIBRARY_PATH="/directory of JVM libraries (libjvm.so) on workers/"), c("one.jar", "two.jar", "three.jar"), c("com.databricks:spark-avro_2.11:2.0.1")) ## End(Not run)## Not run: sc <- sparkR.init("local[2]", "SparkR", "/home/spark") sc <- sparkR.init("local[2]", "SparkR", "/home/spark", list(spark.executor.memory="1g")) sc <- sparkR.init("yarn-client", "SparkR", "/home/spark", list(spark.executor.memory="4g"), list(LD_LIBRARY_PATH="/directory of JVM libraries (libjvm.so) on workers/"), c("one.jar", "two.jar", "three.jar"), c("com.databricks:spark-avro_2.11:2.0.1")) ## End(Not run)
Create a new Java object in the JVM running the Spark driver. The return value is automatically converted to an R object for simple objects. Other values are returned as a "jobj" which is a reference to an object on JVM.
sparkR.newJObject(x, ...)sparkR.newJObject(x, ...)
x |
fully qualified Java class name. |
... |
arguments to be passed to the constructor. |
This is a low level function to access the JVM directly and should only be used for advanced use cases. The arguments and return values that are primitive R types (like integer, numeric, character, lists) are automatically translated to/from Java types (like Integer, Double, String, Array). A full list can be found in serialize.R and deserialize.R in the Apache Spark code base.
the object created. Either returned as a R object if it can be deserialized or returned as a "jobj". See details section for more.
sparkR.newJObject since 2.0.1
sparkR.callJMethod, sparkR.callJStatic
## Not run: sparkR.session() # Need to have a Spark JVM running before calling newJObject # Create a Java ArrayList and populate it jarray <- sparkR.newJObject("java.util.ArrayList") sparkR.callJMethod(jarray, "add", 42L) sparkR.callJMethod(jarray, "get", 0L) # Will print 42 ## End(Not run)## Not run: sparkR.session() # Need to have a Spark JVM running before calling newJObject # Create a Java ArrayList and populate it jarray <- sparkR.newJObject("java.util.ArrayList") sparkR.callJMethod(jarray, "add", 42L) sparkR.callJMethod(jarray, "get", 0L) # Will print 42 ## End(Not run)
SparkSession is the entry point into SparkR. sparkR.session gets the existing
SparkSession or initializes a new SparkSession.
Additional Spark properties can be set in ..., and these named parameters take priority
over values in master, appName, named lists of sparkConfig.
sparkR.session( master = "", appName = "SparkR", sparkHome = Sys.getenv("SPARK_HOME"), sparkConfig = list(), sparkJars = "", sparkPackages = "", enableHiveSupport = TRUE, ... )sparkR.session( master = "", appName = "SparkR", sparkHome = Sys.getenv("SPARK_HOME"), sparkConfig = list(), sparkJars = "", sparkPackages = "", enableHiveSupport = TRUE, ... )
master |
the Spark master URL. |
appName |
application name to register with cluster manager. |
sparkHome |
Spark Home directory. |
sparkConfig |
named list of Spark configuration to set on worker nodes. |
sparkJars |
character vector of jar files to pass to the worker nodes. |
sparkPackages |
character vector of package coordinates |
enableHiveSupport |
enable support for Hive, fallback if not built with Hive support; once set, this cannot be turned off on an existing session |
... |
named Spark properties passed to the method. |
When called in an interactive session, this method checks for the Spark installation, and, if not
found, it will be downloaded and cached automatically. Alternatively, install.spark can
be called manually.
A default warehouse is created automatically in the current directory when a managed table is
created via sql statement CREATE TABLE, for example. To change the location of the
warehouse, set the named parameter spark.sql.warehouse.dir to the SparkSession. Along with
the warehouse, an accompanied metastore may also be automatically created in the current
directory when a new SparkSession is initialized with enableHiveSupport set to
TRUE, which is the default. For more details, refer to Hive configuration at
http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables.
For details on how to initialize and use SparkR, refer to SparkR programming guide at http://spark.apache.org/docs/latest/sparkr.html#starting-up-sparksession.
sparkR.session since 2.0.0
## Not run: sparkR.session() df <- read.json(path) sparkR.session("local[2]", "SparkR", "/home/spark") sparkR.session("yarn", "SparkR", "/home/spark", list(spark.executor.memory="4g", spark.submit.deployMode="client"), c("one.jar", "two.jar", "three.jar"), c("com.databricks:spark-avro_2.12:2.0.1")) sparkR.session(spark.master = "yarn", spark.submit.deployMode = "client", ## End(Not run)## Not run: sparkR.session() df <- read.json(path) sparkR.session("local[2]", "SparkR", "/home/spark") sparkR.session("yarn", "SparkR", "/home/spark", list(spark.executor.memory="4g", spark.submit.deployMode="client"), c("one.jar", "two.jar", "three.jar"), c("com.databricks:spark-avro_2.12:2.0.1")) sparkR.session(spark.master = "yarn", spark.submit.deployMode = "client", ## End(Not run)
Stop the Spark Session and Spark Context.
sparkR.session.stop() sparkR.stop()sparkR.session.stop() sparkR.stop()
Also terminates the backend this R session is connected to.
sparkR.session.stop since 2.0.0
sparkR.stop since 1.4.0
Get the URL of the SparkUI instance for the current active SparkSession.
sparkR.uiWebUrl()sparkR.uiWebUrl()
the SparkUI URL, or NA if it is disabled, or not started.
sparkR.uiWebUrl since 2.1.1
## Not run: sparkR.session() url <- sparkR.uiWebUrl() ## End(Not run)## Not run: sparkR.session() url <- sparkR.uiWebUrl() ## End(Not run)
Get version of Spark on which this application is running.
sparkR.version()sparkR.version()
a character string of the Spark version
sparkR.version since 2.0.1
## Not run: sparkR.session() version <- sparkR.version() ## End(Not run)## Not run: sparkR.session() version <- sparkR.version() ## End(Not run)
This function creates a HiveContext from an existing JavaSparkContext
sparkRHive.init(jsc = NULL)sparkRHive.init(jsc = NULL)
jsc |
The existing JavaSparkContext created with SparkR.init() |
Starting SparkR 2.0, a SparkSession is initialized and returned instead. This API is deprecated and kept for backward compatibility only.
sparkRHive.init since 1.4.0
## Not run: sc <- sparkR.init() sqlContext <- sparkRHive.init(sc) ## End(Not run)## Not run: sc <- sparkR.init() sqlContext <- sparkRHive.init(sc) ## End(Not run)
This function creates a SparkContext from an existing JavaSparkContext and then uses it to initialize a new SQLContext
sparkRSQL.init(jsc = NULL)sparkRSQL.init(jsc = NULL)
jsc |
The existing JavaSparkContext created with SparkR.init() |
Starting SparkR 2.0, a SparkSession is initialized and returned instead. This API is deprecated and kept for backward compatibility only.
sparkRSQL.init since 1.4.0
## Not run: sc <- sparkR.init() sqlContext <- sparkRSQL.init(sc) ## End(Not run)## Not run: sc <- sparkR.init() sqlContext <- sparkRSQL.init(sc) ## End(Not run)
Executes a SQL query using Spark, returning the result as a SparkDataFrame.
sql(sqlQuery)sql(sqlQuery)
sqlQuery |
A character vector containing the SQL query |
SparkDataFrame
sql since 1.4.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") new_df <- sql("SELECT * FROM table") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") new_df <- sql("SELECT * FROM table") ## End(Not run)
Determines if entries of x start with string (entries of) prefix respectively, where strings are recycled to common lengths.
startsWith(x, prefix) ## S4 method for signature 'Column' startsWith(x, prefix)startsWith(x, prefix) ## S4 method for signature 'Column' startsWith(x, prefix)
x |
vector of character string whose "starts" are considered |
prefix |
character vector (often of length one) |
startsWith since 1.4.0
Other column_func:
alias(),
between(),
cast(),
endsWith(),
otherwise(),
over(),
substr()
Prints the current status of the query in JSON format.
status(x) ## S4 method for signature 'StreamingQuery' status(x)status(x) ## S4 method for signature 'StreamingQuery' status(x)
x |
a StreamingQuery. |
status(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
awaitTermination(),
explain(),
isActive(),
lastProgress(),
queryName(),
stopQuery()
## Not run: status(sq)## Not run: status(sq)
Stops the execution of this query if it is running. This method blocks until the execution is stopped.
stopQuery(x) ## S4 method for signature 'StreamingQuery' stopQuery(x)stopQuery(x) ## S4 method for signature 'StreamingQuery' stopQuery(x)
x |
a StreamingQuery. |
stopQuery(StreamingQuery) since 2.2.0
experimental
Other StreamingQuery methods:
awaitTermination(),
explain(),
isActive(),
lastProgress(),
queryName(),
status()
## Not run: stopQuery(sq)## Not run: stopQuery(sq)
Get storagelevel of this SparkDataFrame.
## S4 method for signature 'SparkDataFrame' storageLevel(x)## S4 method for signature 'SparkDataFrame' storageLevel(x)
x |
the SparkDataFrame to get the storageLevel. |
storageLevel since 2.1.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") storageLevel(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") storageLevel(df) ## End(Not run)
Display the structure of a SparkDataFrame, including column names, column types, as well as a a small sample of rows.
## S4 method for signature 'SparkDataFrame' str(object)## S4 method for signature 'SparkDataFrame' str(object)
object |
a SparkDataFrame |
str since 1.6.1
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: # Create a SparkDataFrame from the Iris dataset irisDF <- createDataFrame(iris) # Show the structure of the SparkDataFrame str(irisDF) ## End(Not run)## Not run: # Create a SparkDataFrame from the Iris dataset irisDF <- createDataFrame(iris) # Show the structure of the SparkDataFrame str(irisDF) ## End(Not run)
StreamingQuery can be created by using read.stream() and write.stream()
ssq |
A Java object reference to the backing Scala StreamingQuery |
StreamingQuery since 2.2.0
experimental
Create a structField object that contains the metadata for a single field in a schema.
structField(x, ...) ## S3 method for class 'jobj' structField(x, ...) ## S3 method for class 'character' structField(x, type, nullable = TRUE, ...)structField(x, ...) ## S3 method for class 'jobj' structField(x, ...) ## S3 method for class 'character' structField(x, type, nullable = TRUE, ...)
x |
the name of the field. |
... |
additional argument(s) passed to the method. |
type |
The data type of the field |
nullable |
A logical vector indicating whether or not the field is nullable |
A structField object.
structField since 1.4.0
## Not run: field1 <- structField("a", "integer") field2 <- structField("c", "string") field3 <- structField("avg", "double") schema <- structType(field1, field2, field3) df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) ## End(Not run)## Not run: field1 <- structField("a", "integer") field2 <- structField("c", "string") field3 <- structField("avg", "double") schema <- structType(field1, field2, field3) df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) ## End(Not run)
Create a structType object that contains the metadata for a SparkDataFrame. Intended for use with createDataFrame and toDF.
structType(x, ...) ## S3 method for class 'jobj' structType(x, ...) ## S3 method for class 'structField' structType(x, ...) ## S3 method for class 'character' structType(x, ...)structType(x, ...) ## S3 method for class 'jobj' structType(x, ...) ## S3 method for class 'structField' structType(x, ...) ## S3 method for class 'character' structType(x, ...)
x |
a structField object (created with the |
... |
additional structField objects |
a structType object
structType since 1.4.0
## Not run: schema <- structType(structField("a", "integer"), structField("c", "string"), structField("avg", "double")) df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) schema <- structType("a INT, c STRING, avg DOUBLE") df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) ## End(Not run)## Not run: schema <- structType(structField("a", "integer"), structField("c", "string"), structField("avg", "double")) df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) schema <- structType("a INT, c STRING, avg DOUBLE") df1 <- gapply(df, list("a", "c"), function(key, x) { y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) }, schema) ## End(Not run)
Return subsets of SparkDataFrame according to given conditions
subset(x, ...) ## S4 method for signature 'SparkDataFrame,numericOrcharacter' x[[i]] ## S4 replacement method for signature 'SparkDataFrame,numericOrcharacter' x[[i]] <- value ## S4 method for signature 'SparkDataFrame' x[i, j, ..., drop = F] ## S4 method for signature 'SparkDataFrame' subset(x, subset, select, drop = F, ...)subset(x, ...) ## S4 method for signature 'SparkDataFrame,numericOrcharacter' x[[i]] ## S4 replacement method for signature 'SparkDataFrame,numericOrcharacter' x[[i]] <- value ## S4 method for signature 'SparkDataFrame' x[i, j, ..., drop = F] ## S4 method for signature 'SparkDataFrame' subset(x, subset, select, drop = F, ...)
x |
a SparkDataFrame. |
... |
currently not used. |
i, subset
|
(Optional) a logical expression to filter on rows. For extract operator [[ and replacement operator [[<-, the indexing parameter for a single Column. |
value |
a Column or an atomic vector in the length of 1 as literal value, or |
j, select
|
expression for the single Column or a list of columns to select from the SparkDataFrame. |
drop |
if TRUE, a Column will be returned if the resulting dataset has only one column. Otherwise, a SparkDataFrame will always be returned. |
A new SparkDataFrame containing only the rows that meet the condition with selected columns.
[[ since 1.4.0
[[<- since 2.1.1
[ since 1.4.0
subset since 1.5.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
Other subsetting functions:
filter(),
select()
## Not run: # Columns can be selected using [[ and [ df[[2]] == df[["age"]] df[,2] == df[,"age"] df[,c("name", "age")] # Or to filter rows df[df$age > 20,] # SparkDataFrame can be subset on both rows and Columns df[df$name == "Smith", c(1,2)] df[df$age %in% c(19, 30), 1:2] subset(df, df$age %in% c(19, 30), 1:2) subset(df, df$age %in% c(19), select = c(1,2)) subset(df, select = c(1,2)) # Columns can be selected and set df[["age"]] <- 23 df[[1]] <- df$age df[[2]] <- NULL # drop column ## End(Not run)## Not run: # Columns can be selected using [[ and [ df[[2]] == df[["age"]] df[,2] == df[,"age"] df[,c("name", "age")] # Or to filter rows df[df$age > 20,] # SparkDataFrame can be subset on both rows and Columns df[df$name == "Smith", c(1,2)] df[df$age %in% c(19, 30), 1:2] subset(df, df$age %in% c(19, 30), 1:2) subset(df, df$age %in% c(19), select = c(1,2)) subset(df, select = c(1,2)) # Columns can be selected and set df[["age"]] <- 23 df[[1]] <- df$age df[[2]] <- NULL # drop column ## End(Not run)
An expression that returns a substring.
## S4 method for signature 'Column' substr(x, start, stop)## S4 method for signature 'Column' substr(x, start, stop)
x |
a Column. |
start |
starting position. It should be 1-base. |
stop |
ending position. |
substr since 1.4.0
Other column_func:
alias(),
between(),
cast(),
endsWith(),
otherwise(),
over(),
startsWith()
## Not run: df <- createDataFrame(list(list(a="abcdef"))) collect(select(df, substr(df$a, 1, 4))) # the result is `abcd`. collect(select(df, substr(df$a, 2, 4))) # the result is `bcd`. ## End(Not run)## Not run: df <- createDataFrame(list(list(a="abcdef"))) collect(select(df, substr(df$a, 1, 4))) # the result is `abcd`. collect(select(df, substr(df$a, 2, 4))) # the result is `bcd`. ## End(Not run)
Computes specified statistics for numeric and string columns. Available statistics are:
count
mean
stddev
min
max
arbitrary approximate percentiles specified as a percentage (e.g., "75%")
If no statistics are given, this function computes count, mean, stddev, min,
approximate quartiles (percentiles at 25%, 50%, and 75%), and max.
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting Dataset. If you want to
programmatically compute summary statistics, use the agg function instead.
summary(object, ...) ## S4 method for signature 'SparkDataFrame' summary(object, ...)summary(object, ...) ## S4 method for signature 'SparkDataFrame' summary(object, ...)
object |
a SparkDataFrame to be summarized. |
... |
(optional) statistics to be computed for all columns. |
A SparkDataFrame.
summary(SparkDataFrame) since 1.5.0
The statistics provided by summary were change in 2.3.0 use describe for
previous defaults.
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) summary(df) summary(df, "min", "25%", "75%", "max") summary(select(df, "age", "height")) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) summary(df) summary(df, "min", "25%", "75%", "max") summary(select(df, "age", "height")) ## End(Not run)
Returns the names of tables in the given database as an array.
tableNames(databaseName = NULL)tableNames(databaseName = NULL)
databaseName |
(optional) name of the database |
a list of table names
tableNames since 1.4.0
## Not run: sparkR.session() tableNames("hive") ## End(Not run)## Not run: sparkR.session() tableNames("hive") ## End(Not run)
Returns a SparkDataFrame containing names of tables in the given database.
tables(databaseName = NULL)tables(databaseName = NULL)
databaseName |
(optional) name of the database |
a SparkDataFrame
tables since 1.4.0
## Not run: sparkR.session() tables("hive") ## End(Not run)## Not run: sparkR.session() tables("hive") ## End(Not run)
Returns the specified table or view as a SparkDataFrame. The table or view must already exist or have already been registered in the SparkSession.
tableToDF(tableName)tableToDF(tableName)
tableName |
the qualified or unqualified name that designates a table or view. If a database is specified, it identifies the table/view from the database. Otherwise, it first attempts to find a temporary view with the given name and then match the table/view from the current database. |
SparkDataFrame
tableToDF since 2.0.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") new_df <- tableToDF("table") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") new_df <- tableToDF("table") ## End(Not run)
Take the first NUM rows of a SparkDataFrame and return the results as a R data.frame
take(x, num) ## S4 method for signature 'SparkDataFrame,numeric' take(x, num)take(x, num) ## S4 method for signature 'SparkDataFrame,numeric' take(x, num)
x |
a SparkDataFrame. |
num |
number of rows to take. |
take since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) take(df, 2) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) take(df, 2) ## End(Not run)
Date time functions defined for Column.
current_date(x = "missing") current_timestamp(x = "missing") date_trunc(format, x) dayofmonth(x) dayofweek(x) dayofyear(x) from_unixtime(x, ...) hour(x) last_day(x) minute(x) month(x) quarter(x) second(x) timestamp_seconds(x) to_date(x, format) to_timestamp(x, format) unix_timestamp(x, format) weekofyear(x) window(x, ...) year(x) ## S4 method for signature 'Column' dayofmonth(x) ## S4 method for signature 'Column' dayofweek(x) ## S4 method for signature 'Column' dayofyear(x) ## S4 method for signature 'Column' hour(x) ## S4 method for signature 'Column' last_day(x) ## S4 method for signature 'Column' minute(x) ## S4 method for signature 'Column' month(x) ## S4 method for signature 'Column' quarter(x) ## S4 method for signature 'Column' second(x) ## S4 method for signature 'Column,missing' to_date(x, format) ## S4 method for signature 'Column,character' to_date(x, format) ## S4 method for signature 'Column,missing' to_timestamp(x, format) ## S4 method for signature 'Column,character' to_timestamp(x, format) ## S4 method for signature 'Column' weekofyear(x) ## S4 method for signature 'Column' year(x) ## S4 method for signature 'Column' from_unixtime(x, format = "yyyy-MM-dd HH:mm:ss") ## S4 method for signature 'Column' window(x, windowDuration, slideDuration = NULL, startTime = NULL) ## S4 method for signature 'missing,missing' unix_timestamp(x, format) ## S4 method for signature 'Column,missing' unix_timestamp(x, format) ## S4 method for signature 'Column,character' unix_timestamp(x, format = "yyyy-MM-dd HH:mm:ss") ## S4 method for signature 'Column' trunc(x, format) ## S4 method for signature 'character,Column' date_trunc(format, x) ## S4 method for signature 'missing' current_date() ## S4 method for signature 'missing' current_timestamp() ## S4 method for signature 'Column' timestamp_seconds(x)current_date(x = "missing") current_timestamp(x = "missing") date_trunc(format, x) dayofmonth(x) dayofweek(x) dayofyear(x) from_unixtime(x, ...) hour(x) last_day(x) minute(x) month(x) quarter(x) second(x) timestamp_seconds(x) to_date(x, format) to_timestamp(x, format) unix_timestamp(x, format) weekofyear(x) window(x, ...) year(x) ## S4 method for signature 'Column' dayofmonth(x) ## S4 method for signature 'Column' dayofweek(x) ## S4 method for signature 'Column' dayofyear(x) ## S4 method for signature 'Column' hour(x) ## S4 method for signature 'Column' last_day(x) ## S4 method for signature 'Column' minute(x) ## S4 method for signature 'Column' month(x) ## S4 method for signature 'Column' quarter(x) ## S4 method for signature 'Column' second(x) ## S4 method for signature 'Column,missing' to_date(x, format) ## S4 method for signature 'Column,character' to_date(x, format) ## S4 method for signature 'Column,missing' to_timestamp(x, format) ## S4 method for signature 'Column,character' to_timestamp(x, format) ## S4 method for signature 'Column' weekofyear(x) ## S4 method for signature 'Column' year(x) ## S4 method for signature 'Column' from_unixtime(x, format = "yyyy-MM-dd HH:mm:ss") ## S4 method for signature 'Column' window(x, windowDuration, slideDuration = NULL, startTime = NULL) ## S4 method for signature 'missing,missing' unix_timestamp(x, format) ## S4 method for signature 'Column,missing' unix_timestamp(x, format) ## S4 method for signature 'Column,character' unix_timestamp(x, format = "yyyy-MM-dd HH:mm:ss") ## S4 method for signature 'Column' trunc(x, format) ## S4 method for signature 'character,Column' date_trunc(format, x) ## S4 method for signature 'missing' current_date() ## S4 method for signature 'missing' current_timestamp() ## S4 method for signature 'Column' timestamp_seconds(x)
x |
Column to compute on. In |
format |
The format for the given dates or timestamps in Column
|
... |
additional argument(s). |
windowDuration |
a string specifying the width of the window, e.g. '1 second', '1 day 12 hours', '2 minutes'. Valid interval strings are 'week', 'day', 'hour', 'minute', 'second', 'millisecond', 'microsecond'. Note that the duration is a fixed length of time, and does not vary over time according to a calendar. For example, '1 day' always means 86,400,000 milliseconds, not a calendar day. |
slideDuration |
a string specifying the sliding interval of the window. Same format as
|
startTime |
the offset with respect to 1970-01-01 00:00:00 UTC with which to start
window intervals. For example, in order to have hourly tumbling windows
that start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... provide
|
dayofmonth: Extracts the day of the month as an integer from a
given date/timestamp/string.
dayofweek: Extracts the day of the week as an integer from a
given date/timestamp/string.
dayofyear: Extracts the day of the year as an integer from a
given date/timestamp/string.
hour: Extracts the hour as an integer from a given date/timestamp/string.
last_day: Given a date column, returns the last day of the month which the
given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since
July 31 is the last day of the month in July 2015.
minute: Extracts the minute as an integer from a given date/timestamp/string.
month: Extracts the month as an integer from a given date/timestamp/string.
quarter: Extracts the quarter as an integer from a given date/timestamp/string.
second: Extracts the second as an integer from a given date/timestamp/string.
to_date: Converts the column into a DateType. You may optionally specify
a format according to the rules in:
Datetime Pattern
If the string cannot be parsed according to the specified format (or default),
the value of the column will be null.
By default, it follows casting rules to a DateType if the format is omitted
(equivalent to cast(df$x, "date")).
to_timestamp: Converts the column into a TimestampType. You may optionally specify
a format according to the rules in:
Datetime Pattern
If the string cannot be parsed according to the specified format (or default),
the value of the column will be null.
By default, it follows casting rules to a TimestampType if the format is omitted
(equivalent to cast(df$x, "timestamp")).
weekofyear: Extracts the week number as an integer from a given date/timestamp/string.
year: Extracts the year as an integer from a given date/timestamp/string.
from_unixtime: Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC)
to a string representing the timestamp of that moment in the current system time zone in the JVM
in the given format.
See
Datetime Pattern for available options.
window: Bucketizes rows into one or more time windows given a timestamp specifying column.
Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window
[12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in
the order of months are not supported. It returns an output column of struct called 'window'
by default with the nested columns 'start' and 'end'
unix_timestamp: Gets current Unix timestamp in seconds.
trunc: Returns date truncated to the unit specified by the format.
date_trunc: Returns timestamp truncated to the unit specified by the format.
current_date: Returns the current date at the start of query evaluation as a date column.
All calls of current_date within the same query return the same value.
current_timestamp: Returns the current timestamp at the start of query evaluation as
a timestamp column. All calls of current_timestamp within the same query return the same value.
timestamp_seconds: Creates timestamp from the number of seconds since UTC epoch.
dayofmonth since 1.5.0
dayofweek since 2.3.0
dayofyear since 1.5.0
hour since 1.5.0
last_day since 1.5.0
minute since 1.5.0
month since 1.5.0
quarter since 1.5.0
second since 1.5.0
to_date(Column) since 1.5.0
to_date(Column, character) since 2.2.0
to_timestamp(Column) since 2.2.0
to_timestamp(Column, character) since 2.2.0
weekofyear since 1.5.0
year since 1.5.0
from_unixtime since 1.5.0
window since 2.0.0
unix_timestamp since 1.5.0
unix_timestamp(Column) since 1.5.0
unix_timestamp(Column, character) since 1.5.0
trunc since 2.3.0
date_trunc since 2.3.0
current_date since 2.3.0
current_timestamp since 2.3.0
timestamp_seconds since 3.1.0
Other data time functions:
column_datetime_diff_functions
## Not run: dts <- c("2005-01-02 18:47:22", "2005-12-24 16:30:58", "2005-10-28 07:30:05", "2005-12-28 07:01:05", "2006-01-24 00:01:10") y <- c(2.0, 2.2, 3.4, 2.5, 1.8) df <- createDataFrame(data.frame(time = as.POSIXct(dts), y = y)) ## End(Not run) ## Not run: head(select(df, df$time, year(df$time), quarter(df$time), month(df$time), dayofmonth(df$time), dayofweek(df$time), dayofyear(df$time), weekofyear(df$time))) head(agg(groupBy(df, year(df$time)), count(df$y), avg(df$y))) head(agg(groupBy(df, month(df$time)), avg(df$y))) ## End(Not run) ## Not run: head(select(df, hour(df$time), minute(df$time), second(df$time))) head(agg(groupBy(df, dayofmonth(df$time)), avg(df$y))) head(agg(groupBy(df, hour(df$time)), avg(df$y))) head(agg(groupBy(df, minute(df$time)), avg(df$y))) ## End(Not run) ## Not run: head(select(df, df$time, last_day(df$time), month(df$time))) ## End(Not run) ## Not run: tmp <- createDataFrame(data.frame(time_string = dts)) tmp2 <- mutate(tmp, date1 = to_date(tmp$time_string), date2 = to_date(tmp$time_string, "yyyy-MM-dd"), date3 = date_format(tmp$time_string, "MM/dd/yyy"), time1 = to_timestamp(tmp$time_string), time2 = to_timestamp(tmp$time_string, "yyyy-MM-dd")) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, to_unix = unix_timestamp(df$time), to_unix2 = unix_timestamp(df$time, 'yyyy-MM-dd HH'), from_unix = from_unixtime(unix_timestamp(df$time)), from_unix2 = from_unixtime(unix_timestamp(df$time), 'yyyy-MM-dd HH:mm')) head(tmp) ## End(Not run) ## Not run: # One minute windows every 15 seconds 10 seconds after the minute, e.g. 09:00:10-09:01:10, # 09:00:25-09:01:25, 09:00:40-09:01:40, ... window(df$time, "1 minute", "15 seconds", "10 seconds") # One minute tumbling windows 15 seconds after the minute, e.g. 09:00:15-09:01:15, # 09:01:15-09:02:15... window(df$time, "1 minute", startTime = "15 seconds") # Thirty-second windows every 10 seconds, e.g. 09:00:00-09:00:30, 09:00:10-09:00:40, ... window(df$time, "30 seconds", "10 seconds") ## End(Not run) ## Not run: head(select(df, df$time, trunc(df$time, "year"), trunc(df$time, "yy"), trunc(df$time, "month"), trunc(df$time, "mon"))) ## End(Not run) ## Not run: head(select(df, df$time, date_trunc("hour", df$time), date_trunc("minute", df$time), date_trunc("week", df$time), date_trunc("quarter", df$time))) ## End(Not run) ## Not run: head(select(df, current_date(), current_timestamp())) ## End(Not run)## Not run: dts <- c("2005-01-02 18:47:22", "2005-12-24 16:30:58", "2005-10-28 07:30:05", "2005-12-28 07:01:05", "2006-01-24 00:01:10") y <- c(2.0, 2.2, 3.4, 2.5, 1.8) df <- createDataFrame(data.frame(time = as.POSIXct(dts), y = y)) ## End(Not run) ## Not run: head(select(df, df$time, year(df$time), quarter(df$time), month(df$time), dayofmonth(df$time), dayofweek(df$time), dayofyear(df$time), weekofyear(df$time))) head(agg(groupBy(df, year(df$time)), count(df$y), avg(df$y))) head(agg(groupBy(df, month(df$time)), avg(df$y))) ## End(Not run) ## Not run: head(select(df, hour(df$time), minute(df$time), second(df$time))) head(agg(groupBy(df, dayofmonth(df$time)), avg(df$y))) head(agg(groupBy(df, hour(df$time)), avg(df$y))) head(agg(groupBy(df, minute(df$time)), avg(df$y))) ## End(Not run) ## Not run: head(select(df, df$time, last_day(df$time), month(df$time))) ## End(Not run) ## Not run: tmp <- createDataFrame(data.frame(time_string = dts)) tmp2 <- mutate(tmp, date1 = to_date(tmp$time_string), date2 = to_date(tmp$time_string, "yyyy-MM-dd"), date3 = date_format(tmp$time_string, "MM/dd/yyy"), time1 = to_timestamp(tmp$time_string), time2 = to_timestamp(tmp$time_string, "yyyy-MM-dd")) head(tmp2) ## End(Not run) ## Not run: tmp <- mutate(df, to_unix = unix_timestamp(df$time), to_unix2 = unix_timestamp(df$time, 'yyyy-MM-dd HH'), from_unix = from_unixtime(unix_timestamp(df$time)), from_unix2 = from_unixtime(unix_timestamp(df$time), 'yyyy-MM-dd HH:mm')) head(tmp) ## End(Not run) ## Not run: # One minute windows every 15 seconds 10 seconds after the minute, e.g. 09:00:10-09:01:10, # 09:00:25-09:01:25, 09:00:40-09:01:40, ... window(df$time, "1 minute", "15 seconds", "10 seconds") # One minute tumbling windows 15 seconds after the minute, e.g. 09:00:15-09:01:15, # 09:01:15-09:02:15... window(df$time, "1 minute", startTime = "15 seconds") # Thirty-second windows every 10 seconds, e.g. 09:00:00-09:00:30, 09:00:10-09:00:40, ... window(df$time, "30 seconds", "10 seconds") ## End(Not run) ## Not run: head(select(df, df$time, trunc(df$time, "year"), trunc(df$time, "yy"), trunc(df$time, "month"), trunc(df$time, "mon"))) ## End(Not run) ## Not run: head(select(df, df$time, date_trunc("hour", df$time), date_trunc("minute", df$time), date_trunc("week", df$time), date_trunc("quarter", df$time))) ## End(Not run) ## Not run: head(select(df, current_date(), current_timestamp())) ## End(Not run)
Converts a SparkDataFrame into a SparkDataFrame of JSON string.
Each row is turned into a JSON document with columns as different fields.
The returned SparkDataFrame has a single character column with the name value
## S4 method for signature 'SparkDataFrame' toJSON(x)## S4 method for signature 'SparkDataFrame' toJSON(x)
x |
a SparkDataFrame |
a SparkDataFrame
toJSON since 2.2.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.parquet" df <- read.parquet(path) df_json <- toJSON(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.parquet" df <- read.parquet(path) df_json <- toJSON(df) ## End(Not run)
Removes the specified table from the in-memory cache.
uncacheTable(tableName)uncacheTable(tableName)
tableName |
the qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database. |
SparkDataFrame
uncacheTable since 1.4.0
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") uncacheTable("table") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) createOrReplaceTempView(df, "table") uncacheTable("table") ## End(Not run)
Return a new SparkDataFrame containing the union of rows in this SparkDataFrame
and another SparkDataFrame. This is equivalent to UNION ALL in SQL.
Input SparkDataFrames can have different schemas (names and data types).
union(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' union(x, y)union(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' union(x, y)
x |
A SparkDataFrame |
y |
A SparkDataFrame |
Note: This does not remove duplicate rows across the two SparkDataFrames. Also as standard in SQL, this function resolves columns by position (not by name).
A SparkDataFrame containing the result of the union.
union since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) unioned <- union(df, df2) unions <- rbind(df, df2, df3, df4) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) unioned <- union(df, df2) unions <- rbind(df, df2, df3, df4) ## End(Not run)
This is an alias for union.
unionAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' unionAll(x, y)unionAll(x, y) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' unionAll(x, y)
x |
a SparkDataFrame. |
y |
a SparkDataFrame. |
A SparkDataFrame containing the result of the unionAll operation.
unionAll since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) unionAllDF <- unionAll(df1, df2) ## End(Not run)## Not run: sparkR.session() df1 <- read.json(path) df2 <- read.json(path2) unionAllDF <- unionAll(df1, df2) ## End(Not run)
Return a new SparkDataFrame containing the union of rows in this SparkDataFrame
and another SparkDataFrame. This is different from union function, and both
UNION ALL and UNION DISTINCT in SQL as column positions are not taken
into account. Input SparkDataFrames can have different data types in the schema.
unionByName(x, y, ...) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' unionByName(x, y, allowMissingColumns = FALSE)unionByName(x, y, ...) ## S4 method for signature 'SparkDataFrame,SparkDataFrame' unionByName(x, y, allowMissingColumns = FALSE)
x |
A SparkDataFrame |
y |
A SparkDataFrame |
... |
further arguments to be passed to or from other methods. |
allowMissingColumns |
logical |
When the parameter allowMissingColumns is 'TRUE', the set of column names in x and y can differ; missing columns will be filled as null. Further, the missing columns of x will be added at the end in the schema of the union result.
Note: This does not remove duplicate rows across the two SparkDataFrames. This function resolves columns by name (not by position).
A SparkDataFrame containing the result of the union.
unionByName since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() df1 <- select(createDataFrame(mtcars), "carb", "am", "gear") df2 <- select(createDataFrame(mtcars), "am", "gear", "carb") head(unionByName(df1, df2)) df3 <- select(createDataFrame(mtcars), "carb") head(unionByName(df1, df3, allowMissingColumns = TRUE)) ## End(Not run)## Not run: sparkR.session() df1 <- select(createDataFrame(mtcars), "carb", "am", "gear") df2 <- select(createDataFrame(mtcars), "am", "gear", "carb") head(unionByName(df1, df2)) df3 <- select(createDataFrame(mtcars), "carb") head(unionByName(df1, df3, allowMissingColumns = TRUE)) ## End(Not run)
Mark this SparkDataFrame as non-persistent, and remove all blocks for it from memory and disk.
unpersist(x, ...) ## S4 method for signature 'SparkDataFrame' unpersist(x, blocking = TRUE)unpersist(x, ...) ## S4 method for signature 'SparkDataFrame' unpersist(x, blocking = TRUE)
x |
the SparkDataFrame to unpersist. |
... |
further arguments to be passed to or from other methods. |
blocking |
whether to block until all blocks are deleted. |
unpersist since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") unpersist(df) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) persist(df, "MEMORY_AND_DISK") unpersist(df) ## End(Not run)
Create o.a.s.sql.expressions.UnresolvedNamedLambdaVariable, convert it to o.s.sql.Column and wrap with R Column. Used by higher order functions.
unresolved_named_lambda_var(...)unresolved_named_lambda_var(...)
... |
character of length = 1
if length(...) > 1 then argument is interpreted as a nested
Column, for example |
Column object wrapping JVM UnresolvedNamedLambdaVariable
Creates a WindowSpec with the ordering defined.
windowOrderBy(col, ...) ## S4 method for signature 'character' windowOrderBy(col, ...) ## S4 method for signature 'Column' windowOrderBy(col, ...)windowOrderBy(col, ...) ## S4 method for signature 'character' windowOrderBy(col, ...) ## S4 method for signature 'Column' windowOrderBy(col, ...)
col |
A column name or Column by which rows are ordered within windows. |
... |
Optional column names or Columns in addition to col, by which rows are ordered within windows. |
windowOrderBy(character) since 2.0.0
windowOrderBy(Column) since 2.0.0
## Not run: ws <- windowOrderBy("key1", "key2") df1 <- select(df, over(lead("value", 1), ws)) ws <- windowOrderBy(df$key1, df$key2) df1 <- select(df, over(lead("value", 1), ws)) ## End(Not run)## Not run: ws <- windowOrderBy("key1", "key2") df1 <- select(df, over(lead("value", 1), ws)) ws <- windowOrderBy(df$key1, df$key2) df1 <- select(df, over(lead("value", 1), ws)) ## End(Not run)
Creates a WindowSpec with the partitioning defined.
windowPartitionBy(col, ...) ## S4 method for signature 'character' windowPartitionBy(col, ...) ## S4 method for signature 'Column' windowPartitionBy(col, ...)windowPartitionBy(col, ...) ## S4 method for signature 'character' windowPartitionBy(col, ...) ## S4 method for signature 'Column' windowPartitionBy(col, ...)
col |
A column name or Column by which rows are partitioned to windows. |
... |
Optional column names or Columns in addition to col, by which rows are partitioned to windows. |
windowPartitionBy(character) since 2.0.0
windowPartitionBy(Column) since 2.0.0
## Not run: ws <- orderBy(windowPartitionBy("key1", "key2"), "key3") df1 <- select(df, over(lead("value", 1), ws)) ws <- orderBy(windowPartitionBy(df$key1, df$key2), df$key3) df1 <- select(df, over(lead("value", 1), ws)) ## End(Not run)## Not run: ws <- orderBy(windowPartitionBy("key1", "key2"), "key3") df1 <- select(df, over(lead("value", 1), ws)) ws <- orderBy(windowPartitionBy(df$key1, df$key2), df$key3) df1 <- select(df, over(lead("value", 1), ws)) ## End(Not run)
WindowSpec can be created by using windowPartitionBy() or windowOrderBy()
sws |
A Java object reference to the backing Scala WindowSpec |
WindowSpec since 2.0.0
windowPartitionBy, windowOrderBy
Evaluate a R expression in an environment constructed from a SparkDataFrame with() allows access to columns of a SparkDataFrame by simply referring to their name. It appends every column of a SparkDataFrame into a new environment. Then, the given expression is evaluated in this new environment.
with(data, expr, ...) ## S4 method for signature 'SparkDataFrame' with(data, expr, ...)with(data, expr, ...) ## S4 method for signature 'SparkDataFrame' with(data, expr, ...)
data |
(SparkDataFrame) SparkDataFrame to use for constructing an environment. |
expr |
(expression) Expression to evaluate. |
... |
arguments to be passed to future methods. |
with since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: with(irisDf, nrow(Sepal_Width)) ## End(Not run)## Not run: with(irisDf, nrow(Sepal_Width)) ## End(Not run)
Return a new SparkDataFrame by adding a column or replacing the existing column that has the same name.
withColumn(x, colName, col) ## S4 method for signature 'SparkDataFrame,character' withColumn(x, colName, col)withColumn(x, colName, col) ## S4 method for signature 'SparkDataFrame,character' withColumn(x, colName, col)
x |
a SparkDataFrame. |
colName |
a column name. |
col |
a Column expression (which must refer only to this SparkDataFrame), or an atomic vector in the length of 1 as literal value. |
Note: This method introduces a projection internally. Therefore, calling it multiple times,
for instance, via loops in order to add multiple columns can generate big plans which
can cause performance issues and even StackOverflowException. To avoid this,
use select with the multiple columns at once.
A SparkDataFrame with the new column added or the existing column replaced.
withColumn since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- withColumn(df, "newCol", df$col1 * 5) # Replace an existing column newDF2 <- withColumn(newDF, "newCol", newDF$col1) newDF3 <- withColumn(newDF, "newCol", 42) # Use extract operator to set an existing or new column df[["age"]] <- 23 df[[2]] <- df$col1 df[[2]] <- NULL # drop column ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) newDF <- withColumn(df, "newCol", df$col1 * 5) # Replace an existing column newDF2 <- withColumn(newDF, "newCol", newDF$col1) newDF3 <- withColumn(newDF, "newCol", 42) # Use extract operator to set an existing or new column df[["age"]] <- 23 df[[2]] <- df$col1 df[[2]] <- NULL # drop column ## End(Not run)
Adds/replaces field in a struct Column by name.
withField(x, fieldName, col) ## S4 method for signature 'Column,character,Column' withField(x, fieldName, col)withField(x, fieldName, col) ## S4 method for signature 'Column,character,Column' withField(x, fieldName, col)
x |
a Column |
fieldName |
a character |
col |
a Column expression |
withField since 3.1.0
## Not run: df <- withColumn( createDataFrame(iris), "sepal", struct(column("Sepal_Width"), column("Sepal_Length")) ) head(select( df, withField(df$sepal, "product", df$Sepal_Length * df$Sepal_Width) )) ## End(Not run)## Not run: df <- withColumn( createDataFrame(iris), "sepal", struct(column("Sepal_Width"), column("Sepal_Length")) ) head(select( df, withField(df$sepal, "product", df$Sepal_Length * df$Sepal_Width) )) ## End(Not run)
Defines an event time watermark for this streaming SparkDataFrame. A watermark tracks a point in time before which we assume no more late data is going to arrive.
withWatermark(x, eventTime, delayThreshold) ## S4 method for signature 'SparkDataFrame,character,character' withWatermark(x, eventTime, delayThreshold)withWatermark(x, eventTime, delayThreshold) ## S4 method for signature 'SparkDataFrame,character,character' withWatermark(x, eventTime, delayThreshold)
x |
a streaming SparkDataFrame |
eventTime |
a string specifying the name of the Column that contains the event time of the row. |
delayThreshold |
a string specifying the minimum delay to wait to data to arrive late, relative to the latest record that has been processed in the form of an interval (e.g. "1 minute" or "5 hours"). NOTE: This should not be negative. |
Spark will use this watermark for several purposes:
To know when a given time window aggregation can be finalized and thus can be emitted when using output modes that do not allow updates.
To minimize the amount of state that we need to keep for on-going aggregations.
The current watermark is computed by looking at the MAX(eventTime) seen across
all of the partitions in the query minus a user specified delayThreshold. Due to the cost
of coordinating this value across partitions, the actual watermark used is only guaranteed
to be at least delayThreshold behind the actual event time. In some cases we may still
process records that arrive more than delayThreshold late.
a SparkDataFrame.
withWatermark since 2.3.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() schema <- structType(structField("time", "timestamp"), structField("value", "double")) df <- read.stream("json", path = jsonDir, schema = schema, maxFilesPerTrigger = 1) df <- withWatermark(df, "time", "10 minutes") ## End(Not run)## Not run: sparkR.session() schema <- structType(structField("time", "timestamp"), structField("value", "double")) df <- read.stream("json", path = jsonDir, schema = schema, maxFilesPerTrigger = 1) df <- withWatermark(df, "time", "10 minutes") ## End(Not run)
The data source is specified by the source and a set of options (...).
If source is not specified, the default data source configured by
spark.sql.sources.default will be used.
write.df(df, path = NULL, ...) saveDF(df, path, source = NULL, mode = "error", ...) write.df(df, path = NULL, ...) ## S4 method for signature 'SparkDataFrame' write.df( df, path = NULL, source = NULL, mode = "error", partitionBy = NULL, ... ) ## S4 method for signature 'SparkDataFrame,character' saveDF(df, path, source = NULL, mode = "error", ...)write.df(df, path = NULL, ...) saveDF(df, path, source = NULL, mode = "error", ...) write.df(df, path = NULL, ...) ## S4 method for signature 'SparkDataFrame' write.df( df, path = NULL, source = NULL, mode = "error", partitionBy = NULL, ... ) ## S4 method for signature 'SparkDataFrame,character' saveDF(df, path, source = NULL, mode = "error", ...)
df |
a SparkDataFrame. |
path |
a name for the table. |
... |
additional argument(s) passed to the method. |
source |
a name for external data source. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
partitionBy |
a name or a list of names of columns to partition the output by on the file system. If specified, the output is laid out on the file system similar to Hive's partitioning scheme. |
Additionally, mode is used to specify the behavior of the save operation when data already exists in the data source. There are four modes:
'append': Contents of this SparkDataFrame are expected to be appended to existing data.
'overwrite': Existing data is expected to be overwritten by the contents of this SparkDataFrame.
'error' or 'errorifexists': An exception is expected to be thrown.
'ignore': The save operation is expected to not save the contents of the SparkDataFrame and to not change the existing data.
write.df since 1.4.0
saveDF since 1.4.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.df(df, "myfile", "parquet", "overwrite", partitionBy = c("col1", "col2")) saveDF(df, parquetPath2, "parquet", mode = "append", mergeSchema = TRUE) ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.df(df, "myfile", "parquet", "overwrite", partitionBy = c("col1", "col2")) saveDF(df, parquetPath2, "parquet", mode = "append", mergeSchema = TRUE) ## End(Not run)
Save the content of the SparkDataFrame to an external database table via JDBC. Additional JDBC database connection properties can be set (...)
write.jdbc(x, url, tableName, mode = "error", ...) ## S4 method for signature 'SparkDataFrame,character,character' write.jdbc(x, url, tableName, mode = "error", ...)write.jdbc(x, url, tableName, mode = "error", ...) ## S4 method for signature 'SparkDataFrame,character,character' write.jdbc(x, url, tableName, mode = "error", ...)
x |
a SparkDataFrame. |
url |
JDBC database url of the form |
tableName |
the name of the table in the external database. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
... |
additional JDBC database connection properties. |
Also, mode is used to specify the behavior of the save operation when data already exists in the data source. There are four modes:
'append': Contents of this SparkDataFrame are expected to be appended to existing data.
'overwrite': Existing data is expected to be overwritten by the contents of this SparkDataFrame.
'error' or 'errorifexists': An exception is expected to be thrown.
'ignore': The save operation is expected to not save the contents of the SparkDataFrame and to not change the existing data.
write.jdbc since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.json(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() jdbcUrl <- "jdbc:mysql://localhost:3306/databasename" write.jdbc(df, jdbcUrl, "table", user = "username", password = "password") ## End(Not run)## Not run: sparkR.session() jdbcUrl <- "jdbc:mysql://localhost:3306/databasename" write.jdbc(df, jdbcUrl, "table", user = "username", password = "password") ## End(Not run)
Save the contents of a SparkDataFrame as a JSON file ( JSON Lines text format or newline-delimited JSON). Files written out with this method can be read back in as a SparkDataFrame using read.json().
write.json(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.json(x, path, mode = "error", ...)write.json(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.json(x, path, mode = "error", ...)
x |
A SparkDataFrame |
path |
The directory where the file is saved |
... |
additional argument(s) passed to the method. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
write.json since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.orc(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.json(df, "/tmp/sparkr-tmp/") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.json(df, "/tmp/sparkr-tmp/") ## End(Not run)
Saves the MLlib model to the input path. For more information, see the specific MLlib model below.
write.ml(object, path, ...)write.ml(object, path, ...)
object |
a fitted ML model object. |
path |
the directory where the model is saved. |
... |
additional argument(s) passed to the method. |
spark.als, spark.bisectingKmeans, spark.decisionTree,
spark.gaussianMixture, spark.gbt,
spark.randomForest, spark.survreg, spark.svmLinear,
Save the contents of a SparkDataFrame as an ORC file, preserving the schema. Files written out with this method can be read back in as a SparkDataFrame using read.orc().
write.orc(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.orc(x, path, mode = "error", ...)write.orc(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.orc(x, path, mode = "error", ...)
x |
A SparkDataFrame |
path |
The directory where the file is saved |
... |
additional argument(s) passed to the method. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
write.orc since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.parquet(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.orc(df, "/tmp/sparkr-tmp1/") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.orc(df, "/tmp/sparkr-tmp1/") ## End(Not run)
Save the contents of a SparkDataFrame as a Parquet file, preserving the schema. Files written out with this method can be read back in as a SparkDataFrame using read.parquet().
write.parquet(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.parquet(x, path, mode = "error", ...)write.parquet(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.parquet(x, path, mode = "error", ...)
x |
A SparkDataFrame |
path |
The directory where the file is saved |
... |
additional argument(s) passed to the method. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
write.parquet since 1.6.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.stream(),
write.text()
## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.parquet(df, "/tmp/sparkr-tmp1/") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.json" df <- read.json(path) write.parquet(df, "/tmp/sparkr-tmp1/") ## End(Not run)
The data source is specified by the source and a set of options (...).
If source is not specified, the default data source configured by
spark.sql.sources.default will be used.
write.stream(df, source = NULL, outputMode = NULL, ...) ## S4 method for signature 'SparkDataFrame' write.stream( df, source = NULL, outputMode = NULL, partitionBy = NULL, trigger.processingTime = NULL, trigger.once = NULL, ... )write.stream(df, source = NULL, outputMode = NULL, ...) ## S4 method for signature 'SparkDataFrame' write.stream( df, source = NULL, outputMode = NULL, partitionBy = NULL, trigger.processingTime = NULL, trigger.once = NULL, ... )
df |
a streaming SparkDataFrame. |
source |
a name for external data source. |
outputMode |
one of 'append', 'complete', 'update'. |
... |
additional external data source specific named options. |
partitionBy |
a name or a list of names of columns to partition the output by on the file system. If specified, the output is laid out on the file system similar to Hive's partitioning scheme. |
trigger.processingTime |
a processing time interval as a string, e.g. '5 seconds', '1 minute'. This is a trigger that runs a query periodically based on the processing time. If value is '0 seconds', the query will run as fast as possible, this is the default. Only one trigger can be set. |
trigger.once |
a logical, must be set to |
Additionally, outputMode specifies how data of a streaming SparkDataFrame is written to a
output data source. There are three modes:
append: Only the new rows in the streaming SparkDataFrame will be written out. This output mode can be only be used in queries that do not contain any aggregation.
complete: All the rows in the streaming SparkDataFrame will be written out every time there are some updates. This output mode can only be used in queries that contain aggregations.
update: Only the rows that were updated in the streaming SparkDataFrame will be written
out every time there are some updates. If the query doesn't contain aggregations,
it will be equivalent to append mode.
write.stream since 2.2.0
experimental
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.text()
## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) isStreaming(df) wordCounts <- count(group_by(df, "value")) # console q <- write.stream(wordCounts, "console", outputMode = "complete") # text stream q <- write.stream(df, "text", path = "/home/user/out", checkpointLocation = "/home/user/cp" partitionBy = c("year", "month"), trigger.processingTime = "30 seconds") # memory stream q <- write.stream(wordCounts, "memory", queryName = "outs", outputMode = "complete") head(sql("SELECT * from outs")) queryName(q) stopQuery(q) ## End(Not run)## Not run: sparkR.session() df <- read.stream("socket", host = "localhost", port = 9999) isStreaming(df) wordCounts <- count(group_by(df, "value")) # console q <- write.stream(wordCounts, "console", outputMode = "complete") # text stream q <- write.stream(df, "text", path = "/home/user/out", checkpointLocation = "/home/user/cp" partitionBy = c("year", "month"), trigger.processingTime = "30 seconds") # memory stream q <- write.stream(wordCounts, "memory", queryName = "outs", outputMode = "complete") head(sql("SELECT * from outs")) queryName(q) stopQuery(q) ## End(Not run)
Save the content of the SparkDataFrame in a text file at the specified path. The SparkDataFrame must have only one column of string type with the name "value". Each row becomes a new line in the output file. The text files will be encoded as UTF-8.
write.text(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.text(x, path, mode = "error", ...)write.text(x, path, ...) ## S4 method for signature 'SparkDataFrame,character' write.text(x, path, mode = "error", ...)
x |
A SparkDataFrame |
path |
The directory where the file is saved |
... |
additional argument(s) passed to the method. |
mode |
one of 'append', 'overwrite', 'error', 'errorifexists', 'ignore' save mode (it is 'error' by default) |
write.text since 2.0.0
Other SparkDataFrame functions:
SparkDataFrame-class,
agg(),
alias(),
arrange(),
as.data.frame(),
attach,SparkDataFrame-method,
broadcast(),
cache(),
checkpoint(),
coalesce(),
collect(),
colnames(),
coltypes(),
createOrReplaceTempView(),
crossJoin(),
cube(),
dapplyCollect(),
dapply(),
describe(),
dim(),
distinct(),
dropDuplicates(),
dropna(),
drop(),
dtypes(),
exceptAll(),
except(),
explain(),
filter(),
first(),
gapplyCollect(),
gapply(),
getNumPartitions(),
group_by(),
head(),
hint(),
histogram(),
insertInto(),
intersectAll(),
intersect(),
isLocal(),
isStreaming(),
join(),
limit(),
localCheckpoint(),
merge(),
mutate(),
ncol(),
nrow(),
persist(),
printSchema(),
randomSplit(),
rbind(),
rename(),
repartitionByRange(),
repartition(),
rollup(),
sample(),
saveAsTable(),
schema(),
selectExpr(),
select(),
showDF(),
show(),
storageLevel(),
str(),
subset(),
summary(),
take(),
toJSON(),
unionAll(),
unionByName(),
union(),
unpersist(),
withColumn(),
withWatermark(),
with(),
write.df(),
write.jdbc(),
write.json(),
write.orc(),
write.parquet(),
write.stream()
## Not run: sparkR.session() path <- "path/to/file.txt" df <- read.text(path) write.text(df, "/tmp/sparkr-tmp/") ## End(Not run)## Not run: sparkR.session() path <- "path/to/file.txt" df <- read.text(path) write.text(df, "/tmp/sparkr-tmp/") ## End(Not run)