Package: SparkR 3.1.2.9002

Felix Cheung
SparkR: R Front End for 'Apache Spark'
Provides an R Front end for 'Apache Spark' <https://spark.apache.org>. Compatibility updates by Renova.
Authors:
SparkR_3.1.2.9002.tar.gz
SparkR_3.1.2.9002.zip(r-4.7-any)SparkR_3.1.2.9002.zip(r-4.6-any)SparkR_3.1.2.9002.zip(r-4.5-any)
SparkR_3.1.2.9002.tgz(r-4.6-any)SparkR_3.1.2.9002.tgz(r-4.5-any)
SparkR_3.1.2.9002.tar.gz(r-4.7-any)SparkR_3.1.2.9002.tar.gz(r-4.6-any)
SparkR_3.1.2.9002.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION
card.svg |card.png
SparkR/json (API)
| # Install 'SparkR' in R: |
| install.packages('SparkR', repos = c('https://renova-r.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/renova-r/sparkr/issues
Last updated from:08ea32d8bc (on renova-3.1.2.9002). Checks:7 NOTE, 2 OK. Indexed: yes.
| Target | Result | Time | Files | Syslog |
|---|---|---|---|---|
| linux-devel | NOTE | 201 | ||
| source / vignettes | OK | 284 | ||
| linux-release | NOTE | 162 | ||
| macos-release | NOTE | 131 | ||
| macos-oldrel | NOTE | 140 | ||
| windows-devel | NOTE | 118 | ||
| windows-release | NOTE | 108 | ||
| windows-oldrel | NOTE | 104 | ||
| wasm-release | OK | 108 |
Exports:%<=>%%in%add_monthsaggaliasapprox_count_distinctapproxCountDistinctapproxQuantilearrangearray_aggregatearray_containsarray_distinctarray_exceptarray_existsarray_filterarray_forallarray_intersectarray_joinarray_maxarray_minarray_positionarray_removearray_repeatarray_sortarray_to_vectorarray_transformarray_unionarrays_overlaparrays_ziparrays_zip_withas.data.frameas.DataFrameascasc_nulls_firstasc_nulls_lastasciiassert_trueatan2attachavgawaitTerminationbase64betweenbinbitwiseNOTbroadcastbroundcachecacheTablecancelJobGroupcastcbrtceilcheckpointclearCacheclearJobGroupcoalescecollectcollect_listcollect_setcolnamescolnames<-coltypescoltypes<-columncolumnsconcatconcat_wscontainsconvcorrcountcountDistinctcovcovar_popcovar_sampcrc32create_arraycreate_mapcreateDataFramecreateExternalTablecreateOrReplaceTempViewcreateTablecrossJoincrosstabcubecume_distcurrent_datecurrent_timestampcurrentDatabasedapplydapplyCollectdate_adddate_formatdate_subdate_truncdatediffdayofmonthdayofweekdayofyeardecodedegreesdense_rankdescdesc_nulls_firstdesc_nulls_lastdescribedistinctdropdropDuplicatesdropFieldsdropnadropTempTabledropTempViewdtypeselement_atencodeendsWithexceptexceptAllexplainexplodeexplode_outerexprfactorialfillnafilterfirstfittedflattenformat_numberformat_stringfreqItemsfrom_avrofrom_csvfrom_jsonfrom_unixtimefrom_utc_timestampgapplygapplyCollectgetFieldgetItemgetLocalPropertygetNumPartitionsglmgreatestgroup_bygroupBygrouping_bitgrouping_idhashheadhexhinthistogramhourhypotifelseinitcapinput_file_nameinsertIntoinstall.sparkinstrintersectintersectAllisActiveisLocalisnanisNaNisNotNullisNullisStreamingjoinkurtosislaglastlast_daylastProgressleadleastlevenshteinlikelimitlistColumnslistDatabaseslistFunctionslistTableslitloadDFlocalCheckpointlocatelowerlpadltrimmap_concatmap_entriesmap_filtermap_from_arraysmap_from_entriesmap_keysmap_valuesmap_zip_withmd5meanmergeminutemonotonically_increasing_idmonthmonths_betweenmutatenn_distinctna.omitnanvlncolnegatenext_daynotnrownth_valuentileorderByotherwiseoveroverlaypartitionBypercent_rankpercentile_approxpersistpivotpmodposexplodeposexplode_outerpredictprint.jobjprint.structFieldprint.structTypeprint.summary.DecisionTreeClassificationModelprint.summary.DecisionTreeRegressionModelprint.summary.GBTClassificationModelprint.summary.GBTRegressionModelprint.summary.GeneralizedLinearRegressionModelprint.summary.KSTestprint.summary.RandomForestClassificationModelprint.summary.RandomForestRegressionModelprintSchemaquarterqueryNameradiansraise_errorrandrandnrandomSplitrangeBetweenrankrbindread.dfread.jdbcread.jsonread.mlread.orcread.parquetread.streamread.textrecoverPartitionsrefreshByPathrefreshTableregexp_extractregexp_replaceregisterTempTablerenamerepartitionrepartitionByRangerepeat_stringreverserintrlikerolluprow_numberrowsBetweenrpadrtrimsamplesample_fracsampleBysaveAsTablesaveDFschemaschema_of_csvschema_of_jsonsdsecondselectselectExprsetCheckpointDirsetCurrentDatabasesetJobDescriptionsetJobGroupsetLocalPropertysetLogLevelsha1sha2shiftLeftshiftRightshiftRightUnsignedshowshowDFshufflesignumsizeskewnessslicesort_arraysoundexspark_partition_idspark.addFilespark.alsspark.assignClustersspark.associationRulesspark.bisectingKmeansspark.decisionTreespark.findFrequentSequentialPatternsspark.fmClassifierspark.fmRegressorspark.fpGrowthspark.freqItemsetsspark.gaussianMixturespark.gbtspark.getSparkFilesspark.getSparkFilesRootDirectoryspark.glmspark.isoregspark.kmeansspark.kstestspark.lapplyspark.ldaspark.lmspark.logitspark.mlpspark.naiveBayesspark.perplexityspark.posteriorspark.randomForestspark.survregspark.svmLinearsparkR.callJMethodsparkR.callJStaticsparkR.confsparkR.initsparkR.newJObjectsparkR.sessionsparkR.session.stopsparkR.stopsparkR.uiWebUrlsparkR.versionsparkRHive.initsparkRSQL.initsplit_stringsqlstartsWithstatusstddevstddev_popstddev_sampstopQuerystorageLevelstrstructstructFieldstructField.characterstructField.jobjstructTypestructType.characterstructType.jobjstructType.structFieldsubsetsubstrsubstring_indexsumDistinctsummarizesummarytableNamestablestableToDFtaketimestamp_secondsto_avroto_csvto_dateto_jsonto_timestampto_utc_timestamptoDegreestoJSONtoRadianstransformtransform_keystransform_valuestranslatetrimunbase64uncacheTableunhexunionunionAllunionByNameuniqueunix_timestampunpersistuppervarvar_popvar_sampvariancevector_to_arrayweekofyearwhenwherewindowwindowOrderBywindowPartitionBywithwithColumnwithColumnRenamedwithFieldwithWatermarkwrite.dfwrite.jdbcwrite.jsonwrite.mlwrite.orcwrite.parquetwrite.streamwrite.textxxhash64year
Dependencies:
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| %<=>% | %<=>% %<=>%,Column-method |
| Match a column with given values. | %in% %in%,Column-method |
| S4 class that represents a AFTSurvivalRegressionModel | AFTSurvivalRegressionModel-class |
| summarize | agg agg,GroupedData-method agg,SparkDataFrame-method summarize summarize,GroupedData-method summarize,SparkDataFrame-method |
| alias | alias alias,Column-method alias,SparkDataFrame-method |
| S4 class that represents an ALSModel | ALSModel-class |
| Calculates the approximate quantiles of numerical columns of a SparkDataFrame | approxQuantile approxQuantile,SparkDataFrame,character,numeric,numeric-method |
| Arrange Rows by Variables | arrange arrange,SparkDataFrame,character-method arrange,SparkDataFrame,Column-method orderBy,SparkDataFrame,characterOrColumn-method |
| Download data from a SparkDataFrame into a R data.frame | as.data.frame as.data.frame,SparkDataFrame-method |
| A set of operations working with SparkDataFrame columns | asc asc_nulls_first asc_nulls_last columnfunctions contains desc desc_nulls_first desc_nulls_last getField getItem isNaN isNotNull isNull like rlike |
| Attach SparkDataFrame to R search path | attach attach,SparkDataFrame-method |
| avg | avg avg,Column-method |
| awaitTermination | awaitTermination awaitTermination,StreamingQuery-method |
| between | between between,Column-method |
| S4 class that represents a BisectingKMeansModel | BisectingKMeansModel-class |
| broadcast | broadcast broadcast,SparkDataFrame-method |
| Cache | cache cache,SparkDataFrame-method |
| Cache Table | cacheTable |
| Cancel active jobs for the specified group | cancelJobGroup |
| Casts the column to a different data type. | cast cast,Column-method |
| checkpoint | checkpoint checkpoint,SparkDataFrame-method |
| Clear Cache | clearCache |
| Clear current job group ID and its description | clearJobGroup |
| Coalesce | coalesce coalesce,SparkDataFrame-method |
| Collects all the elements of a SparkDataFrame and coerces them into an R data.frame. | collect collect,SparkDataFrame-method |
| Column Names of SparkDataFrame | colnames colnames,SparkDataFrame-method colnames<- colnames<-,SparkDataFrame-method columns columns,SparkDataFrame-method names names,SparkDataFrame-method names<- names<-,SparkDataFrame-method |
| coltypes | coltypes coltypes,SparkDataFrame-method coltypes<- coltypes<-,SparkDataFrame,character-method |
| S4 class that represents a SparkDataFrame column | column column,character-method column,jobj-method Column-class |
| Aggregate functions for Column operations | approxCountDistinct approxCountDistinct,Column-method approx_count_distinct approx_count_distinct,Column-method collect_list collect_list,Column-method collect_set collect_set,Column-method column_aggregate_functions countDistinct countDistinct,Column-method grouping_bit grouping_bit,Column-method grouping_id grouping_id,Column-method kurtosis kurtosis,Column-method max max,Column-method mean mean,Column-method min min,Column-method n_distinct n_distinct,Column-method percentile_approx percentile_approx,characterOrColumn,numericOrColumn-method percentile_approx,Column-method sd sd,Column-method skewness skewness,Column-method stddev stddev,Column-method stddev_pop stddev_pop,Column-method stddev_samp stddev_samp,Column-method sum sum,Column-method sumDistinct sumDistinct,Column-method var var,Column-method variance variance,Column-method var_pop var_pop,Column-method var_samp var_samp,Column-method |
| Avro processing functions for Column operations | column_avro_functions from_avro from_avro,characterOrColumn-method from_avro,Column-method to_avro to_avro,characterOrColumn-method to_avro,Column-method |
| Collection functions for Column operations | arrays_overlap arrays_overlap,Column,Column-method arrays_overlap,Column-method arrays_zip arrays_zip,Column-method arrays_zip_with arrays_zip_with,characterOrColumn,characterOrColumn,function-method array_aggregate array_aggregate,characterOrColumn,Column,function-method array_contains array_contains,Column-method array_distinct array_distinct,Column-method array_except array_except,Column,Column-method array_except,Column-method array_exists array_exists,characterOrColumn,function-method array_filter array_filter,characterOrColumn,function-method array_forall array_forall,characterOrColumn,function-method array_intersect array_intersect,Column,Column-method array_intersect,Column-method array_join array_join,Column,character-method array_join,Column-method array_max array_max,Column-method array_min array_min,Column-method array_position array_position,Column-method array_remove array_remove,Column-method array_repeat array_repeat,Column,numericOrColumn-method array_sort array_sort,Column-method array_transform array_transform,characterOrColumn,characterOrColumn,function-method array_transform,characterOrColumn,function-method array_union array_union,Column,Column-method array_union,Column-method column_collection_functions concat concat,Column-method element_at element_at,Column-method explode explode,Column-method explode_outer explode_outer,Column-method flatten flatten,Column-method from_csv from_csv,Column,characterOrstructTypeOrColumn-method from_json from_json,Column,characterOrstructTypeOrColumn-method map_concat map_concat,Column-method map_entries map_entries,Column-method map_filter map_filter,characterOrColumn,function-method map_from_arrays map_from_arrays,Column,Column-method map_from_arrays,Column-method map_from_entries map_from_entries,Column-method map_keys map_keys,Column-method map_values map_values,Column-method map_zip_with map_zip_with,characterOrColumn,characterOrColumn,function-method posexplode posexplode,Column-method posexplode_outer posexplode_outer,Column-method reverse reverse,Column-method schema_of_csv schema_of_csv,characterOrColumn-method schema_of_json schema_of_json,characterOrColumn-method shuffle shuffle,Column-method size size,Column-method slice slice,Column-method sort_array sort_array,Column-method to_csv to_csv,Column-method to_json to_json,Column-method transform_keys transform_keys,characterOrColumn,function-method transform_values transform_values,characterOrColumn,function-method |
| Date time arithmetic functions for Column operations | add_months add_months,Column,numeric-method column_datetime_diff_functions datediff datediff,Column-method date_add date_add,Column,numeric-method date_format date_format,Column,character-method date_sub date_sub,Column,numeric-method from_utc_timestamp from_utc_timestamp,Column,character-method months_between months_between,Column-method next_day next_day,Column,character-method to_utc_timestamp to_utc_timestamp,Column,character-method |
| Math functions for Column operations | abs abs,Column-method acos acos,Column-method acosh acosh,Column-method asin asin,Column-method asinh asinh,Column-method atan atan,Column-method atan2 atan2,Column-method atanh atanh,Column-method bin bin,Column-method bround bround,Column-method cbrt cbrt,Column-method ceil ceil,Column-method ceiling ceiling,Column-method column_math_functions conv conv,Column,numeric,numeric-method cos cos,Column-method cosh cosh,Column-method degrees degrees,Column-method exp exp,Column-method expm1 expm1,Column-method factorial factorial,Column-method floor floor,Column-method hex hex,Column-method hypot hypot,Column-method log log,Column-method log10 log10,Column-method log1p log1p,Column-method log2 log2,Column-method pmod pmod,Column-method radians radians,Column-method rint rint,Column-method round round,Column-method shiftLeft shiftLeft,Column,numeric-method shiftRight shiftRight,Column,numeric-method shiftRightUnsigned shiftRightUnsigned,Column,numeric-method sign sign,Column-method signum signum,Column-method sin sin,Column-method sinh sinh,Column-method sqrt sqrt,Column-method tan tan,Column-method tanh tanh,Column-method toDegrees toDegrees,Column-method toRadians toRadians,Column-method unhex unhex,Column-method |
| Miscellaneous functions for Column operations | assert_true assert_true,Column-method column_misc_functions crc32 crc32,Column-method hash hash,Column-method md5 md5,Column-method raise_error raise_error,characterOrColumn-method sha1 sha1,Column-method sha2 sha2,Column,numeric-method xxhash64 xxhash64,Column-method |
| ML functions for Column operations | array_to_vector array_to_vector,Column-method column_ml_functions vector_to_array vector_to_array,Column-method |
| Non-aggregate functions for Column operations | bitwiseNOT bitwiseNOT,Column-method coalesce,Column-method column_nonaggregate_functions create_array create_array,Column-method create_map create_map,Column-method expr expr,character-method greatest greatest,Column-method ifelse ifelse,Column-method input_file_name input_file_name,missing-method is.nan is.nan,Column-method isnan isnan,Column-method least least,Column-method lit lit,ANY-method monotonically_increasing_id monotonically_increasing_id,missing-method nanvl nanvl,Column-method negate negate,Column-method rand rand,missing-method rand,numeric-method randn randn,missing-method randn,numeric-method spark_partition_id spark_partition_id,missing-method struct struct,characterOrColumn-method when when,Column-method |
| String functions for Column operations | ascii ascii,Column-method base64 base64,Column-method column_string_functions concat_ws concat_ws,character,Column-method decode decode,Column,character-method encode encode,Column,character-method format_number format_number,Column,numeric-method format_string format_string,character,Column-method initcap initcap,Column-method instr instr,Column,character-method length length,Column-method levenshtein levenshtein,Column-method locate locate,character,Column-method lower lower,Column-method lpad lpad,Column,numeric,character-method ltrim ltrim,Column,character-method ltrim,Column,missing-method overlay overlay,Column,Column,numericOrColumn-method overlay,Column-method,numericOrColumn-method regexp_extract regexp_extract,Column,character,numeric-method regexp_replace regexp_replace,Column,character,character-method repeat_string repeat_string,Column,numeric-method repeat_string,Column-method rpad rpad,Column,numeric,character-method rtrim rtrim,Column,character-method rtrim,Column,missing-method soundex soundex,Column-method split_string split_string,Column,character-method split_string,Column-method substring_index substring_index,Column,character,numeric-method translate translate,Column,character,character-method trim trim,Column,character-method trim,Column,missing-method unbase64 unbase64,Column-method upper upper,Column-method |
| Window functions for Column operations | column_window_functions cume_dist cume_dist,missing-method dense_rank dense_rank,missing-method lag lag,characterOrColumn-method lead lead,characterOrColumn,numeric-method nth_value nth_value,characterOrColumn,numeric-method nth_value,characterOrColumn-method ntile ntile,numeric-method percent_rank percent_rank,missing-method rank rank,ANY-method rank,missing-method row_number row_number,missing-method |
| corr | corr corr,Column-method corr,SparkDataFrame-method |
| Count | count count,Column-method count,GroupedData-method n n,Column-method |
| cov | cov cov,characterOrColumn-method cov,SparkDataFrame-method covar_pop covar_pop,characterOrColumn,characterOrColumn-method covar_samp covar_samp,characterOrColumn,characterOrColumn-method |
| Create o.a.s.sql.expressions.LambdaFunction corresponding to transformation described by func. Used by higher order functions. | create_lambda |
| Create a SparkDataFrame | as.DataFrame createDataFrame |
| (Deprecated) Create an external table | createExternalTable |
| Creates a temporary view using the given name. | createOrReplaceTempView createOrReplaceTempView,SparkDataFrame,character-method |
| Creates a table based on the dataset in a data source | createTable |
| CrossJoin | crossJoin crossJoin,SparkDataFrame,SparkDataFrame-method |
| Computes a pair-wise frequency table of the given columns | crosstab crosstab,SparkDataFrame,character,character-method |
| cube | cube cube,SparkDataFrame-method |
| Returns the current default database | currentDatabase |
| dapply | dapply dapply,SparkDataFrame,function,characterOrstructType-method |
| dapplyCollect | dapplyCollect dapplyCollect,SparkDataFrame,function-method |
| S4 class that represents a DecisionTreeClassificationModel | DecisionTreeClassificationModel-class |
| S4 class that represents a DecisionTreeRegressionModel | DecisionTreeRegressionModel-class |
| describe | describe describe,SparkDataFrame,ANY-method describe,SparkDataFrame,character-method describe,SparkDataFrame-method |
| Returns the dimensions of SparkDataFrame | dim dim,SparkDataFrame-method |
| Distinct | distinct distinct,SparkDataFrame-method unique unique,SparkDataFrame-method |
| drop | drop drop,ANY-method drop,SparkDataFrame-method |
| dropDuplicates | dropDuplicates dropDuplicates,SparkDataFrame-method |
| dropFields | dropFields dropFields,Column-method |
| A set of SparkDataFrame functions working with NA values | dropna dropna,SparkDataFrame-method fillna fillna,SparkDataFrame-method na.omit na.omit,SparkDataFrame-method |
| (Deprecated) Drop Temporary Table | dropTempTable |
| Drops the temporary view with the given view name in the catalog. | dropTempView |
| DataTypes | dtypes dtypes,SparkDataFrame-method |
| endsWith | endsWith endsWith,Column-method |
| except | except except,SparkDataFrame,SparkDataFrame-method |
| exceptAll | exceptAll exceptAll,SparkDataFrame,SparkDataFrame-method |
| Explain | explain explain,SparkDataFrame-method explain,StreamingQuery-method |
| Filter | filter filter,SparkDataFrame,characterOrColumn-method where where,SparkDataFrame,characterOrColumn-method |
| Return the first row of a SparkDataFrame | first first,characterOrColumn-method first,SparkDataFrame-method |
| Get fitted result from a k-means model | fitted fitted,KMeansModel-method |
| S4 class that represents a FMClassificationModel | FMClassificationModel-class |
| S4 class that represents a FMRegressionModel | FMRegressionModel-class |
| S4 class that represents a FPGrowthModel | FPGrowthModel-class |
| Finding frequent items for columns, possibly with false positives | freqItems freqItems,SparkDataFrame,character-method |
| gapply | gapply gapply,GroupedData-method gapply,SparkDataFrame-method |
| gapplyCollect | gapplyCollect gapplyCollect,GroupedData-method gapplyCollect,SparkDataFrame-method |
| S4 class that represents a GaussianMixtureModel | GaussianMixtureModel-class |
| S4 class that represents a GBTClassificationModel | GBTClassificationModel-class |
| S4 class that represents a GBTRegressionModel | GBTRegressionModel-class |
| S4 class that represents a generalized linear model | GeneralizedLinearRegressionModel-class |
| Get a local property set in this thread, or 'NULL' if it is missing. See 'setLocalProperty'. | getLocalProperty |
| getNumPartitions | getNumPartitions getNumPartitions,SparkDataFrame-method |
| Generalized Linear Models (R-compliant) | glm glm,formula,ANY,SparkDataFrame-method |
| GroupBy | groupBy groupBy,SparkDataFrame-method group_by group_by,SparkDataFrame-method |
| S4 class that represents a GroupedData | groupedData GroupedData-class |
| Compute the hashCode of an object | hashCode |
| Head | head head,SparkDataFrame-method |
| hint | hint hint,SparkDataFrame,character-method |
| Compute histogram statistics for given column | histogram histogram,SparkDataFrame,characterOrColumn-method |
| insertInto | insertInto insertInto,SparkDataFrame,character-method |
| Download and Install Apache Spark to a Local Directory | install.spark |
| Intersect | intersect intersect,SparkDataFrame,SparkDataFrame-method |
| intersectAll | intersectAll intersectAll,SparkDataFrame,SparkDataFrame-method |
| Invokes higher order function expression identified by name, (relative to o.a.s.sql.catalyst.expressions) | invoke_higher_order_function |
| isActive | isActive isActive,StreamingQuery-method |
| isLocal | isLocal isLocal,SparkDataFrame-method |
| S4 class that represents an IsotonicRegressionModel | IsotonicRegressionModel-class |
| isStreaming | isStreaming isStreaming,SparkDataFrame-method |
| Join | join join,SparkDataFrame,SparkDataFrame-method |
| S4 class that represents a KMeansModel | KMeansModel-class |
| S4 class that represents an KSTest | KSTest-class |
| last | last last,characterOrColumn-method |
| lastProgress | lastProgress lastProgress,StreamingQuery-method |
| S4 class that represents an LDAModel | LDAModel-class |
| Limit | limit limit,SparkDataFrame,numeric-method |
| S4 class that represents a LinearRegressionModel | LinearRegressionModel-class |
| S4 class that represents an LinearSVCModel | LinearSVCModel-class |
| Returns a list of columns for the given table/view in the specified database | listColumns |
| Returns a list of databases available | listDatabases |
| Returns a list of functions registered in the specified database | listFunctions |
| Returns a list of tables or views in the specified database | listTables |
| localCheckpoint | localCheckpoint localCheckpoint,SparkDataFrame-method |
| S4 class that represents an LogisticRegressionModel | LogisticRegressionModel-class |
| Merges two data frames | merge merge,SparkDataFrame,SparkDataFrame-method |
| S4 class that represents a MultilayerPerceptronClassificationModel | MultilayerPerceptronClassificationModel-class |
| Mutate | mutate mutate,SparkDataFrame-method transform transform,SparkDataFrame-method |
| S4 class that represents a NaiveBayesModel | NaiveBayesModel-class |
| Returns the number of columns in a SparkDataFrame | ncol ncol,SparkDataFrame-method |
| ! | !,Column-method not not,Column-method |
| Returns the number of rows in a SparkDataFrame | count,SparkDataFrame-method nrow nrow,SparkDataFrame-method |
| Ordering Columns in a WindowSpec | orderBy orderBy,WindowSpec,character-method orderBy,WindowSpec,Column-method |
| otherwise | otherwise otherwise,Column-method |
| over | over over,Column,WindowSpec-method |
| partitionBy | partitionBy partitionBy,WindowSpec-method |
| Persist | persist persist,SparkDataFrame,character-method |
| Pivot a column of the GroupedData and perform the specified aggregation. | pivot pivot,GroupedData,character-method |
| S4 class that represents a PowerIterationClustering | PowerIterationClustering-class |
| Makes predictions from a MLlib model | predict |
| S4 class that represents a PrefixSpan | PrefixSpan-class |
| Print a JVM object reference. | print.jobj |
| Print a Spark StructField. | print.structField |
| Print a Spark StructType. | print.structType |
| Print Schema of a SparkDataFrame | printSchema printSchema,SparkDataFrame-method |
| queryName | queryName queryName,StreamingQuery-method |
| S4 class that represents a RandomForestClassificationModel | RandomForestClassificationModel-class |
| S4 class that represents a RandomForestRegressionModel | RandomForestRegressionModel-class |
| randomSplit | randomSplit randomSplit,SparkDataFrame,numeric-method |
| rangeBetween | rangeBetween rangeBetween,WindowSpec,numeric,numeric-method |
| Union two or more SparkDataFrames | rbind rbind,SparkDataFrame-method |
| Load a SparkDataFrame | loadDF read.df |
| Create a SparkDataFrame representing the database table accessible via JDBC URL | read.jdbc |
| Create a SparkDataFrame from a JSON file. | read.json |
| Load a fitted MLlib model from the input path. | read.ml |
| Create a SparkDataFrame from an ORC file. | read.orc |
| Create a SparkDataFrame from a Parquet file. | read.parquet |
| Load a streaming SparkDataFrame | read.stream |
| Create a SparkDataFrame from a text file. | read.text |
| Recovers all the partitions in the directory of a table and update the catalog | recoverPartitions |
| Invalidates and refreshes all the cached data and metadata for SparkDataFrame containing path | refreshByPath |
| Invalidates and refreshes all the cached data and metadata of the given table | refreshTable |
| (Deprecated) Register Temporary Table | registerTempTable registerTempTable,SparkDataFrame,character-method |
| rename | rename rename,SparkDataFrame-method withColumnRenamed withColumnRenamed,SparkDataFrame,character,character-method |
| Repartition | repartition repartition,SparkDataFrame-method |
| Repartition by range | repartitionByRange repartitionByRange,SparkDataFrame-method |
| rollup | rollup rollup,SparkDataFrame-method |
| rowsBetween | rowsBetween rowsBetween,WindowSpec,numeric,numeric-method |
| Sample | sample sample,SparkDataFrame-method sample_frac sample_frac,SparkDataFrame-method |
| Returns a stratified sample without replacement | sampleBy sampleBy,SparkDataFrame,character,list,numeric-method |
| Save the contents of the SparkDataFrame to a data source as a table | saveAsTable saveAsTable,SparkDataFrame,character-method |
| Get schema object | schema schema,SparkDataFrame-method |
| Select | $ $,SparkDataFrame-method $<- $<-,SparkDataFrame-method select select,SparkDataFrame,character-method select,SparkDataFrame,Column-method select,SparkDataFrame,list-method |
| SelectExpr | selectExpr selectExpr,SparkDataFrame,character-method |
| Set checkpoint directory | setCheckpointDir |
| Sets the current default database | setCurrentDatabase |
| Set a human readable description of the current job. | setJobDescription |
| Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared. | setJobGroup |
| Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. | setLocalProperty |
| Set new log level | setLogLevel |
| show | show show,Column-method show,GroupedData-method show,SparkDataFrame-method show,StreamingQuery-method show,WindowSpec-method |
| showDF | showDF showDF,SparkDataFrame-method |
| Add a file or directory to be downloaded with this Spark job on every node. | spark.addFile |
| Alternating Least Squares (ALS) for Collaborative Filtering | predict,ALSModel-method spark.als spark.als,SparkDataFrame-method summary,ALSModel-method write.ml,ALSModel,character-method |
| PowerIterationClustering | spark.assignClusters spark.assignClusters,SparkDataFrame-method |
| Bisecting K-Means Clustering Model | fitted,BisectingKMeansModel-method predict,BisectingKMeansModel-method spark.bisectingKmeans spark.bisectingKmeans,SparkDataFrame,formula-method summary,BisectingKMeansModel-method write.ml,BisectingKMeansModel,character-method |
| Decision Tree Model for Regression and Classification | predict,DecisionTreeClassificationModel-method predict,DecisionTreeRegressionModel-method print.summary.DecisionTreeClassificationModel print.summary.DecisionTreeRegressionModel spark.decisionTree spark.decisionTree,SparkDataFrame,formula-method summary,DecisionTreeClassificationModel-method summary,DecisionTreeRegressionModel-method write.ml,DecisionTreeClassificationModel,character-method write.ml,DecisionTreeRegressionModel,character-method |
| PrefixSpan | findFrequentSequentialPatterns,PrefixSpan,SparkDataFrame-method spark.findFrequentSequentialPatterns spark.findFrequentSequentialPatterns,SparkDataFrame-method |
| Factorization Machines Classification Model | predict,FMClassificationModel,SparkDataFrame-method predict,FMClassificationModel-method spark.fmClassifier spark.fmClassifier,SparkDataFrame,formula-method summary,FMClassificationModel-method write.ml,FMClassificationModel,character-method |
| Factorization Machines Regression Model | predict,FMRegressionModel,SparkDataFrame-method predict,FMRegressionModel-method spark.fmRegressor spark.fmRegressor,SparkDataFrame,formula-method summary,FMRegressionModel-method write.ml,FMRegressionModel,character-method |
| FP-growth | associationRules,FPGrowthModel-method freqItemsets,FPGrowthModel-method predict,FPGrowthModel-method spark.associationRules spark.associationRules,FPGrowthModel-method spark.fpGrowth spark.fpGrowth,SparkDataFrame-method spark.freqItemsets spark.freqItemsets,FPGrowthModel-method write.ml,FPGrowthModel,character-method |
| Multivariate Gaussian Mixture Model (GMM) | predict,GaussianMixtureModel,SparkDataFrame-method predict,GaussianMixtureModel-method spark.gaussianMixture spark.gaussianMixture,SparkDataFrame,formula-method summary,GaussianMixtureModel-method write.ml,GaussianMixtureModel,character-method |
| Gradient Boosted Tree Model for Regression and Classification | predict,GBTClassificationModel-method predict,GBTRegressionModel-method print.summary.GBTClassificationModel print.summary.GBTRegressionModel spark.gbt spark.gbt,SparkDataFrame,formula-method summary,GBTClassificationModel-method summary,GBTRegressionModel-method write.ml,GBTClassificationModel,character-method write.ml,GBTRegressionModel,character-method |
| Get the absolute path of a file added through spark.addFile. | spark.getSparkFiles |
| Get the root directory that contains files added through spark.addFile. | spark.getSparkFilesRootDirectory |
| Generalized Linear Models | predict,GeneralizedLinearRegressionModel-method print.summary.GeneralizedLinearRegressionModel spark.glm spark.glm,SparkDataFrame,formula-method summary,GeneralizedLinearRegressionModel-method write.ml,GeneralizedLinearRegressionModel,character-method |
| Isotonic Regression Model | predict,IsotonicRegressionModel,SparkDataFrame-method predict,IsotonicRegressionModel-method spark.isoreg spark.isoreg,SparkDataFrame,formula-method summary,IsotonicRegressionModel-method write.ml,IsotonicRegressionModel,character-method |
| K-Means Clustering Model | predict,KMeansModel-method spark.kmeans spark.kmeans,SparkDataFrame,formula-method summary,KMeansModel-method write.ml,KMeansModel,character-method |
| (One-Sample) Kolmogorov-Smirnov Test | print.summary.KSTest spark.kstest spark.kstest,SparkDataFrame-method summary,KSTest-method |
| Run a function over a list of elements, distributing the computations with Spark | spark.lapply |
| Latent Dirichlet Allocation | spark.lda spark.lda,SparkDataFrame-method spark.perplexity spark.perplexity,LDAModel,SparkDataFrame-method spark.perplexity,LDAModel-method spark.posterior spark.posterior,LDAModel,SparkDataFrame-method summary,LDAModel-method write.ml,LDAModel,character-method |
| Linear Regression Model | predict,LinearRegressionModel,SparkDataFrame-method predict,LinearRegressionModel-method spark.lm spark.lm,SparkDataFrame,formula-method summary,LinearRegressionModel-method write.ml,LinearRegressionModel,character-method |
| Logistic Regression Model | predict,LogisticRegressionModel,SparkDataFrame-method predict,LogisticRegressionModel-method spark.logit spark.logit,SparkDataFrame,formula-method summary,LogisticRegressionModel-method write.ml,LogisticRegressionModel,character-method |
| Multilayer Perceptron Classification Model | predict,MultilayerPerceptronClassificationModel-method spark.mlp spark.mlp,SparkDataFrame,formula-method summary,MultilayerPerceptronClassificationModel-method write.ml,MultilayerPerceptronClassificationModel,character-method |
| Naive Bayes Models | predict,NaiveBayesModel-method spark.naiveBayes spark.naiveBayes,SparkDataFrame,formula-method summary,NaiveBayesModel-method write.ml,NaiveBayesModel,character-method |
| Random Forest Model for Regression and Classification | predict,RandomForestClassificationModel-method predict,RandomForestRegressionModel-method print.summary.RandomForestClassificationModel print.summary.RandomForestRegressionModel spark.randomForest spark.randomForest,SparkDataFrame,formula-method summary,RandomForestClassificationModel-method summary,RandomForestRegressionModel-method write.ml,RandomForestClassificationModel,character-method write.ml,RandomForestRegressionModel,character-method |
| Accelerated Failure Time (AFT) Survival Regression Model | predict,AFTSurvivalRegressionModel-method spark.survreg spark.survreg,SparkDataFrame,formula-method summary,AFTSurvivalRegressionModel-method write.ml,AFTSurvivalRegressionModel,character-method |
| Linear SVM Model | predict,LinearSVCModel,SparkDataFrame-method predict,LinearSVCModel-method spark.svmLinear spark.svmLinear,SparkDataFrame,formula-method summary,LinearSVCModel-method write.ml,LinearSVCModel,character-method |
| S4 class that represents a SparkDataFrame | SparkDataFrame-class |
| Call Java Methods | sparkR.callJMethod |
| Call Static Java Methods | sparkR.callJStatic |
| Get Runtime Config from the current active SparkSession | sparkR.conf |
| (Deprecated) Initialize a new Spark Context | sparkR.init |
| Create Java Objects | sparkR.newJObject |
| Get the existing SparkSession or initialize a new SparkSession. | sparkR.session |
| Stop the Spark Session and Spark Context | sparkR.session.stop sparkR.stop |
| Get the URL of the SparkUI instance for the current active SparkSession | sparkR.uiWebUrl |
| Get version of Spark on which this application is running | sparkR.version |
| (Deprecated) Initialize a new HiveContext | sparkRHive.init |
| (Deprecated) Initialize a new SQLContext | sparkRSQL.init |
| SQL Query | sql |
| startsWith | startsWith startsWith,Column-method |
| status | status status,StreamingQuery-method |
| stopQuery | stopQuery stopQuery,StreamingQuery-method |
| StorageLevel | storageLevel storageLevel,SparkDataFrame-method |
| Compactly display the structure of a dataset | str str,SparkDataFrame-method |
| S4 class that represents a StreamingQuery | StreamingQuery-class |
| structField | structField structField.character structField.jobj |
| structType | structType structType.character structType.jobj structType.structField |
| Subset | subset subset,SparkDataFrame-method [ [,SparkDataFrame-method [[ [[,SparkDataFrame,numericOrcharacter-method [[<- [[<-,SparkDataFrame,numericOrcharacter-method |
| substr | substr substr,Column-method |
| summary | summary summary,SparkDataFrame-method |
| Table Names | tableNames |
| Tables | tables |
| Create a SparkDataFrame from a SparkSQL table or view | tableToDF |
| Take the first NUM rows of a SparkDataFrame and return the results as a R data.frame | take take,SparkDataFrame,numeric-method |
| Date time functions for Column operations | column_datetime_functions current_date current_date,missing-method current_timestamp current_timestamp,missing-method date_trunc date_trunc,character,Column-method dayofmonth dayofmonth,Column-method dayofweek dayofweek,Column-method dayofyear dayofyear,Column-method from_unixtime from_unixtime,Column-method hour hour,Column-method last_day last_day,Column-method minute minute,Column-method month month,Column-method quarter quarter,Column-method second second,Column-method timestamp_seconds timestamp_seconds,Column-method to_date to_date,Column,character-method to_date,Column,missing-method to_timestamp to_timestamp,Column,character-method to_timestamp,Column,missing-method trunc trunc,Column-method unix_timestamp unix_timestamp,Column,character-method unix_timestamp,Column,missing-method unix_timestamp,missing,missing-method weekofyear weekofyear,Column-method window window,Column-method year year,Column-method |
| toJSON | toJSON toJSON,SparkDataFrame-method |
| Uncache Table | uncacheTable |
| Return a new SparkDataFrame containing the union of rows | union union,SparkDataFrame,SparkDataFrame-method |
| Return a new SparkDataFrame containing the union of rows. | unionAll unionAll,SparkDataFrame,SparkDataFrame-method |
| Return a new SparkDataFrame containing the union of rows, matched by column names | unionByName unionByName,SparkDataFrame,SparkDataFrame-method |
| Unpersist | unpersist unpersist,SparkDataFrame-method |
| Create o.a.s.sql.expressions.UnresolvedNamedLambdaVariable, convert it to o.s.sql.Column and wrap with R Column. Used by higher order functions. | unresolved_named_lambda_var |
| windowOrderBy | windowOrderBy windowOrderBy,character-method windowOrderBy,Column-method |
| windowPartitionBy | windowPartitionBy windowPartitionBy,character-method windowPartitionBy,Column-method |
| S4 class that represents a WindowSpec | WindowSpec-class |
| Evaluate a R expression in an environment constructed from a SparkDataFrame | with with,SparkDataFrame-method |
| WithColumn | withColumn withColumn,SparkDataFrame,character-method |
| withField | withField withField,Column,character,Column-method withField,Column-method |
| withWatermark | withWatermark withWatermark,SparkDataFrame,character,character-method |
| Save the contents of SparkDataFrame to a data source. | saveDF saveDF,SparkDataFrame,character-method write.df write.df,SparkDataFrame-method |
| Save the content of SparkDataFrame to an external database table via JDBC. | write.jdbc write.jdbc,SparkDataFrame,character,character-method |
| Save the contents of SparkDataFrame as a JSON file | write.json write.json,SparkDataFrame,character-method |
| Saves the MLlib model to the input path | write.ml |
| Save the contents of SparkDataFrame as an ORC file, preserving the schema. | write.orc write.orc,SparkDataFrame,character-method |
| Save the contents of SparkDataFrame as a Parquet file, preserving the schema. | write.parquet write.parquet,SparkDataFrame,character-method |
| Write the streaming SparkDataFrame to a data source. | write.stream write.stream,SparkDataFrame-method |
| Save the content of SparkDataFrame in a text file at the specified path. | write.text write.text,SparkDataFrame,character-method |
