Spark@Stackoverflow

Skewed dataset join in Spark?: I ma joining two big datasets using Spark RDD, one dataset is very much skewed... bit.ly/2eTxtez

Structured streaming in spark for streaming machine learning algorithms: I am beginner for spark streaming. S... bit.ly/2f8ECsc

How to set spark.local.dir property from spark shell?: I'm trying to set spark.local.dir from spark-shell usi... bit.ly/2f8IARl

how to encrypt & Decrypt database credentials in spark sql: Here i have mentioned my user name and password e... bit.ly/2f8y10O

How to read and write data to Google Bigtable in pyspark application ?: I am using Cloudproc spark cluster wh... bit.ly/2f8wszL

How to specify sql dialect when creating spark dataframe from JDBC?: I'm having an issue reading data via cus... bit.ly/2fcdoig

Talend and Apache Spark?: I am confused as to where Talend and Apache spark fit in the big data ecosystem as ... bit.ly/2fccxxR

Spark Pivot - generates a matrix: I need to pivot a two-column dataframe to one row (long to wide) so that I ... bit.ly/2elr8Il

Using python scipy from spark scala: Is it possible to use python "scipy" linear algebra library from spark/s... bit.ly/2exXGw9

aggregating existing data by ids from mongodb in spark: I'm wondering if Mongo-spark is able to handle a scen... bit.ly/2eloGRX

java.lang.UnsupportedOperationException: Schema for type MyClass is not supported: I am using Spark 1.5.0 and... bit.ly/2elxHL1

Registering Hive Custom UDF with Spark (Spark SQL) 2.0.0: I am working on a spark 2.0.0 piece where my requir... bit.ly/2ebnqO9

Creating Pyspark DataFrame column that coalesces two other Columns, why am I getting error of 'unicode' objec... bit.ly/2ebk9Oy

How to get rid of the row wrapper object in pyspark dataframes without using RDD api?: I'm issuing the follow... bit.ly/2ebigl5

When does spark-env.sh runs?: I am assuming that spark-env.sh runs when I start the spark cluster with sbin... bit.ly/2f7Dgh3

Spark Cassandra String not written in Hadoop file: I am Java Developer and learning Spark while working on a ... bit.ly/2eRmNgo

Partitioning in Spark: I created an RDD by parallelizing the following array: var arr: Array[(Int,Char)] = A... bit.ly/2feidpC

Running EMR Spark with multiple S3 accounts: I have an EMR Spark Job that needs to read data from S3 on one a... bit.ly/2f6zg0h

Why does a Spark Application launch with only a single executor on DC/OS?: I have Spark installed, but when I... bit.ly/2eahDIv

Hive/SparkSQL - Auto typecast date to timestamp in expression: When I run Hive queries in Spark SQL, LHS (tim... bit.ly/2f6ApVQ