I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing :sc.textFile('file.csv') .map(lambda line: (line.split(','), line.split(',')))... moreI'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing :sc.textFile('file.csv') .map(lambda line: (line.split(','), line.split(','))) .collect()I would expect this call to give me a list of the two first columns of my file but I'm getting this error :File "", line 1, in IndexError: list index out of rangealthough my CSV file as more than one column.
I am using CDH 5.2. I am able to use spark-shell to run the commands.How can I run the file(file.spark) which contain spark commands.Is there any way to run/compile the scala... moreI am using CDH 5.2. I am able to use spark-shell to run the commands.How can I run the file(file.spark) which contain spark commands.Is there any way to run/compile the scala programs in CDH 5.2 without sbt?Thanks in advance
I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be... more
I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get some snippet in python to find the probability of labels.