建筑人才网官网网址_百度商标查询_佛山网站优化_关键词查询神器

新建scala 项目

创建lib目录

将spark  jars/   路径下所有jar 复制到  lib目录

 添加依赖

创建scala 程序

 

 

package sparkimport org.apache.spark.{SparkConf, SparkContext}object WordCount {def main(args: Array[String]): Unit = {val conf = new SparkConf().setAppName("WordCount").setMaster("local") //本地模式//    val conf = new SparkConf().setAppName("WordCount") //集群模式sval sc = new SparkContext(conf)val result = sc.textFile("hdfs://bigdata112:9000/user/test001.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collectresult.foreach(println)println("Hello, World!")}}

准备测试数据

上传hdfs

hdfs dfs -put test001.txt /user/

 运行程序