spark1.统计句子中特定内容

标签：imp   name   length   sim   map   contain   pre   oca   cat   

    val logFile = "./README.md"  // Should be some file on your server.
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
//    val numAs = logData.filter(line => line.contains("h")).count()
//    val numBs = logData.filter(line => line.contains("j")).count()
    var  params = List("h","j","c","w");

    var searchAnylisay =   params.map(item => logData.filter(line => line.contains(item)).count() )

    println("searchAnylisay length  : %s,".format(searchAnylisay.length))

    searchAnylisay.foreach( x => println(x))

spark1.统计句子中特定内容

标签：imp   name   length   sim   map   contain   pre   oca   cat   

原文地址：http://www.cnblogs.com/wcLT/p/6623635.html