标签：
最近使用SparkSQL做数据的打平操作，就是把多个表的数据经过关联操作导入到一个表中，这样数据查询的过程中就不需要在多个表中查询了，在数据量大的情况下，这样大大提高了查询效率。
if (className.equals("org.apache.spark.deploy.master.Master")) { javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS"); javaOptsKeys.add("SPARK_MASTER_OPTS"); memKey = "SPARK_DAEMON_MEMORY"; } else if (className.equals("org.apache.spark.deploy.worker.Worker")) { javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS"); javaOptsKeys.add("SPARK_WORKER_OPTS"); memKey = "SPARK_DAEMON_MEMORY"; } else if (className.equals("org.apache.spark.deploy.history.HistoryServer")) { javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS"); javaOptsKeys.add("SPARK_HISTORY_OPTS"); memKey = "SPARK_DAEMON_MEMORY"; } else if (className.equals("org.apache.spark.executor.CoarseGrainedExecutorBackend")) { javaOptsKeys.add("SPARK_JAVA_OPTS"); javaOptsKeys.add("SPARK_EXECUTOR_OPTS"); memKey = "SPARK_EXECUTOR_MEMORY"; } else if (className.equals("org.apache.spark.executor.MesosExecutorBackend")) { javaOptsKeys.add("SPARK_EXECUTOR_OPTS"); memKey = "SPARK_EXECUTOR_MEMORY"; } else if (className.equals("org.apache.spark.deploy.ExternalShuffleService") || className.equals("org.apache.spark.deploy.mesos.MesosExternalShuffleService")) { javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS"); javaOptsKeys.add("SPARK_SHUFFLE_OPTS"); memKey = "SPARK_DAEMON_MEMORY"; } else if (className.startsWith("org.apache.spark.tools.")) { String sparkHome = getSparkHome(); File toolsDir = new File(join(File.separator, sparkHome, "tools", "target", "scala-" + getScalaVersion())); checkState(toolsDir.isDirectory(), "Cannot find tools build directory."); Pattern re = Pattern.compile("spark-tools_.*\\.jar"); for (File f : toolsDir.listFiles()) { if (re.matcher(f.getName()).matches()) { extraClassPath = f.getAbsolutePath(); break; } } checkState(extraClassPath != null, "Failed to find Spark Tools Jar in %s.\n" + "You need to run \"build/sbt tools/package\" before running %s.", toolsDir.getAbsolutePath(), className); javaOptsKeys.add("SPARK_JAVA_OPTS"); } else { javaOptsKeys.add("SPARK_JAVA_OPTS"); memKey = "SPARK_DRIVER_MEMORY"; }
看46.47的代码，1.5可以通过SPARk_JAVA_OPTS和SPARK_DRIVER_MEMORY来设置beeline的内存
Spark 1.4.1中Beeline使用的gc overhead limit exceeded
标签：
原文地址：http://www.cnblogs.com/gaoxing/p/4714235.html
