QBoard » Big Data » Big Data - Spark » Spark : how to run spark file from spark shell

Spark : how to run spark file from spark shell

  • I am using CDH 5.2. I am able to use spark-shell to run the commands.

    How can I run the file(file.spark) which contain spark commands.
    Is there any way to run/compile the scala programs in CDH 5.2 without sbt?
    Thanks in advance
      June 12, 2019 11:51 AM IST
    0
  • In command line, you can use

    spark-shell -i file.scala
    ​


    to run code which is written in file.scala
      August 12, 2021 1:58 PM IST
    0
  • To load an external file from spark-shell simply do
    :load PATH_TO_FILE

    This will call everything in your file.

    I don't have a solution for your SBT question though sorry :-)

      December 25, 2020 12:15 PM IST
    0
  • You can run as you run your shell script. This example to run from command line environment example

    ./bin/spark-shell :- this is the path of your spark-shell under bin /home/fold1/spark_program.py :- This is the path where your python program is there.

      January 15, 2022 12:50 PM IST
    0
  • Just to give more perspective to the answers

    Spark-shell is a scala repl

    You can type :help to see the list of operation that are possible inside the scala shell

    scala> :help
    All commands can be abbreviated, e.g., :he instead of :help.
    :edit <id>|<line>        edit history
    :help [command]          print this summary or command-specific help
    :history [num]           show the history (optional num is commands to show)
    :h? <string>             search the history
    :imports [name name ...] show import history, identifying sources of names
    :implicits [-v]          show the implicits in scope
    :javap <path|class>      disassemble a file or class name
    :line <id>|<line>        place line(s) at the end of history
    :load <path>             interpret lines in a file
    :paste [-raw] [path]     enter paste mode or paste a file
    :power                   enable power user mode
    :quit                    exit the interpreter
    :replay [options]        reset the repl and replay all previous commands
    :require <path>          add a jar to the classpath
    :reset [options]         reset the repl to its initial state, forgetting all session entries
    :save <path>             save replayable session to a file
    :sh <command line>       run a shell command (result is implicitly => List[String])
    :settings <options>      update compiler options, if possible; see reset
    :silent                  disable/enable automatic printing of results
    :type [-v] <expr>        display the type of an expression without evaluating it
    :kind [-v] <expr>        display the kind of expression's type
    :warnings                show the suppressed warnings from the most recent line which had any
      January 17, 2022 1:47 PM IST
    0