Working on Databricks Community Edition

By Freshlyne

  •  After logging in, the first thing to do is to create a cluster (If a cluster has already been created, proceed to the next step).
    • For cluster creation, click the “Clusters” menu from the side ribbon, or use the “New Cluster” from “Common Tasks”:



  • Enter the cluster name and keep the rest of the settings as it is , for now, and click “Create Cluster”. This will create the cluster and spin it up. Starting a cluster can take some time, so be patient.


  •            The next step is to create a notebook where the magic happens.   For this, Go to Workspace > YourUsername > Create (From the Drop-down menu) > Notebook.



  • Give the notebook a name, select the default language (Don’t worry, any of the four languages can be used inside the notebook regardless of the default language), and choose the newly created cluster:



  • Once that’s done, it’ll look something like this (Default Notebook theme can be Light, can be changed from View):


  •      On top right hand, make sure the cluster is attached to the notebook. (In above step, it is Detached)
  •           Now, %python, %scala, %sql, %r can be used in different cell blocks to run the commands from those languages:



Note: Variables from one cell can be shared with others, but variables can’t be shared directly between two different languages.

  • Once we are done with our task, it is a good practice to terminate the cluster (The Square icon): 


Comments

Post a Comment

Popular posts from this blog

Reading a file into a dataframe using PySpark in Databricks

By
Image
The very first thing we need is a file. For this demo, this is our sample file :  A comma-separated file with 3 columns and 3 rows. So now that we have that, the first step would be to actually upload the file over to databricks. Get the cluster running. Go to Data tab Click on "Create Table" Drag the file to the specified space. At this point, the file will be uploaded to the DBFS storage. After this, we can work with the file. Copy the location of the file. (Here: /FileStore/tables/SampleData-1.csv) Now, let's get back to the notebook to work with the file.  Creating a dataframe from the file dataframe_var=spark.read.option("header","true").csv('/FileStore/tables/SampleData-1.csv')   Display the dataframe    We can see that the file data is successfully displayed in the dataframe. We will take a look at how to load files to tables in the next blog.
Read more

Creating tables from dataframes

By
Image
 Continuing from our last post: Reading file into dataframe Let's create a table from the dataframe: Dataframe.write.saveAsTable("Database.tablename") We need to remember that we are creating a managed table with the above syntax, and to create an unmanaged table we can add the path option.   Managed Table Unmanaged Table Now we can Query the tables using SQL. Reference for the difference between managed and unmanaged tables: Click here  
Read more