Getting Started
Spark aims to make it easy to work with data. One way they achieve this is by working with spark data as if you were working on a SQL database * Spark SQL enables querying of DataFrames as database tables * Temporary per-session and global tables * The Catalyst optimizer makes SQL queries
Python
Spark DataFrames Previously we looked at RDDs, and were the primary data set in Spark 1. In Spark 2 we rarely use RDDs only for low level transformations and control over the dataset. If the data is unstructured or streaming data we then have to rely on RDDs, for everything
Python
I will keep my data in a folder named 'datasets' If PySpark is not already loaded up, go ahead and start PySpark and create a new Jupyter notebook View information about the SparkContext by inputing sc If we were running a cluster of nodes the output would be a bit
Python
Introduction This article will look at connecting Python and MySQL database. With the help of some sql connection tools, transfering data between Python and MySQL will be simplified. Finally, we will migrate and run the database on a cloud platform prerequisites * database with some data to work on * aws developer
AWS
Overview In a previous module we use the boto3 library to connect our python script with an AWS service. Here, we are going continue down this path by taking a look at some operations we can perform using AWS S3 and python. We will create a storage service with python
AWS
Introduction AWS (Amazon Web Services) is an ecosystem with an abundace of services to fufill many of our development needs. There are also some great ways for us to interact with these services. We can simply log in to the web browser console(https://aws.amazon.com ). Or we can