Python - Virtual Artifact

Python

A collection of 6 posts

Getting Started

PySpark - Spark SQL Context

Spark aims to make it easy to work with data. One way they achieve this is by working with spark data as if you were working on a SQL database * Spark SQL enables querying of DataFrames as database tables * Temporary per-session and global tables * The Catalyst optimizer makes SQL queries

PySpark - Using DataFrames

Spark DataFrames Previously we looked at RDDs, and were the primary data set in Spark 1. In Spark 2 we rarely use RDDs only for low level transformations and control over the dataset. If the data is unstructured or streaming data we then have to rely on RDDs, for everything

PySpark - Using RDDs

I will keep my data in a folder named 'datasets' If PySpark is not already loaded up, go ahead and start PySpark and create a new Jupyter notebook View information about the SparkContext by inputing sc If we were running a cluster of nodes the output would be a bit

Pandas and MySql with a hit of AWS RDS

Introduction This article will look at connecting Python and MySQL database. With the help of some sql connection tools, transfering data between Python and MySQL will be simplified. Finally, we will migrate and run the database on a cloud platform prerequisites * database with some data to work on * aws developer

Creating a AWS S3 service with Python

Overview In a previous module we use the boto3 library to connect our python script with an AWS service. Here, we are going continue down this path by taking a look at some operations we can perform using AWS S3 and python. We will create a storage service with python

Install and Configure Boto3 with AWS

Introduction AWS (Amazon Web Services) is an ecosystem with an abundace of services to fufill many of our development needs. There are also some great ways for us to interact with these services. We can simply log in to the web browser console(https://aws.amazon.com ). Or we can