Data Analysis With Python: All You Need To Know

Describe Python data acquisition and analysis techniques.

Every data analyst should have a good understanding of the below techniques. So the goal of this article is to take the readers through these techniques and to explain these on a basic level.

These are the topics we will go through and discuss:

1. Basic filtering-When you want to get a subset of your data based on the values in a column, we are talking about filtering data.

   In pandas we have multiple ways to do that, for now we look at the most common ones:

   1. Using boolean indexing with square brackets []
   2. Using boolean indexing with.loc
2. Filtering with multiple conditions-

   We applied our first filter, which was pretty straight forward. But let’s say you want to apply a filter with multiple conditions. How would we do that in pandas? For that we have look at Python operators.

   The & operator, The | operator

3. Aggregation- Sometimes there’s the need to aggregate data so you can create certain overviews or to do some  calculation. In pandas we use groupby ( we are referring to a process involving splitting , applying or combining ) for this.

4. Joins-  Joins are combining two dataframes on a side by side manner based on a common column. Most of the time these columns are referred to askey columns .
   The term joinis originated from the database language SQL, and was needed because the data modelling of SQL databases is mostly done by using relational modelling.
   There are many types of joins, and your output will be based on which type of join your perform. Because this is an introductionary tutorial, we will look at the most common one: inner join.The inner join is derived from venn diagrams which represents inner (intersection) part of both sets.

“Experience the power of our web development course with a free demo – enroll now!”

Analyze Python data using a dataset.

For our analysis, we will make use of the pandas library in Python. After downloading the dataset, you will need to read the .csv file as a data frame in Python. You can do this using the Pandas library.

If you do not have it installed, you can do so with a simple “pip install pandas” in your terminal.

Pandas Profiling

This is a very useful tool that can be used by analysts. It generates an analysis report on the data frame, and helps you better understand the correlation between variables.

To generate a Pandas Profiling report, run the following lines of code:

import pandas_profiling as pp
pp.ProfileReport(df)

This report will give you some overall statistical information on the dataset, which looks like this:

By just glancing at the dataset statistics, we can see that there are no missing or duplicate cells in our data frame.

The information provided above usually requires us to run a few lines of codes to find, but is generated a lot more easily with Pandas Profiling.

Pandas Profiling also provides more information on each variable. I will show you an example:

This is information generated for the variable called “Pregnancies.”

As an analyst, this report saves a lot of time, as we don’t have to go through each individual variable and run too many lines of code.

From here, we can see that:

• The variable “Pregnancies” has 17 distinct values.
• The minimum number of pregnancies a person has is 0, and the maximum is 17.
• The number of zero values in this column is pretty low (only 14.5%). This means that above 80% of the patients in the dataset are pregnant.

In the report, there is information like this provided for each variable. This helps us a lot in our understanding of the dataset and all the columns in it.

The plot above is a correlation matrix. It helps us gain a better understanding of the correlation between the variables in the dataset.

There is a slight positive correlation between the variables “Age” and “Skin Thickness”, which can be looked into further in the visualization section of the analysis.

Since there are no missing or duplicate rows in the data frame as seen above, we don’t need to do any additional data cleaning.

“Get hands-on with our python course – sign up for a free demo!”

Read data using Python’s Pandas package.

To read the data frame into Python, you will need to import Pandas first. Then, you can read the file and create a data frame with the following lines of code:

import pandas as pd
df = pd.read_csv('diabetes.csv')

To check the head of the data frame, run:

df.head()

From the screenshot above, you can see 9 different variables related to a patient’s health.

As an analyst, you will need to have a basic understanding of these variables:

• Pregnancies: The number of pregnancies the patient had
• Glucose: The patient’s glucose level
• Blood Pressure
• Skin Thickness: The thickness of the patient’s skin in mm
• Insulin: Insulin level of the patient
• BMI: Body Mass Index of patient
• DiabetesPedigreeFunction: History of diabetes mellitus in relatives
• Age
• Outcome: Whether or not a patient has diabetes

As an analyst, you will need to know the difference between these variable types — Numeric and Categorical.

Numeric variables are variables that are a measure, and have some kind of numeric meaning. All the variables in this dataset except for “outcome” are numeric.

Categorical variables are also called nominal variables, and have two or more categories that can be classified.

The variable “outcome” is categorical — 0 represents the absence of diabetes, and 1 represents the presence of diabetes.