Situatie
Solutie
NumPy
The backbone of Python stats
NumPy is useful for data analysis because it includes alot of built-in functions, including the classic descriptive statistics functions like mean, median, and standard deviation.
I can demonstrate by using the function to create a random number generator and draw 50 samples from the normal distribution.
import numpy as np
rng = np.random.default_rng()
a = rng.standard_normal(50)First I’ll take the average:
a.mean()And the median:
np.median(a)And the sample standard deviation (the ddof reduces the number of items in the array considered by one):
np.std(a,ddof=1)
pandas
Finding the right frame for data
NumPy is useful on its own, but it’s suited to working with arrays and matrices. pandas is a library that works on “DataFrames”, which are rectangular data structures that are similar to spreadsheets and relational databases. It can even import many of these formats, such as SQL, Excel, and CSV files.
I’ll demonstrate the latter by importing a dataset of tips that one New York City waiter collected over a weekend.a dataset of tips that one New York City waiter collected over a weekend:
import pandas as pd
tips = pd.read_csv('stats/data/tips.csv')I can examine the dataset with the head() method:
tips.describe()
SciPy
Even more stats functions
While NumPy and pandas have some useful statistical functions, it’s often not enough. SciPy has a lot of useful numerical functions for scientific work, but it’s the statstical functions that are most useful. They’re contained in the stats submodule.
If you paid attention during the mention of descriptive statistics, you might have noticed that mention of the mode was missing from Numpy and pandas. These libraries don’t have a function for mode, but SciPy does. I’ll use it on the array from the normal distribution mentioned earlier:
from scipy import stats
stats.mode(a)
There are also a number of statistical tests, including the T-test of statistical significance. This would be used for determining if the means of two indepdendent samples were different to a sufficient degree. This is used in things like A/B testing or clinical researching. I’ll simulate two samples with the NumPy random number generator and use the ttest_ind() method:
sample_a = 2 + 3 * rng.standard_normal(15)
sample_b = 2 + 5 * rng.standard_normal(15)
stats.ttest_ind(sample_a,sample_b)
The code sets a mean for both of them to two and a standard deviation of 3 for the first sample and a standard deviation of 5 for the second.
The P-value, noted as pvalue determines the significance. If it’s lower than a certain threshold, such as .05 for a 95% confidence level, we’ll reject the null hypothesis that there’s no relationship between the two. In this case, since the means are the same, the p-value os greater than .05, so we can’t reject the null hypothesis in this case.
Seaborn
Great-looking visualizations
A lot of modern statistical computation is based around visualization. It’s easy to see statistical relationships easily when plotted. While Matplotlib has been the standard visualization library in Python, it can be difficult to get started with. Seaborn is a statistical visualization library that acts as a front end to Matplotlib and allows for many common statistical visualizations, including the classic histograms, box plots, scatterplots, and regression plots.
import seaborn as sns
sns.set_theme()
sns.lmplot(x='total_bill',y='tip',hue='smoker',markers=['o','^'],data=tips)
It’s interesting that the nonsmokers seem more generous than the smokers, except for one outlier at the top right. There seems to be a linear relationship for tips vs. the total bill. The higher the bill, the higher the tip.
statsmodels
Powerful statistical modeling at my fingertips
While you can make a regression plot in Seaborn, you can’t get the values for the equation line. For that, you have to go to another library. statsmodels is one of the libraries of choice. One of the biggest uses is for linear regression like the one I showed above:
import statsmodels.formula.api as smf
results = smf.ols('tip ~ total_bill',data=tips).fit()
results.summary()This uses a formula system popularized by R to create a regression object and display a summary. The results in the coef column are what we want for the regression equation of tips vs. bill.
There’s a lot more in statsmodels, including analysis of variance, or ANOVA.
IPython and Jupyter
Easy interactive Python use
While these aren’t traditional “libraries,” IPython and Jupyter are intimately connected to data analysis.
IPython is a major upgrade to the interactive Python intepreter, and Jupyter is an attractive notebook interface. I tend to use the former for experimentation, while I use the latter when I want to save my results for later and share them with other people.
Leave A Comment?