📘 P2: Titanic¶
1. References¶¶
Datasets:¶
In this project I was working with the data set "Titanic Data" from the Udacity website.
It contains demographics and passenger information from 891 of the 2224 passengers and crew on board the Titanic.
This link allows to see the description of this dataset on the Kaggle website, where the data was obtained.
https://www.kaggle.com/c/titanic/data
Articles:¶
http://matplotlib.org/examples/pylab_examples/
http://people.duke.edu/~ccc14/pcfb/numpympl/MatplotlibBarPlots.html
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.linregress.html
https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
Resources :¶
Online Statistics Education: An Interactive Multimedia Course of Study. Project Leader: David M. Lane, Rice University.
The section below is for the code libraries.¶
In [1]:
import pandas as pd
import numpy as np
import scipy
In [8]:
%pylab inline
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
In [3]:
from scipy import stats
from pylab import plot,show
import seaborn as sns
from operator import truediv
In [4]:
def convert_list_to_int(x):
y =[]
for element in x:
el = element[0]
y.append(el)
return y
In [5]:
def percentages_xy(x,y):
return 100.0*x/y
In [6]:
def pieplot(x,xlabel):
figure(1, figsize=(5,5))
ax = axes([0.1, 0.1, 0.8, 0.8])
labels = '1', '2', '3'
fracs = np.array(x)
explode=(0, 0.05, 0)
pie(fracs, explode=explode, labels=labels,
autopct='%1.0f%%', shadow=True, startangle=0)
title(xlabel)
show()
In [7]:
def pearson_stat(x,y,ylabel):
x1 = np.array(x)
y1 = np.array(y)
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
print ' slope r standard deviation '
print ' '
print ' ', slope, " ", r_value, " ", std_err
line = slope*x1+intercept
plot(x1,line,'m-',x1,y1,'o')
pylab.xlim([x[0]-0.5,x[-1]+0.5])
if x == pclass_list:
pylab.xlabel('Pclass')
elif x == index0:
pylab.xlabel('Group by fare')
else:
pylab.xlabel('Age category')
pylab.ylabel(ylabel)
show()
In [9]:
titanic_df = pd.read_csv('/Users/olgabelitskaya/Downloads/titanic_data.csv')
titanic_df.head()
Out[9]:
The most interesting variable we could observe here is "Survived". The main question is: what factors made people more likely to survive? Let us investigate them step by step.¶
I propose to consider whether the number of survivors was dependent on their age, sex, social status (class and fare) or not.¶
The total number of passengers in the sample:¶
In [10]:
len(titanic_df)
Out[10]:
The number of survivors in the sample:¶
In [11]:
titanic_df['Survived'].sum()
Out[11]:
(including this indicator in percentage):¶
In [12]:
100*titanic_df['Survived'].mean()
Out[12]:
We note that in the database there is no information on the age, the cabins and the ports of embarkation for the part of the ship passengers in this sample. This will prevent the qualitative analysis in some cases.¶
In [13]:
titanic_df.info()
In [14]:
pclass = pd.Series(titanic_df['Pclass'])
pclass_list = list(set(pclass.values))
pclass_list
Out[14]:
and the number of passengers for each class:¶
In [15]:
number_by_pclass = titanic_df.groupby('Pclass').count()['PassengerId']
We can estimate the percentage composition of how many passengers these classes have:¶
In [16]:
number_by_pclass_percent = percentages_xy(number_by_pclass,len(titanic_df))
number_by_pclass_df = pd.DataFrame(data={'Number by Pclass': number_by_pclass,
'Number by Pclass in percentages':number_by_pclass_percent})
number_by_pclass_df
Out[16]:
In [18]:
pieplot(number_by_pclass,'Number by Pclass in percentages')
And the number of survivors for all classes and the percentage of survivors for each class:¶
In [103]:
survived_by_pclass = titanic_df.groupby('Pclass').sum()['Survived']
survived_by_pclass_percent1 = percentages_xy(titanic_df.groupby('Pclass').sum()['Survived'],
titanic_df['Survived'].sum())
survived_by_pclass_percent2 = percentages_xy(titanic_df.groupby('Pclass').sum()['Survived'],
number_by_pclass)
survived_by_pclass_df = pd.DataFrame(data={'Survived by Pclass': survived_by_pclass,
'Survived by Pclass in percentages I':survived_by_pclass_percent1,
'Survived by Pclass in percentages II':survived_by_pclass_percent2,})
survived_by_pclass_df
Out[103]:
The variable "Survived by Pclass in percentages I" determines the percentage of passengers who survived in this class in relation to the total number of survivors.¶
The variable "Survived by Pclass in percentages II" determines the percentage of passengers who survived in this class in relation to the total number of passengers in this class.¶
In [20]:
pieplot(survived_by_pclass,'Survived by Pclass in percentages I')
A certain tendency is immediately detected: the higher class of passengers (the lower number of the class), the greater percentage of passengers who survived in this class in relation to the total number of passengers in this class.¶
In this situation we can use the Pearson product-moment correlation coefficient as a measure of the strength of the linear relationship between two variables. Looking at the data, it can be assumed that between the independent variable "Pclass" and the dependent variable "Survived by Pclass in percentages II" there is a negative linear relationship.¶
In [100]:
plt.style.use('seaborn-pastel')
plt.rcParams['figure.figsize'] = (8, 4)
pearson_stat(pclass_list,survived_by_pclass_df['Survived by Pclass in percentages II'] ,
'Survived by Pclass in percentages II')
This picture and the calculations show a plot for which Pearson's r = -0.994024355227. It's extremly close to r = -1, so the relationship between the variable "Pclass" and the variable "Survived by Pclass in percentages II" is a strong negative linear regression.¶
3.3 Fare¶
Let us investigate possible links between the variable "Fare" and other indicators.¶
We can guess that the lower the price of the ticket, the more passengers on board with a ticket at this price. It's also easy to assume that the cost of travel depends on the class. Therefore (bearing in mind paragraph 3.2) it's possible to detect the relationship between the number of survived people and the cost of their tickets.¶
3.3.1 At first, we can find the average fare for passengers of each class.¶
In [29]:
fare_mean_by_class = titanic_df.groupby('Pclass').mean()['Fare']
fare_mean_by_class
Out[29]:
The calculations below show a plot for which Pearson's r = -0.907494151378. It's enough close to r = -1, so the relationship between the independent variable "Pclass" and the dependent variable "Fare mean by class" is a strong negative linear regression.¶
In [101]:
pearson_stat(pclass_list,fare_mean_by_class,'Fare mean by Pclass')
3.3.2 Let us now consider the general indicators of the fare (minimum, maximum and average values).¶
In [31]:
fare = pd.Series(titanic_df['Fare'])
In [32]:
print 'max =', max(fare)
print 'min =', min(fare)
print 'mean =', mean(fare)
We construct a histogram distributing the passengers by intervals of the fare.¶
In [99]:
plt.rcParams['figure.figsize'] = (6, 3)
titanic_df.Fare.hist()
plt.xlabel("Fare")
plt.ylabel("Number by fare")
Out[99]:
At the intervals of about 50 units the large spread the number of passengers with minimum and maximum level of payment is observed.¶
In [36]:
number_by_fare = pd.Series(titanic_df.groupby('Fare').count()['PassengerId'])
For better understanding of the distribution of passengers in the group, I have also constructed the line. It looks like an interval about 10 units will be a better decision.¶
In [102]:
plt.rcParams['figure.figsize'] = (8, 4)
number_by_fare.plot(color='steelblue', linewidth=1.5, linestyle="-")
plt.xlabel("Fare")
plt.ylabel("Number by fare")
Out[102]:
For the next analisis all the passengers had been divided into categories depending on the fare with the difference approximately equal to 10 units.¶
In [43]:
bins1 = np.linspace(fare.min(), fare.max(), 52)
groups = fare.groupby(pd.cut(fare, bins1)).count()
groups1 = groups.tolist()
index1 = [i for i in range(0,510) if i%10 == 0]
groups_df = pd.DataFrame(data={'Number in group by fare': groups1}, index=index1)
We can construct a graph showing the number of the passengers fall into each category.¶
In [98]:
groups_df.plot(color='darkred', linewidth=1.5, linestyle="-")
plt.xlabel("Fare")
plt.ylabel("Number in group by fare")
Out[98]:
Let us exclude outliers. First 27 groups include the majority of passengers, the remaining groups - only three people.¶
In [64]:
groups0 = groups1[:27]
index0 = index1[:27]
number_in_rest = sum(groups1[27:])
print number_in_rest
Now we can estimate the existence of dependence.¶
In [94]:
pearson_stat(index0,groups0,'Number in group by fare')
In this case Pearson's r = -0.617051133707. It's not so close to r = -1, so the relationship between the independent variable "Group by fare" and the dependent variable "Number in group by fare" is a weak negative relationship.¶
3.3.3 Let us have a look at the variable "Fare" and the variable "Survived". Here is a built database for these categories.¶
In [66]:
survived_by_fare = titanic_df.groupby('Fare').sum()['Survived']
survived_groups_df = pd.DataFrame(data={'Survived by fare': survived_by_fare})
In [67]:
survived_groups_df.reset_index(inplace=True)
As in the paragraph 3.3.2 passengers had been divided into several dozen categories according to the price of the ticket and the number of survivors in each category was counted. Outliers were also excluded.¶
In [104]:
survived_groups = survived_groups_df.groupby(pd.cut(survived_groups_df["Fare"], bins1)).sum()
survived_groups1 = survived_groups.fillna(0)
survived_groups2 = survived_groups1['Survived by fare'][:27].tolist()
survived_groups2 = [int(x) for x in survived_groups2]
fare_groups_df = pd.DataFrame(data={'Number by group': groups0,
'Survived by group': survived_groups2},
index=index0)
fare_groups_df.head()
Out[104]:
In [70]:
m = np.array(survived_groups2)
n = np.array(groups0)
# survived_groups_in_percentages0 = 100.0*np.true_divide(m,n)
# survived_groups_in_percentages1 = pd.Series(survived_groups_in_percentages0).fillna(0)
Let us estimate the existence of dependence between "Group by fare" and "Survived in group by fare in percentages" t¶
In [93]:
pearson_stat(index0,survived_groups_in_percentages1,'Survived in group by fare in percentages')
In this case Pearson's r = -0.157400468005. It's enough close to r = 0, so between the independent variable "Group by fare" and the dependent variable "Survived in group by fare in percentages" there is no relationship.¶
In [105]:
number_by_sex = titanic_df.groupby('Sex').count()['PassengerId']
number_by_sex_in_percentages = percentages_xy(titanic_df.groupby('Sex').count()['PassengerId'],
len(titanic_df))
In [106]:
survived_by_sex = titanic_df.groupby('Sex').sum()['Survived']
survived_by_sex_in_percentages1=percentages_xy(titanic_df.groupby('Sex').sum()['Survived'],
titanic_df['Survived'].sum())
survived_by_sex_in_percentages2 = percentages_xy(titanic_df.groupby('Sex').sum()['Survived'],
number_by_sex)
For clarity, the data are combined into one table.¶
In [107]:
survived_by_sex_df = pd.DataFrame(data={'Number by sex': number_by_sex,
'Number by sex in percentages':number_by_sex_in_percentages,
'Survived by sex':survived_by_sex,
'Survived by sex in percentages I':survived_by_sex_in_percentages1,
'Survived by sex in percentages II':survived_by_sex_in_percentages2})
survived_by_sex_df
Out[107]:
The variable "Survived by sex in percentages I" determines the percentage of survived passengers of this sex in relation to the total number of survivors.¶
The variable "Survived by sex in percentages II" determines the percentage of survived passengers of this sex in relation to the total number of passengers of the same sex.¶
In [88]:
plt.rcParams['figure.figsize'] = (8, 4)
fig = plt.figure()
ax = fig.add_subplot(111)
## Data
Number_by_Sex_in_percentages = pd.Series(survived_by_sex_df['Number by sex in percentages'])
Survived_by_Sex_in_percentages1 = pd.Series(survived_by_sex_df['Survived by sex in percentages I'])
Survived_by_Sex_in_percentages2 = pd.Series(survived_by_sex_df['Survived by sex in percentages II'])
## Necessary variables
ind = np.array([1,2]) # the x locations for the pclass
width = 0.2 # the width of the bars
## Bars
rects1 = ax.bar(ind, Number_by_Sex_in_percentages, width, color='red', alpha = 0.7)
rects2 = ax.bar(ind+width, Survived_by_Sex_in_percentages1, width, color='green', alpha = 0.7)
rects3 = ax.bar(ind+2*width, Survived_by_Sex_in_percentages2, width, color='blue', alpha = 0.7)
# Axes and labels
ax.set_xlim(-width+1,len(ind)+width+0.6)
ax.set_ylim(0,90)
ax.set_ylabel('Values by sex in percentages')
ax.set_title('Number of passengers and Survived passengers by sex in percentages')
xTickMarks = ['female','male']
ax.set_xticks(ind+width)
xtickNames = ax.set_xticklabels(xTickMarks)
plt.setp(xtickNames, rotation=10, fontsize=30)
## Legend
ax.legend((rects1[0], rects2[0], rects3[0]),
('Number by sex in percentages', 'Survived by sex in percentages I', 'Survived by sex in percentages II') )
plt.show()
3.5 Age¶
The next step - we have a look at the variable "Age" and the variable "Survived".¶
In [82]:
age = pd.Series(titanic_df['Age'])
We can consider the general indicators of the age of passengers (minimum, maximum and average values).¶
In [83]:
print 'max =', max(age)
print 'min =', min(age)
print 'mean =', mean(age)
We will group the passengers on the basis of age and consider their distribution.¶
In [84]:
number_age = titanic_df.groupby('Age').count()['PassengerId']
In [87]:
plt.rcParams['figure.figsize'] = (8, 4)
titanic_df.Age.hist()
plt.xlabel("Age")
plt.ylabel("Number by age")
Out[87]:
Let us divide passengers into age categories and estimate the number of survivors in each of them. In this analysis, we do not count the passengers whose age is unknown.¶
In [89]:
age = pd.cut(titanic_df['Age'], [0,13,19,35,60,85], labels=['child', 'teenager', 'young','middle-aged','old'])
df_age = pd.DataFrame(data={'Age category': age,'Survived': titanic_df['Survived']})
number_by_age = df_age.groupby('Age category').count()
number_by_age1 = number_by_age.values.tolist()
survived_by_age = df_age.groupby('Age category').sum()
survived_by_age1 = survived_by_age.values.tolist()
survived_by_age_in_percentages = percentages_xy(survived_by_age,number_by_age)
survived_by_age_in_percentages1 = survived_by_age_in_percentages.values.tolist()
Here is a built database for these categories.¶
In [91]:
df_survived_by_age = pd.DataFrame(data={'Number by age category': convert_list_to_int(number_by_age1),
'Survived by age category': convert_list_to_int(survived_by_age1),
'Survived by age category in percentages':
convert_list_to_int(survived_by_age_in_percentages1)},
index=['child', 'teenager', 'young','middle-aged','old'])
df_survived_by_age
Out[91]:
We assign each age category ['child', 'teenager', 'young','middle-aged','old'] the corresponding number [1, 2, 3, 4, 5] and analyze the dependence of the variable "Survived by age category in percentages" on the variable "Age category".¶
In [92]:
pearson_stat([1, 2, 3, 4, 5],
convert_list_to_int(survived_by_age_in_percentages1),
'Survived by age category in percentages')
