# Supervised Learning on Census Dataset

In this project, I employ several supervised learning algorithms to model individuals' income using data collected from the 1994 US Census. The goal of the project is to construct a model that accurately predicts whether an individual makes more than $50,000.

The result of the project is a model that is **87.1%** accurate (75.4% F-score) at predicting whether an individual makes more than $50,000 based on 13 datapoints. Training the identical model on the most important 5 datapoints results in a model that is **85.9%** accurate (72.8% F-score).

This is page **1** of **3** for the project, where I

# Explore and Preprocess the Data

The specific tasks break down as follows:

**Explore the Data**

a.**Plot Histograms**for Continuous (Numerical) Feature Distributions

b.**Plot Bar Charts**for Discrete (Categories) Feature Distributions**Preprocess the Data**

a. Split into**Features and Labels**

b.**Transform Skewed**Continuous Features

c.**Normalize Continuous**Features

d.**One-Hot Encode Categorical**Features

e.**Suffle and Split**the Data**Write**the Processed Data to File

# Exploring the Data

The dataset for this project was donated by Ron Kohavi and Barry Becker, after originally being published in the article “Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid”. The article by Ron Kohavi is linked here. The data I investigate in this project consists of small changes to the original dataset, which is now available from the UCI Machine Learning Repository. Specifically, these changes include removing the ‘fnlwgt’ feature and records with missing or ill-formatted entries.

```
import numpy as np
import pandas as pd
from time import time
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
c = pd.read_csv('data/census.csv')
```

The outcome (label) is contained in the final column, `income`

.

```
n_records = c.shape[0]
n_greater_50k = c[c['income']=='>50K'].shape[0]
n_at_most_50k = c[c['income']=='<=50K'].shape[0]
greater_50k_percent = n_greater_50k/n_records
print(' Total records: {}'.format(n_records))
print(' Individuals with income >$50K: {}'.format(n_greater_50k))
print('Individuals with income <=$50K: {}'.format(n_at_most_50k))
print(' %age with income >$50K: {:.1%}'.format(greater_50k_percent))
```

```
Total records: 45222
Individuals with income >$50K: 11208
Individuals with income <=$50K: 34014
%age with income >$50K: 24.8%
```

Among the features, there are:

- 5
**continuous**, numerical features and - 8
**discrete**, categorical features

```
c.info()
```

```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 45222 entries, 0 to 45221
Data columns (total 14 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 age 45222 non-null int64
1 workclass 45222 non-null object
2 education_level 45222 non-null object
3 education-num 45222 non-null float64
4 marital-status 45222 non-null object
5 occupation 45222 non-null object
6 relationship 45222 non-null object
7 race 45222 non-null object
8 sex 45222 non-null object
9 capital-gain 45222 non-null float64
10 capital-loss 45222 non-null float64
11 hours-per-week 45222 non-null float64
12 native-country 45222 non-null object
13 income 45222 non-null object
dtypes: float64(4), int64(1), object(9)
memory usage: 4.8+ MB
```

## Plot Histograms for Continuous Feature Distributions

```
def plot_continuous_distributions(field, bins=None, xlim_list=None, xtick_list=None):
fig, (b, h) = plt.subplots(ncols=1, nrows=2,
sharex=True,
gridspec_kw={'height_ratios':[1,3]},
figsize=(12,3))
b.boxplot(c[field],
widths=0.6,
vert=False)
b.set_title(field,
fontsize=14)
b.set_yticks([])
b.set_xticks(xtick_list)
h.hist(c[field],
bins=bins,
align='left')
h.set_xlim(xlim_list)
h.set_xticks(xtick_list)
for ax in [b, h]:
for spine in ax.spines.values():
spine.set_visible(False)
ax.tick_params(
axis='x',
bottom=False)
ax.tick_params(
axis='y',
left=False,
right=False)
```

```
params = [('age',np.arange(15,95,5),[10,95],np.arange(10,100,10)),
('education-num',np.arange(0,17,1),[0,17],np.arange(2,18,2)),
('capital-gain',np.arange(0,100000,5000),[-5000,105000],
np.arange(0,100000,10000)),
('capital-loss',np.arange(0,4500,250),[-250,5000],np.arange(0,4500,500)),
('hours-per-week',np.arange(0,100,5),[-5,105],np.arange(0,110,10))]
for field, bins, xlim_list, x_tick_list in params:
plot_continuous_distributions(field, bins, xlim_list, x_tick_list)
```

## Plot Bar Charts for Discrete Feature Distributions

```
def plot_discrete_distributions(field):
summary = (pd.DataFrame(c[field].value_counts())
.reset_index())
if summary.shape[0] > 14:
other_total = summary[14:][field].sum()
summary = summary[:14]
summary = summary.append({'index':'Other',
field:other_total},
ignore_index=True)
summary['percentage'] = round(summary[field]/summary[field].sum()*100.,
0).astype(int)
labels = summary['index']
position = summary.index
counts = summary[field]
plt.figure(figsize=(12,3))
plt.bar(position, counts, align='center')
if summary.shape[0] > 6:
rotation = 20
else:
rotation = 0
plt.xticks(position, labels, rotation=rotation)
rects = plt.gca().patches
for n, r in enumerate(rects):
height = r.get_height()
plt.gca().text(r.get_x() + r.get_width() / 2,
height + plt.gca().get_ylim()[1]*0.08,
str(summary['percentage'][n]) + '%\n' + str(height),
ha='center',
va='center')
for spine in plt.gca().spines.values():
spine.set_visible(False)
plt.gca().tick_params(
axis='x',
bottom=False)
plt.gca().tick_params(
axis='y',
left=False,
labelleft=False,
right=False)
plt.title(field,
fontsize=14);
```

```
for field in ['workclass', 'education_level', 'marital-status', 'occupation',
'relationship', 'race', 'sex', 'native-country', 'income']:
plot_discrete_distributions(field)
```

# Preprocessing the Data

The data must be preprocessed before it can be used as input for machine learning algorithms. This “**preprocessing**” consists of handling invalid or missing entries, a broad term for this is **cleaning**, as well as other **formatting** and **restructuring**.

## Split into Features and Labels

```
labels_raw = c['income']
features_raw = c.drop(columns=['income'])
```

## Transforming Skewed Continuous Features

`capital-gain`

and `capital-loss`

are both highly-skewed features. The common practice is to apply a **logarithmic transformation** so that the very large and very small values do not negatively affect the performance of a learning algorithm.

Note that since the logarithm of `0`

is undefined, I translate the values by a small amount above `0`

in order to apply the logarithm successfully.

Note that the only features that are log transformed, below, are the features in the list of skewed fields.

```
skewed = ['capital-gain', 'capital-loss']
features_log_transformed = pd.DataFrame(data = features_raw)
features_log_transformed[skewed] = features_raw[skewed].apply(lambda x: np.log(x + 1))
```

Visualize the change.

```
c['log-capital-gain'] = features_log_transformed['capital-gain']
c['log-capital-loss'] = features_log_transformed['capital-loss']
params = [('capital-gain',np.arange(0,100000,5000),[-5000,105000],
np.arange(0,100000,10000)),
('log-capital-gain',np.arange(0,12,1),[0,12],np.arange(0,14,2)),
('capital-loss',np.arange(0,4500,250),[-250,5000],np.arange(0,4500,500)),
('log-capital-loss',np.arange(0,8,1),[0,9],np.arange(0,10,2))]
for field, bins, xlim_list, x_tick_list in params:
plot_continuous_distributions(field, bins, xlim_list, x_tick_list)
c = c.drop(columns=['log-capital-gain','log-capital-loss'])
```

### Normalizing Numerical Features

In addition to transforming highly skewed features, it is also good practice to scale numerical features. This does not change the shape of the distribution like a log transform. It does, however, ensure that each feature is treated equally when applying supervised learning.

Note that the `capital-gain`

and `capital-loss`

fields were already log-transformed and are now being min-max-scaled.

```
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler() # default=(0, 1)
numerical = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']
features_log_minmax_transform = \
pd.DataFrame(data = features_log_transformed)
features_log_minmax_transform[numerical] = \
scaler.fit_transform(features_log_transformed[numerical])
```

Visualize the change.

```
for col in numerical:
c['norm-' + col] = features_log_minmax_transform[col]
params = [('age',np.arange(15,95,5),[10,95],np.arange(10,100,10)),
('norm-age',np.arange(0,1.05,0.05),[-0.05,1.05],np.arange(0,1.1,0.1)),
('education-num',np.arange(0,17,1),[0,17],np.arange(2,18,2)),
('norm-education-num',np.arange(0,1.05,0.05),[-0.05,1.05],np.arange(0,1.1,0.1)),
('capital-gain',np.arange(0,100000,5000),[-5000,105000],
np.arange(0,100000,10000)),
('norm-capital-gain',np.arange(0,1.05,0.05),[-0.05,1.05],np.arange(0,1.1,0.1)),
('capital-loss',np.arange(0,4500,250),[-250,5000],np.arange(0,4500,500)),
('norm-capital-loss',np.arange(0,1.05,0.05),[-0.05,1.05],np.arange(0,1.1,0.1)),
('hours-per-week',np.arange(0,100,5),[-5,105],np.arange(0,110,10)),
('norm-hours-per-week',np.arange(0,1.05,0.05),[-0.05,1.05],np.arange(0,1.1,0.1))]
for field, bins, xlim_list, x_tick_list in params:
plot_continuous_distributions(field, bins, xlim_list, x_tick_list)
c = c.drop(columns=c.columns[c.columns.str.contains('norm')])
```

### One-Hot Encode Categoricals

Learning algorithms generally expect numeric input. This dataset has 8 features that are categorical, which will need to be converted. This is generally accomplished with **one-hot encoding** the data. The `pandas.get_dummies()`

method will be used for one-hot encoding.

someFeature | |
---|---|

0 | B |

1 | C |

2 | A |

**After One-Hot Encode**

someFeature_A | someFeature_B | someFeature_C |
---|---|---|

0 | 1 | 0 |

0 | 0 | 1 |

1 | 0 | 0 |

The target label, `income`

, must also be converted. Since there are only two possible categories, we can avoid using one-hot encoding and simply encode those categories as `0`

or `1`

.

```
features_final = pd.get_dummies(features_log_minmax_transform)
before_encoding_cols = len(features_log_minmax_transform.columns)
after_encoding_cols = len(features_final.columns)
print("Before one-hot encode: {} features".format(before_encoding_cols))
print(" After one-hot encode: {} features".format(after_encoding_cols))
```

```
Before one-hot encode: 13 features
After one-hot encode: 103 features
```

```
labels_final = (labels_raw
.replace('<=50K',0)
.replace('>50K',1))
```

### Shuffle and Split the Data

Use an 80/20 Training/Testing split.

```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(features_final,
labels_final,
test_size = 0.2,
random_state = 0)
print("Training set: {} samples".format(X_train.shape[0]))
print(" Testing set: {} samples".format(X_test.shape[0]))
```

```
Training set: 36177 samples
Testing set: 9045 samples
```

## Write to File

```
for data_subset, name in [(X_train, 'X_train'),
(X_test, 'X_test'),
(y_train, 'y_train'),
(y_test, 'y_test')]:
data_subset.to_csv('processed/' + name + '.csv')
```