TL;DR

When using tf.estimator , use the normalizer_fn argument in tf.feature_column.numeric_feature to normalize using the same parameters (mean, std, etc.) for training, evaluation, and serving.

def zscore(col):

mean = 3.04

std = 1.2

return (col — mean)/std feature_name = ‘total_bedrooms’

normalized_feature = tf.feature_column.numeric_column(

feature_name,

normalizer_fn=zscore)

Normalization for the tf.estimator API

Neural network activations generally like their inputs to be normalized. Normalizing inputs to nodes in a network helps prevent the so-called vanishing (and exploding) gradients. Batch normalization is the most comprehensive approach for normalization, but it incurs an extra cost and may be overkill for your problem. Thus you may just want to normalize your inputs.

There are two approaches to normalizing inputs when using the tf.estimator API (which is the easiest way to build a TensorFlow model): inside the input_fn and while creating a feature_column . I will show you an example to perform the ladder, then I will show you to train multiple models using ML Engine.

Boilerplate code: Normalize using the `normalizer_fn` argument

Check out this notebook for a full working example.

Normalizing inside the input_fn allows for more flexibility (you can also perform feature engineering here), but I find using the normalizer_fn with tf.feature_column.numeric_column to be more elegant. Here is a basic example:

def zscore(col):

mean = 3.04

std = 1.2

return (col — mean)/std feature_name = ‘total_bedrooms’

normalized_feature = tf.feature_column.numeric_column(

feature_name,

normalizer_fn=zscore)

Below, I will show an end-to-end example, to get the normalization parameters, then normalize all numeric columns in my dataset.

You should compute the normalization parameters ahead of time on the training set. In this case, I’m using Pandas to get the mean and standard deviation for each numeric column:

Alternatively, TensorFlow Transform provides a scalable approach for normalizing inputs and is particularly suitable for larger datasets. Check out an example here.

After running the above, we return parameters for each columns:

{‘households’: {‘mean’: 501.34073416222617, ‘std’: 382.81658748493305},

‘housing_median_age’: {‘mean’: 28.5441089402013, ‘std’: 12.610144469735635},

‘median_income’: {‘mean’: 3.8814239564831365, ‘std’: 1.9061255708184284},

‘population’: {‘mean’: 1428.941163410302, ‘std’: 1150.5106244960523},

‘total_bedrooms’: {‘mean’: 539.6057578448787, ‘std’: 418.76075045523334},

‘total_rooms’: {‘mean’: 2642.2929988158676, ‘std’: 2162.649970020439}}

Now you can create the feature columns, using the training means and standard deviations computed above.

NUMERIC_FEATURES = [‘housing_median_age’, ‘total_rooms’, ‘total_bedrooms’, ‘population’, ‘households’, ‘median_income’] feature_columns = create_feature_cols(NUMERIC_FEATURES, use_normalization=True)

And finally you can build the estimator using the feature columns:

model = tf.estimator.DNNRegressor(hidden_units=[10,4],

model_dir = outdir,

feature_columns = feature_columns)

Training multiple models in the cloud using ML Engine

Normalization is a hyperparameter, and in practice, it would be useful to evaluate different normalization schemes. For example you may want to try training a model just normalizing your features and comparing it to also normalizing inputs to hidden layers using batch normalization.

You can fire off multiple experiments in parallel using Cloud ML Engine. This will be particularly valuable for larger datasets, where you want to leverage the cloud to scale model training. To train with ML Engine, you need to package up the model code, creating task.py and model.py model files. See the repo for an example.

It’s best practice to test the model package locally first to ensure there are no syntactic or semantic errors:

OUTPUT_DIR=’trained_model’

export PYTHONPATH=${PYTHONPATH}:${PWD}/model_code

python -m trainer.task — outdir $OUTPUT_DIR — normalize_input 1

After, you can submit a job to ML Engine using `gcloud ml-engine jobs submit training`:

JOBNAME=my_ml_job_$(date -u +%y%m%d_%H%M%S)

REGION=’us-central1'

BUCKET=’gs://crawles-sandbox’

OUTPUT_DIR=$BUCKET/’housing_trained_model’

PACKAGE_PATH=$PWD/model_code/trainer gcloud ml-engine jobs submit training $JOBNAME \

-- package-path=$PACKAGE_PATH \

-- module-name=trainer.task \

-- region=$REGION \

--staging-bucket=$BUCKET\

-- scale-tier=BASIC \

-- runtime-version=1.8 \

-- \

-- outdir=$OUTPUT_DIR\

-- normalize_input=0

In my case, I trained with and without computing the z-score for the features:

As expected, normalizing the inputs improves the final model performance.

That’s it! If you found this post helpful, please give it a clap so others can find it.

Additional Resources