Package deepnetts.net.train

Class BackpropagationTrainer

java.lang.Object

deepnetts.net.train.BackpropagationTrainer

All Implemented Interfaces:

Trainer, Serializable

public class BackpropagationTrainer
extends Object
implements Trainer, Serializable

Backpropagation training algorithm for feed forward and convolutional neural networks. Backpropagation is a supervised machine learning algorithm which iteratively reduces prediction error, by looking for the minimum of loss function.

See Also:

• FeedForwardNetwork
• ConvolutionalNetwork
• LossFunction
• OptimizerType
• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static final String	PROP_BATCH_MODE	Name of the batchMode property
static final String	PROP_BATCH_SIZE	Name of the batchSize property
static final String	PROP_LEARNING_RATE	Name of the learningRate property
static final String	PROP_MAX_EPOCHS	Name of the maxEpochs property
static final String	PROP_MAX_ERROR	Name of the maxError property
static final String	PROP_MOMENTUM	Name of the momentum property
static final String	PROP_OPTIMIZER_TYPE	Name of the optimizer property

Constructor Summary

Constructors

Constructor	Description
BackpropagationTrainer(NeuralNetwork neuralNet)	Creates an instance of BackpropagationTrainer for the given neural network to train.
BackpropagationTrainer(Properties prop)	Creates an instance of BackpropagationTrainer with the given properties.

Method Summary

Modifier and Type	Method	Description
void	addListener(TrainingListener listener)	Adds training listener to this trainer.
boolean	createsTrainingSnaphots()	Returns true if network creates training snapshots, false otherwise.
int	getBatchSize()	Batch size is number of training examples after which network's weights are adjusted.
int	getCheckpointEpochs()	On how many epochs the snapshots of the trained network should be created.
int	getCurrentEpoch()	Returns the current training epoch(iteration) of this trainer.
float	getDropout()	Dropout is a technique to prevent overfitting, which skips adjusting weights for some neurons with given probability.
boolean	getEarlyStopping()	Early stopping stops training if it starts converging slow, and prevents overfitting.
float	getEarlyStoppingMinLossChange()	Early stopping stops training if the error/loss start converging to slow.
int	getEarlyStoppingPatience()	How many epochs to wait to see if the loss is lowering to slow.
boolean	getExtendedLogging()	Extended logging includes additional info for debugging the training.
float	getLearningRate()	Learning rate controls the step size as a percent of the error to use for adjusting internal parameters(weights) of the neural network.
long	getMaxEpochs()	Returns the setting for maximum number of training epochs(iterations).
float	getMaxError()	Returns the setting for the stopping error threshold.
float	getMomentum()	Momentum settings helps to avoid oscillations in weight changes and get more stable and faster training.
NeuralNetwork<?>	getNeuralNetwork()	Returns a neural network trained by this trainer.
OptimizerType	getOptimizer()
boolean	getShuffle()	Returns shuffle flag which determines if training set should be shuffled before each epoch.
int	getSnapshotEpochs()	On how many epochs to make training snapshots.
String	getSnapshotPath()	Path to use for making snapshots - saving the current state of trained network during the training in order to be able to restore it from a training point if needed.
float	getStopAccuracy()
float	getStopError()	Alias for getMaxError().
javax.visrec.ml.data.DataSet<?>	getTestSet()	Test set is used after the training to estimate performance of the trained model and generalization ability with new data.
float	getTrainingAccuracy()	Accuracy metric which tells us a percent of correct predictions for training set.
float	getTrainingLoss()	Total training error/loss at the current epoch.
float	getValidationAccuracy()	Accuracy metric which tells us a percent of correct predictions for validation set.
float	getValidationLoss()	Validation loss is an error calculated using validation set, used to prevent overfitting, and validate architecture and training settings.
boolean	isBatchMode()	In batch mode weights are adjusted after the pass of all examples from the training set, while in online mode weights are adjusted after each training example.
void	removeListener(TrainingListener listener)	Removes training listener from this trainer.
BackpropagationTrainer	setBatchMode(boolean batchMode)	Sets flag whether to use batch mode during the training.
BackpropagationTrainer	setBatchSize(int batchSize)	Batch size is number of training examples after which network's weights are adjusted.
BackpropagationTrainer	setCheckpointEpochs(int checkpointEpochs)	On how many epochs the snapshots of the trained network should be created.
BackpropagationTrainer	setDropout(float dropout)	Dropout is a technique to prevent overfitting, which skips adjusting weights for some neurons with given probability.
BackpropagationTrainer	setEarlyStopping(boolean earlyStopping)	Early stopping stops training if it starts converging slow, and prevents overfitting.
BackpropagationTrainer	setEarlyStoppingMinLossChange(float earlyStoppingMinLossChange)	Early stopping stops training if the error/loss start converging to slow.
BackpropagationTrainer	setEarlyStoppingPatience(int earlyStoppingPatience)	How many epochs to wait to see if the loss is lowering to slow.
void	setExtendedLogging(boolean extendedLogging)	Extended logging includes additional info for debugging the training.
BackpropagationTrainer	setL1Regularization(float regL1)	L1 regularization (sum of abs values) is used to prevent overfitting and too large weights.
BackpropagationTrainer	setL2Regularization(float regL2)	L2 regularization (sum of squares) is used to prevent overfitting and too large weights.
BackpropagationTrainer	setLearningRate(float learningRate)	Learning rate controls the step size as a percent of the error to use for adjusting internal parameters(weights) of the neural network.
BackpropagationTrainer	setLearningRateDecay(float decayRate)	Learning rate decay lowers the learning rate with each epoch by devayRate factor, which may improve error lowering the error.
BackpropagationTrainer	setMaxEpochs(long maxEpochs)	Deprecated. Use setStopEpochs instead
BackpropagationTrainer	setMaxError(float maxError)	Deprecated. Use setStopError instead
BackpropagationTrainer	setMomentum(float momentum)	Momentum settings helps to avoid oscillations in weight changes and get more stable and faster training.
BackpropagationTrainer	setOptimizer(OptimizerType optimizer)
final void	setProperties(Properties prop)	Sets properties from available keys in specified prop object.
BackpropagationTrainer	setShuffle(boolean shuffle)	Sets shuffle flag which determines if training set should be shuffled before each epoch.
void	setSnapshotEpochs(int snapshotEpochs)	On how many epochs to make training snapshots.
BackpropagationTrainer	setSnapshotPath(String snapshotPath)	Path to use for making snapshots - saving the current state of trained network during the training in order to be able to restore it from a training point.
BackpropagationTrainer	setStopAccuracy(float stopAccuracy)
BackpropagationTrainer	setStopEpochs(long stopEpochs)	Sets number of epochs/iterations to run the training.
BackpropagationTrainer	setStopError(float stopError)	The training stops when/if training error reach this value.
void	setTestSet(javax.visrec.ml.data.DataSet<MLDataItem> testSet)	Test set is used after the training to estimate performance of the trained model and generalization ability with new data.
void	setTrainingSnapshots(boolean trainingSnapshots)	Training snapshots save the current state of the trained neural network during the training in order to be able to restore it from a training point if needed.
void	stop()	Stops the training.
void	train(javax.visrec.ml.data.DataSet<?> trainingSet, double valSplit)	Run training using given training set, and split part of it to use as a validation set.
void	train(javax.visrec.ml.data.DataSet<? extends MLDataItem> trainingSet)	Runs training using specified training set.
void	train(javax.visrec.ml.data.DataSet<MLDataItem> trainingSet, javax.visrec.ml.data.DataSet<MLDataItem> validationSet)	Runs training using given training and validation sets.
void	updateLearningRate(float learningRate)	Updates learning rate for all layers during the learning rate decay.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

PROP_MAX_ERROR

public static final String PROP_MAX_ERROR

Name of the maxError property

See Also:

• Constant Field Values

PROP_MAX_EPOCHS

public static final String PROP_MAX_EPOCHS

Name of the maxEpochs property

See Also:

• Constant Field Values

PROP_LEARNING_RATE

public static final String PROP_LEARNING_RATE

Name of the learningRate property

See Also:

• Constant Field Values

PROP_MOMENTUM

public static final String PROP_MOMENTUM

Name of the momentum property

See Also:

• Constant Field Values

PROP_BATCH_MODE

public static final String PROP_BATCH_MODE

Name of the batchMode property

See Also:

• Constant Field Values

PROP_BATCH_SIZE

public static final String PROP_BATCH_SIZE

Name of the batchSize property

See Also:

• Constant Field Values

PROP_OPTIMIZER_TYPE

public static final String PROP_OPTIMIZER_TYPE

Name of the optimizer property

See Also:

• Constant Field Values

Constructor Details

BackpropagationTrainer

public BackpropagationTrainer(NeuralNetwork neuralNet)

Creates an instance of BackpropagationTrainer for the given neural network to train.

Parameters:

neuralNet - neural network to train using this instance of backpropagation algorithm

BackpropagationTrainer

public BackpropagationTrainer(Properties prop)

Creates an instance of BackpropagationTrainer with the given properties.

Parameters:

prop - key,value pairs of properties for backpropagation

Method Details

train

public void train(javax.visrec.ml.data.DataSet<MLDataItem> trainingSet,
 javax.visrec.ml.data.DataSet<MLDataItem> validationSet)

Runs training using given training and validation sets. Training set is used to train model, while validation set is used to check model evaluation metrics during the training. with unseen data in order to prevent over-fitting. Note that validation set is different from test set which is used after the training in order to evaluate trained model.

Parameters:

trainingSet - set of example data to train the network

validationSet - set of example data to validate the network during the training

train

public void train(javax.visrec.ml.data.DataSet<?> trainingSet,
 double valSplit)

Run training using given training set, and split part of it to use as a validation set.

Parameters:

trainingSet - set of example data to train the network

valSplit - percent of training set to use as a validation set, value between 0 and 1, commonly something like 0.1, 0.2

train

public void train(javax.visrec.ml.data.DataSet<? extends MLDataItem> trainingSet)

Runs training using specified training set. Training is an iterative procedure during which network's internal parameters(weights) are adjusted in order to minimize prediction error for the given example data in training set.

Specified by:

train in interface Trainer

Parameters:

trainingSet - set of example data to train the network

getMaxEpochs

public long getMaxEpochs()

Returns the setting for maximum number of training epochs(iterations). Training stops when the specified number of training epochs or error threshold (stopError) is reached

Returns:

max training epochs

setMaxEpochs

public BackpropagationTrainer setMaxEpochs(long maxEpochs)

Deprecated.

Use setStopEpochs instead

Sets maximum number of training epochs(iterations) for training the network. Epoch is a single pass of all trainings examples from the training set. The training will stop after the specified number of epochs, if the network does not reach some other stopping condition before (like error threshold).

Parameters:

maxEpochs - the maximum number of training epochs(iterations) for training the network

Returns:

this trainer

See Also:

• stopError

setStopEpochs

public BackpropagationTrainer setStopEpochs(long stopEpochs)

Sets number of epochs/iterations to run the training. When this number of epochs is reached the training will stop, if target accuracy has not been reached before.

Parameters:

stopEpochs - number of epochs after which training will stop

Returns:

this trainer

See Also:

• stopError

getMaxError

public float getMaxError()

Returns the setting for the stopping error threshold. The training stops when total network error reaches this value.

Returns:

stop error threshold

getStopError

public float getStopError()

Alias for getMaxError().

Returns:

setMaxError

public BackpropagationTrainer setMaxError(float maxError)

Deprecated.

Use setStopError instead

Sets stopping error threshold for this training. The training will stop when/if training error reach this value. This method will be deprecated and setStopError method should be used instead, as more intuitive.

Parameters:

maxError - maximum error threshold

Returns:

this trainer

setStopError

public BackpropagationTrainer setStopError(float stopError)

The training stops when/if training error reach this value.

Parameters:

stopError - value of training error to stop the training

Returns:

this trainer

getStopAccuracy

public float getStopAccuracy()

setStopAccuracy

public BackpropagationTrainer setStopAccuracy(float stopAccuracy)

setLearningRate

public BackpropagationTrainer setLearningRate(float learningRate)

Learning rate controls the step size as a percent of the error to use for adjusting internal parameters(weights) of the neural network. With too large values training may cannot be completed and error will grow, while with too small values training might last too long or get stuck in local minimum. Commonly used default value for this setting is 0.01, which practically means that 1% of the error will be used for weight modification.

Parameters:

learningRate - a value in range (0, 1), where 0.01 is being used as a default initial value

Returns:

this trainer

getLearningRate

public float getLearningRate()

Learning rate controls the step size as a percent of the error to use for adjusting internal parameters(weights) of the neural network. With too large values training may cannot be completed and error will grow, while with too small values training might last too long or get stuck in local minimum. Commonly used default value for this setting is 0.01, which practically means that 1% of the error will be used for weight modification.

Returns:

getNeuralNetwork

public NeuralNetwork<?> getNeuralNetwork()

Returns a neural network trained by this trainer.

Returns:

instance of a neural network trained by this trainer

updateLearningRate

public void updateLearningRate(float learningRate)

Updates learning rate for all layers during the learning rate decay. Used by LearningRateDecay technique.

Parameters:

learningRate - a value of learning rate to set for all layers

See Also:

• LearningRateDecay

setLearningRateDecay

public BackpropagationTrainer setLearningRateDecay(float decayRate)

Learning rate decay lowers the learning rate with each epoch by devayRate factor, which may improve error lowering the error.

Parameters:

decayRate -

Returns:

this trainer

setL2Regularization

public BackpropagationTrainer setL2Regularization(float regL2)

L2 regularization (sum of squares) is used to prevent overfitting and too large weights.

Parameters:

regL2 - coefficient for L2 regularization

Returns:

this trainer

setL1Regularization

public BackpropagationTrainer setL1Regularization(float regL1)

L1 regularization (sum of abs values) is used to prevent overfitting and too large weights.

Parameters:

regL1 - coefficient for L1 regularization

Returns:

this trainer

getShuffle

public boolean getShuffle()

Returns shuffle flag which determines if training set should be shuffled before each epoch.

Returns:

value of the shuffle flag

setShuffle

public BackpropagationTrainer setShuffle(boolean shuffle)

Sets shuffle flag which determines if training set should be shuffled before each epoch.

Parameters:

shuffle -

Returns:

this trainer

addListener

public void addListener(TrainingListener listener)

Adds training listener to this trainer.

Parameters:

listener - object that listens for the events in this trainer

removeListener

public void removeListener(TrainingListener listener)

Removes training listener from this trainer.

Parameters:

listener - listener to remove

isBatchMode

public boolean isBatchMode()

In batch mode weights are adjusted after the pass of all examples from the training set, while in online mode weights are adjusted after each training example.

See Also:

• setBatchMode(boolean)

setBatchMode

public BackpropagationTrainer setBatchMode(boolean batchMode)

Sets flag whether to use batch mode during the training. In batch mode weights are adjusted after the pass of all examples from the training set, while in online mode weights are adjusted after each training example.

Parameters:

batchMode -

Returns:

this trainer

getBatchSize

public int getBatchSize()

Batch size is number of training examples after which network's weights are adjusted.

Returns:

setBatchSize

public BackpropagationTrainer setBatchSize(int batchSize)

Batch size is number of training examples after which network's weights are adjusted.

Parameters:

batchSize -

Returns:

setMomentum

public BackpropagationTrainer setMomentum(float momentum)

Momentum settings helps to avoid oscillations in weight changes and get more stable and faster training. It has effect only if momentum optimizer is used.

Parameters:

momentum - a decimal value greater than zero and less than one

Returns:

getMomentum

public float getMomentum()

Momentum settings helps to avoid oscillations in weight changes and get more stable and faster training. It has effect only if momentum optimizer is used.

Returns:

stop

public void stop()

Stops the training.

getTrainingLoss

public float getTrainingLoss()

Total training error/loss at the current epoch. The error is calculated using loss function and is referred to also as a loss.

Returns:

total training error/loss at the current epoch.

getValidationLoss

public float getValidationLoss()

Validation loss is an error calculated using validation set, used to prevent overfitting, and validate architecture and training settings.

Returns:

error/loss calculated usng validation set

getTrainingAccuracy

public float getTrainingAccuracy()

Accuracy metric which tells us a percent of correct predictions for training set.

Returns:

classification accuracy for the training examples

getValidationAccuracy

public float getValidationAccuracy()

Accuracy metric which tells us a percent of correct predictions for validation set.

Returns:

classification accuracy for examples in validation set

getCurrentEpoch

public int getCurrentEpoch()

Returns the current training epoch(iteration) of this trainer. Epoch is one pass of all examples from a training set.

Returns:

current training epoch

getOptimizer

public OptimizerType getOptimizer()

setOptimizer

public BackpropagationTrainer setOptimizer(OptimizerType optimizer)

getTestSet

public javax.visrec.ml.data.DataSet<?> getTestSet()

Test set is used after the training to estimate performance of the trained model and generalization ability with new data. Examples (data) from test set should never be used during the training. Tests set is commonly generated by splitting all available data in training and test sets in some ratio.

Returns:

test set - example data not used during the training, that will be used for evaluation/testing of the trained model

setTestSet

public void setTestSet(javax.visrec.ml.data.DataSet<MLDataItem> testSet)

Test set is used after the training to estimate performance of the trained model and generalization ability with new data. Examples (data) from test set should never be used during the training. Tests set is commonly generated by splitting all available data in training and test sets in some ratio.

Parameters:

testSet - example data not used during the training, that will be used for evaluation/testing of the trained model

getEarlyStopping

public boolean getEarlyStopping()

Early stopping stops training if it starts converging slow, and prevents overfitting.

Returns:

setEarlyStopping

public BackpropagationTrainer setEarlyStopping(boolean earlyStopping)

Early stopping stops training if it starts converging slow, and prevents overfitting.

Parameters:

earlyStopping -

Returns:

this trainer

setSnapshotPath

public BackpropagationTrainer setSnapshotPath(String snapshotPath)

Path to use for making snapshots - saving the current state of trained network during the training in order to be able to restore it from a training point.

Parameters:

snapshotPath -

Returns:

this trainer

getSnapshotPath

public String getSnapshotPath()

Path to use for making snapshots - saving the current state of trained network during the training in order to be able to restore it from a training point if needed.

Returns:

directory to store snapshots of the neural networks during the training

getSnapshotEpochs

public int getSnapshotEpochs()

On how many epochs to make training snapshots.

Returns:

setSnapshotEpochs

public void setSnapshotEpochs(int snapshotEpochs)

On how many epochs to make training snapshots.

Parameters:

snapshotEpochs -

setTrainingSnapshots

public void setTrainingSnapshots(boolean trainingSnapshots)

Training snapshots save the current state of the trained neural network during the training in order to be able to restore it from a training point if needed.

Parameters:

trainingSnapshots -

createsTrainingSnaphots

public boolean createsTrainingSnaphots()

Returns true if network creates training snapshots, false otherwise. Training snapshots save the current state of the trained neural network during the training in order to be able to restore it from a training point if needed.

Returns:

getEarlyStoppingMinLossChange

public float getEarlyStoppingMinLossChange()

Early stopping stops training if the error/loss start converging to slow. If the loss change is lower than given value for patience epochs the training will stop.

Returns:

setEarlyStoppingMinLossChange

public BackpropagationTrainer setEarlyStoppingMinLossChange(float earlyStoppingMinLossChange)

Early stopping stops training if the error/loss start converging to slow. If the loss change is lower than given value for patience epochs the training will stop.

Parameters:

earlyStoppingMinLossChange -

Returns:

this trainer

getEarlyStoppingPatience

public int getEarlyStoppingPatience()

How many epochs to wait to see if the loss is lowering to slow.

Returns:

setEarlyStoppingPatience

public BackpropagationTrainer setEarlyStoppingPatience(int earlyStoppingPatience)

How many epochs to wait to see if the loss is lowering to slow.

Parameters:

earlyStoppingPatience -

Returns:

getCheckpointEpochs

public int getCheckpointEpochs()

On how many epochs the snapshots of the trained network should be created.

Returns:

setCheckpointEpochs

public BackpropagationTrainer setCheckpointEpochs(int checkpointEpochs)

On how many epochs the snapshots of the trained network should be created.

Parameters:

checkpointEpochs -

Returns:

setProperties

public final void setProperties(Properties prop)

Sets properties from available keys in specified prop object.

Parameters:

prop -

setDropout

public BackpropagationTrainer setDropout(float dropout)

Dropout is a technique to prevent overfitting, which skips adjusting weights for some neurons with given probability.

Parameters:

dropout - value between 0.2 and 0.8 which represents probability to skip adjusting weights

Returns:

this trainer

getDropout

public float getDropout()

Dropout is a technique to prevent overfitting, which skips adjusting weights for some neurons with given probability.

Returns:

value between 0.2 and 0.8 which represents probability to skip adjusting weights

getExtendedLogging

public boolean getExtendedLogging()

Extended logging includes additional info for debugging the training.

Returns:

setExtendedLogging

public void setExtendedLogging(boolean extendedLogging)

Extended logging includes additional info for debugging the training.

Parameters:

extendedLogging -