You can create a custom loss function and metrics in Keras by defining a TensorFlow/Theano symbolic function that returns a scalar for each data-point and takes the following two arguments: tensor of true values, tensor of the corresponding predicted values.
Note that the loss/metric (for display and optimization) is calculated as the mean of the losses/metric across all datapoints in the batch.
Keras loss functions are defined in losses.py
Additional loss functions for Keras can be found in keras-contrib repository.