Here are many things that one can think of that could improve the performance of the learning algorithm. This tips were provided by professor Andrew Ng. in his Machine Learning course at Coursera.
- Get more training examples
- Try smaller sets of features
- Try getting additional features
- Try adding Polynomials features ( x1², x2², x1, x2 etc)
- Try decreasing λ
- Try increasing λ
Some people may try one technique & others may try another technique but we should really research a lot before jumping into the above mention techniques to improve our Machine Learning algorithm.