Hints when starting on a new problem Start by getting better than change results, as a baseline for improvements Strip the problem space down to a simpler version, e.g. just learn to classify 0 and 1, rather than all the digits of MNIST Focus on getting decent values hyperparameters one by one (e.g. $\lambda$ or $\eta$), rather than randomly jumping around hyperparameter space Start with getting decent learning rates etc. before scaling up the number of neurons Initially jump about by largish...
