Hamiltonian formalism in higher derivative calculus of variations