Two-particle angular correlations between trigger particles in the forward pseudorapidity range (2.5 < |η| < 4.0) and associated particles in the central range (|η| < 1.0) are measured with the ALICE detector in p-Pb collisions at a nucleon-nucleon centre-of-mass energy of 5.02 TeV. The trigger particles are reconstructed using the muon spectrometer, and the associated particles by the central barrel tracking detectors. In high-multiplicity events, the double-ridge structure, previously discovered in two-particle angular correlations at midrapidity, is found to persist to the pseudorapidity ranges studied in this Letter. The second-order Fourier coefficients for muons in high-multiplicity events are extracted after jet-like correlations from low-multiplicity events have been subtracted. The coefficients are found to have a similar transverse momentum (pT) dependence in p-going (p-Pb) and Pb-going (Pb-p) configurations, with the Pb-going coefficients larger by about 16 ± 6%, rather independent of pT within the uncertainties of the measurement. The data are compared with calculations using the AMPT model, which predicts a different pT and η dependence than observed in the data. The results are sensitive to the parent particle v2 and composition of reconstructed muon tracks, where the contribution from heavy flavour decays is expected to dominate at pT>2 GeV/c.