While restrictions on new arrivals, including caps on people from any new EU member state, are necessary, reforming the jobs market is just as important, he argued.
Mr Miliband distanced himself from the rhetoric of his predecessor Gordon Brown, saying: "I am not going to promise 'British jobs for British workers'. But we need an economy which offers working people a fair crack of the whip. The problem we need to address is in those areas and sectors where local talent is locked out of opportunity."
He said Labour had to change its approach to immigration and recognise "the costs as well as the benefits", arguing the last Labour government under Mr Brown became "too disconnected from the concerns of working people".
"We too easily assumed those who worried about immigration were stuck in the past, unrealistic about how things could be different, even prejudiced. But Britain was experiencing the largest peacetime migration in recent history, and people's concerns were genuine. Why didn't we listen more? At least by the end of our time in office, we were too dazzled by globalisation and too sanguine about its price.
"By focusing too much on globalisation and migration's impact on growth, we lost sight of who was benefiting from that growth - and the people who were being squeezed. And, to those who lost out, Labour was too quick to say 'Like it or lump it'."