Few fathers are taking advantage of additional paternity leave, mainly because of the low statutory rate of pay they would receive, according to a new report.
A study by the TUC showed that of 285,000 men eligible to take up to 26 weeks leave, just 1,650 did so in 2011/12.
The low take up is because men can't afford to live on the statutory weekly rate of £136 a week - which is rarely topped up by employers, said the TUC.
In contrast, most fathers take the first two weeks of paternity leave, which is usually topped up by employers.
TUC General Secretary Frances O'Grady said: "A good gift for fathers this Sunday would be for ministers to increase statutory paternity pay rates and for employers to top it up for longer, so that new dads can spend more time with their children.
"Poor levels of financial support are preventing new dads from taking extra time off and are particularly affecting low-paid fathers who simply cannot afford to take leave.
"Extending paternity pay from two to six weeks and paying a better statutory rate would make a massive difference, as has been shown in other countries."
A Business Department spokesman said: "The current system for parental leave is old-fashioned and too rigid. This is why we are introducing a system of shared parental leave from April 2015 so that fathers can take more leave if they want to in the early days of a child's life.
"We want to challenge the myth that it is the mother's role to stay at home and care for children.
"Men will be more able to get better involved with the caring of their children from the earliest stages and evidence shows this sort of involvement has significant benefits for children's educational and emotional development in later life."