The 79-year-old married Haines when he was 22-year-old and accepts that he could not cope up with things after he became a father.
"Pat became pregnant and we had our wonderful daughter Dominique, who is now 56. But it was the wonderful straw that broke an already very weak camel's back," thesun.co.uk quoted Caine as saying.
"I walked out of the marriage. The marriage breakdown was entirely my fault. I was too young and immature to take on the poverty and personal and professional failure," he added.
Caine is currently married to actress Shakira Baksh. He was recently seen in, 'The Dark Knight Rises'.