Willie Nelson Recalls Having to Admit Cheating to Wife When Mistress Had Their Baby: I 'Was Caught'

Willie Nelson is the first to admit that he was never the perfect husband.

The country music icon, 87, looks back at his marriages and infidelities in his new joint memoir Me and Sister Bobbie: True Tales of the Family Band, which he co-wrote with his sister Bobbie Nelson.

In the book, the singer recalls the moment his second wife Shirley Collie learned that he fathered a child with another woman. "Shirley saw a bill from a Houston hospital. I tried to play it off as no big deal. I told her that I had to go to the hospital for something minor," he writes.

"Shirley wasn't buying that for one simple reason: The bill said the charges were for the birth of a baby girl, Paula Carlene, born to a Mrs. Connie Nelson," he says.

Understandably, "Shirley wanted to know who in the hell was Connie Nelson," he notes, adding, "Couldn't lie. Was caught flatfooted. Had to get the words out of my mouth. 'Connie’s my girlfriend and Paula's our daughter.' "

Nelson was married to his first wife Martha Matthews from 1952-62 and they had three children together: daughters Lana and Susie and son Willie "Billy" Hugh Jr., who died in an accident on Christmas Day 1991.

The star and second wife Shirley, who wed in 1963 and divorced in 1971, the same year he wed Connie Kopeke. In addition to daughter Paula, they later had daughter Amy.

RELATED: How Willie Nelson and His 'Best Friend' Big Sister Bobbie Helped Each Other Through Tragedy

Then, during his marriage with third wife Kopeke, the singer fell in love with makeup artist Annie Marie D'Angelo on the set of Stagecoach, the 1986 film he starred in with Johnny Cash, Kris Kristofferson and Waylon Jennings.

"I messed up another marriage. My wandering ways were too much for any woman to put up with. I'll always love Connie. I'll always love all my wives. I've always said that there's no such thing as a 'former' wife. Once in your life, a wife never leaves. I regret the pain I caused Connie — and Martha and Shirley before her — and have no excuses. But love is love, and in the mideighties I fell head over heels in love with Ann Marie D'Angelo, called Annie. Never had met a woman like her before," Nelson writes in the book.

In 1988, Nelson and Kopeke divorced and he wed D'Angelo in 1991.

"My love for Annie was all-consuming, but winning her over wasn't easy. She had to be sure my marriage was over and that I was truly free. She didn't care about me being a celebrity. She was whip-smart, with a keen appreciation for all forms of art. It didn't hurt that she was pretty and radiated enough energy to light up any room she entered. She also had a sharp political sense," Nelson writes of his fourth wife, to whom he is still married and had two sons Lukas and Jacob Micah.

"Later she'd be a great partner in pushing the progressive causes so important to me. You've already seen that, when it came to romance, I had a gift for complicating things. But marrying Annie wasn't complicated at all. It's about the smartest thing I ever did. I can say that because, 34 years after we first met, we're still together and going strong," he says.

Over two decades into his fourth marriage, he said in a 2012 Express article that he learned he had another child, daughter Renee, whose mother is his friend Mary Haney.

Though his kids have different mothers, Nelson has a close relationship with his family.

Amid the ongoing coronavirus pandemic, the country icon told PEOPLE recently that he has been spending quality time with loved ones at his 700-acre Luck Ranch in Spicewood, Texas.

"My sister Bobbie lives close by, and we talk all the time. Annie and I are on the ranch, with the horses, and we're doing okay," Nelson said. "We're a very close family."

To read an excerpt from their book Me and Sister Bobbie: True Tales of the Family Band, pick up this week's issue of PEOPLE, on newsstands Friday.