The 30-year-old poet had been quietly dating the Led Zeppelin rocker for several months before their union was revealed to the world in January 2015, and though she now understands why their 46-year age gap caused headlines, she admitted the attention initially made her feel "uncomfortable".
She told the latest issue of Tatler magazine: "Now I understand it. I guess some people will say I've made an unusual choice.
"At the time it felt very uncomfortable. I felt like my life was going to be over.
"I had such a sense of shame about the whole thing."
And Scarlett insisted she is "lucky" to be in a relationship with the 76-year-old guitarist.
She said: "I'm in an amazing relationship. I think a lot of people make assumptions but Jimmy's an exceptional man and beautiful.
"There are a lot of women of all ages who are excited to meet him.
"And I feel lucky, he really is my best friend and the person who makes me laugh the most...
"It's funny because on paper there is a massive difference.
"But when he's standing in front of me, it's not something I feel It's not something that impedes us."
The couple lead a quiet life at home in London as they rarely go out to parties and don't like entertaining at their abode.
Scarlett said: "We don't go out to parties or clubs.
"Obviously Jimmy has lived but now he doesn't drink either, so that's another thread that binds us.
"He's seen people who have died, he's lost best friends. He's seen all sides of that coin...
"We don't entertain in the house.
"Jimmy's children come to stay, which is lovely.
"My siblings come too...
"Other than that, we like it to be just us. I feel so safe here, it's such an inspiring house."
The couple met through mutual friends and though there wasn't anything romantic between them at first, Scarlett was wary of Jimmy's "dark reputation".
She said: "At first we were just friends. Jimmy has this dark reputation and certainly people play up the Casanova image.
"With me, he wasn't remotely flirty. I kept bumping into him.
"One time, I was walking to Whole Foods and saw him with his son and daughter and he introduced us.
"Then he bumped into me and my parents on our way to dinner. It was strange. "
The pair's relationship began in August 2014 and Scarlett soon moved into Jimmy's home, and she often wishes she could recapture the early days of their romance.
She said: "I've said this to Jimmy but if there was any moment in life I could relive, it would be those days when I was falling in love with him."
Despite the attention their relationship received when it was made public, Scarlett insisted they didn't do anything to keep it hidden.
She said: "For months it was like we had a cloak of invisibility. We'd walk down the street together and no one would notice."
