Mitt Romney’s California neighbors are not exactly his biggest fans

Click image to see more of Mitt Romney's home in La Jolla, California (Lenny Ignelzi/AP)

To claim the presidency, Mitt Romney is faced with the challenge of winning over voters who may have supported President Barack Obama four years ago, especially independents in key swing states. But it's making nice with the people in his own backyard, literally, that could prove to be the most problematic for the Republican nominee.

The New York Times' Michael Barbaro takes a deep dive into Romney's relationship—or lack thereof—with the people who live near him in La Jolla, Calif., where presidential politics is coloring a debate over the candidate's plans to quadruple the size of his vacation home there.

[Related: Romney to put his fortune in blind trust if elected]

But that's not the only complaint people in this wealthy beach enclave have about Romney. Some neighbors complain about Romney's new presidential entourage—including dozens of Secret Service agents who now guard the candidate 24 hours a day—and its effect on their quiet street.

Yet most of the angst about Romney seems to be driven by disagreement with the candidate's politics. At least six homes near Romney's house are owned by gay couples, including the home four doors up the street where the couple are trying to organize a fundraiser at their home for Obama because of his support of same sex marriage.

According to Barbaro, many people on Romney's street were surprised that Romney chose to move into a "progressive" neighborhood, as one neighbor puts it. Adding to the tensions, the candidate, who was spotted painting his fence a few weekends ago, rarely engages with his neighbors, which has irked many of them.

[Related: Romney's rich donors 'buying tax relief']

Some in the neighborhood simply cast Romney as a fuddy-duddy. An unidentified man tells the Times Romney confronted him about smoking pot on the beach last summer and asked him to stop. Recently, the issue came up again, when a police officer asked Mark Quint, a Democrat who lives a few doors down from Romney, to report any pot smoking on the beach because of neighbor complaints.

"He was pretty clear it was the Romneys, " Quint told Barbaro.