Undoubtedly, Google's biggest mistake was its failed social media strategy.
Google’s first entry into social media was Orkut, launched in 2004, which was the side project of a single developer and not part of any holistic social strategy. Orkut managed to gain solid traction in India and especially Brazil, apparently just by being an early entrant to those markets. However, Orkut never gained traction globally. It didn't have much to differentiate it from other social networks. It was never really invested in by Google, perhaps because of its origin as a side project.
Google's first strategic entry into the social media market was Google Wave[1]. Wave merged several different kinds of communication, like email, IM and social media, into a single application. Wave never left the experimental phase. It was a promising concept that never had the chance to realize its potential.
Instead, it was replaced by Google Buzz. Buzz turned your Gmail contact list into a social network, so that Google didn't have to get you to sign up separately - an attempt to achieve a critical mass of users instantly. However, in doing so, it exposed every user's contacts to all of their other contacts. This was a big problem for clients of prostitutes and domestic violence shelters. Google was immediately sued and forced to settle with the FTC. Buzz was discontinued 18 months after launch.
Buzz was succeeded by Google+. G+ was similar to Facebook, with a few extra features. Google+ also failed. For one explanation why, from someone who worked on the product, see Why Google+ Failed. The place looked like a ghost town. For bonus points, the Plus team used internal politics and financial incentives to force other Google products to integrate with it, even when that didn't make sense. Many Google employees are still bitter about it. Google eventually shuttered Google+ without a replacement.