Squirrels are probably the worst pest of fruit trees in this country. They can damage fruit at all stages, and the damage is frequently misattributed to birds.
Squirrel damage is very easy to identify. If you look carefully you can see two identical upper incisor marks, with a 'spooned out' mark above made by the lower incisors. Other fruitlets in this cluster just show tooth marks which match. Squirrels rarely eat the whole fruit at this stage, but can fatally damage a large number by 'mouthing' them out of curiousity.
Two year ago I was not at home to monitor the problem at this stage, and lost the entire crop from my 50-odd cultivars. One pear, two apples and no plums, the entire result from a sizeable, mature orchard. The problem is largely due to a particular neighbour who has numerous bird feeders that are not squirrel proof, and the population had grown from a single pair to numerous competing ones over the exact time period that has been excess supply food in the environment. I trapped and disposed of 10 squirrels last year, and this had a very positive effect on the crop which was very large despite a particularly poor growing season.
This is not something I like doing, but it is impossible to get worthwhile results from fruit growing where there are significant numbers of squirrels.