I think it's at the referees discretion as to wether the ball is under the control of the goalkeeper. Generally though, the keepers are given such a high level of protection these days and almost always given the benefit of the doubt.
Certainly, kicking the ball out of the goalkeepers hands is a big no-no and almost certainly would be given as a foul against the attackers.
[Edit]...and I've been watching some of the roughest Scottish football for decades and witnessed a player on one team who got into an on pitch fight with another player. He picked up a grassy divot and tried to shove it down his opponents throat - that was deemed 'alright & to 'Get up you big baby''. By contrast, in the same game, a player farted in the general direction of the opposing goalkeeper was sent off for serious foul play!*
*I might have exaggerated that a bit :-)