The following definition pasted from dictionary.com

–noun

1. an athletic activity requiring skill or physical prowess and often of a competitive nature, as racing, baseball, tennis, golf, bowling, wrestling, boxing, hunting, fishing, etc.

2. a particular form of this, esp. in the out of doors.

Now, I would probably argue against hunting and fishing, and I know some people would argue against racing, but this is how the word is defined. Just because you don't find golf to be very strenuous has nothing to do with it. When played competitively there is no doubt that golf is a sport. When played as relaxation or for fun, much like shooting baskets or playing 21 instead of a basketball game, I can agree that some golf is not sporting, and some "golfers" are surely not sportsmen.