For some, the outlying island is more than just a weekend getaway, it's a way of life