My rule of thumb is build a site for a user, not a spider.