I have nothing useful to add to this... we had that direction complement in the Chinese course we did last year, and there were always cases where it didn't work as expected.

Especially if you're talking about "taking things somewhere" you have to distinguish if you're taking them "out of sight" or just "moving them a bit" or whatever. It confused me more than it helped me.

My personal gut feeling about movement of yourself is similar to presumenothing's below. When I move, I always 去 unless I return/go back to somewhere/someone, then I would use 来. I suspect both are okay? That's also judging from the grammar site not being 100% consistent, itself.