YOU SAID:
they went over there to go to their house
INTO JAPANESE
彼らは家に行くためにあそこに行った
BACK INTO ENGLISH
they went there to go home
INTO JAPANESE
彼らは家に帰るためにそこに行きました
BACK INTO ENGLISH
they went there to go home
That didn't even make that much sense in English.