According to traditional western astrology, the 4th house is the father. The modern connection with one's home is because in past centuries property passed from father to son through the paternal line.
Yes, it was sexist, but then the mother is the 7th house from the 4th, or the wife (7th) of the father (4th) or radix 10th house.
Modern astrologers often conflate signs and houses, such that they equate the 4th sign to the 4th house. With Cancer ruled by the moon, the 4th began to seem more like Mom than Dad. Similarly, with the 10th sign Capricorn ruled by Saturn, the 10th began to seem more like Dad.
But this isn't the traditional system.