go against, be contrary to
牾 centers on opposition and conflict: going against something or being contrary to it.
牾 combines 牜 (cow/ox) with 吾, which likely contributes the sound. The exact historical development is uncertain, but the character is associated with opposition or going against.
An ox (牜) and the sound component 吾 together suggest a stubborn ox pushing against a barrier: going against and being contrary.
For ゴ, imagine a contrary ox that won't go where you want: go -> ゴ, and the ox goes against your command.