UFC fighter Jason "Mayhem" Miller was arrested last week for assault ... TMZ has learned -- after the fighter allegedly put his sister into a headlock and refused to let her go.

Law enforcement tells TMZ, Miller and his sis were at some house party in North Carolina -- when his sister decided she wanted to take off.

According to law enforcement, Jason's sister claims he wouldn't let her leave -- and put her into a headlock.

We're told Jason's sister managed to escape and notified the Chatham County Sheriff -- who had Jason arrested later that evening for simple assault and false imprisonment.