Victims and defendants facing long delays waiting for their cases to come to court say their lives are being "left on hold".