The transmission was not shifting real nicely when it was running. You could go up and down through the gears with no problems when the engine was off.
I have the cases apart now, the only part thats under spec according to the service manual is the nubs on the shift forks that fit into the shift drum. It makes sense that this is the source of my shifting problem, the way there worn, there not going to move as freely in the shift drum as they should be to shift properly. The gears are all in good shape and moving on the gear shafts nicely.