Tony,
throuhput drop indicates one of the two things, server isnt sending anything, your scripts are erroring out.
you have indicated timeouts, what timeouts are they? stepdownload timeouts? if stepdownloadtimeout, what was your rationale in letting them timeout vs increasing it and letting the script continue and not terminate it?
interms of load balancer, you said two of them didnt update any logs? did you findout why?
since a fileupload is involved, do you have necessary space left in the SAN or harddrive? it will also be helpful if you provide some server monitor screenshots.
from a tool perspective here is something in your power to do immediately
what is your caching setting in vugen? can you try a run without caching, simulate a new user every iteration?
502's typically are cookie and cache related. among the many other reasons is your loadtest happening in the same domain and subnet as the servers are in?
one more question is what is the loadgenerator doing at that time in terms of cpu,mem,q lengths, etc?