Hi,
There is lack of info to give you definite advise. So there are more question raiser than answers.
1. Is I properly understand that your application server is custom made?
2. Is application you launch is text terminal aplication or it graphical UI application just launched by terminal emulator?
3. What type and model of terminal emulates your application server, e.g. vt420-7/vt220, IBM 3270, IBM 5250, etc.?
4. Do you use server or/and client sertificates during login?
5. Can you try to connect to you application server using standalone PowerTerm terminal used by LR to record RTE. To do so, please go to the LR installation folder, e.g. "C:\Program Files (x86)\HP\LoadRunner\bin" and run "rtegate -bypass" command. If stanalone terminal can't work with your application server, it useless to try to record application in LR.
6. LR 11.52 contain not last version of PowerTerm terminal, so ff previous attempt is unsuccessful, you can try to download latest trial version of Ericom PowerTerm Interconnect and try it to connect. This may show if upgrade of LR terminal may sense.
7. How other more common terminals work with your application server, e.g. PuTTY.