Hi Thiago,
It is not normal. The issues could be caused by: Data Execution Prevention, User Account Control, Antivirus Software, Firewall, or some other software which is preventing proper work of VuGen. Can you please disable them and try to record the script again?
If there are issues with your environment you can try using Proxy recording. Please check:
http://h30499.www3.hp.com/t5/LoadRunner-Support-Customer/Proxy-Recording-with-LoadRunner-11-52/m-p/6304007#M998
http://h30499.www3.hp.com/t5/HP-LoadRunner-and-Performance/Using-proxy-recording-for-local-AUTs-in-LoadRunner-12-00/ba-p/6416362#.UzJYI4UkT00
http://h30499.www3.hp.com/t5/HP-LoadRunner-and-Performance/Using-proxy-recording-for-local-AUTs-in-LoadRunner-12-00/ba-p/6416362#.UzJYI4UkT00
Regarding the Java pop ups, it is not a VuGen issue. It is a security setting implemented in the newer versions of Java. You have an option to disable Java content to be displayed in the browser during working with WebTours from Java Control Pannel -> Security, Enable Java Content in the browser checkbox should not be checked.
Kind regards,
Vesela