I recommend you to refer the VuGen help, section related to .Net protocol recording options. There you can learn how to manage the recording filters (filters are intended to specify which activity, classes, and methods to record).
First steps I'd take - is to create custom filter, add reference to RabbitMQ.Client and RabbitMQ.ServiceModel, include the very basic top-level interfaces in those dlls (like RabbitMQ.Client.IConnection, RabbitMQ.Client.IConnectionFactory) and the main classes that are responsible for client-server interaction. You have to refer RabbitMQ documentation here, I'm sorry I can't provide more info, because don't know much about this framework.