Utilizing RDP also helps reduce the number of tools that you are introducing to an environment and in some cases will help you blend in. Will application white-listing stop built-in Microsoft tools?
Another interesting feature with RDP is the ability to mount local resources on the remote machine. This gives the attacker the ability to drag-and-drop files to and from the victim machine. The transfers are done through the encrypted session and are relatively safe from the admin's prying eyes (unless you are being MiTM'd). Once you start mstsc.exe, select options and click the "Local Resources" tab.
Under "Local devices and resources," click the more button. Now you will see the local drives available to mount in the RDP session. Select the drive that you want to mount and click ok and connect.
Once you connect you will see a drive mapped similar to a network drive. You can drag-and-drop between the machines to introduce whatever tools you would like to run or use it as an easy and reliable way to offload any valuable data.