iSpy : a real time application of shoulder surfing attacks

As far as all the digital and physical ways of obtaining information about a person, spying on a person is the most reliable way of doing so. Even if the person is protected from all digital attacks by protection software, he/she has very little to do against being spied on. The mobile application I developed focuses on the idea of digitalising and automating these kind of spying(or shoulder-surfing attacks). It is based on another eavesdropping attack[1], where they implemented the idea of eavesdropping on the input that is being entered to an iPhone, exploiting the magnification of the keys on the keyboard, with a video that is post processed, yielding reliable results. My application focuses on providing these results in real time on a mobile platform, with only the power of another mobile device. This way, it increases the usability and mobility of the attack, as it may be performed while the victim is unaware that such an attack is being performed on him/her. The application relies on the computer vision and image processing techniques. With the help of the in-built camera of the device, it inputs a video to send to processing. Based on three different stages of image processing, it first detects the screen, distinguishes and separates it from the rest of the frame that is being captured by the camera. As the screen of the phone is probably tilted, or captured from an angle, the screen is transformed by the help of a feature detection algorithm and rectified to make use of the input. After this process, the screen goes through several processes to distinguish the background (i.e., the keyboard without any input or occlusion) from the fluctuations that may happen on the keyboard. These may be the enlarged keys or occlusions (fingers on the keyboard, etc). Later these fluctuations are processed to distinguish the actual input (enlarged keys) from occlusions. After these eliminations of the input, the application matches the template of the keys to the input according to their position on the keyboard. The user sees the estimated keys through the application in-real time, as well as the processed frames. My implementation process was done with C++ and Objective-C, which uses the camera input to get the each frame, forward it to implementations of SURF algorithm to obtain the screen, plot the homography to rectify it, see if this yields a solution that corresponds with the optical flow of the screen, get the result and try to see if it yields a foreground which can be used by the image processing methods. If so, continue with thresholding the image to make use of the high pass filters and feed to the blob detectors that detect keys according to their size and area. Later on, finding the location of the key and matching the template of the key to the found area, to make sure we have the right output. The tests were done with a random text and with changing conditions(lighting, angles, threshold of the blob finding and sampling frame rate). These conditions show to be in a very delicate equilibrium since with different settings, the correctly detected key rate may change very quickly. They show that the sampling rate is still not enough for recording all keystrokes, while from the ones that the device can distinguish, given the right conditions, the application is able to distinguish the keys at a decent success percentage at quasi real-time. As this application is quite the first step for such shoulder surfing attacks via mobile, this is a promising one in mobile shoulder surfing attacks, while there is a lot of room to improve.