The primary goal of the KType project is to improve communication for people with disabilities. The KType iPad app presents the user with an on-screen keyboard with large keys that can be re-arranged easily for comfort and efficiency. The problem is that many users find it difficult to navigate a touch-screen user-interface due to unsteady hands.

After watching users try to use iPad's multi-touch screen, the primary problem I identified was too many unintentional touches from fingers, thumb, and sides or bottom of the palm. Pictured below is an example with the intentional touch point marked in green, unintentional touch points marked in red.

Depending on whether they are left-handed or right-handed and position their hand upright or turned, the index or middle finger used to point could be in one of the four corners as shown below.

Additionally, the finger or touch-point that is extended the most in one direction (highlighted in yellow) might not be the index finger (highlighted in green) that the user is pointing with.

As the user touches one or more points on the iPad, the KType application gets notified of the touch positions. The core algorithmic problem we have to solve is to correctly identify which touch-point should be used as the primary selection point. In other words, given a set of 2-10 coordinates, how do we make one green and all others red?

Without any user-specific configuration, this is a very difficult problem to solve because they could be right-handed or left-handed and position their hand in any angle from upright to inverted. So KType provides a per-user configuration option to specify hand-alignment in one of the five positions: Normal, Right-Hand Upright, Left-Hand Upright, Right-Hand Turned, Left-Hand Turned. The Normal option is the default setting which positions the cursor in the middle of all the touch-points and works well for users who can avoid unintentional touches. For all other users, we need to be able to identify the correct touch point to use based on which hand-alignment setting has been specified.

Once we know how the hand is aligned, all we have to do is find the touch point nearest to the selected corner as shown below. The green line is the shortest and so we put the KType on-screen cursor at the position highlighted in green and ignore all other points.

hand_align = one of [righthand-upright, lefthand-upright,
                     righthand-turned , lefthand-turned]

min_distance = min_x = min_y = 9999999
max_x = max_y = 0

for each touchpoint p
  min_x = min(min_x, p.x)
  max_x = max(max_x, p.x)
  min_y = min(min_y, p.y)
  max_y = max(max_y, p.y)

corner_x = if hand_align is righthand then min_x else max_x
corner_y = if hand_align is upright   then min_y else max_y

best_position = [corner_x, corner_y]

for each touchpoint p
  new_distance = (corner_x - p.x)^2 + (corner_y - p.y)^2 # no need to take the square-root, save CPU cycles
  if new_distance < min_distance
    min_distance = new_distance
    best_position = [p.x, p.y]

return best_position