Research
The Human Sensing Group focuses on advancing technologies in eye tracking, tactile sensing, high-speed sensing, and active acoustic sensing. Our research aims to develop innovative solutions for human-computer interaction, assistive systems, and intelligent sensing applications. Explore our projects below to learn more about our contributions to these cutting-edge fields.
Corneal Imaging
Estimating Focused Object using Corneal Surface Image
Researchers are exploring the use of eye tracking in head-mounted camera systems. Traditional methods require detailed calibration in advance, but prolonged use can disrupt the calibration between the eye and the scene camera. Additionally, even if the point-of-regard is estimated using a portable eye-tracker, the focused object might not be identified. Therefore, we propose a novel method for estimating the object a user is focusing on by capturing reflections on the corneal surface with an eye camera. This approach allows simultaneous extraction of eye and environmental information. We use inverse ray tracing to rectify the reflected image and a scale-invariant feature transform to estimate the object where the point-of-regard is located. Continuous unwarped images can also be generated from corneal surface images. We believe our proposed method could be applied to guidance systems, and experiments estimating the focused object and point-of-regard have confirmed its feasibility.
Remote Corneal Imaging by Integrating a 3D Face Model and an Eyeball Model
In corneal imaging methods, utilizing a 3D eyeball model is essential for generating undistorted images. Typically, the relationship between the eye and eye camera is fixed using a head-mounted device. However, remote corneal imaging has potential applications in areas like surveillance systems and driver monitoring. To facilitate remote corneal imaging, we integrated a 3D eyeball model with a 3D face model. Evaluation experiments confirmed the feasibility of this approach. We demonstrated that the center of the eyeball can be estimated based on face tracking, enabling continuous remote eye tracking through corneal imaging.
Tracking Iris and Pupil Simultaneously using RGB-IR Camera
We propose a novel eye-tracking method that utilizes a multispectral camera to simultaneously track the pupil and recognize the iris. Our hybrid approach leverages existing methods, combining them to compensate for the weaknesses present in each individual method when used alone. Notably, our method accounts for movements of the eye's center of rotation, which has often been treated as a static point in earlier studies. Additionally, our method allows for the measurement of pupil diameter using just a single camera. To validate the effectiveness of our method, we conducted experiments estimating the area and shape of the iris, the point-of-gaze, and the size of the pupil. The results indicate that our proposed method offers improved performance compared to previous methods, particularly in situations where the eye moves to the extreme inner corner of its socket.
Eye-tracking for Real World
Estimating 3D Point-of-regard and Visualizing Gaze Trajectories under Natural Head Movements
The portability of an eye-tracking system encourages us to develop a technique for estimating 3D point-of-regard. Unlike conventional methods, which estimate the position in the 2D image coordinates of the mounted camera, such a technique can represent richer gaze information of the human moving in the larger area. In this paper, we propose a method for estimating the 3D point-of-regard and a visualization technique of gaze trajectories under natural head movements for the head-mounted device. We employ the visual SLAM technique to estimate head configuration and extract environmental information. Even in cases where the head moves dynamically, the proposed method could obtain a 3D point-of-regard. Additionally, gaze trajectories are appropriately overlaid on the scene camera image.
Estimating Point-of-Gaze using Smooth Pursuit Eye Movements without Implicit and Explicit User-Calibration
Detecting the point-of-gaze in the real world is a challenging problem in eye-tracking applications. The point-of-gaze is estimated using geometry constraints, and user-calibration is required. In addition, the distances of the focused targets are variable and large in the real world. Therefore, a calibration-free approach without geometry constraints is needed to estimate the point-of-gaze. Recent studies have investigated smooth pursuit eye movements (smooth pursuits) for human-computer interaction applications, and we consider that these smooth pursuits can also be employed in eye tracking. Therefore, we developed a method for estimating the point-of-gaze using smooth pursuits without any requirement for implicit and explicit user-calibration. In this method, interest points are extracted from the scene image, and the point-of-gaze is detected using these points, which are strongly correlated with eye movements. We performed a comparative experiment in a real environment and demonstrated the feasibility of the proposed method.
Estimating Focused Pedestrian using Smooth-Pursuits Eye Movements and Point Cloud toward Assistive System for Wheelchair
Intelligent electric wheelchairs have been developed for personal mobility, and eye-gaze measurement is essential for comprehending the user's attention and for assisting operations. In previous studies, the gaze vector (i.e., the visual or optical axis of the eye) was projected onto the environmental map; hence, an eye tracker was installed on the wheelchair. In addition, hardware calibration, which determines the geometric relationship between the eye tracker and other sensors, such as LiDAR, was performed beforehand. Recently, wearable eye-trackers are expected to employ a daily-use device; therefore, the cooperation between sensors is essential without geometric constraints. Accordingly, we propose a method for estimating focused pedestrians using smooth-pursuit eye movements in the real world. Pedestrians are tracked using a point cloud obtained with 3D LiDAR, and the trajectories of the focused pedestrian are recorded on an environmental map constructed with simultaneous localization and mapping. Several experiments were conducted to evaluate the computational methods for the correlation between eye movements and moving objects, and we confirmed the feasibility and the current issues through these experiments.
Polarization Sensing
Cross-Ratio Based Gaze Estimation using Polarization Camera System
In eye-tracking, near-infrared light is often emitted, and at least four LEDs are located at the corners of displays for detecting the screen plane in the cross-ratio based method. However, long-time radiation of near-infrared light can make a user fatigued. Therefore, in this study, we attempted to extract the screen area correctly without near-infrared radiation emission. A polarizing filter is included in the display, and thus, visibility of the screen can be controlled by the light’s polarization direction of the external polarized light filter. We propose gaze estimation based on the cross-ratio method using a developed polarization camera system, which can capture two polarized images of different angles simultaneously. Further, we confirmed that the point-of-gaze could be estimated using the screen reflection detected by computing the differences between two images without near-infrared emission.
Screen corner detection using polarization camera for cross-ratio based gaze estimation
Eye tracking, which measures line of sight, is expected to advance as an intuitive and rapid input method for user interfaces, and a cross-ratio based method that calculates the point-of-gaze using homography matrices has attracted attention because it does not require hardware calibration to determine the geometric relationship between an eye camera and a screen. However, this method requires near-infrared (NIR) light-emitting diodes (LEDs) attached to the display in order to detect screen corners. Consequently, LEDs must be installed around the display to estimate the point-of-gaze. Without these requirements, cross-ratio based gaze estimation can be distributed smoothly. Therefore, we propose the use of a polarization camera for detecting the screen area reflected on a corneal surface. The reflection area of display light is easily detected by the polarized image because the light radiated from the display is polarized linearly by the internal polarization filter. With the proposed method, the screen corners can be determined without using NIR LEDs, and the point-of-gaze can be estimated using the detected corners on the corneal surface. We investigated the accuracy of the estimated point-of-gaze based on a cross-ratio method under various illumination and display conditions. Cross-ratio based gaze estimation is expected to be utilized widely in commercial products because the proposed method does not require infrared light sources at display corners.
Polarized Near-Infrared Light Emission for Eye Gaze Estimation
The number of near-infrared light-emitting diodes (LEDs) is increasing to improve the accuracy and robustness of eye-tracking methods, and it is necessary to determine the identifiers (IDs) of the LEDs when applying multiple light sources. Therefore, we propose polarized near-infrared light emissions for an eye gaze estimation. We succeeded in determining the IDs of LEDs using polarization information. In addition, we remove glints from the cornea for correctly detecting the pupil center. We confirmed the effectiveness of using polarized near-infrared light emissions through evaluation experiments.
High-Speed Sensing
Eye Gaze Estimation using Imperceptible Marker Presented on High-Speed Display
Advanced eye-tracking methods require a dedicated display equipped with near-infrared LEDs (light-emitting diodes). However, this requirement hinders the widespread adoption of such methods. Additionally, some glints may pass undetected when a large display is employed. To avoid these problems, we propose eye gaze estimation using imperceptible markers presented on a commercially available high-speed display. The marker reference points reflected on the cornea are extracted instead of glints, and the point-of-gaze can be estimated using the cross-ratio method.
Event-Based Pupil Tracking Using Bright and Dark Pupil Effect
Real-time high-speed gaze estimation can enable next-generation gaze-based interaction. The event camera, which captures intensity variations at high frequency, has been employed to this end. However, pupil tracking based only on events is difficult because events are sparse and limited. We propose high-speed pupil tracking using an event camera based on the bright and dark pupil effect. Two illumination sources generate events in the pupil area, and the pupil center is determined in real time at over 2000 Hz without requiring complete image from the events. We implemented gaze target estimation using smooth-pursuit eye movements and confirmed high-speed pupil tracking that may reduce the delay in gaze-based interaction.
Tactile Sensing
Vision-based tactile sensing using multiple contact images generated by RFTIR
Current vision-based tactile sensors have several limitations, such as their size and measurable surface. Therefore, we propose a novel vision-based tactile sensor based on the re-propagated frustrated total internal reflection (FTIR). The part of the FTIR generated by the contact is re-propagated through the medium, and the FTIR are observed from the side of the medium. We validate the physical principle of observation, including multiple contact images by simulations. In addition, a prototype system is developed to estimate the contact position through observations and regression algorithms. Finally, several experiments were performed to confirm the feasibility of the proposed contact estimation based on the repropagated FTIR.
RFTIRTouch: Touch Sensing Device for Dual-sided Transparent Plane Based on Repropagated Frustrated Total Internal Reflection
Frustrated total internal reflection (FTIR) imaging is widely applied in various touch-sensing systems. However, vision-based touch sensing has structural constraints, and the system size tends to increase. Although a sensing system with reduced thickness has been developed recently using repropagated FTIR (RFTIR), it lacks the property of instant installation anywhere because observation from the side of a transparent medium is required. Therefore, this study proposes an "RFTIRTouch" sensing device to capture RFTIR images from the contact surface. RFTIRTouch detects the touch position on a dual-sided plane using a physics-based estimation and can be retrofitted to existing transparent media with simple calibration. Our evaluation experiments confirm that the touch position can be estimated within an error of approximately 2.1 mm under optimal conditions. Furthermore, several application examples are implemented to demonstrate the advantages of RFTIRTouch, such as its ability to measure dual sides with a single sensor and waterproof the contact surface.
Active Acoustic Sensing
Active Bone-Conducted Sound Sensing for Estimating Joint Angle
We propose a wearable sensor system that measures joint angles of an elbow and finger using vibration which is emitted actively. A novelty of this research is to use active sensing for measuring a joint angle. The active sensing means to emit vibration and sounds to a bone, and a microphone receives the propagated vibration and sounds.
Haptic-enabled Active Bone-Conducted Sound Sensing
We propose active bone-conducted sound sensing for estimating a joint angle of a finger and simultaneous use as a haptic interface. For estimating the joint angle, an unnoticeable vibration is input to the finger, and a perceptible vibration is additionally inputted to the finger for providing haptic feedback. The joint angle is estimated by switching the estimation model depending on the haptic feedback and the average error of the estimation is within about seven degrees.
Hand Pose Estimation based on Active Bone-Conducted Sound Sensing
Estimating hand poses is essential to achieve intuitive user interfaces. In Virtual Reality, an infrared (IR) camera is used for hand tracking, and direct manipulation can be accomplished by using hands. Additionally, wearable devices have also attracted attention because of their portability. We have developed a method based on the use of a wearable device to estimate the joint angle, which can be determined using the amplitude of vibration. However, the joint angle, which can be estimated, is limited to particular joints. Therefore, our proposed method determines the hand pose based on active bone-conducted sound sensing toward intuitive user interfaces. We employed the power spectral density as a feature, thereby enabling the hand pose to be classified with a support vector machine. We confirmed the recognition accuracy and the feasibility of our proposed method through evaluation experiments.
Estimating contact force of fingertip
This study proposes a method for estimating the contact force of the fingertip by inputting vibrations actively. The use of active bone-conducted sound sensing has been limited to es- timating the joint angle of the elbow and the finger. We ap- plied it to the method for estimating the contact force of the fingertip. Unlike related works, it is not necessary to mount the device on a fingertip, and tactile feedback is enabled using tangible vibrations.
Grip Force Estimation based on Active Bone-Conducted Sound Sensing
We propose a method for determining grip force based on active bone-conducted sound sensing, which is an active acoustic sensing. In our previous studies, we estimated the joint angle, hand pose, and contact force by emitting a vibration to the body. We aspired to expand to an additional application of an active bone-conducted sound sensing, thus, we tried to estimate the grip force by creating a wrist-type device. The grip force was determined by using the power spectral density as the features, and gradient boosted regression trees (GBRT). Through evaluation experiments, the average error of the estimated grip force was around 15 N. Moreover, we confirmed that the grip strength could be determined with high accuracy.
Contact Point Estimation using Pneumatic System Noise
Active acoustic sensing is being widely used in various fields, with applications including shape estimation of soft pneumatic actuators. In a pneumatic system, air tubes are frequently adopted, and thus it is essential to detect failures along the air path. Although acoustic sensing has been used for detecting contact and identifying the contact position along a tube, it has not been applied to pneumatic systems. We devised an acoustic sensing method to this end for air tubes in a pneumatic system. As pneumatic system noise propagates through the air tube, we employed this type of noise instead of the conventional method of using a sound source or emitting vibration with an additional oscillator.