1. Introduction

Augmented reality (AR) is a technology that combines virtual contents and the physical world in real time by adding computer-generated (CG) images to the view of the physical environment (Azuma, 1997). Contrary to virtual reality (VR), in which a user’s view consists entirely of CG imagery, users can interact with virtual objects created with CG images in realistic contexts (Arena et al., 2022). AR has therefore been expected to extend and enrich visual experiences and actions in the physical environment.

The motivation of our present study was to show behavioral evidence that AR can be used to change and research human weight perception. For this purpose, we considered a well-known phenomenon called the size-weight illusion (SWI) and addressed whether it could occur even with AR objects (i.e., physical objects with superimposed CG images). In the SWI, individuals perceive a smaller object as heavier when lifted than a larger object of the same weight (Flanagan and Beltzner, 2000). Several studies have already examined the effect of the size of superimposed CG images on the subjective heaviness of a physical object in the VR environment (Buckingham, 2019) and AR environment (Rohrbach et al., 2021). However, the findings were controversial. Buckingham (2019) asked participants to use their hand to grasp and lift a physical object with fully overlaid CG images. The results showed that the participants felt a virtually smaller object to be heavier than a virtually larger object, suggesting that the size of superimposed CG images affected the subjective heaviness. In contrast, Rohrbach et al. (2021) superimposed an identical CG image onto smaller and larger objects and found that participants rated the physically smaller object heavier than the physically larger object for the latter half of four trials. This suggests that a CG image superimposition did not have a significant impact.

We speculated that one reason for the failure to show a significant impact of CG imagery in Rohrbach et al. (2021) could be a methodological issue in presenting the objects. In their study, semi-translucent CG images were superimposed on the physical objects such that participants saw both the virtual and physical objects (Rohrbach et al., 2021). It is therefore possible that the conflicting information about the virtual and physical sizes may have reduced the effect of the CG images. If this was the case, then we expect that the SWI would occur when a non-translucent CG image fully overlaid on a physical object was used.

The main purpose of the present study was to address whether our speculation on the findings of Rohrbach et al. (2021) was correct. More specifically, we tested whether the size of a superimposed CG image affected the subjective heaviness of an object, irrespective of its physical size. For this purpose, we used three objects: a virtually small CG image superimposed on a physically small object (PSVS), a virtually large CG image superimposed on a physically small object (PSVL), and a virtually large CG image superimposed on a physically large object (PLVL). If the SWI occurred based on the size of the CG images, then objects with virtually small CG images (i.e., PSVS) would be perceived as heavier than those with virtually large CG images (i.e., PSVL, PLVL). In contrast, if the findings of Rohrbach et al. (2021) were the case even for an object with a non-translucent CG image, then the physically smaller object (i.e., PSVS, PSVL) would be perceived as heavier than the physically larger object (i.e., PLVL).

The secondary purpose was to investigate whether subjective heaviness would be altered with the size of the superimposed CG images even when the size was altered after a series of movements to lift an object (i.e., reaching and lifting an object off the table) has already been initiated. Previous studies using physical objects showed that the SWI occurred even when the object was observable only just after the moment participants lifted the object off a table (Plaisier et al., 2019). We newly investigated what would happen when the size of the superimposed CG image changed suddenly during a series of movements to lift an object.

2. Methods

Participants
Twenty-six individuals participated. All participants were right-hand dominant and had not reported any history of musculoskeletal or neurological disorders. This experiment was approved by the Ethics Committee of Tokyo Metropolitan University (approval number: H3-131) and the Sony Bioethics Committee (approval number: 22-F-0015). All participants provided written informed consent to participate in this study. The data obtained from four participants were excluded from the following analysis because the data collection was incomplete due to technical issues. We used data obtained from twenty-two participants for the following analysis (ten males and twelve females; age: 37.0 ± 10.9 years).

Apparatus
The experimental setup to perform the object lifting in the AR environment was as follows. A desktop computer (OMEN by HP Obelisk Desktop 875-1xxx, HP, USA) was used for data collection and stimulus presentation. Participants wore a head–mounted display (HMD) (Oculus Rift S, Meta, USA) with an attached stereo camera (ZED mini, Stereo Lab, USA). The HMD was fixed on the metal frame, which was attached to the desk, to keep the center of the participant’s view on the object. The setup used in this experiment is shown in Figure 1.

The spatial locations of the HMD, the object to be lifted, and the wrist strap were captured by eighteen cameras for three-dimensional motion analyses (OQUS and MIQUS, Qualisys, Sweden). Five passive retro-reflective markers were attached to the HMD. Five markers were attached to each object. Four markers were attached to the wrist strap. Two markers were attached to thumb and index finger of each participant’s right hand. These markers were tracked at a sampling frequency of 100 Hz and processed by motion-capture software (Qualisys Track Manager, QTM; Qualisys, Sweden). The positions of the markers were streamed from the QTM to the Unity game engine (Unity v2019.4.16f1, Unity Technologies, USA) using the software development kit for Unity (Qualisys-Unity SDK; https://www.qualisys.com/software/integrations/unity/). This SDK enabled us to render a CG image onto a physical object in real time based on the spatial locations of the markers attached to the object.

Three cubes (6.0 cm × 6.0 cm × 6.0 cm) were used as physically small (PS) objects, and three cuboids (6.0 cm × 9.0 cm × 6.0 cm) were used as physically large (PL) objects. These objects were made of plastic and were adjusted so that the center of mass was located at the center of each object. Three cubes and cuboids were different in weight (160 g, 210 g, and 260 g). Objects of 210 g were used in the main trials, whereas the others were used in dummy trials. The objects used in the dummy trials were inserted randomly between the main trials to prevent participants from learning the weight of the objects in the main trials and giving the same heaviness rating for all trials. Participants used a handle (approximately 130 g) to grasp and lift each object. In practice trials, a wooden object (9.0 cm × 9.0 cm × 9.0 cm) was used.

A virtual object was overlaid on the physical object and tracked in real time according to the movements of the physical object. The virtual objects applied to the cubes and cuboids were referred to as virtually small (VS) and virtually large (VL), respectively. The size of the virtually small objects was 7.0 cm × 7.0 cm × 7.0 cm, whereas the size of the virtually large objects was 7.0 cm × 10.5 cm × 7.0 cm (Fig. 1c). These virtual size values were selected so that the virtual object was fully overlaid on the physical object with attached markers.

Task and protocol
A time series of rating the subjective heaviness of a lifted object in each trial is shown in Figure 2. Participants sat on a height-adjustable chair and leaned on the chest pad to keep a front-inclined posture in a comfortable position. Participants wore an HMD. The object to be lifted was placed on a platform so that participants would see it on the center of the HMD screen. Before the start of the trial, participants placed their right hand on the desk, while their vision was occluded by a black screen on the display. Each trial started with a verbal instruction and the removal of occlusion. Participants observed an object presented in front of them for three seconds without any movement. An auditory beep was then emitted for three seconds. While the auditory beep was sounding, participants moved their right hand to pinch the top of the handle of the object with the thumb and the index finger and lifted it vertically until the object was sufficiently above the platform (about 5-10 cm). To avoid obstructing the view of object by participant’s hand during reaching, participants were instructed to reach their right hand from the lateral side of the object. After the auditory beep ceased, participants replaced the object to its original position on the table, and their vision was occluded again. They then verbally rated the subjective heaviness of the lifted object using the absolute magnitude estimation; participants assigned a numerical scale with no upper or lower limits (Zwislocki & Goodman, 1980).

Participants lifted the objects under three conditions: a no-size-change condition, a size-change condition, and a no-vision condition. The no-size-change condition was set to investigate the effect of CG image size on the subjective heaviness of the object, i.e., the main purpose of this study (Fig. 3a). Under this condition, the size of virtual objects remained unchanged within a trial. Three objects were used: the small virtual object superimposed on the small object (PSVS), the large virtual object superimposed on the small object (PSVL), and the large virtual object superimposed on the large object (PLVL).
The size-change condition was set to investigate the effect of changes in the size of a CG image superimposed during movement on the subjective heaviness of the object, i.e., the secondary purpose of this study. Under this condition (Fig. 3b), the size of the virtual object was altered at one of the three predetermined timings (referred to as T1, T2, and T3). T1 was one second before the time the auditory beep was emitted. T2 was the time when the wrist moved vertically by 1 cm. T3 was the time the object was lifted 1 cm off the table. For each of T1, T2, andT3 conditions, the size of the virtual object changed either from small to large (StoL) or from large to small (LtoS).

The no-vision condition was introduced to ensure that the SWI did not occur without vision. This was necessary because several studies showed that haptic information produced at the moment of lifting the object could affect the subjective heaviness of the object (Ellis and Lederman, 1993; Buckingham, 2019). To ensure that such haptic information did not affect the present results, we asked participants to lift the physically small (PS) object and the physically large (PL) object without vision and checked to be sure no SWI was found.

At the introduction of each new trial condition, participants performed two practice trials to become acquainted with the procedure under that condition. To avoid any effect of the appearance of the object used in practice trials, we used a wooden object without CG images for the practice trials.
Participants performed a total of 77 main trials. Under the no-size-change condition, participants performed 15 main trials (one weight (210 g) * three objects (PSVS, PSVL, and PLVL) * five trials) and six dummy trials (two weights (160 and 260 g) * three objects (PSVS, PSVL, and PLVL)). Under the size-change condition, participants performed 30 main trials (one weight (210 g) * two objects (LtoS and StoL) * three timings (T1, T2, and T3) * five trials) and 12 dummy trials (two weights (160 and 260 g) * two objects (LtoS and StoL) * three timings (T1, T2, and T3)). Under the no-vision condition, participants performed 10 main trials (one weight (210 g) * two objects (RS and RL) * five trials) and four dummy trials (two weights (160 and 260 g) * two objects (RS and RL)). The trials under each condition were pseudo-randomized. The order of conditions to be performed was counterbalanced among participants.

Dependent variables and data analysis
An illusion magnitude was used as a dependent variable to quantify how much heavier a visually small object was perceived to be than a visually larger one (Arthur et al., 2020; Buckingham, 2019). Specifically, the illusion magnitude was calculated by subtracting the mean heaviness rating for the large object from that for the small object. A verbal rating of heaviness (i.e., a numeric scale), which was obtained from each trial, was converted into z scores using the following equation (1):

where  represents the heaviness rating for each trial, and  and  represent the mean value and the unbiased standard deviation, respectively, for all main trials of each block. We separately calculated z score values for three experimental blocks to avoid the order effects. Under the no-size-change condition (Fig. 3a), illusion magnitudes were calculated based on three points of view: the effects of both physical and virtual sizes, the effect of virtual size, and the effect of physical size. The illusion magnitudes produced by the effects of both physical and virtual sizes were calculated by subtracting the mean of the z scores for the PLVL object from that for the PSVS object for each participant. The illusion magnitude produced by the effect of virtual size was calculated by subtracting the mean of the z scores for the PSVL object from that for the PSVS object for each participant. The illusion magnitude produced by the effect of physical size was calculated by subtracting the mean of the z scores for the PLVL object from that for the PSVL object for each participant. Under the size-change condition (Fig. 3b), the illusion magnitudes with size-change differences were calculated. The illusion magnitude was calculated by subtracting the mean value of the z scores for the LtoS object from that for the StoL object at three timings for each participant. Under the no-vision condition (Fig. 3c), the illusion magnitude produced by the effects of physical size only on weight perception was calculated by subtracting the mean value of the z scores for the PL object from that for the PS object for each participant.

Statistical analyses
Statistical tests were conducted for each of three conditions separately. For the no-size-change condition, we performed a Wilcoxon signed-rank test to check whether the illusion magnitude for each effect (both physical and virtual sizes, only virtual size, and only physical size) would be higher or lower than zero. This was necessary to verify whether the SWI was produced with the virtual size but not with the physical size. The level of significance was set to 0.05/3 = 0.0167 to avoid alpha errors by applying multiple statistical tests. We conducted statistical tests in both cases that included outliers and excluded them. The reason is that outliers showed the extreme opposite SWI. Outliers were defined based on Tukey’s rules: values were regarded as outliers when they exceeded the upper or lower bounds by 1.5 times the interquartile range compared to the values of other participants. These procedures were the same as those in a previous study of material weight illusion in immersive VR (Naylor et al., 2022).

We used the data of participants whose illusion magnitudes from the effects of both virtual and physical sizes were greater than zero under the no-size-change condition (n = 15) to perform statistical tests of the size-change condition. By focusing on participants who perceived the smaller object as heavier than the larger object (SWI) when the object size was constant, we could test whether the size before the change or after the change mainly affected the subjective heaviness of the object for each participant. For the size-change condition, we performed a Wilcoxon signed-rank test to see whether the illusion magnitude would be higher or lower than zero for each timing (T1, T2, and T3). The level of significance was set to 0.0167. For the no-vision condition, we performed a Wilcoxon signed-rank test to see whether the illusion magnitude produced by the physical size only with haptics only would be higher or lower than zero. The level of significance was set to 0.05. All statistical procedures were performed using a customized Python program (ver. 3.9.12).

3. Results

Effect of CG image size on subjective heaviness
Regarding the effects of both physical and virtual sizes, the median illusion magnitude was not significantly greater than zero (p = .030). When one outlier was excluded from the statistical test, the median illusion magnitude was significantly greater than zero (p = .006), which indicated that the SWIs were induced by the effects of physical and virtual size. Regarding the effect of virtual size, the median illusion magnitude was not significantly greater than zero (p = .036). When three outliers were excluded from the statistical test, the median illusion magnitude was significantly greater than zero (  < .001), which indicated that the SWIs were induced even by the effect of virtual size alone. Regarding the effect of physical size, the median illusion magnitude was not significantly greater than zero (p = .406), indicating that the SWIs were not induced by the effect of physical size.

Effect of changes in CG image size during movement on subjective heaviness
Regarding the change in virtual size at T1, the median illusion magnitude was not significantly different from zero (p = .095). When two outliers under the T1 condition were excluded from these statistical tests, the median illusion magnitude under the T1 condition was significantly greater than zero (p < .001), which indicated that the represented SWI was induced based on the virtual size after the change in size at T1. Regarding the change in virtual size at T2 and T3, the median illusion magnitudes were not significantly higher or lower than zero (p = .804 and p = .315, respectively).

Ensuring no SWI without vision
A Wilcoxon signed-rank test showed the illusion magnitude produced by the effect of physical size with haptics only was not significantly greater than zero (p = .424).

4. Discussion

The main purpose of our study was to address whether the SWI would occur with AR objects. What we initially found was that, under the no-size-change condition, some participants demonstrated the opposite SWI. When these participants were excluded as outliers, we found that the remaining participants rated the virtually smaller object heavier than the virtually larger object. Additionally, the participants rated virtually identically sized objects the same regardless of the physical size of the objects. We also ensured that, under the no-vision condition, no SWI was found, suggesting that haptic information produced at the moment of lifting the object was unlikely to bias the perception of heaviness. These results suggest that the SWI occurred based on the size of the superimposed CG images regardless of the size of the physical object. The secondary purpose was to investigate the effect of size changes in superimposed CG images during a series of movements to lift an object. We found that subjective heaviness was not significantly affected when the size changes occurred after participants initiated movement to reach and lift an object.

The results under the no-size-change condition revealed that virtual objects fully overlaid on the physical objects affected subjective heaviness. This was in contrast to the findings of Rohrbach et al. (2021), in which virtual objects were created with semi-translucent CG images and showed that subjective heaviness was determined based on the size of physical objects rather than that of CG images. Based on the present findings, we propose that fully rendering CG images onto physical objects is necessary to maintain the size effect of virtual objects, possibly by avoiding conflict between the information about the virtual and physical sizes. Notably, some participants, who were categorized as outliers in the present study, showed the opposite SWI: they perceived virtually larger objects to be heavier than virtually smaller objects. A previous study also showed the phenomenon of the opposite SWI after participants repeatedly lifted large objects with low mass and small objects with high mass (Flanagan et al., 2008). These results suggest that prior experience with combinations of sizes and weights affects the SWI pattens. Considering their findings, the opposite SWI obtained from some participants may have been the result of their prior experiences of the relationship between sizes and weights.

With regard to the effect of size changes that occurred while reaching or just after the lifting of the object, SWI patterns were inconsistent among participants. Plaisier et al. showed that the SWI occurred in the case of an object that was observed only 200 ms after being lifted (Plaisier et al., 2019). This suggests that, at least in the physical environment, visual information about the size of an object affects its subjective heaviness even when participants see the object only just after it is lifted. One of the critical differences between Plaisier et al. and the present study is that, while the object size was stationary during observation in Plaisier et al., participants in the present study observed that the object size had changed. It is possible that in the latter case, participant’s expectations of the observed object’s weight were generated based on visual size before changes occurred, and the expectations were maintained because a series of movements to lift the object had already been planned based on that expectation. Further studies are necessary to address the validity of these explanations.

Other explanations are also possible as to why no significant impact of changing the size of an object on weight perception was found. Firstly, participants may have been difficult to pay attention to the changes in the size of object because what they actually reached and pinched was not the object but the upper handle. Secondly, the reaching distance may have been too short. Generally, the greater the difference between the time to plan the action based on the visual information and the time to obtain the sensory feedback as the result of lifting the object is, the more likely the illusion is to occur. So, relatively shorter reaching distance in the present study may have not provided sufficient time to produce SWI. Further studies are also necessary to address the validity of these explanations.

There were relatively large individual differences in the present study. Using physical objects of different weights may have resulted in experience-based fluctuation in subjective heaviness. Van Polanen and Devera showed that an object was perceived as lighter after lifting a relatively heavy object (van Polanen & Davare, 2015). Because objects with three different weights were randomly presented in the present study, individual differences in the subjective heaviness of the object may have been produced based on the weight of the object each participant lifted in the previous trial. To minimize that possible effect, it is necessary to test using single-weight objects.

In conclusion, we showed that presenting virtual objects in an AR environment altered human perception of the object’s weight. We expect that AR can be used to extend and enrich human weight perception in physical environments and to explore the effects of other visual characteristics, such as materials (Naylor et al., 2022) or physics (Schwettmann et al., 2019) on haptics.

Author Contributions
Conceptualization, R.W., J.I., K.Y., T.O., T.I., and T.H.; methodology, R.W., J.I., K.Y., T.O., T.I., and T.H.; investigation, R.W., J.I., and T.U.; formal analysis, R.W.; data curation, R.W., J.I., K.Y., T.U., T.O., T.I., and T.H.; writing-original draft preparation, R.W., J.I., and T.H.; writing-review and editing, R.W., J.I., T.O., T.I., and T.H.; supervision. T.O. and T.H.

Funding
This study was funded by Sony Group Corporation.

Institutional Review Board Statement
This study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Tokyo Metropolitan University (approval number: H3-131) and the Sony Bioethics Committee (approval number: 22-F-0015).

Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.

Data Availability Statement
The authors can make the data available upon request.

Conflicts of Interest
This study was conducted as joint research between Tokyo Metropolitan University and Sony Group Corporation. Takahiro Higuchi is a professor at Tokyo Metropolitan University and a principal investigator. Ryo Watanabe, Koki Yokoyama, and Takuma Umemori are students at Tokyo Metropolitan University and lab members of Takahiro Higuchi. Junki Inoue, Tsuyoshi Ishikawa, and Takanobu Omata are employees of Sony Group Corporation.

References
Arthur, T., Vine, S., Brosnan, M., & Buckingham, G. (2020). Predictive sensorimotor control in autism. Brain, 143(10). https://doi.org/10.1093/brain/awaa243

Buckingham, G. (2019). Examining the size–weight illusion with visuo-haptic conflict in immersive virtual reality. Quarterly Journal of Experimental Psychology, 72(9). https://doi.org/10.1177/1747021819835808

Flanagan, J. R., Bittner, J. P., & Johansson, R. S. (2008). Experience Can Change Distinct Size-Weight Priors Engaged in Lifting Objects and Judging their Weights. Current Biology, 18(22), 1742–1747. https://doi.org/10.1016/j.cub.2008.09.042

Naylor, C. E., Proulx, M. J., & Buckingham, G. (2022). Using Immersive Virtual Reality to Examine How Visual and Tactile Cues Drive the Material-Weight Illusion. Attention, Perception, and Psychophysics, 84(2), 509–518. https://doi.org/10.3758/s13414-021-02414-x

Plaisier, M. A., Kuling, I. A., Brenner, E., & Smeets, J. B. J. (2019). When Does One Decide How Heavy an Object Feels While Picking It Up? Psychological Science, 30(6), 822–829. https://doi.org/10.1177/0956797619837981

Schwettmann, S., Tenenbaum, J. B., & Kanwisher, N. (2019). Invariant representations of mass in the human brain. ELife, 8, e46619. https://doi.org/10.7554/eLife.46619

van Polanen, V., & Davare, M. (2015). Sensorimotor memory biases weight perception during object lifting. Frontiers in Human Neuroscience, 9(DEC). https://doi.org/10.3389/fnhum.2015.00700

Zwislocki, J. J., & Goodman, D. A. (1980). Absolute scaling of sensory magnitudes: A validation. Perception & Psychophysics, 28(1). https://doi.org/10.3758/BF03204312