Home > Open-Source-Tools > Mobile Detection Task – MDT for Android

Mobile Detection Task – MDT for Android

Michael Krause, Antonia S. Conti, Huihui Duan, Moritz Späth

 

https://github.com/InstituteOfErgonomics/MDT


  Source code is available in our GitHub repository: https://github.com/InstituteOfErgonomics/MDT

https://play.google.com/store/apps/details?id=de.tum.mw.lfe.mdt

  A compiled Android app can be installed via the market: https://play.google.com/store/apps/details?id=de.tum.mw.lfe.mdt
 

mdt_a DSCN1743b

 

The Mobile Detection Task (MDT) is a derivative of the Detection Response Tasks (see e.g. http://www.lfe.mw.tum.de/en/arduino-drt) allowing mobile, wireless experimentation through the use of a nomadic device (Android smart phone). Since the MDT is not within the DRT ISO Standard (currently under development), this task is referred to a MDT rather than a DRT, despite similar task qualities. The DRT is being standardized under ISO (ISO/CD 17488) and is used to measure the attentive effects of cognitive workload induced by the performance of a task or tasks of interest (see Conti, Dlugosch, & Bengler, 2014).

The basic idea behind the MDT is to implement the DRT protocol on a smart phone and to use the hardware of the device (touch screen, display, vibration motor, WLAN, etc.) to asses the attentive effects of cognitive workload. The display (visual) and vibration motor (tactile) are used to present participants with signals, and the touch screen or an external button can be used to communicate participant responses. Data can be logged locally on the system or smart phone and can even be wirelessly transmitted (depending on the phones capabilities). These connections can also be used to control and monitor an experiment.

All these capabilities are possible with the use of a single device you already likely possess! In order to get started with this setup, you will need to download the software we provide here on the site listed below (provided free of charge, so fear not). This source is intentionally open source (GPL) so that you can adjust it to your project needs.

 

If you connect wires and equipment to a subject, please do so with caution and take precautionary measures (e.g. beware of electrical safety, etc.).

If experimental setups include a driving task, special care must be taken. e.g. no cables or device parts should interfere with being able to safely perform the driving task. As a general piece of advice, please think about what can fail while preparing the experiment (something becomes loose or gets stuck to other parts,…) and take every measure possible to avoid such occurrences.

 

This documentation is not in the public domain, but the following is valid for it:

THE DOCUMENTATION IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE DOCUMENTATION OR THE USE OR OTHER DEALINGS IN
THE DOCUMENTATION.


Change history / Notes:
2015/June/22 Note: For some devices (e.g. Samsung S4) it is more appropiate to change the 1k resistor for the TRRS adaptor by a capacitor (e.g., 100nF).


 

Operation

The app can e.g. be downloaded, compiled and installed with the source code. Data from the app are logged to a location which is specified by the smart phone as ‘external storage’. On some phones this is a SD card, on some others external storage is an internal storage anyway. The app logs to this storage in the folder ‘MDT’. The files are CSVs. Some phones have problems to do right file handling, if the phone is connected to a PC. If you can not see the files directly after an experiment on your PC, the files are maybe visible after a reconnection or reboot of the phone. Or maybe you get better results with other USB connection protocols like MTP or PTP.

With the long press alarm chechbox an alarm can be enabled, if a subject holds the touch screen longer than 1.5 seconds. The apprehension that led to this implementations was that a test subject unintentionaly can put a finger or hand on the touchscreen to rest. In an experiment with about 30 people and different experimental conditions, the alarm was activated only one time, so very seldom. For the alarm the normal phone rington is used.

With the tactile and visual checkboxes the tactile stimulus via the vibration motor and/or the visual stimulus via a red screen can be enabled.

With the start and stop button an experiment can be started. After an experiment some stats are available at the buttom of the screen (Average reaction time, hit rate and number of stimuli).

In the right lower corner an indicator shows a remote connection. The number to the rigth of this indicator signals that the experimenter has send a marker (in the figures marker ’7′).

With the external button checkbox the use of an external button can be enabled. It is an tempting idea to simply use the headset buttons which are normally used for speech recognition and music player controll, as a response button. Unfortunally a test with these gave unsatisfying result: An Arduino was programmed to get the onset of the vibration motor directly from the phone’s printed circuit board and issue a defined headset button press after a fixed amount of time. In the results these fixed times were affected by different nuisances. The related code is still in the source code, but commented out. If you want to fiddle about, don’t hesitate.

The implemented external button option use another idea: If enabled, a sound signal with 11025Hz is generated, the sound output is set to about 50% volume and the signal is played on and on. If you have no ear phones connected, you will hear a nerve-racking beep. Some smartphones use a 4 pole 3.5mm jack as connector, sometimes referred to as TRRS (tip, ring, ring, sleeve) and the external microphone signal on the sleeve. So it is possible to feed back the generated sound signal of the ear plugs via a closed button as a microphone signal. The schematic is in the following figure

trrs

TRRS connection (only tested on Samsung Galaxy Ace and Y DUOS)

 

and was tested originaly with a Samsung Galaxy Ace and a Samsung Galaxy Y Duos. Some people tested it also on phones/tablets of other manufacturers (e.g. HTC and Asus) and reported it works. For some Sony and LG devices tests were negative. Any attachment and soldering of things to your smartphone is at your own risk. The 1k resistor prevented the tested Samsung smartphones from the misinterpretation as a headset button action (like adjust volume level or start speech recognition). When the app detects the sound signal on the microphone channel it can be used as response-button closed signal. Initialy the threshold must be set. For some deviecs we get better results with a capacitor (e.g., 100nF) instead of the 1k resistor.
Important note: Some smartphones have a DC voltage on the microphone channel that can generate a high audio signal when the reponse button is released. If the audio threshold is set wrongly (too high) the phones measures not the time when the button is pressed, else the time when the button is released. Therefore, make sure the green bar and the ‘external button close’-indicator act like in the pictures below, when you PRESS the reaction button.

 

 

 

device-2014-01-23-013520

MDT help/about screen shows some commands and IP address

device-2014-01-23-014157

Configuration/status screen before and between an experiment will be started. External button open (low signal, short green bar)

device-2014-01-23-014202

External button closed (high signal, long green bar). The threshold slider is set between these points. The “external button pressed”-indicator also helps.

device-2014-01-23-013936

Experiment currently running.

device-2014-01-23-013942

Experiment currently running. Visual stimulus.

DSCN1743b

Setup with external button

mdt_telnet2

The reponse is binary, so not realy human readable.

mdt_telnet

For test purposes, or if you just need to controll the experiment remotely, you can simply telnet into the app. (Hint: different telnet apps are also available for Android). At the bottom of this article is also a SILAB snippet, that demonstartes how to connect the MDT to the SILAB driving simulation.

 

Timing

There are several issues with the timing. In generell the scheduler on the android phones is not for for realtime purposes, nevertheless it is optimized for fast responiveness of apps, to get a good user experience. Another problem is the Java garbage collector (GC). Depending on which programs are running, the garbage collector comes up from time to time. The function of the GC requires that are all programms stopped for a (hopefully) short time, so the GC can do his work. The MDT app simply changes the background color of a GUI element as a visual stimulus. It doesn’t track when this changed color will be drawn to a framebuffer, nor when this framebuffer will be displayed. To make things not even worse, it can be advisable to kill any other user app. Thus the phone e.g. does not play music or download app updates in parallel to a MDT. So, lower the CPU load. Maybe you can also remove the SIM card from the phone or disable GSM/UMTS connectivity. Thus, no phone call or SMS will disturb an experiment. If you consider the various nuisances as systematical and random measurement errors, there is one countermeasure to get good results: Increase the data basis. The smartphone has the benefit to get an easy to handle and cheap experimental setup, but to the expense of timing accuracy. If an experiment needs accurate timing the Arduino DRT can be a low cost alternative (see http://www.lfe.mw.tum.de/en/arduino-drt ).

To get a clue about what times we are talking about, a test circuit was set up: An Arduino Uno read in the onset voltage of the vibration stimulus (direct from the phone circuit board) or the visual stimulus via an photo diode on the display and issued an external button press via the phone jack after a defined delays of time (300ms). So the measured times and the deviation from 300 ms is what we are interested in. The smart phone was a Samsung S5830 Android 2.3.6. The first histogram shows the measurement of N=1651 reference reaction time (300ms). The mean is 289.5ms (SD= 7.9ms; MIN=199ms MAX=316ms). During the measurements the visual stimulus was also enabled. Time taking in code starts after preparing both tactile and visual stimuli (depending on enabled or not). The second histogram shows the same measurement for N=1895 visual stimuli (M= 357.4ms; SD=7.2ms; MIN=334; MAX=413ms).

Take away message: Don’t interpret single reaction time values if you use MDT. Use a broader data basis. If you need accurate timing look at the Arduino DRT. Do no analysis on the absolute measured reaction times, interpret your results relative: E.g. system A has a reaction time 20% greater system B.

image006

image002

 

Evaluation Experiment

Intro

There were 2 basic aims of this experiment: 1) to determine whether the n-back task (MIT, AgeLab), a cognitively loading task, is more or less loading when performed as a system-paced or user-paced task; 2) to evaluate the whether the differences in technology that exist between the MDT and DRT lead to different results. This section discusses the latter of the experimental aims.

Participants and Procedure

Twenty participants were tested in a between-subjects design experiment. Participants performed a baseline (operating only the MDT) the n-back in 3 levels (MDT+0-back, MDT+1-back, MDT+2-back). As well as a combination with a Continuous Tracking Task (CTT with joystick): MDT+ CTT, MDT+CTT+0-back, MDT+CTT+1-back, MDT+CTT+2-back. Half of the participants performed the n-back tasks as system-paced and the other half as user-paced. Two tailed t-tests for the relevant conditions showed no significant difference between the user-paced and system-paced means. Therefore, as we deal with the differences between the MDT and DRT measurements, the measures of both n-back groups are combined for this comparison.
For each experimental condition, one minute of MDT was recorded by the smartphone (Galaxy Ace S5830) and, in parallel, a microcontroller stopped when the smartphone’s vibration motor started and the button press of the participant. The average difference of about 8ms between microcontroller and smartphone (see table) is quite precise the difference measured technically, without a subject in the loop.

mdt_a

Results

In the table the experimental conditions are ranked by increasing mean reaction time (ms) and the effect size (Cohens d) between adjacent experimental conditions is calculated, based on the microcontroller values and based on the smartphone values. The difference in effect size between the values measured with smartphone and microcontroller is negligible. Thus, one minute MDT (~15 stimuli; 20 subjects) is more than enough, to compensate the lack of real-time capability of the smartphone.
The conditions baseline (MDT), MDT+0-back and MDT+2-back from this experiment (Experiment A; N=20) are also compared to the outcome of another experiment (Experiment B; N=35) with classical DRT equipment and vibrating motor on the shoulder. In this comparison the MDT would even provide a higher effect size.

mdtDataTabelle

In Krause et al. (2014) we used the correlation between experimental conditions as a quality measure for the hardware and experimental setup. For Krause et al. (2014) the MDT was mounted to the steering wheel and the subject responded via the touchscreen. The correlations for this setup were between 0.2-0.3 and therefore the setup was denoted a low quality setup. The following tables show that mounting the MDT on the arm reveals correlation of about 0.5-0.8. This is comparable to correlations we found with a DRT setup in real traffic (Krause et al., 2015).

mdtCorrelation1
mdtCorrelation2

Conclusion

If carefully applied (appropiate subject size, task duration and mounting), the MDT can be a low cost (free), alternative, non-standardized, derrivative DRT setup. It is especially valuable to illustrate the idea of DRT procedures, e.g., when teaching courses about driver distraction. The audience can attend with the app on their own phones.

File Logging Format

In Version 1 the following CSV logging format is used.

count;timestampStimulusOnset;diffToPlanedOnset;timestampNextOnset;rt;result;buttonDownCount;visual;tactile;longPressAlarmEnabled;longPressAlarmActivated;externalButton;marker

 

  • count up counting number of presented stimuli, start with 1
  • timestampStimulusOnset unix timestamp [ms] of stimulus onset
  • diffToPlanedOnset difference [ms] between the stimulus onset and the planned stimulus onset
  • timestampNextOnset unix timestamp [ms] of the next planned stimulus onset
  • rt reaction time [ms]
  • results the results can be ‘H’it, ‘C’heat and ‘M’iss (if ‘M’iss rt = ”)
  • buttonDownCount how often the test subject pressed the recation button and/or the touch screen in this experiment
  • visual if this stimulus was presented visual [=true] or visual was disabled [=false]
  • tactile if this stimulus was presented tactile[=true] or tactile was disabled [=false]
  • longPressAlarmEnabled if long press alarm was enabled [=true] or disabled [=false]
  • longPressAlarmActivated how often the long press alarm was activated in this experiment
  • externalButton if this reaction came from the externalt button [=true] or via the touch screen [=false]
  • holdTime how long [ms] the external button was hold by the user during the last interaction (previous reaction (n-1)). The hold time of the last reaction (before experiment stop) is dismissed. This value is typically between 50-200ms.
  • marker experimental notation (number) set by the examiner to mark a condition or event

Example of a recorded file “1389223127477.TXT”:

count;timestampStimulusOnset;diffToPlanedOnset;timestampNextOnset;rt;result;buttonDownCount;visual;tactile;longPressAlarmEnabled;longPressAlarmActivated;externalButton;holdTime;marker;V1.0.1
1;1389223131294;12;1389223134315;221;H;1;false;true;false;0;false;0;7;
2;1389223134326;11;1389223137737;238;H;2;false;true;false;0;false;0;7;
3;1389223137748;11;1389223141499;191;H;3;false;true;false;0;false;0;7;
4;1389223141509;10;1389223144615;195;H;4;false;true;false;0;false;0;7;
5;1389223144625;10;1389223148097;248;H;5;false;true;false;0;false;0;7;
6;1389223148104;7;1389223152274;303;H;6;false;true;false;0;false;0;7;
7;1389223152284;10;1389223157025;222;H;7;false;true;false;0;false;0;7;
8;1389223157035;10;1389223161401;288;H;8;false;true;false;0;false;0;7;
9;1389223161415;14;1389223164783;202;H;9;false;true;false;0;false;0;7;
10;1389223164798;15;1389223169049;;M;9;false;true;false;0;false;0;7;
11;1389223169067;18;1389223173280;;M;9;false;true;false;0;false;0;7;
12;1389223173308;28;1389223176468;251;H;10;false;true;false;0;false;0;7;
13;1389223176474;6;1389223179542;20;C;12;false;true;false;0;false;0;7;
14;1389223179550;8;1389223183621;28;C;37;false;true;false;0;false;0;7;
15;1389223183629;8;1389223187713;124;H;65;false;true;false;0;false;0;7;
16;1389223187722;9;1389223191595;221;H;69;false;true;false;0;false;0;7;

or formated as table:

count timestamp-StimulusOnset diffTo-PlanedOnset timestamp-NextOnset rt result buttonDown Count visual tactile longPress-AlarmEnabled longPress-AlarmActivated externalButton holdTime marker V1.0.1
1 1389223131294 12 1389223134315 221 H 1 false true false 0 false 0 7
2 1389223134326 11 1389223137737 238 H 2 false true false 0 false 0 7
3 1389223137748 11 1389223141499 191 H 3 false true false 0 false 0 7
4 1389223141509 10 1389223144615 195 H 4 false true false 0 false 0 7
5 1389223144625 10 1389223148097 248 H 5 false true false 0 false 0 7
6 1389223148104 7 1389223152274 303 H 6 false true false 0 false 0 7
7 1389223152284 10 1389223157025 222 H 7 false true false 0 false 0 7
8 1389223157035 10 1389223161401 288 H 8 false true false 0 false 0 7
9 1389223161415 14 1389223164783 202 H 9 false true false 0 false 0 7
10 1389223164798 15 1389223169049 M 9 false true false 0 false 0 7
11 1389223169067 18 1389223173280 M 9 false true false 0 false 0 7
12 1389223173308 28 1389223176468 251 H 10 false true false 0 false 0 7
13 1389223176474 6 1389223179542 20 C 12 false true false 0 false 0 7
14 1389223179550 8 1389223183621 28 C 37 false true false 0 false 0 7
15 1389223183629 8 1389223187713 124 H 65 false true false 0 false 0 7
16 1389223187722 9 1389223191595 221 H 69 false true false 0 false 0 7

The results shows that 16 stimuli were presented. The test subject achieved 12 hits, 2 misses and 2 cheats. All stimuli were presented tactile. The experimenter set the marker ’7′. A closer look at the buutonDownCount column reveals that around line 13 to 15 the test subject tried to cheat and pressed very often and very fast.

Transmission Protocol

The app opens a port (by default port 7007, TCP) and can be reached e.g. via telnet or TCP sockets. The remote control of the app is implemented in Version 1 via a one byte protocol the transmission in the other directions has some more bytes.

For the one byte control simple (ASCII) codes i.e. each one byte are send to the app:

  • send ‘#‘ or code 32 (Space Bar) to start an experiment
  • send ‘$‘ or code 27 (ESC code) to stop an experiment
  • send ‘T‘ to enable the ‘T’actile vibrator for stimulus presentation
  • send ‘M‘ to ‘M’ute (disable) the tactile vibrator for stimulus presentation
  • send ‘V‘ to enable the ‘V’isual stimulus presentation
  • send ‘H‘ to switch disable the visual stimulus presentation ‘H’ide
  • send ‘L‘ to enable the ‘L’ong press alarm
  • send ‘S‘ to disable the long press alarm (‘S’ilent)
  • send ‘E‘ to enable the ‘E’xternal button
  • send ‘I‘ to disable the extrnal button and use the ‘I’nternal touch screen
  • send ‘0’, ‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘8’, ‘9‘ to set experimental triggers also called marker

 

The app send the following packets (little endian):

//Version1 packets
-------------------------------------
//int count (4 byte)
//long onset (8 byte)
//int onsetDiff (4 byte)
//long nextOnset (8 byte)
//int long rt (4 byte)
//byte result (1 byte)
//int buttonDownCount (4 byte)
//byte bool visual (1 byte)
//byte bool tactile (1 byte)
//byte bool longPressAlarmEnabled (1 byte)
//int longPressAlarmCount (4 byte)
//byte bool externalButton (1 byte)
//byte marker (1 byte)
//int queuedPackets (4 byte)
  • count is a 4 byte int, counting the stimuli
  • onset is a long int holding the unix timestamp of stimulus onset
  • onsetDiff is a 4 byte int, representing the difference between onset and planned onset. Thus the planned stimulus onset in the packet of the last stimulus (n-1) plus the onsetDiff of the current stimulus = the current onset
  • nextOnset is a long int holding the unix timestamp when the next stimulus onset is planned to be presented
  • rt is a 4 byte int for the reaction time in milliseconds of the test subject
  • result is 1 byte that can have in the current version five values:
    • public static final byte B_CHEAT = 0; // enum byte for cheat
    • public static final byte B_HIT = 1; // enum for hit
    • public static final byte B_MISS = 2; // enum for miss
    • public static final byte B_START = 3; // enum for start of experfiment
    • public static final byte B_STOP = 4; // enum for stop of experiment
  • buttonDownCount is a 4 byte int that counts how often the test subject pressed the response button and/or touch screen
  • visual is 1 byte (bool) that represents if the stimulus was presented visual (=1) or visual was disabled (=0)
  • tactile is 1 byte (bool) that represents if the stimulus was presented tactile(=1) or tactile was disabled (=0)
  • longPressAlarmEnabled is 1 byte (bool) that represents if the long press alarm was enabled (=1) or disabled (=0)
  • longPressAlarmCount is a 4 byte int that count how often the long press alarm was activated in this experiment
  • externalButton is 1 byte (bool) that represents if the subject reacted via an external button (=1) or via the touch screen (=0)
  • marker is 1 byte that can be set by the experimenter to mark experimental conditions
  • queuedPackets is a 4 byte int that holds how many MDT packets were in the send queque before this packet was added to the queue. If you get a packet and this count is >0 you know, the packet was not directly delivered.

 

 

#SILAB snippet to connect MDT smartphone to SILAB driving simulation and control via the operator keyboard

DPUSocket mySmartphoneMDT
{
    Computer = { OPERATOR };
    Index = 100;
    #BigEndian [true/false]
    BigEndian = false;
    #Socket_IsTCPIP TCP/IP [=true] or UDP [=false]
    Socket_IsTCPIP = true;
    Socket_IP = "192.168.2.165";
    Socket_IsServer = false;
    Socket_PortSend = 7007;
    Socket_PortReceive = 7000;
    #SendDefinition packet definition
    SendDefinition = (
        (remoteControl,ubyte)
    );
    #ReceiveDefinition packet definition
    ReceiveDefinition = (
        (count, ulong), #count, counts up on every stimulus
        (onsetL, ulong), #onset milliseconc unix timestamp, lower 4 bytes
        (onsetH, ulong), #onset milliseconc unix timestamp, upper 4 bytes
        (onsetDiff, long), #onsetDifference milliseconds: diff = (planedNextOnset - whatReallyHappened)
        (nextOnsetL, ulong), #next onset milliseconc unix timestamp, lower 4 bytes
        (nextOnsetH, ulong), #next onset milliseconc unix timestamp, upper 4 bytes
        (rt, ulong), #reaction time milliseconds
        (result, byte), #0=cheat 1=hit 2=miss 3=expStart 4=expStop
        (buttonDownCount, ulong),#how often screen was touched
        (visual, ubyte), #uses visual mode
        (tactile, ubyte), #uses tactile mode
        (longPressAlarmEnabled, ubyte), #is long press alarm enabled
        (mLongPressAlarmCount, ulong), #how often long press alarm was activated
        (externalButton, ubyte), #reaction comes from external button
        (marker, ubyte), #which marker is currently set on smartphone
        (queuedPackets, ulong) #queued packets on the smartphone to send
    );

};
Connections =
{
    myKeyboard.Key mySmartphoneMDT.remoteControl #keyboard can start/stop and controll the experiment
    # myHedgehogKiller.Out2 mySmartphoneMDT.remoteControl #hedgehog can also start/stop and controll the experiment
};
#--------------------------------------------------------------
DPUKeyboard myKeyboard
{
    Computer = { OPERATOR };
    Index = 100;
};
#-----------------------------------------------------------

 

References

  • Conti A.S., Dlugosch, C., & Bengler, K. (2014). The Effect of Task Set Instruction on Detection Response Task Performance. In: De Waard, D., Brookhuis, K., Wiczorek, R., Di Nocera, F., Barham, P., Weikert, C., Kluge, A., Gerbino, W., and Toffetti, A., (Eds.), Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2013 Annual Conference (pp. 107-117). Available as open source online download, ISSN 2333-4959: http://www.hfes-europe.org/wp-content/uploads/2014/06/Conti.pdf
  • Krause, M., Conti, A., Späth, M., & Bengler, K. (2014). Testing Open-Source Implementations for Detection Response Tasks. In C. C. Gonzalez, C. Ordonez, & H. Fardoun (Eds.), Proceedings of the XV International Conference on Human Computer Interaction http://dx.doi.org/10.1145/2662253.2662313
  • Krause, M. , Knott, V., Bengler, K. (2015). Implementing the Tactile Detection Task in a Real Road Experiment to Assess a Traffic Light Assistant. ACHI 2015

If you used the project in an experiment, we would be thankful for your feedback.

Also if you altered the source code, or even made a new project out of it. Maybe we can set a link.