Home > Research > Open-Source-Tools > Mobile Detection Task – MDT for Android

Mobile Detection Task – MDT for Android

Michael Krause, Antonia Conti

 

The Mobile Detection Task (MDT) is a derivative of the Detection Response Tasks (see e.g. http://www.lfe.mw.tum.de/en/arduino-drt) allowing mobile, wireless experimentation through the use of a nomadic device (Android smart phone). Since the MDT is not within the DRT ISO Standard (currently under development), this task is referred to a MDT rather than a DRT, despite similar task qualities.

The basic idea behind the MDT is to implement the DRT protocol on a smart phone and to use the hardware of the device (touch screen, display, vibration motor, WLAN, etc.) to asses the attentive effects of cognitive workload. The display (visual) and vibration motor (tactile) are used to present participants with signals, and the touch screen or an external button can be used to communicate participant responses. Data can be logged locally on the system or smart phone and can even be wirelessly transmitted (depending on the phones capabilities). These connections can also be used to control and monitor an experiment.

All these capabilities are possible with the use of a single device you already likely possess! In order to get started with this setup, you will need to download the software we provide here on the site listed below (provided free of charge, so fear not). This source is intentionally open source (GPL) so that you can adjust it to your project needs.

 

Important note: The absolute reaction time values are not as accurate as you may expect. Nevertheless, sometimes useful to compare relatively experimental conditions or different systems under test.

 

For the impatient reader:

The source code is open source and can be found at GitHub: https://github.com/InstituteOfErgonomics/MDT/

The compiled app can be downloaded at the Playstore, for free: https://play.google.com/store/apps/details?id=de.tum.mw.lfe.mdt

Note: Please report bugs via the GitHub repository or mail to krause@tum.de

 

If you connect wires and equipment to a subject, please do so with caution and take precautionary measures (e.g. beware of electrical safety, etc.).

If experimental setups include a driving task, special care must be taken. e.g. no cables or device parts should interfere with being able to safely perform the driving task. As a general piece of advice, please think about what can fail while preparing the experiment (something becomes loose or gets stuck to other parts,…) and take every measure possible to avoid such occurrences.

 

This documentation is not in the public domain, but the following is valid for it:

THE DOCUMENTATION IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE DOCUMENTATION OR THE USE OR OTHER DEALINGS IN
THE DOCUMENTATION.


 

Operation

The app can e.g. be downloaded, compiled and installed with the source code or send by email to a device. Data from the app are logged to a location which is specified by the smart phone as ‘external storage’. On some phones this is a SD card, on some others external storage is an internal storage anyway. The app logs to this storage in the folder ‘MDT’. In this folder subfolders are created with the convention ‘year-month-day’ e.g. 2014-01-09. In these folders data of single experiments are logged to files with the convention ‘unixtimestamp.txt’ e.g. ’1389223127477.txt’. The files are CSVs. Thus if you rename it to ‘.csv’, on some computer systems these are already associated to an appropiate program. Some phones have problems to do right file handling, if the phone is connected to a PC. If you can not see the files directly after an experiment on your PC, the files are maybe visible after a reconnection or reboot of the phone. Or maybe you get better results with other USB connection protocols like MTP or PTP.

With the long press alarm chechbox an alarm can be enabled, if a subject holds the touch screen longer than 1.5 seconds. The apprehension that led to this implementations was that a test subject unintentionaly can put a finger or hand on the touchscreen to rest. In an experiment with about 30 people and different experimental conditions, the alarm was activated only one time, so very seldom. For the alarm the normal phone rington is used.

With the tactile and visual checkboxes the tactile stimulus via the vibration motor and/or the visual stimulus via a red screen can be enabled.

With the start and stop button an experiment can be started. After an experiment some stats are available at the buttom of the screen (Average reaction time, hit rate and number of stimuli).

In the right lower corner an indicator shows a remote connection. The number to the rigth of this indicator signals that the experimenter has send a marker (in the figures marker ’7′).

With the external button checkbox the use of an external button can be enabled. It is an tempting idea to simply use the headset buttons which are normally used for speech recognition and music player controll, as a response button. Unfortunally a test with these gave unsatisfying result: An Arduino was programmed to get the onset of the vibration motor directly from the phone’s printed circuit board and issue a defined headset button press after a fixed amount of time. In the results these fixed times were affected by different nuisances. The related code is still in the source code, but commented out. If you want to fiddle about, don’t hesitate.

The implemented external button option use another idea: If enabled, a sound signal with 11025Hz is generated, the sound output is set to about 50% volume and the signal is played on and on. If you have no ear phones connected, you will hear a nerve-racking beep. Some smartphones use a 4 pole 3.5mm jack as connector, sometimes referred to as TRRS (tip, ring, ring, sleeve) and the external microphone signal on the sleeve. So it is possible to feed back the generated sound signal of the ear plugs via a closed button as a microphone signal. The schematic is in the following figure

trrs

TRRS connection (only tested on Samsung Galaxy Ace and Y DUOS)

 

and was tested originaly with a Samsung Galaxy Ace and a Samsung Galaxy Y Duos. Some people tested it also on phones/tablets of other manufacturers (e.g. HTC and Asus) and reported it works. For some Sony and LG devices tests were negative. Any attachment and soldering of things to your smartphone is at your own risk. The 1k resistor prevented the tested Samsung smartphones from the misinterpretation as a headset button action (like adjust volume level or start speech recognition). When the app detects the sound signal on the microphone channel it can be used as response-button closed signal. Initialy the threshold must be set.
Important note: Some smartphones have a DC voltage on the microphone channel that can generate a high audio signal when the reponse button is released. If the audio threshold is set wrongly (too high) the phones measures not the time when the button is pressed, else the time when the button is released. Therefore, make sure the green bar and the ‘external button close’-indicator act like in the pictures below, when you PRESS the reaction button.

 

 

 

device-2014-01-23-013520

MDT help/about screen shows some commands and IP address

device-2014-01-23-014157

Configuration/status screen before and between an experiment will be started. External button open (low signal, short green bar)

device-2014-01-23-014202

External button closed (high signal, long green bar). The threshold slider is set between these points. The “external button pressed”-indicator also helps.

device-2014-01-23-013936

Experiment currently running.

device-2014-01-23-013942

Experiment currently running. Visual stimulus.

DSCN1743b

Setup with external button

mdt_telnet2

The reponse is binary, so not realy human readable.

mdt_telnet

For test purposes, or if you just need to controll the experiment remotely, you can simply telnet into the app. (Hint: different telnet apps are also available for Android). At the bottom of this article is also a SILAB snippet, that demonstartes how to connect the MDT to the SILAB driving simulation.

 

 

How to apply the stimuli and reponse possibilities in your experiment is up to you. Maybe you mount the phone on the dashboard and the subject reacts via an external button. Or the phone is attached to the upper arm and vibrates. In an experiment we directly mounted it to the steering wheel. The phone issued visual or tactile stimuli and the subjects reacted via the touchscreen

 

Timing

There are several issues with the timing. In generell the scheduler on the android phones is not for for realtime purposes, nevertheless it is optimized for fast responiveness of apps, to get a good user experience. Another problem is the Java garbage collector (GC). Depending on which programs are running, the garbage collector comes up from time to time. The function of the GC requires that are all programms stopped for a (hopefully) short time, so the GC can do his work. The MDT app simply changes the background color of a GUI element as a visual stimulus. It doesn’t track when this changed color will be drawn to a framebuffer, nor when this framebuffer will be displayed. To make things not even worse, it can be advisable to kill any other user app. Thus the phone e.g. does not play music or download app updates in parallel to a MDT. So lower the CPU load. Maybe you can also remove the SIM card from the phone or disable GSM/UMTS connectivity. Thus, no phone call or SMS will disturb an experiment. If you consider the various nuisances as systematical and random measurement errors, there is one countermeasure to get good results: Increase the data basis. I.e., test more subjects, do more repeated measurements or test for a longer period of time.

The smartphone has the benefit to get an easy to handle and cheap experimental setup, but to the expense of accuracy. This can also be counterbalanced by using more smartphone setups with more test subjects in parallel to get a wider data basis. If an experiment needs accurate timing the Arduino DRT can be a low cost alternative (see http://www.lfe.mw.tum.de/en/arduino-drt ).

We tested a smartphone setup in comparision to an Arduino DRT setup: The smartphone was also able to detect the workload differences between the used experimental conditions, but on a different absolute level. So no absolute validity but relative validity.

The smartphone setup is not the right choice, if you need an accurate timing for every single response and if the absolute reaction times are of interest. Nevertheless, if you compare e.g. two systems (A and B), the smartphone setup can be an alternative.

To get a clue about what times we are talking about, a test circuit was set up: An Arduino Uno read in the onset voltage of the vibration stimulus (direct from the phone circuit board) or the visual stimulus via an photo diode on the display and issued an external button press via the phone jack after a defined delays of time (300ms). So the measured times and the deviation from 300 ms is what we are interested in. The smart phone was a Samsung S5830 Android 2.3.6. The first histogram shows the measurement of N=1651 reference reaction time (300ms). The mean is 289.5ms (SD= 7.9ms; MIN=199ms MAX=316ms). During the measurements the visual stimulus was also enabled. Time taking in code starts after preparing both tactile and visual stimuli (depending on enabled or not). The second histogram shows the same measurement for N=1895 visual stimuli (M= 357.4ms; SD=7.2ms; MIN=334; MAX=413ms).

Take away message: Don’t interpret single reaction time values if you use MDT. Use a broader data basis. If you need accurate timing look at the Arduino DRT. Do no analysis on the absolute measured reaction times, interpret your results relative: E.g. system A has a reaction time 20% greater system B.

image006

image002

 

Measured Vibration

We measured the vibration from a Samsung Galaxy Ace S5830 by softly attaching an piezo sensor several times at different locations of the housing. The results vary between 2m/s² and 15m/s² depending on location and direction.

 

File Logging Format

In Version 1 the following CSV logging format is used.

count;timestampStimulusOnset;diffToPlanedOnset;timestampNextOnset;rt;result;buttonDownCount;visual;tactile;longPressAlarmEnabled;longPressAlarmActivated;externalButton;marker

 

  • count up counting number of presented stimuli, start with 1
  • timestampStimulusOnset unix timestamp [ms] of stimulus onset
  • diffToPlanedOnset difference [ms] between the stimulus onset and the planned stimulus onset
  • timestampNextOnset unix timestamp [ms] of the next planned stimulus onset
  • rt reaction time [ms]
  • results the results can be ‘H’it, ‘C’heat and ‘M’iss (if ‘M’iss rt = ”)
  • buttonDownCount how often the test subject pressed the recation button and/or the touch screen in this experiment
  • visual if this stimulus was presented visual [=true] or visual was disabled [=false]
  • tactile if this stimulus was presented tactile[=true] or tactile was disabled [=false]
  • longPressAlarmEnabled if long press alarm was enabled [=true] or disabled [=false]
  • longPressAlarmActivated how often the long press alarm was activated in this experiment
  • externalButton if this reaction came from the externalt button [=true] or via the touch screen [=false]
  • holdTime how long [ms] the external button was hold by the user during the last interaction (previous reaction (n-1)). The hold time of the last reaction (before experiment stop) is dismissed. This value is typically between 50-200ms.
  • marker experimental notation (number) set by the examiner to mark a condition or event

Example of a recorded file “1389223127477.TXT”:

count;timestampStimulusOnset;diffToPlanedOnset;timestampNextOnset;rt;result;buttonDownCount;visual;tactile;longPressAlarmEnabled;longPressAlarmActivated;externalButton;holdTime;marker;V1.0.1
1;1389223131294;12;1389223134315;221;H;1;false;true;false;0;false;0;7;
2;1389223134326;11;1389223137737;238;H;2;false;true;false;0;false;0;7;
3;1389223137748;11;1389223141499;191;H;3;false;true;false;0;false;0;7;
4;1389223141509;10;1389223144615;195;H;4;false;true;false;0;false;0;7;
5;1389223144625;10;1389223148097;248;H;5;false;true;false;0;false;0;7;
6;1389223148104;7;1389223152274;303;H;6;false;true;false;0;false;0;7;
7;1389223152284;10;1389223157025;222;H;7;false;true;false;0;false;0;7;
8;1389223157035;10;1389223161401;288;H;8;false;true;false;0;false;0;7;
9;1389223161415;14;1389223164783;202;H;9;false;true;false;0;false;0;7;
10;1389223164798;15;1389223169049;;M;9;false;true;false;0;false;0;7;
11;1389223169067;18;1389223173280;;M;9;false;true;false;0;false;0;7;
12;1389223173308;28;1389223176468;251;H;10;false;true;false;0;false;0;7;
13;1389223176474;6;1389223179542;20;C;12;false;true;false;0;false;0;7;
14;1389223179550;8;1389223183621;28;C;37;false;true;false;0;false;0;7;
15;1389223183629;8;1389223187713;124;H;65;false;true;false;0;false;0;7;
16;1389223187722;9;1389223191595;221;H;69;false;true;false;0;false;0;7;

or formated as table:

count timestamp-StimulusOnset diffTo-PlanedOnset timestamp-NextOnset rt result buttonDown Count visual tactile longPress-AlarmEnabled longPress-AlarmActivated externalButton holdTime marker V1.0.1
1 1389223131294 12 1389223134315 221 H 1 false true false 0 false 0 7
2 1389223134326 11 1389223137737 238 H 2 false true false 0 false 0 7
3 1389223137748 11 1389223141499 191 H 3 false true false 0 false 0 7
4 1389223141509 10 1389223144615 195 H 4 false true false 0 false 0 7
5 1389223144625 10 1389223148097 248 H 5 false true false 0 false 0 7
6 1389223148104 7 1389223152274 303 H 6 false true false 0 false 0 7
7 1389223152284 10 1389223157025 222 H 7 false true false 0 false 0 7
8 1389223157035 10 1389223161401 288 H 8 false true false 0 false 0 7
9 1389223161415 14 1389223164783 202 H 9 false true false 0 false 0 7
10 1389223164798 15 1389223169049 M 9 false true false 0 false 0 7
11 1389223169067 18 1389223173280 M 9 false true false 0 false 0 7
12 1389223173308 28 1389223176468 251 H 10 false true false 0 false 0 7
13 1389223176474 6 1389223179542 20 C 12 false true false 0 false 0 7
14 1389223179550 8 1389223183621 28 C 37 false true false 0 false 0 7
15 1389223183629 8 1389223187713 124 H 65 false true false 0 false 0 7
16 1389223187722 9 1389223191595 221 H 69 false true false 0 false 0 7

The results shows that 16 stimuli were presented. The test subject achieved 12 hits, 2 misses and 2 cheats. All stimuli were presented tactile. The experimenter set the marker ’7′. A closer look at the buutonDownCount column reveals that around line 13 to 15 the test subject tried to cheat and pressed very often and very fast.

Transmission Protocol

The app opens a port (by default port 7007, TCP) and can be reached e.g. via telnet or TCP sockets. The remote control of the app is implemented in Version 1 via a one byte protocol the transmission in the other directions has some more bytes.

For the one byte control simple (ASCII) codes i.e. each one byte are send to the app:

  • send ‘#‘ or code 32 (Space Bar) to start an experiment
  • send ‘$‘ or code 27 (ESC code) to stop an experiment
  • send ‘T‘ to enable the ‘T’actile vibrator for stimulus presentation
  • send ‘M‘ to ‘M’ute (disable) the tactile vibrator for stimulus presentation
  • send ‘V‘ to enable the ‘V’isual stimulus presentation
  • send ‘H‘ to switch disable the visual stimulus presentation ‘H’ide
  • send ‘L‘ to enable the ‘L’ong press alarm
  • send ‘S‘ to disable the long press alarm (‘S’ilent)
  • send ‘E‘ to enable the ‘E’xternal button
  • send ‘I‘ to disable the extrnal button and use the ‘I’nternal touch screen
  • send ‘0’, ‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘8’, ‘9‘ to set experimental triggers also called marker

 

The app send the following packets (little endian):

//Version1 packets
-------------------------------------
//int  count (4 byte)
//long onset (8 byte)
//int onsetDiff (4 byte)
//long nextOnset (8 byte)
//int long rt (4 byte)
//byte result (1 byte)
//int buttonDownCount (4 byte)
//byte bool visual (1 byte)
//byte bool tactile (1 byte)
//byte bool longPressAlarmEnabled (1 byte)
//int longPressAlarmCount (4 byte)
//byte bool externalButton (1 byte)
//byte marker (1 byte)
//int queuedPackets (4 byte)
  • count is a 4 byte int, counting the stimuli
  • onset is a long int holding the unix timestamp of stimulus onset
  • onsetDiff is a 4 byte int, representing the difference between onset and planned onset. Thus the planned stimulus onset in the packet of the last stimulus (n-1) plus the onsetDiff of the current stimulus = the current onset
  • nextOnset is a long int holding the unix timestamp when the next stimulus onset is planned to be presented
  • rt is a 4 byte int for the reaction time in milliseconds of the test subject
  • result is 1 byte that can have in the current version five values:
    • public static final byte B_CHEAT = 0; // enum byte for cheat
    • public static final byte B_HIT = 1; // enum for hit
    • public static final byte B_MISS = 2; // enum for miss
    • public static final byte B_START = 3; // enum for start of experfiment
    • public static final byte B_STOP = 4; // enum for stop of experiment
  • buttonDownCount  is a 4 byte int that counts how often the test subject pressed the response button and/or touch screen
  • visual is 1 byte (bool) that represents if the stimulus was presented visual (=1) or visual was disabled (=0)
  • tactile is 1 byte (bool) that represents if the stimulus was presented tactile(=1) or tactile was disabled (=0)
  • longPressAlarmEnabled is 1 byte (bool) that represents if the long press alarm was enabled (=1) or disabled (=0)
  • longPressAlarmCount is a 4 byte int that count how often the long press alarm was activated in this experiment
  • externalButton is 1 byte (bool) that represents if the subject reacted via an external button (=1) or via the touch screen (=0)
  • marker is 1 byte that can be set by the experimenter to mark experimental conditions
  • queuedPackets is a 4 byte int that holds how many MDT packets were in the send queque before this packet was added to the queue. If you get a packet and this count is >0 you know, the packet was not directly delivered.

 

 

#SILAB snippet to connect MDT smartphone to SILAB driving simulation and control via the operator keyboard

DPUSocket mySmartphoneMDT
{
    Computer = { OPERATOR };
    Index = 100;
    #BigEndian [true/false]
    BigEndian = false;
    #Socket_IsTCPIP TCP/IP [=true] or UDP [=false]
    Socket_IsTCPIP = true;
    Socket_IP = "192.168.2.165";
    Socket_IsServer = false;
    Socket_PortSend = 7007;
    Socket_PortReceive = 7000;
    #SendDefinition packet definition
    SendDefinition = (
        (remoteControl,ubyte)
    );
    #ReceiveDefinition packet definition
    ReceiveDefinition = (
        (count, ulong), #count, counts up on every stimulus
        (onsetL, ulong), #onset milliseconc unix timestamp, lower 4 bytes
        (onsetH, ulong), #onset milliseconc unix timestamp, upper 4 bytes
        (onsetDiff, long), #onsetDifference milliseconds: diff = (planedNextOnset - whatReallyHappened)
        (nextOnsetL, ulong), #next onset milliseconc unix timestamp, lower 4 bytes
        (nextOnsetH, ulong), #next onset milliseconc unix timestamp, upper 4 bytes
        (rt, ulong), #reaction time milliseconds
        (result, byte), #0=cheat 1=hit 2=miss 3=expStart 4=expStop
        (buttonDownCount, ulong),#how often screen was touched
        (visual, ubyte), #uses visual mode
        (tactile, ubyte), #uses tactile mode
        (longPressAlarmEnabled, ubyte), #is long press alarm enabled
        (mLongPressAlarmCount, ulong), #how often long press alarm was activated
        (externalButton, ubyte), #reaction comes from external button
        (marker, ubyte), #which marker is currently set on smartphone
        (queuedPackets, ulong) #queued packets on the smartphone to send
    );

};
Connections =
{
    myKeyboard.Key mySmartphoneMDT.remoteControl #keyboard can start/stop and controll the experiment
    # myHedgehogKiller.Out2 mySmartphoneMDT.remoteControl #hedgehog can also start/stop and controll the experiment
};
#--------------------------------------------------------------
DPUKeyboard myKeyboard
{
    Computer = { OPERATOR };
    Index = 100;
};
#-----------------------------------------------------------

 

References

…coming

If you used the project in an experiment, we would be thankful for your feedback.

Also if you altered the source code, or even made a new project out of it. Maybe we can set a link.