Data Sources

[mojtaba]

Data sources enable passive data collection from participants’ phones, wearables, and other devices. They collect information such as movement, location, physiological measurements, device usage, and environmental conditions without requiring participants to actively enter data during collection.

Avicenna supports several types of data sources, each with its own requirements, supported platforms, and collection method. This guide provides an overview of the available data sources, explains how to add and configure them in a study, and describes how data is collected, uploaded, and made available for analysis.

For details about a particular data source, including its available fields and platform-specific behavior, see its individual documentation linked below.

Available Data Sources

The following table summarizes the data sources available in Avicenna, what they measure, the platforms they support, and how data reaches Avicenna.

Data source	Measures	Platforms	Collection method
Motion Sensors	Accelerometer, Gyroscope, Magnetic Field, Gravity, Linear Acceleration, Orientation	Android, iOS	Phone collection, burst-based
Pedometer	Steps	Android, iOS	Phone collection, continuous
Motion-Based Activity Recognition	Activity type, such as walking or being in a vehicle	Android, iOS	Phone collection
GPS	Location	Android, iOS	Phone collection, burst-based
Wi-Fi	Nearby Wi-Fi networks	Android	Phone collection, burst-based
Bluetooth	Nearby Bluetooth devices	Android	Phone collection, burst-based
Bluetooth Beacons	Proximity to study beacons	Android, iOS	Phone collection
Environmental Sensors	Temperature, light, air pressure, proximity, humidity	Android; pressure also on iOS	Phone collection, burst-based
Battery	Battery level and charging state	Android, iOS	Phone collection
Screen State	Screen on/off and lock/unlock events	Android, iOS	Phone collection, on each change
App Usage	Apps used and when	Android, iOS	Android: phone collection. iOS: Screen Time reports collected through an activity
Keyboard	Typing and keystroke events	Android, iOS	Avicenna research keyboard
Web Activity	Web browsing	Android	Firefox Nightly add-on
Weather	Weather at the participant’s location	Android, iOS	Avicenna servers retrieve hourly weather from OpenWeather using the participant’s GPS location
HealthKit	Activity, vital signs, sleep, state of mind	iOS	Health platform
SensorKit	Phone and Apple Watch sensors, device usage, speech and keyboard metrics	iOS, after Apple approval	Health platform
Garmin SDK	High-resolution heart rate, beat-to-beat intervals, motion, sleep, stress	Garmin watch with Android or iOS	Watch sync
Garmin API	Daily summaries, heart rate, sleep, stress, respiration	Garmin watch	Provider push
Google Health	Steps, heart rate, sleep, activity, nutrition from Fitbit and Pixel Watch devices	Google account	Provider push, with an hourly pull
Oura	Sleep, readiness, activity, heart rate	Oura Ring	Provider push, with an hourly heart rate pull
WHOOP	Sleep, recovery, workouts	WHOOP band	Provider push
Polar	Training sessions, sleep, continuous heart rate, nightly recharge	Polar device	Scheduled pull
Hexoskin	Cardiorespiratory, activity, and sleep data	Hexoskin shirt	Scheduled pull
Buzud	Interstitial glucose	Buzud CGM sensor	Scheduled pull, every 30 minutes
Mira	Hormone test results, menstrual cycle, daily logs	Mira analyzer	Scheduled pull, daily
BACtrack Skyn (on request)	Transdermal alcohol concentration	BACtrack Skyn with iOS	File upload
SwitchBot (on request)	Indoor and outdoor temperature and humidity	SwitchBot sensor	File upload

Data sources marked on request do not appear in the dashboard’s data source list. To add one to a study, contact Avicenna support.

Retired data sources, such as App Usage (Legacy), Fitbit, and earlier Polar data sources, cannot be added to new studies. Data already collected through them remains available for export.

Understanding Collection Methods

Data sources use different methods to deliver records to Avicenna. The collection method affects where data comes from and when it becomes available.

• Phone collection: The Avicenna app reads the phone’s sensors and system services during each collection cycle and uploads the records.
• Health platform: The Avicenna app reads data recorded by Apple HealthKit or Apple SensorKit on the iPhone and uploads it.
• Watch sync: The watch records data and synchronizes it with the Avicenna app over Bluetooth. The app then uploads the records.
• Provider push: The participant connects their provider account to Avicenna. The provider notifies Avicenna’s servers when new data is available, and the servers retrieve it.
• Scheduled pull: The participant connects their provider account, and Avicenna’s servers request new data on a fixed schedule.
• File upload: The participant exports a file from the device manufacturer’s app and uploads it through a File question in a survey. Avicenna’s servers then process the file.

Choosing Data Sources for Your Study

When selecting data sources, consider the information your study needs and how participants will provide it.

• Measures: Check which variables each source provides and whether they match your research questions.
• Platform compatibility: Confirm that the source supports the operating systems and devices your participants use.
• Device requirements: Some sources require a particular wearable, account, or additional hardware.
• Permissions and setup: Consider which phone permissions or provider-account connections participants must grant.
• Collection and availability: Phone sensors, wearable synchronization, and provider-based collection have different timing and data availability characteristics.

Each data source has its own documentation with further details about its requirements, supported devices, and collected fields. Review these details before adding a source to your study.

Adding Data Sources to a Study

To add a data source:

1. Open your study dashboard.
2. Go to Data Sources.
3. Click Add New Data Source.
4. Browse the available data sources or use the search box to find one by name, category, or description. Sources are organized into categories to help you find the type of data you need.
5. Select the data source and configure its settings:
   • Mandatory or Optional: Decide whether participants can disable the data source in the app. Mandatory sources do not have an opt-out toggle in the app. Optional sources do.
   • Name and Description: Enter the name and description that participants will see during enrollment and on the study homepage. Explain what the source collects and why your study needs it.
   • Collect Historical Data: Shown only for some cloud-based data sources. See Provider Collection.
6. Click Add to save the data source to your study.

Participant-Facing Names and Descriptions

The name and description you configure are displayed in two places:

• In the consent material when participants join the study.
• In the Data Sources section of the study homepage.

Use clear, participant-friendly language to explain what information is collected and why. You can provide additional details in your study’s informed consent material.

If your study supports multiple languages, you can translate data source names and descriptions using localization.

Viewing, Editing, and Removing Data Sources

After adding data sources, you can view and manage them on the study dashboard’s Data Sources page. You can sort the list by type, category, name, and mandatory status.

Use the menu next to a data source to access the following actions:

• Edit: Change the data source’s configuration, including its participant-facing settings.
• Go to Data Export: Open the Data Export page to export records collected through that source.
• Remove: Remove the data source from the study.

Removing a data source stops its collection immediately. The collected data is retained unless you select Delete the data from the data source when confirming removal.

If you remove a source without selecting this option and later decide that its data should also be deleted, contact Avicenna support.

Some data sources cannot be removed while they are used by activities in your study:

• Bluetooth Beacons cannot be removed while an activity uses Proximity triggering logic.
• GPS cannot be removed while an activity uses Geofencing triggering logic.

Study Settings

Each data source has settings that determine how it behaves in the study.

Mandatory and Optional Data Sources

When configuring a data source, you can mark it as mandatory or optional.

• Mandatory: Participants cannot turn off the data source through its toggle in the Avicenna app.
• Optional: Participants can enable or disable the data source using a toggle in the app.

Mandatory status does not prevent participants from revoking the underlying phone permissions. If a participant revokes a required permission through their phone settings, collection may stop. This is recorded in the Audit Trail as a Data Collection Failed event.

Participant Setup and Permissions

Some data sources require specific phone permissions or device connections before collection can begin.

When participants join a study that includes data sources requiring setup, the app checks whether the necessary steps have been completed. If something is missing, a Your study setup is incomplete banner appears on the study homepage.

Participants must complete the outstanding setup steps, such as granting the required permissions or connecting a provider account. For wearable data sources, participants can select Don’t have this device if they do not have the required device. This disables that data source for the participant.

Participants can revisit the Data Sources page in the app to update permissions. They can also revoke permissions through their phone settings at any time.

Phone Collection

Phone-collected data comes from the device’s sensors and system services. The Avicenna app reads this information in collection cycles and uploads the records.

Collection Cycle

By default, the app runs a collection cycle every 5 minutes. Each cycle reads the phone collection data sources enabled in the study.

Actual timing depends on the operating system:

• Android: The operating system schedules background work, so cycles may run earlier or later than the configured interval.
• iOS: Cycles run at the configured interval while iOS keeps the app running in the background. See Background Data Collection for more information.

If a participant is enrolled in more than one study, the app uses a single collection cycle for all their studies, following the shortest interval configured among those studies.

Continuous and Burst-Based Collection

Phone-collected data sources use different collection behaviors.

Continuous collection

The phone’s operating system continuously records or updates information, and the Avicenna app reads the accumulated data during each collection cycle.

For example:

• Pedometer: The operating system continuously counts steps. When the app requests pedometer data, it receives the steps recorded since the previous request.
• Screen State: The operating system tracks screen state changes and notifies the app when the screen turns on or off or the device is locked or unlocked.

The collection cycle determines when the app retrieves the data; it does not mean that these measurements are captured only during the cycle.

Burst-based collection

For burst-based sources, the app asks the operating system to collect sensor readings during a short period in each collection cycle. This period is called the burst length.

No readings are collected between bursts, so the resulting data represents periodic samples rather than continuous measurements.

The following table shows the burst lengths for phone sensors on Android:

Data source	Burst length
Accelerometer, Gyroscope, Magnetic Field, Gravity, Linear Acceleration, Orientation	60 seconds
GPS	Up to 60 seconds, ending once three accurate locations are read
Bluetooth	60 seconds
Wi-Fi	30 seconds
Motion-Based Activity Recognition	10 seconds
Ambient Temperature, Light, Pressure, Proximity, Relative Humidity	10 seconds
Battery	One reading per cycle

At the default 5-minute interval, a 60-second burst covers 20% of the participant’s time. Shorter intervals increase sampling coverage but also increase battery use.

Collection behavior on iOS differs for some data sources. See each data source’s individual documentation for platform-specific details.

Permissions

Some phone data sources require permissions such as location or Motion & Fitness.

Until the required permission is granted, the corresponding source cannot collect data. The study homepage displays the incomplete setup banner while required setup steps remain outstanding.

For more information about platform-specific requirements, see the individual data source documentation and Background Data Collection.

Upload and Storage

Phone-collected records are stored on the participant’s device before being uploaded in the background.

On Android, the upload runs at the end of each collection cycle. If Wi-Fi only is enabled, records remain on the phone until it connects to Wi-Fi.

The app also needs sufficient free storage to save new records. If the phone runs out of storage, the app stops storing new records. The participant receives no warning, and the dashboard does not indicate that storage is the cause.

Records that were successfully stored can be uploaded later when storage is available and the app synchronizes. Records that were never stored cannot be recovered.

Provider Collection

Provider-based data sources use a different collection process from phone sensors. They do not depend on the phone’s collection cycle.

After a participant connects their account with a provider, data becomes available to Avicenna once the participant’s device has synchronized with the provider and the provider has made the records available.

The time it takes for data to reach Avicenna depends on both the device’s synchronization and the provider’s data availability. Some sources use provider push, while others use scheduled pulls or a combination of methods. See each source’s documentation for its specific data collection behavior.

By default, Avicenna collects provider data from the moment the participant connects their account. For Buzud CGM and the Google Health data sources, you can turn on Collect Historical Data when you add or edit the data source. Avicenna then also collects the data the participant recorded earlier in their participation period, from its start up to the moment they connected. Data from before the participation period is never collected. The option is off by default.

Collection stops if the participant revokes Avicenna’s access through their provider account.

Data Collection Configuration

Researchers can configure how frequently Avicenna collects and uploads data from non-cloud-based data sources in their study.

To access these settings:

1. Open your study dashboard.
2. Go to Data Sources.
3. Open Settings.
4. Configure the following options.

Data Collection Interval

Choose how frequently the app runs collection cycles:

• Standard: Every 5 minutes.
• Frequent: Every 2 minutes.
• Custom: An interval between 1 and 60 minutes.

The selected interval applies to all non-cloud-based data sources in the study.

Wi-Fi Only

Enable Wi-Fi only to restrict data uploads to Wi-Fi connections.

• Off: Records can be uploaded over Wi-Fi or mobile data, depending on the available connection.
• On: Records are uploaded only when the participant’s phone is connected to Wi-Fi.

Restricting uploads to Wi-Fi helps participants conserve mobile data, but records remain on their phones until a Wi-Fi connection is available. Use this setting when participants are expected to connect to Wi-Fi regularly.

Participants can also enable Wi-Fi-only uploads in the app.

Settings for Test Participants

Test participants can adjust their own collection interval in the Avicenna app. Their settings override the interval configured by the researcher, allowing them to test different collection frequencies without changing the study-wide configuration.

To change the interval:

1. Open Settings in the Avicenna app.
2. Go to Data Collection Configuration.
3. Select Standard, Frequent, or Custom and enter an interval between 1 and 60 minutes if needed.

GPS has additional collection settings, described in the Location documentation.

Common Data Fields

Each data source has its own set of fields, described in its individual documentation. Most raw data exports also include common fields that help identify and organize records.

• Study ID (study_id): Identifies the study associated with the record.
• User ID (user_id): Identifies the participant who provided the record.
• Device ID (device_id): Identifies the device that collected the record. This field is available for phone-collected data but may be absent from provider-based data sources.
• Record Time (record_time): The time the record was captured, in UTC. See Handling of Timezones.

Records available in Kibana also include:

• Relative Record Time (rel_record_time): The time in milliseconds since the participant joined the study. For example, 3600000 represents one hour after the participant joined. This field is not updated if the participant’s start time is changed.

The fields available depend on the data source and where its records are stored. See Raw Data and Kibana Basics for more information.

Troubleshooting

Low or Missing Records

If you think some data records are missing or there are fewer data records than you expected, whether after exporting/downloading the data or by viewing the In-Operation or other plots on the Participation page:

1. Check the general steps to diagnose participation issues.
2. Check if the study setup is complete. If the pink Your study setup is incomplete banner is still shown at the top of the study’s homepage in the app, the participant has outstanding setup steps, which are described under Permissions and Setup Flow. You can check the Application State logs on Kibana to see which permissions are granted/revoked or not granted at all. Note that some data sources might not need specific permission. On the other hand, the participant should prevent the mobile app from being restricted/terminated by the operating system. Check these pages for more details:
   • Background Data Collection
   • Android Battery Optimization Workarounds
   • Android Background Permission Workarounds
   • Don’t Kill My App
3. Check whether the participant’s phone has run out of storage. Data waiting to upload is stored on the device first, so a phone with no free space stops collecting, with no warning in the app and nothing on the dashboard to distinguish it from a participant who has simply stopped engaging. Ask the participant how much free space they have, ask them to free some if needed, then have them open Avicenna and use Sync Data under Settings. Data that was already recorded is uploaded once there is room to work; data that was never written cannot be recovered.
4. If you haven’t marked your data source as Mandatory, the participant might have opted out of its data collection. You can check that by the Study Data Sources field under Application State logs on Kibana.
5. Check the data collection behavior of Avicenna to understand possible limitations and differences between Android and iOS, and see what you can expect in general. For each data source, check the additional details, if any, on the collection behavior under their own pages. For example, for GPS, see this section. This is especially useful if you checked the In Operation status and thought it was low; a value less than 100% does not necessarily translate to data loss.
6. Check if the participant is using the web app by checking their devices. Some data sources (e.g., GPS) collect data using mobile apps only.
7. Check for the “Data Collection Failed” and “Data Collection Cycle Did Not Start” audit logs. The first log type includes the reason behind the failure too, which might be, for example, due to the device not supporting the corresponding sensors.
8. Check if the sensor is working properly on the device. You can test that by collecting similar data using other apps or, in the case of wearable data sources (e.g., Garmin), see if you or the participant can see any data under the corresponding accounts.
9. When working with the exported CSV files (e.g., survey responses), cell values with line breaks may appear as extra lines in some text editors. The CSV format handles this by enclosing such content in double quotes ("). Most data analysis tools (e.g., Excel, Google Sheets, R, Python) correctly interpret these quoted fields.
   Try importing the CSV into a spreadsheet application like Google Sheets and see if the issue persists.