Module contents

Setup and initialization for the QC Dashboard.

class dashboard.RegexConverter(url_map, *items)

Bases: BaseConverter

A ‘regex’ type for URL routes.

For whatever reason Flask (as of this writing) does not appear to have a built-in way to allow regular expressions in URL routes. RegexConverter adds this capability. For more info see this post.

Example

@app.route('/someroute/<regex("*.png"):varname>')
def some_view(varname):
    ...

This would add a URL endpoint for the pattern /someroute/*.png. The part of the URL that matches the regex will be passed in to the view without the prefix. e.g. accessing the URL ‘/someroute/my_picture.png’ would set varname to ‘my_picture.png’

dashboard.configure_scheduler(app, csrf)
dashboard.connect_db()

Push an application context to allow external access to database models.

Anything that uses a flask extension, or accesses the app config, needs to operate inside of an application context. This function can be called to push the context.

dashboard.create_app(config=None)

Generate an application instance from the given configuration.

This will load the application configuration, initialize all extensions, and register all blueprints.

dashboard.load_blueprints(app)

Register all blueprints for the app.

A blueprint must be in the dashboard’s blueprint folder and must implement the ‘register_bp’ function to be loaded by this function.

dashboard.setup_devel_ext(app)

Set up extensions only used within development environments.

Submodules

dashboard.datman_utils module

Dashboard functionality that requires datman’s config module.

Some parts of the dashboard need datman.config to locate things on the file system but datman.config uses the dashboard’s models. So to help prevent circular references (and debug them when they happen) isolate all the datman.config related code here.

This can safely be imported elsewhere but no dashboard related imports should ever appear here

Note

This currently does not import correctly outside of an app context.

One consequence of this is that sphinx’s automodule can’t add it to the docs. We have to solve the datman.config circular reference issues to fix this. :(

dashboard.datman_utils.delete(config, key, folder=None, files=None)
dashboard.datman_utils.delete_bids(config, subject, session, scan=None)
dashboard.datman_utils.delete_scan(scan)
dashboard.datman_utils.delete_session(session)
dashboard.datman_utils.delete_timepoint(timepoint)
dashboard.datman_utils.get_manifests(timepoint)

Collects and organizes all QC manifest files for a timepoint.

Parameters:

timepoint (dashboard.models.Timepoint) – A timepoint from the database.

Returns:

A dictionary mapping session numbers to a dictionary of input nifti files and their manifest contents.

For example: {

1: {nifti_1: nifti_1_manifest,

nifti_2: nifti_2_manifest},

2: {nifti_3: nifti_3_manifest}

}

dashboard.datman_utils.get_study_path(study, folder=None)

Returns the full path to the study on the file system.

If folder is supplied and is defined in study config then path to the folder is returned instead.

dashboard.datman_utils.read_json(in_file)

Read a json file.

Parameters:

in_file (str) – The full path the the json file to load.

Returns:

A dictionary of the json contents or an error message.

Return type:

dict

dashboard.datman_utils.update_header_diffs(scan)

dashboard.emails module

Functions for sending email notifications.

If an email message must be sent from the server side, and it isn’t used exclusively by view functions (which only run on the server), then it needs to be given to the scheduler. That means it needs a monitor and a check function. To learn more about monitor/check functions see dashboard.monitors

Any email notifications that might be submitted to the scheduler must only receive arguments that are JSON serializable. (see here for info on serializable types).

dashboard.emails.async_exec(f)

Allow a given function to execute in the background.

dashboard.emails.missing_redcap_email(session, study=None, dest_emails=None)

Notify that a session that requires a REDCap survey did not receive one.

Parameters:

  • session (str) – A session ID.

  • study (str, optional) – The study that the session belongs to.

  • dest_emails (str or list of str, optional) – Email address(es) to relay the notification to.

dashboard.emails.send_async_email(app, email)

Send an email in the background.

Parameters:

  • app (flask.Flask) – The current application instance.

  • email (flask_mail.Message) – The message to send.

dashboard.emails.send_email(subject, body, html_body=None, recipient=None)

Organize email contents into a message and send it in the background.

Parameters:

  • subject (str) – The subject line.

  • body (str) – The plain text body of the email.

  • html_body (str, optional) – An optional HTML formatted version of the plain text body. Some email clients are plain text only. If the recipient’s client can’t render HTML they will only receive the plain text version.

  • recipient (str or list of str, optional) – An email address (or list of email address) to send the message to. If none is provided the message will be sent to the address(es) configured as the dashboard admin(s).

dashboard.exceptions module

exception dashboard.exceptions.InvalidDataException

Bases: Exception

An exception for attempts to add incorrect data to the database.

exception dashboard.exceptions.InvalidUsage(message, status_code=None, payload=None)

Bases: Exception

An exception for incorrect usage of the URL endpoints.

status_code = 400
to_dict()
exception dashboard.exceptions.MonitorException

Bases: Exception

An exception for scheduled jobs that have encountered problems.

exception dashboard.exceptions.RedcapException

Bases: Exception

An exception for REDCap interface issues.

exception dashboard.exceptions.SchedulerException

Bases: Exception

An exception for problems while interacting with the scheduler.

dashboard.forms module

Web forms used by the dashboard.

Forms are defined using the ` Flask-WTForms <https://wtforms.readthedocs.io/en/latest/>`_ extension. This allows us to create HTML forms in python without having to worry writing HTML or avoiding CSRF vulnerabilities.

class dashboard.forms.AnalysisForm(*args, **kwargs)

Bases: FlaskForm

Add a new analysis to the dashboard.

This feature has not yet been fully implemented.

description = <UnboundField(TextAreaField, ('Description',), {'validators': [<wtforms.validators.DataRequired object>]})>
name = <UnboundField(StringField, ('Brief name',), {'validators': [<wtforms.validators.DataRequired object>]})>
software = <UnboundField(TextAreaField, ('Software',), {})>
class dashboard.forms.SelectMetricsForm(*args, **kwargs)

Bases: FlaskForm

Choose metrics from the database.

This form needs to be updated when we fix our QC metric integrations.

is_phantom = <UnboundField(HiddenField, (), {'default': False})>
metrictype_id = <UnboundField(SelectMultipleField, ('Metric type',), {'coerce': <class 'int'>})>
metrictype_vals = []
query_complete = <UnboundField(HiddenField, (), {'default': False})>
scan_id = <UnboundField(SelectMultipleField, ('Scan',), {'coerce': <class 'int'>})>
scan_vals = []
scantype_id = <UnboundField(SelectMultipleField, ('Scan type',), {'coerce': <class 'int'>})>
scantype_vals = []
session_id = <UnboundField(SelectMultipleField, ('Session',), {'coerce': <class 'int'>})>
session_vals = []
site_id = <UnboundField(SelectMultipleField, ('Site',), {'coerce': <class 'int'>})>
site_vals = []
study_id = <UnboundField(SelectMultipleField, ('Study',), {'coerce': <class 'int'>})>
study_vals = []
class dashboard.forms.StudyOverviewForm(*args, **kwargs)

Bases: FlaskForm

Make edits to a study’s README form.

readme_txt = <UnboundField(TextAreaField, ('README',), {'id': 'readme_editor'})>
study_id = <UnboundField(HiddenField, (), {})>

dashboard.monitors module

Add and run scheduled jobs.

Each scheduled job requires two pieces:

  1. A ‘check’ function that will run at the scheduled time and that does the actual work.

  2. A ‘monitor’ function that packages up the check function, any arguments it needs, and a date/time and submits it to the server.

Database queries can be made from within the check function to reduce the number of arguments that must be passed. Remember also to use the check function to verify that the scheduled task still makes sense to run at the time it is executed. i.e. make sure data hasn’t been deleted, notifications are still relevant, etc.

Warning

Any inputs submitted to the scheduler must be JSON serializable. Check functions, therefore, must only accept these types as input.

dashboard.monitors.add_monitor(check_function, input_args, input_kwargs=None, job_id=None, days=None, hours=None, minutes=None)

Add a job to be run on the server at a scheduled time.

Parameters:

  • check_function (function) – The function that will run at the scheduled day and time.

  • input_args (Any) – Arguments to pass to check_function at runtime

  • input_kwargs (Any, optional) – Optional args to pass to check_function at runtime.

  • job_id (str, optional) – A unique identifier for the job

  • days (int, optional) – Number of days to add to the current time when setting the run date.

  • hours (int, optional) – Number of hours to add to the current time when setting the run date

  • minutes (int, optional) – Number of minutes to add to the current time when setting the run date

Raises:

dashboard.exceptions.SchedulerException – If the job cannot be added to the server.

Returns:

The HTTP reply sent by the server

Return type:

str

dashboard.monitors.check_redcap(name, num, recipients=None)

Emails a notification if the given session doesnt have a redcap record.

Parameters:

  • name (str) – A session name

  • num (int) – A session number

  • recipients (list of str, optional) – A list of email address to notify.

Raises:

dashboard.exceptions.MonitorException – If a matching session can’t be found.

dashboard.monitors.get_emails(users)

Retrieve a list of emails for the given users (without duplicates).

Parameters:

users (list of dashboard.models.User) – A list of users to extract email addresses for.

Returns:

A str email address for each user.

Return type:

list

dashboard.monitors.monitor_redcap_import(name, num, users=None, study=None)

Add a scheduled job to run check_redcap().

This adds a scheduled job that will run check_redcap two days after job submission and notify either the given list of users or all staff contacts and study RAs if a redcap record has not found at that time.

Parameters:

  • name (str) – A session name

  • num (int) – A session number

  • users (list of dashboard.models.User, optional) – A list of users to notify

  • study (dashboard.models.Study, optional) – The study to monitor if the session belongs to more than one.

Raises:

dashboard.queries module

Reusable database queries.

dashboard.queries.find_scans(search_str)

Used by the dashboard’s search bar and so must work around fuzzy user input.

dashboard.queries.find_sessions(search_str)

Used by the dashboard’s search bar and so must work around fuzzy user input.

dashboard.queries.find_subjects(search_str)

Used by the dashboard’s search bar

dashboard.queries.get_redcap_config(project, instrument, url, create=False)
dashboard.queries.get_scan(scan_name, timepoint=None, session=None, bids=False)

Used by datman. Return a list of matching scans or an empty list

dashboard.queries.get_scan_qc(approved=True, blacklisted=True, flagged=True, study=None, site=None, tag=None, include_phantoms=False, include_new=False, comment=None, user_id=None, sort=False)

Get a set of QC records matching the given search terms.

Parameters:

  • approved (bool, optional) – If True scan QC records that have been approved will be included in the result. Defaults to True.

  • blacklisted (bool, optional) – If True scan QC records that have been blacklisted will be included in the result. Defaults to True.

  • flagged (bool, optional) – If True scan QC records that have been flagged will be included in the result. Defaults to True.

  • study (str or list(str), optional) – A study ID or list of study IDs to restrict the search to. Defaults to None.

  • site (str or list(str), optional) – A site ID or list of site IDs to restrict the search to. Defaults to None.

  • tag (str or list(str), optional) – A tag or list of tags to restrict the search to. Defaults to None.

  • include_phantoms (bool, optional) – Whether to include phantom QC records in the result. Defaults to False.

  • include_new (bool, optional) – Whether to include scans that have not yet been reviewed in the output. Defaults to False.

  • comment (str, optional) – A semi-colon delimited list of QC comments to search for. Defaults to None.

  • user_id (int, optional) – The ID of a valid user. If this is given the records returned will be restricted to those that the user has permission to view. Note that this does not take into account permissions for dashboard admins. That is, if the user ID given is for a dashboard admin, the results will be overly restrictive.

  • sort (bool, optional) – Whether to sort the results. Sorting is done by scan name. Defaults to False.

Returns:

A list of tuples of the format

{name: str, approved: bool, comment: str}, where ‘status’ is a boolean value that represents whether the scan was approved or flagged/blacklisted.

Return type:

list(dict)

dashboard.queries.get_scantypes(tag_id=None, create=False)

Get all tags (or one specific tag) defined in the database.

Parameters:

  • tag_id (str, optional) – A single tag to look up. Defaults to None.

  • create (bool, optional) – Whether to create a new record if tag_id doesnt exist. Defaults to False.

Returns:

A list of Scantype records.

Return type:

list

dashboard.queries.get_session(name, num)

Used by datman. Return a specific session or None

dashboard.queries.get_studies(name=None, tag=None, site=None, create=False)

Find a study or studies based on search terms.

If no terms are provided all studies in the database will be returned.

Parameters:

  • name (str, optional) – The name of a specific study. If given other search terms will be ignored. Defaults to None.

  • tag (str, optional) – A study tag / code (e.g. SPN01) as found in the first part of datman style subject IDs. Defaults to None.

  • site (str, optional) – A site tag (e.g. CMH) as found in the second part of datman style subject IDs. Defaults to None.

  • create (bool, optional) – Whether to create the study if it doesnt exist. This option is ignored if ‘name’ isn’t provided. Defaults to False.

Returns:

A list of matching dashboard.models.Study records. May

be empty if no matches found.

Return type:

list

dashboard.queries.get_study_timepoints(study, site=None, phantoms=False)

Obtains all timepoints from Studies model

Parameters:

  • study – Study codename used in DATMAN

  • site – Additionally apply a filter on the timepoints for a specific site

  • phantoms – Optional argument to keep phantoms in record

Returns:

[‘PACTMD_CMH_ABCD’, ‘PACTMD_CMH_DEFG’, … ]

If the site optional arugment is used, then only timepoints belonging to site will be returned in a list

Return type:

A list of timepoint names from the specified study. For example

dashboard.queries.get_timepoint(name, bids_ses=None, study=None)

Used by datman. Return one timepoint or None

dashboard.queries.get_user(username)
dashboard.queries.query_metric_types(**kwargs)

Query the database for metric types fitting the specifications

dashboard.queries.query_metric_values_byid(**kwargs)

Queries the database for metrics matching the specifications. Arguments are lists of strings containing identifying names

Example: rows = query_metric_value(Studies=[‘ANDT’,’SPINS’],

ScanTypes=[‘T1’], MetricTypes=[‘SNR’])

dashboard.queries.query_metric_values_byname(**kwargs)

Queries the database for metrics matching the specifications. Arguments are lists of strings containing identifying names

Example: rows = query_metric_value(Studies=[‘ANDT’,’SPINS’],

ScanTypes=[‘T1’], MetricTypes=[‘SNR’])

dashboard.task_scheduler module

class dashboard.task_scheduler.ContextThreadExecutor(max_workers=10, pool_kwargs=None)

Bases: ThreadPoolExecutor

Runs all scheduler jobs within the app context.

By default ThreadPoolExecutor does not propagate the app context correctly when it submits jobs to its pool to execute. This class fixes the problem by replacing the ‘run_job’ function submitted to the thread pool with the ‘context_run’ wrapper which ensures a context has been pushed before ‘run_job’ executes.

class dashboard.task_scheduler.RemoteScheduler(app=None)

Bases: object

A client scheduler that submits jobs to a scheduler server’s API.

This scheduler adds jobs via the dashboard server’s scheduler API instead of directly interfacing with the job store. This is done to ensure that all jobs run from the server side only and never from an instance of the dashboard that has been imported.

If more of the scheduler API needs to be exposed a list of all built in end points can found in flask_apscheduler/scheduler.py in ‘APScheduler._load_api’

add_job(job_id, job_function, **extra_args)
init_app(app)
start()
dashboard.task_scheduler.context_run(app, job, jobstore_alias, run_times, logger_name)
dashboard.task_scheduler.disable_scheduler_csrf(app, csrf)

Disable csrf for scheduler views.

Enabling csrf for an app is a good idea, however the scheduler doesnt use it and will raise ‘CSRF token not found’ errors if they arent disabled for its views. This turns it off only for these views.

Parameters:

  • app (flask.app.Flask) – The current app object.

  • csrf (flask_wtf.csrf.CSRFProtect) – The initialized csrf object.

dashboard.task_scheduler.format_job_function(job_function)

dashboard.utils module

Helper functions for views.

dashboard.utils.dashboard_admin_required(f)

Verifies a user is a dashboard admin before granting access

dashboard.utils.get_scan(scan_id, study_id, current_user, fail_url=None)
dashboard.utils.get_session(timepoint, session_num, fail_url)
dashboard.utils.get_timepoint(study_id, timepoint_id, current_user)
dashboard.utils.is_safe_url(target)
dashboard.utils.prev_url()

Returns the referring page if it is safe to do so, otherwise directs the user to the index.

dashboard.utils.read_bool(value)

Converts a value to boolean.

bool(value) misreads “false”/”False” as True, so we use this instead when parsing booleans from JSON requests.

dashboard.utils.report_form_errors(form)
dashboard.utils.study_admin_required(f)

Verifies a user is a study admin or a dashboard admin. Any view function this wraps must have ‘study_id’ as an argument

Subpackages