Psychometrics, as we all know, is a field of study concerned with the theory and technique involved behind psychological measurement. This field is primarily concerned with testing, measurement, assessment, and related activities. The field entails two key aspects for research purposes – a) construction of instruments, b) revolves around the development of procedures for measurement.

This age-old domain is largely based upon the work of Charles Spearman, but the technological changes across the entire landscape of instruments, data, applications, and domains associated have brought machine learning techniques into psychometrics.

Existing online assessments

Online assessments today mostly involve students attempting Multiple Choice Questions (MCQs). In other words, it largely replicates in structure, style and content the traditional “pencil and paper” based test.

Sometimes, the assessment additionally encapsulates capabilities of modern HTML, CSS, and Javascript. This helps in creating something called Technology Enhanced Items, where response options could appear in the form of a drop-down menu, or users might have to order and drag the options to the appropriate location on the screen.

Just like any other MCQ-based assessment, examinees choose an answer to each MCQ, proceeding on to the next, until there are no more questions left. Based on the assessment taken, the candidate will receive performance feedback then, or at a later stage.

The feedback obtained can be in the form of total number of responses answered correctly, if the test is a linear Computer Based Test (CBT). However, in case of CAT-based assessment, the candidate will receive a transformed IRT scale score. The system might be also able to furnish descriptive feedback to the candidate, providing details about the syllabus content that the examinee has mastered.

In 1980s, psychologists developed a model that could assess human beings based on five personality traits, known as the “Big Five,” which included openness, conscientiousness, extroversion, agreeableness, neuroticism. The problem with this approach was the data collection bit.

Renowned psychologist, Michael Kosinski developed a solution to this problem in 2008. Back then, as a student in Warsaw, he had collaborated with fellow student David Stillwell to launch a Facebook application, called MyPersonaity app. Users could fill out several psychometric questionnaires using the app, based on which they were evaluated and furnished with a “personality profile.”

Use Case: Understanding the elements of Griffin’s platform

The landscape has witnessed an influx of precursor technologies that disrupt the conventional approach for online assessment. Professor Patrick Griffin has made significant strides in the same direction. His work revolved around collaborative problem solving.

His assessment model presents a problem-solving task to at least two candidates, who must use separate computers. The problem appears incomplete to each individual user. To solve the problem, users must work and collaborate using a messaging app.

As a part of the test, the platform records in detail all the onscreen activity, including the messaging bit, for both the candidates during the task, as opposed to simply recording their correct and wrong answers. The data collected are appended to log files for analysis.

An interesting fact to note, certain data within the logs are generally thought to be related with social and cognitive components of collaborative problem solving, viz., perspective taking and task regulation. The process involves data being coded into sets of positive integers (or scores) for each component. These scores are based on the strength of the relationship between the data and component. Generally, the higher the integer, the stronger is the association.

The data is finally analyzed leveraging the Rasch Partial Credit Model, a psychometric model which helps in estimating the difficulty of the components, and the abilities of the participants. An interesting fact to take note of Griffin’s platform is that though the system leverages cognitive abilities, it achieves so without having to resemble the traditional psychometric test. Griffin’s platform teaches students to solve problems collaboratively with other people, in real time.

Use Case: Cambridge Analytica paves the road for Trump’s victory using Big Data-driven Psychometry

Cambridge Analytica had been involved with US election campaign for quite sometime. In the most recent election, which saw Donald Trump win, the organization had a significant role to play. Alexander Nix, the CEO for Cambridge Analytica focused on using psychometrics in place of demographics, which was the conventional norm, until now.

The success of the firm can be attributed to the combination of three core techniques, behavioral science using the OCEAN Model, Big Data analysis, and ad targeting. At the beginning, the firm purchases personal data from a range of different sources, like land registries, automotive data, shopping data, bonus cards, club memberships, and more. Next, the firm aggregates this data with the electoral rolls of the Republican party and online data, to calculate a Big Five personality profile.

Nix shows how psychographically categorized voters can be differently addressed. The messages differed for the most part only in microscopic details, to target the recipients in the optimal psychological way by including different headings, colors, captions, with a photo or video. This fine-tuning reaches helps in reaching down to the smallest groups. “Pretty much every message that Trump put out was data-driven,” mentions Nix.

The embedded Cambridge Analytica team comprised of only a dozen people. The firm received $100,000 from Trump in July, $250,000 in August, and $5 million in September. According to Nix, the company earned over $15 million overall.

The decision to focus on Michigan and Wisconsin in the final weeks of the campaign was made on the basis of data analysis done by the organization.

Machine Learning revolutionizes Psychometry

Machine Learning will revolutionize Psychometrics. IRT psychometrics are usually based upon logistic regression techniques. However, the technique fails to promise best class of models for classification anymore.

Machine Learning can be utilized to reveal candidate’s strengths in the in the social components of collaborative problem solving, such as perspective taking, participation, and social regulation. This can be achieved by simply analyzing a participant’s social media data, leveraging Machine Learning techniques.

Machine Learning techniques can extend to incorporate Virtual Reality technology. This will help towards expanding the scope of problem solving tasks, while enriching the resulting data stream. These techniques can also be applied to study difficult real life scenarios.

Last Word

Machine Learning techniques are revolutionizing assessments in an innovative and purposeful manner trumping old psychometric models.

Moreover, assessment platforms like Griffin’s generate data that more closely resemble Big Data, and is different compared to what is obtained from a conventional psychometric test. All these facts point out to Machine Learning clearly replacing Psychometrics soon.