User Story Classification with Machine Learning and LLMs

We address the problem of classifying Capability, Task, Hard-goal, and Soft-goal in user stories. Such a classification is essential for generating Rationale Tree. Several articles have attempted to classify different aspects of user stories in the past. However, classifying the Capability, Task, Hard-goal, and Soft-goal class has been largely overlooked. To this aim, we present three pipelines. The first two pipelines rely on standard machine learning methods. They differ in how they represent features, i.e. bag-of-word vs. embedding from deep learning methods. Our third pipeline explores a recent NLP development, viz. few-shot classification with two LLMs, Mistral and Llama. Our experiments reveal that using deep learning embedding as a feature of classical machine learning methods significantly improves performance, even for minority classes. Thus, such features could help alleviate class imbalance and data sparsity issues. We also found out that Mistral outperformed Llama. However, its performance was still far below that achieved by classical machine learning methods. We believe that our is novel as we are the first to study the problem of classifying Capability, Task, Hard-goal, and Soft-goal, and as we investigate how LLMs perform in this problem.