The following steps are proposed for implementing Case Classification:
- Start by verifying if the given text has been classified previously, for instance, by using GPTCache. If a classification exists, reuse the same labels. If not, continue with step 2.
- Summarize the text or case note and extract symptoms. Acronyms and abbreviations are expanded for clarity. Use the summarized text and extracted data for the rest of the steps. Summarization using multiple agents, followed by compiling the best summary based on those individual summaries, may yield better results.
- Identify major issue themes, such as core disjointed problems—"slow performance and keyboard backlight not working” compared with “slow boot time and slow software loading time.” This step should focus on identifying core root causes and encompassing symptoms rather than individual lexical phrases.
- Obtain relevant candidate T1, T1-T2 and T1-T2-T3 labels for each major issue, theme, or symptom. The process emphasizes recall to ensure that all applicable labels are retrieved. This is achieved by using lexical and semantic search and applying business rules for specific labels such as "fee based support." Agents must know the entire label hierarchy to present all T1, T1-T2, and T3s so that they can determine which level to choose.
- Multiple agents independently select the most suitable labels for each symptom, considering the overall context and the extracted symptom. By validating labels agreed by agents, hallucination is prevented, meaning that no new labels are introduced at this stage. Business knowledge can be incorporated in prompts through few-shots prompting; for example, consider [fee based support] labels for scenarios such as A, B, and C.
- Majority voting is implemented for the final selection of labels for each symptom. The final selection of labels is validated. If labels such as [boot] and [boot]-[no boot] are both predicted, only the most detailed label is retained.
- For each predicted T1 or T2 label, the process checks whether the label can be extended to a T1-T2 or T1-T2-T3. Tools that recall similar labels are given to aid LLMs in creating consistent label parts or reusing existing ones.
- An actor-critic prompt determines whether it is possible to extend a new label or not. This is followed by summarization of the discussion to get the finally agreed label.
- Unique recognizers are implemented for AI-created labels, for example, [boot]-[no boot]-{when USB connected}. Optionally, incorporate human in-the-loop verification for new labels that are created so that a workflow can be implemented to replace the AI label with a human label and start using the human-recommended label.
- New labels and associated label parts are indexed in the database. This ensures that the labels created earlier are reused if the same issue recurs.