Refining the Frontend 'Getting to Know You' Stage: Reflecting Knowledge Level Over Conversation Volume

Frontend 'Still Learning' Stage: Improving User Level Reflection from Knowledge to Conversation Volume

Have you ever encountered a problem where a user's level isn't accurately reflecting their actual knowledge, but is simply determined by the volume of their conversations? In such cases, users might feel frustrated being classified at a lower level than they actually are. In this post, I want to share how I tackled this issue and what points to be mindful of to avoid falling into the same trap.

Attempts and Pitfalls

Initially, I stuck with the existing logic of the user level management system. The system determined a user's level based on how many conversations they had on a specific topic. However, I quickly realized this was far from reflecting their actual knowledge level.

For example, a user might have already acquired significant knowledge after just a few questions on a particular topic. Yet, the system would still classify them as 'Beginner' simply because the conversation volume was low.

// Existing Logic (Hypothetical Example)
function getUserLevelByConversation(user, topic) {
  const conversationCount = user.getConversationCount(topic);
  if (conversationCount < 5) {
    return 'Beginner';
  } else if (conversationCount < 20) {
    return 'Intermediate';
  } else {
    return 'Advanced';
  }
}

Measuring only the conversation volume like this continuously led to problems where the actual knowledge level wasn't being properly reflected. I dug into this for 3 hours, but ultimately, the limitations of using just conversation volume became clear.

The Root Cause

The fundamental reason for the problem was that the criteria for determining user levels were solely focused on 'activity volume'. There was a lack of metrics that could objectively measure the user's 'actual knowledge level'. While conversation volume can indicate user engagement, it doesn't directly show the extent of their learning.

The Solution

So, I changed the user level criteria from 'conversation volume' to 'actual knowledge level'. To achieve this, I modified the relevant UI components, hooks, and library logic.

The new approach comprehensively considers how many concepts a user understands on a particular topic, how well they perform on related quizzes, and so on.

// Modified Logic (Hypothetical Example)
function getUserLevelByKnowledge(user, topic) {
  const knowledgeScore = user.getKnowledgeScore(topic); // New logic to measure knowledge score
  const quizAccuracy = user.getQuizAccuracy(topic);    // Quiz accuracy

  if (knowledgeScore < 0.4 || quizAccuracy < 0.5) {
    return 'Beginner';
  } else if (knowledgeScore < 0.8 || quizAccuracy < 0.8) {
    return 'Intermediate';
  } else {
    return 'Advanced';
  }
}

By introducing metrics that reflect the user's actual learning outcomes in this way, I was able to improve the accuracy of level classification.

Results

• Established level criteria that more accurately reflect users' actual knowledge.
• Increased satisfaction among users in the 'Still Learning' stage. (Qualitative change)
• Improved the accuracy of content recommendations per level, leading to increased learning efficiency. (Qualitative change)

Summary — How to Avoid the Same Pitfalls

• [ ] When calculating user levels, be sure to include metrics that can measure 'actual performance' in addition to 'activity volume'.
• [ ] When introducing new metrics, verify their accuracy through comparative tests against existing logic.
• [ ] Continuously collect user feedback to consistently improve level criteria.