In this chapter you'll combine embeddings, similarity, and nearest neighbors, in order to create a news article recommender, which will recommend articles to a reader based on the articles they've read before. Furthermore, you'll use classification to determine the articles' category, and then generation to extracct tags from the articles.

The repository for this project is here, and we encourage you to follow the code along with this tutorial.

Often, we run across a news article that sparks our interest in a particular topic or theme, leaving us curious, or even hungry, to learn more. We want to build on that momentum by reading more articles on the topic right away. Article recommendation apps make this easier by surfacing similar articles to the one we are reading, so we can expand our understanding as much as we want.

Our News Article Recommender app is based on the concept of text embeddings, which capture the meaning of a piece of text beyond simple keyword matching. It compares the embeddings similarity between a selected article to all other available articles. The app refines the list of candidate articles by using text classification to only recommend articles within the same category (e.g., sports or business). It then extracts a list of tags from each recommended article and displays them alongside the article to help the reader scan and choose the next article to read.

All of this is done via three Cohere API endpoints stacked together: Embed, Classify, and Generate. The steps to build the News Article Recommender are:

• Step 1: Get a list of articles
• Step 2: Embed articles
• Step 3: Create article selector
• Step 4: Find the most similar articles
• Step 5: Classify each article’s category
• Step 6: Extract tags from these articles
• Step 7: Put everything together

Read on for more details on each of these steps.

First, we’ll need to load the articles. In this example, we’re using the BBC news article dataset [source], which consists of articles from a few categories: business, politics, tech, entertainment, and sports, and we’re focusing on a subset of this dataset containing 100 articles.

# Load the dataset to a dataframe
@st.cache
def load_df(csv_file):
    return pd.read_csv(csv_file, delimiter=',')

df = load_df('bbc_news_test.csv')
articles = df["news"].tolist()
titles = df["title"].tolist()

Next, we’ll turn each article's text into embeddings. An embedding is a list of numbers that a model uses to represent a piece of text, capturing its context and meaning. We do this by calling the Cohere Embed endpoint, which takes in texts as input and returns embeddings as output.

# Get text embeddings via the Embed endpoint
def embed_text(texts):
    embeddings = co.embed(
                model='large',
                texts = texts)
    embeddings = np.array(embeddings.embeddings)
    return embeddings

We’ll then create a Streamlit selectbox where a reader can select the article that he or she is currently reading (let's call this the target).

# Drop down selectbox
col1, col2 = st.columns(2)
with col1:
   current_id = st.selectbox(
   "Select an Article",
   range(len(titles)),
   format_func=lambda x: titles[x])

Next, we’ll create a function to find other articles with the most similar embeddings (let's call these candidates) using cosine similarity.

Cosine similarity is a metric that measures how similar two sequences of numbers are (embeddings in our case), and we compute it for each target-candidate pair.

# Calculate cosine similarity between the target and candidate articles
def get_similarity(target,candidates):
   # Turn list into array
   candidates = np.array(candidates)
   target = np.expand_dims(np.array(target),axis=0)
 
   # Calculate cosine similarity
   similarity_scores = cosine_similarity(target,candidates)
   similarity_scores = np.squeeze(similarity_scores).tolist()
 
   # Sort by descending order in similarity
   similarity_scores = list(enumerate(similarity_scores))
   similarity_scores = sorted(similarity_scores, key=lambda x:x[1], reverse=True)
 
   # Return similarity scores
   return similarity_scores

# Get the similarity between the target and candidate articles
embs = rec.embed_text(articles)
current_emb = embs[current_id]
similarity_scores = rec.get_similarity(current_emb,embs)

In the context of news articles, it’s important to note that two articles may have similar meanings but can fall under totally different categories. For example, an article about a player’s anger over a racism fine (Sports) can appear to be similar to an article about a politician’s disagreement over an apology (Politics) because both articles revolve around confrontations and the clash of individuals.

In our case, we don’t want to recommend articles from a different category even if they appear to be similar to the current article. For this, we need to build a news category classifier that will only select articles from the same category.

We’ll use the Cohere Classify endpoint to build a news category classifier that classifies articles into five classes: Business, Politics, Tech, Entertainment, and Sports. For this app, we’ll train a custom model using another subset of the BBC news article. Find more information in the training a representation model documentation.

def classify_text(input_text):
   text = truncate_text(input_text)
   classifications = co.classify(
   model='504b1b30-4927-464d-9d4c-412f9771775b-ft',
   inputs=[text]
   )
   return classifications.classifications[0].prediction
  
current_class = rec.classify_text(articles[current_id])

Next, we’ll want to enrich the recommended articles with more information to help users scan the list for key information and discover content. The outcome: each recommended article will be displayed with a list of tags containing the key topics mentioned in the article.

We’ll build this tags extractor via the Cohere Generate endpoint. First, we create a prompt with a few examples of text and its tags. Then we call the Generate endpoint using the following prompt, appending each recommended article to get the recommended tags.

Refer to the prompt engineering documentation if you’d like to know more about creating prompts for the Generate endpoint.

Article: france starts digital terrestrial france has become the last big european country to launch a digital terrestrial tv (dtt) service.  initially  more than a third of the population will be able to receive 14 free-to-air channels. despite the long wait for a french dtt roll-out
Tags: france, digital terrestrial
--
Article: apple laptop is  greatest gadget  the apple powerbook 100 has been chosen as the greatest gadget of all time  by us magazine mobile pc.  the 1991 laptop was chosen because it was one of the first  lightweight  portable computers and helped define the layout of all future notebook pcs.
Tags: apple, apple powerbook 100, laptop
--
Article: {text}
Tags:"""
 
# Get extractions via the Generate endpoint
def extract_tags(complete_prompt):
   # Truncate the complete prompt if too long
   token_check = co.tokenize(text=complete_prompt)
   if len(token_check.tokens) > 2000:
       complete_prompt = co.detokenize(token_check.tokens[:2000]).text
 
   prediction = co.generate(
       model='xlarge',
       prompt=complete_prompt,
       max_tokens=10,
       temperature=0.3,
       k=0,
       p=1,
       frequency_penalty=0,
       presence_penalty=0,
       stop_sequences=["--"],
       return_likelihoods='NONE')
   tags_raw =  prediction.generations[0].text
 
   if "\n" in tags_raw:
       tags_clean = re.search(".+?(?=\\n)",tags_raw).group()
   else:
       tags_clean = tags_raw
 
   if tags_clean:
       tags = tags_clean.split(",")
       tags = list(dict.fromkeys(tags)) # remove duplicates
       tags = [tag.strip() for tag in tags] # remove empty spaces
       tags = [tag for tag in tags if tag] # remove empty tags
       tags = [tag for tag in tags if len(tag) > 3] # remove short tags
       tags = [f"`{tag}`" for tag in tags] # format tag string
       tags = ",".join(tags)
   else:
       tags = "None"
 
   return tags

tags = rec.extract_tags(complete_prompt)

Finally, we’ll tie together all the steps that we’ve completed so far: embedding, finding the most similar articles, news category classification, and tags extraction.

We then render the Streamlit components to display the current and recommended articles.

# Current article
with col3:
   st.markdown("***You are currently reading...***")
   current_class = rec.classify_text(articles[current_id])
   st.markdown(f"###### {current_class.capitalize()}")
   st.subheader(f"{titles[current_id]}")
   current_article = articles[current_id]
   st.write(current_article.replace("$","\$"))
 
# Recommended article
with col4:
   st.markdown("***You might also like...***")
   SHOW_TOP = 3
   count = 0
   for candidate_id,_ in similarity_scores:
       candidate_article = articles[candidate_id]
       candidate_title = titles[candidate_id]
 
       candidate_class = rec.classify_text(candidate_article)
 
       if candidate_class ==  current_class and candidate_id != current_id:
           # Show title
           st.markdown(f"###### {candidate_class.capitalize()}")
           st.subheader(candidate_title)
 
           # Show article
           expander = st.expander("Read article")
           expander.write(candidate_article.replace("$","\$"))
          
           # Show tags
           complete_prompt = rec.create_prompt(candidate_article)
           tags = rec.extract_tags(complete_prompt)
           st.markdown(tags)
           st.write("------")
       
           count += 1
 
       if count == SHOW_TOP:
           break

And that concludes the process of creating our News Article Recommender app, built via three Cohere API endpoints stacked together: Embed, Classify, and Generate. To get started building your own version, create a free Cohere account.