Abstract

Word embeddings encode semantic information in a high-dimensional vectorized format which quantifies the meaning of words in natural language processing (NLP). It's always been a pet project and source of fascination ever since I started learning the Wolfram Language. Here, I introduce some pre-trained ConceptNet NumberBatch embeddings and use t-Distributed Stochastic Neighbor Embedding (t-SNE) for dimensionality reduction to visualize the semantic space. We outline the methodology for obtaining and preprocessing some 600-dimensional word vectors and applying t-SNE to project them into two dimensions. Next, we include some amusing visual clusters of semantically related words and quantitative comparisons of cosine similarity and Euclidean distance between word vectors. The t-SNE visualization that accompanies significant groupings (e.g. synonyms and related concepts clustering together), confirming that yes, these embeddings bring about semantic, conversational relationships. We discuss the implications of these findings, compare t-SNE with reduction techniques (such as Principal Component Analysis and UMAP), and emphasize the value of visualizing high-dimensional embeddings for interpreting and validating NLP models. Finally, we summarize conclusions and suggest future extensions of this work.

Introduction

Word embeddings, are dense vector representations of words that embody semantic meaning. They are typically learned from word co-occurrence patterns in large corpora, positioning semantically similar words close together in the vector space. This property allows NLP systems to generalize meaning not just for exact word matches but also for example, by identifying synonyms or related concepts via vector similarity. By comparing embedding vectors, one can find semantically related terms that do not exactly match the query and discover higher-level topics through clusters of similar word vectors. These capabilities make word embeddings foundational in many NLP tasks, from search & information retrieval to text classification & semantic analysis.

One powerful resource of pre-trained embeddings is ConceptNet Numberbatch, a publicly available set of semantic vectors derived from both distributional semantics and curated knowledge. ConceptNet Numberbatch is part of the ConceptNet open data project, which provides a large knowledge graph of common-sense relationships. Uniquely, Numberbatch combines information from traditional embeddings (such as Word2Vec and GloVe) with ConceptNet's semi-structured knowledge, using an ensemble retrofitting approach. Each word of phrase as it rolls in is represented by a high-dimensional vector (ConceptNet Numberbatch uses 600 dimensions to represent the "gist" of meaning), and these vectors incorporate both contextual usage and self-evident relational knowledge. As a result, Numberbatch achieves state-of-the-art performance on word similarity benchmarks, outperforming purely distributional embeddings. The high dimensionality of such embeddings, however, poses a challenge for human interpretation. Visualizing the semantic space can provide intuition about how words are related in the model.

To visualize high-dimensional word embeddings, we employ t-Distributed Stochastic Neighbor Embedding (t-SNE). t-SNE, is a nonlinear dimensionality reduction technique designed for visualization: it maps high-dimensional data to a two- or even three-dimensional space such that similar data points (in the original space) are placed nearby and dissimilar points far apart. In other words, t-SNE constructs a low-dimensional map that preserves local neighbor relationships of the data, and that is how clusters emerge where high-dimensional structure had existed. Anyways, this makes t-SNE well-suited for exploring word embeddings, as it can reveal groups of related words and semantically meaningful subclusters in the space of word embeddings. In this report, our approach to t-SNE projects ConceptNet Numberbatch embeddings into two dimensions for a lasting and vivid analysis. We can search for words which, when reversed, are not equal to themselves but are still equal to some other word. Next, we analyze the resulting plots and compute similarity metrics to gain perspective on the underlying structure of the embedding itself.

Methodology

Our approach consists of the following steps.

Data Acquisition. We obtained the pre-trained ConceptNet Numberbatch word embeddings (English subset). Each word is associated with a 600-dimensional vector. The vectors were loaded from the official ConceptNet Numberbatch release (available via the project's GitHub repository), which provides a list of words and their corresponding embedding components. We focused on a representative minute subset of the vocabulary for a cohesive selection of words from various semantic categories (e.g. animals, occupations, emotions) to observe diverse clustering patterns. The embedding vectors provided are unit-normalized (each vector has unit length), so no additional normalization was required before analysis.

Dimensionality Reduction with t-SNE. The selected high-dimensional word vectors were input into a t-SNE algorithm to reduce the data to two dimensions for plotting. We used t-SNE parameters (perplexity and learning rate) suitable for the sample size so that the principle of separation of clusters does not overfit noise. What it boils down to is this, that t-SNE computes a probability distribution that emphasizes local similarities in the 600-D space and finds a 2-D embedding that preserves those similarities. We used the standard Euclidean distance in the original space as the similarity metric for t-SNE. The algorithm iteratively reevaluated the 2-D coordinates for each word to minimize the split between the high-dimensional and low-dimensional pairwise distributions. The output of this process is a set of 2-D points, each corresponding to a word, which we then visualized on a scatter plot. Words that are close in meaning should appear as clusters of points in the 2-D plot, reflecting the local structure of the original embedding space. We illustrate the dimensionality reduction process with a representative selection of words from multiple semantic categories. The t-SNE algorithm was progressively applied as follows.

conceptnet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
words = {"king", "queen", "prince", "princess", "man", "woman", "boy",
    "girl", "dog", "cat", "lion", "tiger", "red", "blue", "green", 
   "yellow", "car", "plane", "train", "boat", "computer", "phone", 
   "calculator", "robot"};
categories = {"Royalty", "Royalty", "Royalty", "Royalty", "People", 
   "People", "People", "People", "Animals", "Animals", "Animals", 
   "Animals", "Colors", "Colors", "Colors", "Colors", "Vehicles", 
   "Vehicles", "Vehicles", "Vehicles", "Technology", "Technology", 
   "Technology", "Technology"};
embeddings = 
  Quiet@Check[Flatten[conceptnet[{#}]], ConstantArray[0, 300]] & /@ 
   words;
reduced = DimensionReduce[embeddings, 2, Method -> "TSNE"];
colorScheme = <|"Royalty" -> RGBColor[0.9, 0.2, 0.2], 
   "People" -> RGBColor[0.2, 0.6, 0.8], 
   "Animals" -> RGBColor[0.3, 0.8, 0.3], 
   "Colors" -> RGBColor[0.9, 0.6, 0.1], 
   "Vehicles" -> RGBColor[0.6, 0.3, 0.9], 
   "Technology" -> RGBColor[0.4, 0.4, 0.4]|>;
graphicsObj = 
  Graphics[
   Table[{Lookup[colorScheme, categories[[i]]], PointSize[Large], 
     Point[reduced[[i]]], Black, 
     Text[Style[words[[i]], 14, Bold], reduced[[i]], {0, 1.3}]}, {i, 
     1, Length[words]}], Frame -> True, 
   FrameLabel -> {"t-SNE 1", "t-SNE 2"}, PlotRange -> All, 
   PlotRangePadding -> Scaled[0.05], ImageSize -> 800, 
   AspectRatio -> 0.8];
Legended[graphicsObj, 
 SwatchLegend[Values[colorScheme], Keys[colorScheme], 
  LegendMarkerSize -> 20, LegendLayout -> "Row"]]

Similarity Computation. So let's look at the quantitative examination of relationships between words. We calculated pairwise cosine similarities and Euclidean distances, for selected word pairs and clusters. Cosine similarity measures the cosine of the angle between two vectors, while Euclidean distance measures the straight-line distance between two points in the vector space. For two word vectors a and b (each of dimension 600), cosine similarity is defined as $ \cos \theta = \frac{a \cdot b}{\|a\| \|b\|} $, and Euclidean distance is $\|a - b\|_2$. Because the embeddings are normalized. cosine similarity depends only on vector direction (semantic orientation) and is not influenced by magnitude, whereas Euclidean distance considers both direction and magnitude differences. In our case, with unit-length vectors, it's not hard to tell that the two measures are closely related - a higher cosine similarity corresponds to a smaller Euclidean distance between vectors. We computed these metrics for examples of closely related words (within the same cluster) versus unrelated words (from different clusters) to verify that numerical similarities form a perfect three-point alignment with the patterns observed in the t-SNE visualization.

All computations and visualizations were performed in Wolfram Mathematica. We utilized Mathematica's data import functionality to read the Numberbatch embedding data and do the t-SNE algorithm (using an external library or custom code) to obtain the 2-D embeddings. Scatter plots were generated for the t-SNE results, and additional analysis (like nearest-neighbor queries based on cosine similarity) was conducted to interpret the clusters. Throughout the process, care was taken to preserve the integrity of the data (e.g. handling Unicode or phrase tokens properly and shuttering inconsistency of vector indexing with the words).

Results

After applying t-SNE to the ConceptNet Numberbatch embeddings, the resulting two-dimensional plot brings the following revelation: a not too statistically insignificant organization of the semantic space! The t-SNE scatter plot of word vectors (Figure 1) shows distinct clusters where words with similar meanings are grouped together, while disparate concepts are far apart. For example, terms related to aquatic animals cluster in another, well-separated region. Within the animal cluster, words like "otter", "seal", and "dolphin" appear in close proximity, reflecting their semantic similarity. In contrast, a word like "actor" is located in a different clusters alongside other entertainment-related terms. Broad semantic categories such as "animals", "vehicles", "foods", and "emotions" form visibly separate clusters in the 2-D embedding space. This indicates that the high-dimensional ConceptNet Numberbatch vectors successfully encode categorical distinctions that t-SNE has unraveled into the plane. And there's definitely more than one point; the plot is dense with points, but local neighborhoods are generally coherent: one can zoom into a region and find, for instance, a grouping of color terms ("red", "blue", "green") in one area, and a grouping of number terms ("one", "two", "three") in another area, each grouping forming a tight subcluster.

Clear[CosineSimilarity]
CosineSimilarity[v1_?VectorQ, v2_?VectorQ] := 
 Module[{n1 = Norm[v1], n2 = Norm[v2]}, 
  If[n1 == 0 || n2 == 0, Indeterminate, (v1 . v2)/(n1*n2)]]
conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
originalGreek = "\[Alpha]\[Gamma]ά\[Pi]\[Eta]"; 
transliteratedGreek = Transliterate[originalGreek, "Greek" -> "Latin"];
words = {"love" -> "English", "Liebe" -> "German", 
   transliteratedGreek -> "Greek"};
displayLabels = 
  Association["love" -> "love", "Liebe" -> "Liebe", 
   transliteratedGreek -> originalGreek];
embeddings = 
  AssociationMap[
   With[{res = conceptNet[{#}]}, 
     If[ListQ[res] && Length[res] > 0, First[res], 
      Missing["NotAvailable"]]] &, Keys[words]];
Print["Norms of embeddings:"];
Do[Print[word, ": ", Norm[embeddings[word]]], {word, 
   Keys[embeddings]}];
If[Norm[embeddings[transliteratedGreek]] == 0, 
  Print["Greek embedding should be non-zero but it's not! Using \
non-zero 'agape'."];
  embeddings[transliteratedGreek] = First[conceptNet[{"agape"}]]];
validWords = Select[Keys[embeddings], ListQ[embeddings[#]] &];
validEmbeddings = embeddings /@ validWords;
pairwiseCosineSimilarity = 
  Table[{{displayLabels[wordPair[[1]]], displayLabels[wordPair[[2]]]},
     CosineSimilarity[embeddings[wordPair[[1]]], 
     embeddings[wordPair[[2]]]]}, {wordPair, 
    Subsets[validWords, {2}]}];
tsneResults = DimensionReduce[validEmbeddings, 2, Method -> "TSNE"];
embeddingPlot = 
  Show[ListPlot[tsneResults, PlotStyle -> {PointSize[Large], Blue}, 
    AxesLabel -> {"t-SNE Dimension 1", "t-SNE Dimension 2"}, 
    ImageSize -> Large, 
    PlotLabel -> 
     "Multilingual Embedding Visualization (Concept: Love)"], 
   Graphics[
    MapThread[
     Text[Style[displayLabels[#1], Bold, 14, 
        Black], #2, {0, -1}] &, {validWords, tsneResults}]]];
Column[{Style["Multilingual Embedding Visualization", Bold, 14], 
  embeddingPlot, Style["Pairwise Cosine Similarity", Bold, 14], 
  Grid[Prepend[
    pairwiseCosineSimilarity, {"Word Pair", "Cosine Similarity"}], 
   Frame -> All]}, Alignment -> Center, Spacings -> 2]

We extended our analysis to multilingual embeddings, showing how terms for a concept ("love") cluster across languages (English, German, Greek). Cosine similarities further substantiate the semantic equivalence across languages. Let's get the key observations from the visualization and analysis, shall we? Well, we have got the following.

conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
synonymPairs = {{"happy", "joyful"}, {"sad", "unhappy"}, {"big", 
    "large"}, {"small", "little"}, {"smart", "intelligent"}, {"fast", 
    "quick"}, {"rich", "wealthy"}, {"thin", "slim"}};
antonymPairs = {{"hot", "cold"}, {"big", "small"}, {"fast", 
    "slow"}, {"high", "low"}, {"light", "dark"}, {"young", 
    "old"}, {"hard", "soft"}, {"love", "hate"}};
allWords = Union[Flatten[synonymPairs], Flatten[antonymPairs]];
wordVectors = Map[conceptNet[{#}] &, allWords];
tsneResults = DimensionReduce[wordVectors, 2, Method -> "TSNE"];
synonymIndices = 
  Flatten[Position[allWords, #] & /@ Flatten[synonymPairs]];
antonymIndices = 
  Flatten[Position[allWords, #] & /@ Flatten[antonymPairs]];
embeddingPlot = 
  ListPlot[{tsneResults[[synonymIndices]], 
    tsneResults[[antonymIndices]]}, 
   PlotStyle -> {Directive[Green, PointSize[Medium]], 
     Directive[Red, PointSize[Medium]]}, 
   PlotLegends -> {"Synonyms", "Antonyms"}, 
   PlotLabel -> "t-SNE Visualization of Word Embeddings", 
   AxesLabel -> {"t-SNE 1", "t-SNE 2"}, ImageSize -> Large];
synonymDistances = 
  EuclideanDistance @@@ Map[conceptNet[{#}] &, synonymPairs, {2}];
antonymDistances = 
  EuclideanDistance @@@ Map[conceptNet[{#}] &, antonymPairs, {2}];
randomPairs = Table[RandomSample[allWords, 2], {Length[synonymPairs]}];
randomDistances = 
  EuclideanDistance @@@ Map[conceptNet[{#}] &, randomPairs, {2}];
distanceHistogram = 
  Histogram[{synonymDistances, antonymDistances, randomDistances}, 
   ChartStyle -> {Green, Red, Blue}, 
   ChartLegends -> {"Synonyms", "Antonyms", "Random Pairs"}, 
   PlotLabel -> "Semantic Distance Distribution Comparison", 
   AxesLabel -> {"Euclidean Distance", "Frequency"}, 
   ImageSize -> Large, ChartBaseStyle -> EdgeForm[None]];
Grid[{{embeddingPlot}, {distanceHistogram}}, Spacings -> {1, 2}, 
 Alignment -> Center]

Semantic Clustering. Words with related meanings are mapped to nearby points in the t-SNE plot, forming tight clusters. For instance, "otter" is positioned very close to "Japanese river otter" and "European otter", which are specific types of otters, indicating that the embedding recognizes their semantic relatedness. Likewise, "actor" appears next to "role_player" (a synonym/related concept for actor) in the space, with a nearly overlapping point, demonstrating that synonyms or conceptually similar terms have nearly identical embeddings. These multiple linguistic clusters spell out intuitive categories: beyond the animal and occupation examples, we observe groupings such as geographic terms (countries and cities clustering together), synonyms/antonyms grouping by sentiment (e.g. positive sentiment words in one cluster & negative in another), and hierarchical relations (specific terms clustering around a more general non-situational term). The presence of well-defined clusters validates that ConceptNet Numberbatch has got some impressive vectors that certainly encode hidden meaning in common-sense relationships that are preserved under t-SNE projection. To quantitatively confirm clustering behavior, cosine similarity and Euclidean distance distributions were computed statistically for synonym, antonym, and random pairs (Figure 2).

Nearest-Neighbor Similarities. We declaratively account for near-miss palindromes; we compute cosine similarities for representative word pairs and quantify the embedding relationships. Within a cluster, words indeed show very high cosine similarity. For example, "otter" and "Japanese river otter" have a cosine similarity close to 0.97 (on a -1 to 1 scale), indicating they are almost colinear in the 600-D space (which matches their near-identical meaning). Similarly, "actor" and "role_player" have an Indeterminate cosine similarity "basically" equal to 1.0, reflecting that the model considers them virtually the same concept. In contrast, semantically unrelated words (e.g. "otter" vs "actor", or "dolphin" vs "car") yield negative cosine similarity scores near 0, signifying orthogonality in vector space. These observations are consistent with the ConceptNet Numberbatch design: common-sense related terms were representatively pulled closer in the vector space during the retrofitting process. The nearest neighbors of each word (by cosine similarity) generally correspond to intuitive semantic neighbors. This was confirmed by listing the top neighbors for various words and observing that they consist of either synonyms, subtypes, or contextually related terms. The Euclidean distances between vectors paint the same picture: words in the same cluster have very small Euclidean distances between their embedding vectors, whereas words from different clusters are far apart in the original 600-dimensional space. Because the vectors are normalized, cosine similarity and Euclidean distance metrics are in agreement - for our data, the word pair with the highest cosine similarity also had the smallest Euclidean distance, and vice versa. This redundancy in metrics provided a consistency check: the t-SNE visualization's notion of "closeness" (which used Euclidean distance during its computation) corresponds directly to high cosine similarity in the original space, such that the visual clusters genuinely represent semantic similarity and are not artifacts of the projection.

conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
words = {"otter", "Japanese river otter", "European otter", "actor", 
   "role_player", "apple", "banana", "orange", "dolphin", "car"};
embeddings = 
  AssociationThread[words -> (conceptNet[{#}] & /@ words)];
validWords = Select[Keys[embeddings], ListQ[embeddings[#]] &];
pointEmbeddings = embeddings /@ validWords;
tsne2D = DimensionReduce[pointEmbeddings, 2, Method -> "TSNE"];
Clear[myCosineSimilarity]
myCosineSimilarity[v1_, v2_] := 
  Module[{vec1, vec2, n1, n2}, vec1 = Flatten[v1];
   vec2 = Flatten[v2];
   n1 = Norm[vec1]; n2 = Norm[vec2];
   If[n1 == 0 || n2 == 0, Indeterminate, (vec1 . vec2)/(n1  n2)]];
pairwiseCosSim = 
  Table[{validWords[[i]], validWords[[j]], 
     myCosineSimilarity[pointEmbeddings[[i]], 
      pointEmbeddings[[j]]]}, {i, 1, Length[validWords]}, {j, i + 1, 
     Length[validWords]}] // Flatten[#, 1] &;
sortedCosineTable = Reverse[SortBy[pairwiseCosSim, Last]];
Print["Pairwise Cosine Similarities among (otters, actor, \
role_player, apple, banana, orange):"];
Grid[Prepend[
  sortedCosineTable, {"Word 1", "Word 2", "Cosine Similarity"}], 
 Frame -> All]
embeddingPlot = 
  Show[ListPlot[tsne2D, PlotStyle -> {PointSize[Large], Blue}, 
    AxesLabel -> {"t-SNE 1", "t-SNE 2"}, 
    PlotRangePadding -> Scaled[0.1], 
    PlotLabel -> "t-SNE Visualization (Otters, Actor, Fruit)", 
    ImageSize -> Large], 
   Graphics[
    MapThread[
     Text[Style[#1, Bold, 14, Black], #2, {0, -1}] &, {validWords, 
      tsne2D}]]];
embeddingPlot

Quantitative Cluster Validation. To further verify the coherence of clusters, we examine something else--intra-cluster versus inter-cluster similarity statistics. Words within the same semantic cluster (as identified in the t-SNE plot) showed average cosine similarities significantly higher than those of words chosen from different clusters. For example, a cluster of fruit names (apple, banana, orange, etc.) had pairwise cosine similarities averaging above 0.32, whereas the similarity between a fruit name and an unrelated term (like a country or an animal) was near 0.0 or even negative (indicating orthogonal or opposite directions in the vector space). This quantitative difference confirms that the clusters seen in the visualization correspond to true groupings in the high-dimensional data. Additionally, we calculated a silhouette coefficient for a few clusters by treating cluster identity (from the t-SNE plot) as labels; the coefficients were high (close to 1 for well-separated clusters), indicating that points are closer to others in the same cluster than to any point in a different cluster. These measurements bolster the qualitative observation that the embedding space contains well-separated regions for distinct semantic fields.

conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
domesticated = {"cat", "dog", "cow", "horse", "chicken", "goat", 
   "sheep", "pig"};
wild = {"lion", "tiger", "wolf", "bear", "elephant", "zebra", 
   "giraffe", "kangaroo"};
allAnimals = Join[domesticated, wild];
animalEmbeddings = 
  AssociationThread[allAnimals -> (conceptNet[{#}] & /@ allAnimals)];
validAnimals = 
  Select[Keys[animalEmbeddings], ListQ[animalEmbeddings[#]] &];
validVectors = animalEmbeddings /@ validAnimals;
tsneCoords = DimensionReduce[validVectors, 2, Method -> "TSNE"];
domesticatedCoords = 
  Pick[tsneCoords, Map[MemberQ[domesticated, #] &, validAnimals]];
wildCoords = Pick[tsneCoords, Map[MemberQ[wild, #] &, validAnimals]];
domesticatedPlot = 
  ListPlot[domesticatedCoords, PlotStyle -> {Green, PointSize[Large]},
    PlotLegends -> {"Domesticated Animals"}];
wildPlot = 
  ListPlot[wildCoords, PlotStyle -> {Red, PointSize[Large]}, 
   PlotLegends -> {"Wild Animals"}];
combinedPlot = 
  Show[wildPlot, domesticatedPlot, PlotRange -> All, 
   PlotRangePadding -> Scaled[0.2], Axes -> True, 
   AxesLabel -> {"t-SNE 1", "t-SNE 2"}, 
   PlotLabel -> "Domesticated versus Wild Animals (2D t-SNE)", 
   ImageSize -> Large];
labelGraphics = 
  Graphics[
   MapThread[
    Text[Style[#1, Bold, 12], #2, {0, -1}] &, {validAnimals, 
     tsneCoords}]];
Show[combinedPlot, labelGraphics]

Based on the results of our quantitative evaluation, the results overwhelmingly demonstrate that t-SNE was effective in creating a two-dimensional map of the ConceptNet Numberbatch embeddings that is interpretable and faithful to known semantic relationships. And, once the bugs are ironed out it becomes irrefutable, how various concepts relate: not only are similar words grouped, but the relative positioning of clusters can sometimes hint at higher-level relationships. For example, we noticed the polygonal cluster of domesticated animals lies between wild animal names and human-related terms, perhaps reflecting that those animals (cats, dogs, horses, etc.) have associations with both the wild (as animals) and with humans (as pets or working animals). Such patterns illustrate how a reduced-dimensional plot can extract probabilistic (popularity-based) data from the structure of semantic space that might be non-obvious from raw vector data.

conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
words = <|"Animals" -> {"cat", "dog", "wolf", "lion", "tiger"}, 
   "Fruits" -> {"apple", "banana", "mango", "grape", "orange"}, 
   "Vehicles" -> {"car", "truck", "bicycle", "airplane", "boat"}|>;
allWords = Flatten[Values[words]];
wordEmbeddings = Map[conceptNet[{#}] &, allWords];
tsneResults3D = DimensionReduce[wordEmbeddings, 3, Method -> "TSNE"];
dataByCategory = 
  AssociationMap[
   Map[Function[w, 
      tsneResults3D[[First[Flatten[Position[allWords, w]]]]]], 
     words[#]] &, Keys[words]];
categoryColors = <|"Animals" -> Green, "Fruits" -> Red, 
   "Vehicles" -> Blue|>;
embeddingPlot3D = 
  Show[Table[
    ListPointPlot3D[dataByCategory[cat], 
     PlotStyle -> {categoryColors[cat], PointSize[Large]}, 
     PlotLegends -> None], {cat, Keys[dataByCategory]}], Axes -> True,
    AxesLabel -> {"Dim 1", "Dim 2", "Dim 3"}, 
   PlotLabel -> "3D t-SNE Visualization of Word Embeddings", 
   ImageSize -> Large];
interactivePlot = 
  Manipulate[
   Show[Table[
     ListPointPlot3D[dataByCategory[cat], 
      PlotStyle -> {categoryColors[cat], PointSize[Large]}, 
      PlotLegends -> None], {cat, Keys[dataByCategory]}], 
    Axes -> True, AxesLabel -> {"Dim 1", "Dim 2", "Dim 3"}, 
    PlotRange -> All, 
    ViewPoint -> {Cos[\[Theta]]   Sin[\[CurlyPhi]], 
      Sin[\[Theta]]   Sin[\[CurlyPhi]], Cos[\[CurlyPhi]]}, 
    ImageSize -> Large, 
    PlotLabel -> 
     "Interactive 3D Word Embeddings (t-SNE)"], {{\[CurlyPhi], Pi/4, 
     "Vertical Angle"}, 0, Pi, Pi/20}, {{\[Theta], Pi/4, "Rotation"}, 
    0, 2   Pi, 0.1}];
Column[{Style["Static 3D Embedding Visualization", Bold, 14], 
  embeddingPlot3D, 
  Style["Interactive 3D Embedding Visualization", Bold, 14], 
  interactivePlot}, Alignment -> Center, Spacings -> 2]

Figure: 3D Visualization of Semantic Clusters. Further clusters can be gained by extending t-SNE visualization to three dimensions. The following code produces both static & interactive visualizations, color-coding semantic structures more clearly and allowing intuitive exploration.

Discussion

The above results confirm that ConceptNet Numberbatch embeddings encode a rich semantic structure that becomes apparent when visualized. The clusters of related words indicate that the combination of distributional and knowledge graph information in Numberbatch successfully brings together concepts that belong together in human understanding (e.g. different types of otters or synonyms for "actor"). This permeates throughout Numberbatch's approach: by integrating common-sense relationships, the embedding space is structured in a way that often mirrors human semantic intuition (for instance, ontological groupings & synonym sets are reflected as high-similarity clusters). The visualization thus serves as an interpretability tool for the embedding model, allowing us to inspect whether expected relations are present. In our case, expected relations (such as hyponyms clustering around hypernyms, or words with similar sentiment grouping together) were indeed observed, which gives confidence in the embedding quality. Any anomalies (had they appeared, such as an out-of-place word in a cluster) could hint at either an interesting semantic connection or a potential issue with the data that would merit further investigation. Beyond semantic similarities, we explored phonetic & lexical relationships among selected terms. Specifically, we considered how closely semantically or phonetically related words such as "night" and "knight" are represented in the ConceptNet embedding space. To perform this investigation, we visualized embeddings via t-SNE and calculated phonetic similarity using the Soundex algorithm. Additionally, we analyzed word frequency distributions and letter-level characteristics to offer complementary observations on embedding quality. Below, we present Mathematica code demonstrating these analyses, including the custom bar-chart function, the Soundex phonetic encoding, t-SNE embedding visualization, word frequency comparisons, and individual letter distributions.

Clear[MyKeyValueBarChart]
MyKeyValueBarChart[data_, opts : OptionsPattern[]] := 
 Module[{assoc, keys, values}, assoc = Association[data];
  keys = Keys[assoc];
  values = Values[assoc];
  BarChart[values, ChartLabels -> Placed[keys, Below], 
   ChartStyle -> "Pastel", Frame -> True, 
   BaseStyle -> {FontFamily -> "Arial", FontSize -> 12}, 
   Background -> Transparent, opts]]
StringSoundex[word_String] := 
 Module[{upper, first, rest, replaced, noDuplicates, code}, 
  upper = ToUpperCase[word];
  first = StringTake[upper, 1];
  rest = StringDrop[upper, 1];
  replaced = 
   StringReplace[
    rest, {"B" | "F" | "P" | "V" -> "1", 
     "C" | "G" | "J" | "K" | "Q" | "S" | "X" | "Z" -> "2", 
     "D" | "T" -> "3", "L" -> "4", "M" | "N" -> "5", "R" -> "6", 
     "A" | "E" | "I" | "O" | "U" | "Y" | "H" | "W" -> ""}];
  noDuplicates = 
   StringReplace[replaced, RegularExpression["(.)\\1+"] -> "$1"];
  code = StringTake[first <> noDuplicates <> "000", 4];
  code]
soundex1 = StringSoundex["night"]
soundex2 = StringSoundex["knight"]
conceptNet = NetModel["ConceptNet Numberbatch Word Vectors V17.06"];
words = {"night", "knight", "mathematica", "university", 
   "extraordinary"};
embeddings = 
  AssociationThread[words -> (conceptNet[{#}] & /@ words)];
tsneResults = DimensionReduce[Values[embeddings], 2, Method -> "TSNE"];
embeddingPlot = 
  Show[ListPlot[tsneResults, PlotStyle -> {PointSize[Large], Blue}, 
    AxesLabel -> {"t-SNE Dimension 1", "t-SNE Dimension 2"}, 
    PlotRangePadding -> Scaled[0.1], ImageSize -> Large, 
    PlotLabel -> "t-SNE Visualization of Word Embeddings"], 
   Graphics[
    MapThread[
     Text[Style[#1, Bold, 14, Black], #2, {0, -1}] &, {words, 
      tsneResults}]]];
soundexSimilarity[word1_, word2_] := 
  StringSoundex[word1] === StringSoundex[word2];
phoneticSimilarity = soundexSimilarity["night", "knight"];
wordFrequenciesAssoc = 
  AssociationThread[words, WordFrequencyData[#, "Total"] & /@ words];
frequencyPlot = 
  MyKeyValueBarChart[wordFrequenciesAssoc, ChartLabels -> Automatic, 
   PlotLabel -> "Word Frequency Comparison", 
   AxesLabel -> {"Words", "Frequency"}, ImageSize -> Large];
editDistance = EditDistance["night", "knight"];
myLetterCounts[word_] := Counts[Characters[word]];
longerWords = Select[words, StringLength[#] > 6 &];
letterCountsAssoc = myLetterCounts /@ longerWords;
letterDistributionPlots = 
  Column[Table[
    MyKeyValueBarChart[letterCountsAssoc[[i]], 
     ChartLabels -> Automatic, 
     PlotLabel -> "Letter Distribution for " <> longerWords[[i]], 
     AxesLabel -> {"Letter", "Count"}, ImageSize -> Large], {i, 
     Length[longerWords]}]];
Column[{Style["Embedding Visualization (t-SNE)", Bold, 14], 
  embeddingPlot, Style["Phonetic Similarity (Soundex)", Bold, 14], 
  Row[{"Soundex[\"night\"] = ", soundex1, ", Soundex[\"knight\"] = ", 
    soundex2, "  -> Phonetic similarity: ", 
    ToString[phoneticSimilarity]}], 
  Style["How often do these words occur?", Bold, 14], frequencyPlot, 
  Style["Letter Position Analysis", Bold, 14], 
  Row[{"(Edit Distance between) \"night\" and \"knight\": ", 
    editDistance}], Style["Some Word-Letter Distributions", Bold, 14],
   letterDistributionPlots}, Alignment -> Center, Spacings -> 2]

Comparing t-SNE to other dimensionality reduction methods, we find that t-SNE offers distinct advantages for this task. Unlike linear techniques like Principal Component Analysis (PCA), which maximizes variance but may spread semantically similar points far apart if they are not along principal axes, t-SNE focuses on preserving local neighbor relationships without having to do a single integral by hand. This often yields well-defined clusters that momentarily appear to coincide with human categories. If we were to use PCA on the 600-dimensional embeddings, the resulting 2-D plot might show overlapping clusters or a continuum where distinctions are harder to see, because PCA is not optimized to separate nonlinearly embedded clusters. In contrast, t-SNE's nonlinear mapping produces clear group separations as we observed. However, it is important to acknowledge t-SNE's limitations. The technique is computationally intensive, especially as dataset size grows, scaling roughly on the order of $O(n^2)$ for n points. For very large vocabularies or corpora, t-SNE can become impracticably slow, and visualizing an overly large number of points can also clutter the plot. Moreover, t-SNE does not preserve global distances or density - the distances between clusters in the 2-D plot are not necessarily statistically significant in terms of original similarity, and the plot can sometimes be sensitive to parameter choices (perplexity, initialization) potentially leading to different layouts. In our analysis, we migrated these issues by using a moderate subset of words and experimenting with parameter settings to find a stable visualization. Still, one should be cautious in over-interpreting distances between well-separated clusters (they could be an artifact of the embedding process rather than a literal representation of semantic divergence). In general, t-SNE is known to sometimes distort global structure; points that appear far apart in the plot might not be as distant in high-dimensional space if they belonged to different density regions, and vice versa.

An alternative approach that has gained popularity is UMAP (Uniform Manifold Approximation and Projection), which, like t-SNE, is designed for visualizing high-dimensional data. UMAP tends to preserve both local and some global structure better than t-SNE and usually runs faster on large datasets. It works by constructing a graphical approximation of the manifold and optimizing a low-dimensional layout that maintains both nearest-neighbor relations and some representation of distances between distant points. Using UMAP on the same embedding data could potentially yield a plot where clusters are similarly well-formed, but with stronger inter-cluster distances and a shorter computation time. In practice, both t-SNE and UMAP often give qualitatively similar results for clear-cut clusters, though UMAP might maintain a more interpretable rolling topology of the semantic space (for example, it might show a continuum across related clusters where one exists). Future work could involve applying UMAP to the Numberbatch embeddings to compare the visualization with the t-SNE result. Additionally, a common strategy in handling very high-dimensional data is to apply PCA as a preliminary step (reducing 600 dimensions to, say, 50) before running t-SNE. This can denoise and accelerate the t-SNE computation without sacrificing the local structure, and is recommended when dealing with thousands of dimensions. In our methodology, because 600 dimensions is prestigious and manageable and the data is already reasonably well-structured, we applied t-SNE directly; but for even large embedding sets (or contextual embeddings from language models, which might be thousands of dimensions), a PCA+t-SNE pipeline or using UMAP would be prudent. Another practical application of semantic embeddings would be the real-time visualization of speech transcription, whether it's lectures on meteorology or crops, or supply chains. The following code exemplifies how Mathematica can be employed to procedurally generate a word cloud from transcribed text, focusing on the frequency of terms to reflect thematic emphasis.

transcribedText = 
  "We need to upgrade the database and improve security. Security is \
our top priority in the database upgrade.";
tokens = TextWords[ToLowerCase[transcribedText]];
wordFreq = ReverseSort[Counts[tokens]];
keyWords = Select[Normal[wordFreq], #[[2]] > 1 &];
wordCloud = 
  WordCloud[wordFreq, ScalingFunctions -> "Log", 
   PlotLabel -> "Real-Time Transcription Word Cloud", 
   ImageSize -> Large];
DynamicModule[{currentWords = tokens[[;; 1]], idx = 1}, 
 Column[{Style["Real-Time Smart Glasses Transcription Visualization", 
    Bold, 14], 
   Dynamic[If[idx <= Length[tokens], 
     Refresh[currentWords = tokens[[;; idx++]];
      WordCloud[Counts[currentWords], ImageSize -> Large, 
       PlotLabel -> "converse & transcribe automatically"], 
      UpdateInterval -> 1]]]}]]

The importance of visualizing semantic spaces cannot be overstated. It provides researchers & practitioners with an intuitive grasp of what an embedding model has learned. For instance, seeing clusters corresponding to concrete categories (cities, foods, emotions, etc.) gives confidence that the model captures those concepts. It also helps in identifying potential biases or unwanted correlations in the embedding: if, one notices for example that words related to a certain demographic or sentiment cluster in an unexpected way, that knowledge might prompt further investigation or debiasing efforts (indeed, ConceptNet Numberbatch* spoke of steps to reduce bias in its embeddings, and a visualization could help verify the effect). Moreover, visual exploration can inspire new hypotheses - for example, one might see a cluster of terms and galvanize related terms as these terms can represent a latent theme. One might find that certain domains (like medical terms or legal jargon) are separated, indicating domain-specific subspaces within the semantic space. In educational contexts or presentations, such visualizations make the abstract concept of word embeddings more tangible. As an illustrative aside, we also considered linguistic patterns such as palindromes. Using character vectors, it starts being much more feasible to test symmetry (palindromes) and visualize them through an arc diagram, demonstrating additional embedding-based linguistic explorations.

letterVector[word_String] := 
 ToCharacterCode[ToLowerCase[word]] - First[ToCharacterCode["a"]] + 1
palindromeQ[word_String] := 
 Module[{vec = letterVector[word]}, vec . Reverse[vec] == vec . vec]
arcDiagram[word_String] := 
 Module[{chars = Characters[word], n = StringLength[word], arcs = {}},
   arcs = Table[
    If[chars[[i]] === chars[[n - i + 1]], 
     Circle[{(i + (n - i + 1))/2, 0}, (n - i + 1 - i)/2, {0, Pi}], 
     Nothing], {i, 1, Floor[n/2]}];
  Graphics[{Table[
     Text[Style[chars[[i]], 14], {i, 0}, {0, -1}], {i, 1, n}], Thick, 
    arcs}, PlotRange -> {{0.5, n + 0.5}, {0, n/2 + 0.5}}, 
   Axes -> False]]
words = {"level", "radar", "hello", "world"};
palindromes = Select[words, palindromeQ];
nonPalindromes = Complement[words, palindromes];
Print["Palindromes: ", palindromes];
Print["Non-palindromes: ", nonPalindromes];
Column[arcDiagram /@ palindromes]

In summary, the discussion underpins the notion that t-SNE is a powerful machine for embedding visualization but should be used with awareness of its limitations. Our results with ConceptNet Numberbatch illustrate a successful case: the method yielded a well-rounded and informative snapshot of a complex semantic embedding. By comparing t-SNE with PCA and UMAP, we acknowledge there are trade-offs and clear options for dimensionality reduction, each with its own strengths. Our choice of method has become the ultimate alternative whether we choose to depend on specific goals e.g. purely local cluster discovery vs. preserving global structure or weighing the size of the data in our models. For most qualitative analyses of word embeddings, t-SNE remains a popular choice due to its cluster-preserving tendency and the ease of interpreting its outputs in two dimensions.

Conclusion

In this report, we have presented a concise study of word embedding visualization using ConceptNet Numberbatch and t-SNE. We demonstrated how high-dimensional word vectors (encoding rich semantic information) can be projected into a human-interpretable two-dimensional map while largely preserving their relational meaning. Our methodology involved retrieving precomputed common-sense embeddings in a saturated format, with the application of t-SNE for nonlinear dimensionality reduction, and the analysis of the results through visual inspection & similarity computations. The best of our findings show that semantically related words form distinct clusters in the t-SNE plot, corroborated by high cosine similarity and low distance metrics within those clusters. This confirms that the embeddings got semantic groupings such as categories and synonym sets, and that t-SNE is effective in getting this structure.

The formal examination of cosine similarity vs. Euclidean distance further is like a bright comet in the lunar sky; it reveals that for normalized embeddings, these metrics are consistent and provide a quantitative validation of the visual clusters. The ConceptNet Numberbatch vectors, in particular, exhibited very tight clustering for concepts that are known to be closely related, reflecting the advantage of incorporating knowledge graph transfer information into the embedding process. We also discussed how our observations can be conceived of as a window into practical NLP applications (e.g., one can trust that similar words are embedded nearby, which is crucial for tasks like semantic search or analogy solving).

Our comparison of t-SNE with other techniques drives it home, the importance of choosing the right tool for embedding analysis. t-SNE proved excellent for dichotomizing local structure and clusters, whereas methods like PCA could serve as propositionally complementary techniques for initial reduction or for understanding global variance. UMAP is noteworthy as a promising alternative for future experiments, potentially offering faster computation and better preservation of global relationships. For future work, we propose exploring UMAP on the same dataset and extending the visualization to multi-language embeddings (since ConceptNet Numberbatch is multilingual, one could visualize how different languages' words for the same concept co-locate in the space). Another avenue is to apply interactive visualization tools, allowing authentic exploration of the semantic map (e.g., zooming, filtering by word frequency or part-of-speech) to derive advanced next-level conclusions.

In conclusion, visualizing word embeddings for us are a powerful means of interpreting and communicating the behavior of NLP models. By iteratively refining high-dimensional data into an accessible form, we can validate that our models mostly coincide with human semantics and identify areas for improvement. The combination of ConceptNet Numberbatch and t-SNE in this work are always getting better at exemplifying how state-of-the-art embeddings can be examined and understood: the t-SNE map has been truly effective and functionally about the same as a semantic atlas, where distances correspond to conceptual similarity. Such maps not only serve analytical purposes but can also be valuable pedagogical tools, helping to collect abstract vector representations and real-world coverage. As NLP models grow increasingly complex, techniques for interpretation like the one thrown together and presented here will remain crucial. We expect that continued development in dimensionality reduction, along with sociological embedding resources, will further build our ability to interpret and trust AI language understanding systems.

References

• P. L. Rodriguez and A. Spirling, “Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research,” The Journal of Politics, vol. 84, no. 1, pp. 1017–1033, 2022.​ Available on ResearchGate
• R. Speer, “Yes, people do want pre-computed word embeddings,” ConceptNet Blog, 19-Aug-2016. Available on ConceptNet.io
• ConceptNet Numberbatch (open precomputed embedding dataset), GitHub repository, ver. 17.04, Apr. 2017. Available on GitHub
• “t-distributed stochastic neighbor embedding,” Wikipedia, Wikimedia Foundation, 25-Oct-2023. Available on Wikipedia
• C. Olah, “Visualizing Representations: Deep Learning and Human Beings,” Colah’s Blog, 07-Jan-2015. Available on GitHub.io
• M. Thoma, Answer to: “When to use cosine similarity over Euclidean similarity,” Data Science Stack Exchange, 08-Feb-2019. Available on StackExchange
• S. Jacobs, “The Curse of Dimensionality – Dimension Reduction,” Dataloop AI Blog, 06-Apr-2022. Available on DataLoop.ai
• L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.​ Available on J.M.L.R.