[Music]
uh hello everyone so H thank
[Applause]
you first of all thank you for coming
today to my presentation so today I hope
to teach you about embeddings what they
are how they work and how you can use
them if I've done my job right I hope to
make you just as excited by embeddings
as I
am so to start off we're going to go
through some of the pre 2013 methods for
embeddings and work our way to state of
the the art neural
networks so first of all what are
embeddings fundamentally embeddings are
a way to capture information and
represent it on a computer this
information is typically language or
images but can really be anything for
the purposes of this talk I'm just going
to be covering
language so when we say we're going to
embed a word that means to take a word
and embed it into a vector space where
similar words are closer
together this in essence captures
semantic meaning and turns it into a
vector of numbers that we can then use
computers
with so how can computers understand
language well maybe we can try and
Define a giant dictionary full of all
the English words and their
definitions but that doesn't really
capture any sort of semantic
meaning well maybe we can then order the
dictionary so that similar words are
closer
together but what happens when the word
changes depending on the context for
example if I'm talking about bugs
am I talking about bugs insects or bugs
computer
errors maybe then we could try and
hardcode all the different types of um
context but I don't think any of us
really want to be um writing over you
know a million if
statements so it turns out the solution
to capturing word embeddings and
capturing word meaning begins outside
the world of computing and instead in
the world of
linguistics John rert fth proposed the
idea that words with similar meanings
will appear in similar
contexts this is called the
distributional hypothesis and is best
described by his 1957 quote you shall
know a word by the company it
keeps the central idea here is that
words with similar meanings will appear
in similar contexts this means that you
can derive a word's meaning by the
context in which it is
used this is the basis uh for the way
modern computer science captures
semantic meaning in modern natural
language
processing so prior to the Advent and
subsequent domination of neuron networks
algorithms in the early
2010s natural language processing relied
heavily on rule based systems and
symbolic systems much like the um ideas
we played around in the slides
before but there's actually another
class of method that were class of
methods that were called the statistical
methods many of these statistical
methods are based on word co-occurrence
and bag of word approaches as a form of
embedding words based on the as a form
of embedding words and text
respectively so for example we can use
the um word co-occurrence matrices so we
can use the frequency of words within a
context window of a certain word to
create embeddings a context window is
just the words in front and behind the
word that you're looking at and can vary
in length depending on the model you're
using so when you count all the times a
word occurs within another words context
window you get what is called a
co-occurrence matrix so this one for
example along the top here I've just
taken a subset of the words and likewise
down the bottom but in reality you're
going to get a matrix that's you know
the same size as it's as long as it is
uh wide so along the top we have a
subset uh and this will often be hundred
you thousands to tens of thousands of
words long depending on how big your
documents are so the word digital for
example has had computer the word the
word computer occur
1,670 times within its context window
and the word data occur
1,683 this makes it very um dissimilar
to words like cherry and strawberry
which have had very few occurrences of
those words but are high in words like
pie and sugar
if you just take two Dimensions out of
this Matrix you can actually use it to
kind of um plot the words so if we just
use the dimensions computer and data we
can see that the words digital and
information are actually very close
together and often times when we're
measuring a sort of closeness with
vectors we're not using how far away the
the dots are but instead we're measuring
the the distance of the angle between
them so the angle between these two
words are very low so this is called
cosine similarity and it would be pretty
high in this
case an adjacent concept to this is bag
of words to create document embeddings
we use a bag of Words which means we can
encode our documents to be a account of
all the words it contains so if we have
these three documents this would be our
vocabulary which is just all the unique
words in our documents and these would
be the word
frequencies what we are left with are
document embeddings and this is a useful
way for comparing documents based on the
words that occur within them but it
doesn't make any attempt to compare the
documents based on the meaning of the
words within the
documents
so both of these um family of
statistical methods use the frequency of
occurrences in order to derive some
notion of similarity the idea here is
that similar words will have a similar
number of Co concurrences and likewise
similar documents will have a similar
number of word
frequencies these methods produce large
dimensions because often the size of the
vocabulary is how long your dimensions
are going to be uh these methods also
produce what are called sparse
embeddings or sparse vectors because
they mostly contain zeros because most
words don't occur next to each
other so by the early 2010s neural
networks were experiencing a bit of a
Renaissance in computer science driven
by advances in compute power and neural
networks demonstrating how good they are
at classifying whether an image is a cat
dog or a
muffin so the introduction of word to
VEC in 2013 by Thomas mikeloff and his
team at Google combin natural language
processing and neuron
networks instead of just counting the
word neighbors like we pre like we did
in previous methods we feed the neuron
Network um examples of words that are
neighbors and examples of words that are
not neighbors and then we force it to
predict
which is which for the predictions to be
accurate the neuron network is forced to
capture some sort of deeper
contextual deeper semantic understanding
of the words so that it can actually
make accurate predictions when it we
force it to
predict so the training data for the
word Toc model is nothing complex all
you need to do is process all your data
to create a bunch of word pairs you can
do this by looking at the words that
occur within the context window of a
specific word and labing labeling them
as neighbors so for example hard has a
context window with the word
implementation in it and we'd want the
neuron Network to predict that those two
words are neighbors we do the same thing
for is until we get a list for all that
for that for that word and then we
iterate to the next word so we go to two
and we do the same thing for all the
words in all of our documents until
eventually we eventually we get a giant
list of all the words and their
neighbors equally as important
is that you
um randomly sample words that don't
occur together and label them as non-
neighbors if you don't do this the ne
Network could get away with um with
labeling everything as or predicting
everything as neighbors and it would be
100%
accurate before we move on to discuss
more about the word algorithm we need to
quickly do a bit of a primer on neuron
Network
so neuron networks are structured in a
way that they take data as input and
transform it via its layers into a
desired output so for example if you
were to train a cat classifying neural
network the inputs would be all the um
pixel values of the image and then the
values would then be transformed through
the layers into maybe a binary output
that represents whether the image is a
cat or
not so essentially neural network are
really good at learning patterns and
much like a child learns it needs to be
exposed to examples and feedback to
become good at recognizing
something unlike a child a neuron
Network stores its learnings in matrices
that are often um referred to as its
weights so in this diagram here these
would be the neuron Network's weights
the weights are responsible for
propagating the data from one set of
layers to the next
layer and each time you show a neur
network an example it adjusts its
weights to minimize its prediction
errors which then improves its ability
to make correct predictions on unseen
data just a bit of a clarification this
is a major simplification but it should
give you just enough inition to work
with so to train a model using the word
Toc algorithm we use a neural network to
predict the most likely context words
for each word in our vocabulary so the
diagram we showed before the it's very
similar to this neuron Network except
I've just um blown up blown up the
weights and made them into squares for
it to be easier for us to
visualize so the first set of Weights in
the word algorithm is a v byn Matrix
where each row corresponds to a word
from the V dimensional
vocabulary the N columns contain
flirting Point numbers that are used to
represent the word's meaning they are
typically around 300 columns for word
models but there's no limit and they may
they may range between you know 50 to a
th000 where smaller dimensions are more
efficient
computationally but they do tend to
sacrifice their semantic resolution much
like you would when you limit the amount
of pixels you can represent an image
with so these particular weights in the
word Matrix are actually referred to as
its embedding Matrix and as it's
learning to maximize its prediction
accuracy the neuronet is slowly nudging
the numbers in there so that similar
words
have similar
numbers as you may have guessed we're
not actually interested in the
predictive ability of the neuron network
but rather the embedding Matrix that is
created as a byproduct of
it once training is complete the Matrix
is harvested which is what we use for
our word em word
embeddings the trained embedding Matrix
provides dense representations of Words
which means every number in the word
embedding is Meaningful and contributes
to its representation unlike the sparse
predecessors we just talked
about these dense embeddings contain
really interesting properties that
actually are the best way to illustrate
them I think is to compress your 300
Dimensions down to two so you can see
them on a graph so if we look at the
first property for example we can see
that embeddings these embeddings are
fantastic at capturing meaning similar
words like fantastic awesome and amazing
are all on the same location whereas
terrible awful and Dreadful are all in
their own group and unrelated words like
bug are all the way by themselves the
second thing that I find really
interesting is the um the ability for
embeddings to preserve semantic
relationships like tents so here we can
see flew is to
Flying as ran is to running and the cool
thing is no one trained the neuron
Network to do this just by show forcing
it to predict its word neighbors it was
able to capture these relationships by
itself
it also captures real world information
so cra is to Australia as Beijing is to
China and a consequence of these
relationships that are captured is the
ability to perform meaning meaningful
Vector arithmetic so if we take the
embedding for Cambra and subtract the
vector for Australia and add the vector
for China you get a vector that is
approximately the same as the vector for
Beijing and to prove this I've used the
most similar function in the uh model
I've been using and I did the
computation in there and you can see the
top five most similar vectors you're
getting Beijing right at the top
followed by China CRA Chinese and then a
typo of
Beijing so up until this point our
embeddings have been static
representations of words pre 2013 the
statistical methods used static vectors
that were long and sparse they were long
because every Dimension literally
represented the count of a specific
word word toc on the other hand has uses
the Learned representations of words
from a neural network meaning that each
dimmension while not directly
interpretable uh is used to represent
the meaning of the words this makes word
tobec much more efficient and effective
at capturing word meanings because it's
the dimensions in it being a lot smaller
but it's not perfect there's a problem
word Toc produces static word
embeddings therefore words with multiple
meanings end up being encoded as some
sort of awkward average between all the
context in which it appears so for
example this is a general model and so
you can see the word bug is stuck stuck
between insect and error SLG glitch so
this model was probably changed on
general information but if you trange
your model on say biological text You'
expect to see bug right over by the word
insect and likewise computer science
textbooks or so on you'd expect to see
bug right next to all the error and
glitch sort of words so what's clear is
that for the best embedding of a word
you really need to have an embedding for
each context that it can appear
in enter Bert in 2018 by Jacob Devin and
colleagues at Google which prod bir
produces Dynamic embeddings making them
much more effective at capturing meaning
ber uses a deep learning uh a deep
neural network based on the Transformer
architector and deep in this context
literally just means that it has more
layers unlike the word to one unlike the
word to neuron Network which just just
had
one so a bit of a tangent and I don't
expect you to take all this in but this
is what the Transformer architecture
looks like on the left you have the
encoder and on the right you have the
decoder
so the Transformer architecture is
foundational to generative models like
chat GPT 4 and representative models
like
Bert Bert uses the encoder part of the
Transformer which takes words and
outputs
embeddings gbt on the other hand gbt 4
utilizes the decoder side which takes
embeddings and outputs words both the
encoder and the decoder use what is
called an attention mechanism which
allows them to focus on Words more
important to the context on the word
more important to the context of the
word than generic words like is thee and
end so this allows the encoder to create
high quality Dynamic
embeddings these AR this Transformer
architecture is behind the recent
explosion in AI capabilities and it's
quite literally in their names so chat
gp4 or GPT 4 stands for generative
pre-trained Transformer 4 but on the
other hand then stands for bidirectional
and cod of representations from
Transformers so bir's a bit different
but it's also much the same so B trains
its equivalent to word to matric ber
trains its equivalent to the word to
embedding matrix by predicting missing
words in a
sentence so explicit is better than
implicit it should it should predict
better if we blank out the word better
and likewise sequences of words like my
dog is Cur he likes playing we would
want to force the model to predict that
those are
sequential because Bert produces
contextual word embeddings it takes
sentences as inputs rather than single
words and then produces an embedding for
each word in the
sentence rather than harvesting the
embedding Matrix like we did with word
DEC we keep the neuronet intact so that
we can continue to produce Dynamic word
embeddings so now that we have context
sensitive word embedding models given a
word a model will produce a vector that
is specific to the word and its
context but what if we want to
understand and embed sentences
paragraphs or even entire
documents well it turns out the easiest
way to do this is just to average all of
the output uh word embeddings that our B
model gave us and this will give you a
single Vector that represents the entire
meaning of your text
the reason we can't use models like word
to V to do this is because the word
embeddings there are static and do not
change depending on the context so
sentences like the cat chased the dog
and the dog chased the cat would be
encoded identically despite the
sentences having different
meaning a bit more on this so static and
beddings produce the same vectors for a
word regardless of the context it's used
in so my code has a bug there's a bug in
my soup would produce the same word um
embedding regardless of their context
whereas a model like bird would um
produce vectors for a word that are then
Modified by the presence of the
neighboring words so in this case the
role of this is in int this is the role
of the attention mechanism which
selectively modifies the vector based on
the relevance of surrounding words for
example my code has a bug the the
presence of the word code would be be
highly relevant to the context of the
word bug and therefore the embedding for
the word bug would be modified so that
it's closer to the word code likewise
for there's a bug in my sup and as you
can see the context on these words bird
is using the context on both sides of
the word this is where it gets the this
is why it's called bir
directional so before we move on to a
quick demo a quick recap on the methods
we've discussed
pre 2013 methods use static and sparse
vectors to represent words whereas word
to V algorithms use static but dense
embeddings this shift away towards dense
embeddings hugely prized the method of
embedding words and it became the go-to
method until B was released in 2018
which provides Dynamic densely
represented word vectors but that
doesn't mean word to can't still be used
and it often still is since it's a lot
more efficient
but it's not used where context is
highly important to what you're wanting
to
do so we're going to do a quick demo
using the descriptions of the Pyon
presentations to see which ones are most
similar we will use the library called
sentence Transformers or esbert which
provides great sentence embedding models
trained using
Bert the first thing we're going to do
is we're going to load in all the Pyon
session titles in their descriptions and
we're also going to load Ascension
transform Transformer model and a cross
encoder model and I'll talk more about
the cross encoder model
later so using the sentence Transformer
model we calculate all of the sentence
Bings for all of our descriptions all in
one line of code and then like we talked
about before we can compress these
embeddings into just two dimensions and
we can plot
them so this plot is pretty cool there's
not really any strong clusters except
for maybe uh the top left we have all
the education and you know teaching
related speeches and in the top right we
have all the D Jango and database
related speeches but in the middle yes
there is clusters there but they're not
very strong and which probably goes to
show that the pon team did a great job
at choosing a very diverse set of
topics so let's say we want to find the
most similar speeches to the speech
teaching digital Technologies in
Australian schools for Python and the
cooker Berry so we get the index for it
here and then we now use P torch to
compare the distance between this is
using coine similarity we compare the
distance between the target embedding
and all of the other embeddings for the
speeches and see and we want to find the
most the top 10 closest embeddings to
our Target so this is just like before
when we had two dimensional angles now
we have 300 dimensional angles and we're
calculating which ones are
closer so under the hood it looks a bit
like this where we take in two espert or
we take in two embeddings and we put
them through a cosine similarity
function and that will return a number
from negative 1 to one where negative 1
means that the embeddings are you
completely in opposite directions and
one means that they're in the same
direction or they're
parallel it seems to have done a pretty
good job because the these speeches are
all mostly related to education and
teaching and all that sort of stuff and
schools but one thing we noticed is
micropython I don't think is about
specifically about education in schools
so we can probably do a bit better here
and that's where the cross encoder comes
in so using just the top 10 speeches We
compare our Target speech
description with one at a time with the
top 10 uh Speech
descriptions what this looks like
is this is what the cross en code is for
and it's it's also a bur model but
instead of producing word embeddings
it's been optimized to produce a score
representing the similarity of its
inputs it does this by comparing each of
the embeddings at a word level rather
than measuring the distance between the
averaged word embedding of the model
prior this makes it much more precise in
judging what text is similar however
it's extremely less efficient and that's
why it's often done as a second step to
kind of rerank the results that the
first step did which is much more
efficient
and it saves you from reranking a much
larger set of
documents so if we go to then these
results we can see that for example
micropython has been reranked all the
way down to the bottom which goes to
show that probably wasn't related to
education so I've actually got a live
web app version of this which will let
you select your speech and see the most
similar speeches and I'll make that
available right at the
end um so embeddings can be used for a
whole range of tasks and I want to just
go through a few more cool applications
this one's probably my favorite it's you
can actually see uh how the meaning of
words have changed throughout time so if
you were to train like a word Toc model
on documents from say the 1850s you'd
see that the word broadcast is Ed in a
context that's very similar to farming
and sewing seeds and you know
broadcasting all the seeds down to the
1900s broadcast becomes about
newspapers down to the 1990s broadcast
quickly becomes about television radio
and even the
BBC another one that I find very cool is
awful in the 1850s awful wasn't a good
word you know full of all amazing
Majestic down to the 1900s and 1990s all
four very becomes very quickly becomes a
negative word meaning terrible or
horrible another application is
visualizing embeddings I know I've
spoken about this before but it's
actually really really good for
explorers atory data analysis and if the
graph you're using is interactive you
can get a quick idea of what makes up
all the Clusters and get a real
understanding of your data if you
haven't seen it before so in this
example you can see that the
non-fiction uh genre of books are on the
complete opposite side of the science
fiction which intuitively makes
sense the final application or not the
final one but one of the bigger ones is
Vector databases so Vector databases
Essen store all the vectors for your
embeddings which allow you to query them
using embeddings as well so for example
if you wanted to make a document search
you could then embed your query and find
the most similar documents to your query
so this is actually used as far as I'm
aware in Google and Yahoo searches as
well for sort of more of a semantic
understanding of what you're Googling
and a common use case in modern
applications is generative llms so if
you hook your generative llm generally
isn't going to be trained on your data
so to give it access to data that's it's
not being trained on you can simply
input the user's prompt into the vect
database retrieve the most relevant
chunks of text or documents and then
append those to your um prompt and then
feed the prompt and the relevant chunks
into your llm llm and that will then um
produce relevant information and that's
actually called retrieval augmented
generation and I believe there's a
presentation or a workshop on that this
Monday so that' be good so that's really
good for chat
Bots another way you can do it is you
can optimize your sentence and betting
models to kind of fit your definition of
similarity so out of the box this uh
model is doing what it should be all all
the product descriptions are close
together but if you then find unun it
you can find unit to be for example um
putting negative and positive uh
sentiment closer together so that's all
the applications I've brought up today
but it really is up to your imagination
with the
Tings so as promised here's a QR code or
a link if you prefer links to the
interactive demo I haven't tested it
really um on anyone except myself so
we'll see how it goes but um uh there's
probably no time for questions but any
questions please feel free to come up
and see me outside thank you
[Applause]