How to getting word embedding from the trained model of TransCoder?

[sushantkumar007007]

I am trying to extract a words embedding of the various tokenized (.tok) files. I have preprocessed the various dataset using preprocessing pipeline suggested in the TransCoder. I have also trained the model and can also used pretrained (TransCoder) to extract embedding matrix and embedding vectors of various tokens of various tokenized file.
Authors have plotted t-SNE visualization of a cross-lingual token embeddings. They obtained by encoding programming language tokens into TransCoder's lookup table.
Can authors explain how you did that? I also want to extract embedding of these tokens.

[showjim]

Hi Kumar,
I took the output of variable "enc1" in translate.py as the embedding to do the t-sne visualization.
But I am not sure the authors did in the same way.

[showjim]

Hi Kumar,

1. I took the output of encoder from the model. In terms of code, it is the variable "enc1" in translate.py.
2. the input is a cpp/py sample I find from website and one I create by hand.
3. set a debug point in the line on "enc1" line in translate.py, save the tensor output with cpp/py sample code respectively, along with the token list as the labels in the figure I sent.
4. the t-sne method I use the library from sklearn and plot them with scatter from matplotlib. I think you can manage the rest yourself, it is not very complex.

Good luck!
MOST IMPORTANT: this method is my approach to get the t-sne visualization, I do not sure about it is the same as anthors', or it may be not in the right way.

[sushantkumar007007]

Hi Kumar,

1. I took the output of encoder from the model. In terms of code, it is the variable "enc1" in translate.py.
2. the input is a cpp/py sample I find from website and one I create by hand.
3. set a debug point in the line on "enc1" line in translate.py, save the tensor output with cpp/py sample code respectively, along with the token list as the labels in the figure I sent.
4. the t-sne method I use the library from sklearn and plot them with scatter from matplotlib. I think you can manage the rest yourself, it is not very complex.

Good luck! MOST IMPORTANT: this method is my approach to get the t-sne visualization, I do not sure about it is the same as anthors', or it may be not in the right way.

Why the embedding vector generated by using enc1 variable are of different sizes for different program, size of the 3d vector should be same for different program ?

[showjim]

Hi Kumar,

Since the size of this Tensor determined by the input token size and model embedding shape.

[sushantkumar007007]

Hi Kumar,

Since the size of this Tensor determined by the input token size and model embedding shape.

Then how can be compare the two different program of two different embeddding shape?
is the embedding vector generated by enc1 is of last layer or last few layers? because I am trying to extract the embedding vector from last few layers of Transcoder.