How to train word2vec model to work better on producing synonym of adjective words?

I need to train word2vec model to produce synonyms of any user input adjetive words using unsupervised learning from some corpus, and ideally, the produced synonyms are also adjective words.

I have removed all punctuations, space, treated all numbers and proper noun as the same term respectively, and done lemmatization when pre-processing the corpus.

I use Skip Gram model(not sure if this is the best soluion for this problem) and generate training batch using generate_batch() function taken from TensorFlow, which generates (center_word, context_word) pairs:

def generate_batch(batch_size, num_skips, skip_window): global data_index assert batch_size % num_skips == 0 assert num_skips <= 2 * skip_window batch = np.ndarray(shape=(batch_size), dtype=np.int32) labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32) span = 2 * skip_window + 1 # [ skip_window target skip_window ] buffer = collections.deque(maxlen=span) if data_index + span > len(data): data_index = 0 buffer.extend(data[data_index:data_index + span]) data_index += span for i in range(batch_size // num_skips): target = skip_window # target label at the center of the buffer targets_to_avoid = [skip_window] for j in range(num_skips): while target in targets_to_avoid: target = random.randint(0, span - 1) targets_to_avoid.append(target) batch[i * num_skips + j] = buffer[skip_window] labels[i * num_skips + j, 0] = buffer[target] if data_index == len(data): buffer[:] = data[:span] data_index = span else: buffer.append(data[data_index]) data_index += 1 # Backtrack a little bit to avoid skipping words in the end of a batch data_index = (data_index + len(data) - span) % len(data) return batch, labels

Basically, the code for training the model is also almost the same as TensorFlow tutorial sample code. I have run it few times with different batch-size, skip-windwo-size, learning rate etc, but the result is far from being acceptable, most produced synonyms are not even adjective. So my questions are:

1. If I only generate training batch when the center word is adjective, and simply slide the window when it is not, is this approach considered as unsupervised?
2. Is there anything that needs to be redesigned in generate_batch() function? I was told that it is better to redesign this function to work better for this specific case, but I have no idea what can be improved except question 1 approach.
3. How to produce adjective synonyms? I used to think that a skip-window of size 4-7 would tend to capture semantic meanings(if I did not understand what I have learned wrong), and distinguish adjective from other type of words, but this is not what I am getting.
4. Regarding the parameters: skip-window-size, batch-size, learning-rate, is there any commonly used values to experiment?

Any advice on how to improve would be appreciated!

1 Answer

Our linguistic concept of a 'synonym' is narrower than the similarity reflected in the word-positions found by word2vec-like algorithms.

In particular, what we consider 'antonyms` generally appear as very similar in word-vectors, because the words are very similar in most aspects and the contexts in which they appear – only contrasting in some specific, topic-related way.

As a result, most-similar (nearest-neighbor) word lists tend to include synonyms, but also other related words.

Possible directions for better results would include:

• labeling words with part-of-speech info before training, and then filtering neighbor-lists to only include adjectives

• testing different context-window sizes - often small windows emphasize functional similarities ("can this word be used in the same places?") and larger windows topical similarities ("are these words used in the same discussions?")

(Unverified thought: the best adjectival synonyms might appear near the top of most-similar lists based on small-context-windows, and large-context-windows.)