In recent research, a team of researchers from Mistral AI has presented Mixtral 8x7B, a language model based on the new Sparse Mixture of Experts (SMoE) model with open weights. Licensed under the Apache 2.0 license and as a sparse network of a mixture of experts, Mixtral serves just as a decoder model. The team has shared that Mixtral's feedforward block has been chosen from eight different parameter groups. Every layer and token has two parameter groups, called experts, that are dynamically selected by the router network to process the token and combine their results additively. As only a portion