The process of drug development is costly, time-consuming, and high-risk, typically requiring 10 to 15 years and investments up to $2 billion, with most candidates failing clinical trials. Successful therapies must satisfy stringent criteria such as precise target interactions, non-toxicity, and appropriate pharmacokinetics. Current AI models, while specialized in specific tasks within this process, may have limited scope that affects their performance.
To address this issue, the Therapeutics Data Commons (TDC) offers datasets that assist AI models in predicting drug attributes. However, these models typically operate independently. Large Language Models (LLMs) with multitasking capabilities hold the potential to enhance therapeutic methods by learning across multiple tasks using a unified approach.
Transformer-based LLMs have made significant strides in natural language processing. By undergoing self-supervised learning on large datasets, these models perform exceptionally across diverse tasks. Recent studies indicate that LLMs can also handle various tasks, including regression, utilizing parameterized text representations.
In therapeutics, specialized models like Graph Neural Networks (GNNs) represent molecules as graphs for applications such as drug discovery. Protein and nucleic acid sequences are also encoded to predict attributes like binding and structure. As LLMs find broader applications in biology and chemistry, models like LlaSMol and protein-specific models have shown promising results in tasks such as drug synthesis and protein engineering.
To advance this field, researchers from Google Research and Google DeepMind introduced Tx-LLM. This finely-tuned, general-purpose LLM is designed for diverse therapeutic tasks, building upon PaLM-2. Tx-LLM is trained on 709 datasets encompassing 66 functions within the drug discovery pipeline, utilizing a set of weights to handle various chemical and biological entities, including small molecules, proteins, and nucleic acids.
To train Tx-LLM, researchers compiled a data collection named TxT. This collection comprises 709 drug discovery datasets from the TDC database, spanning 66 tasks. Each dataset includes four components: instructions, background, questions, and answers, suitable for instruction tuning. These tasks encompass binary classification, regression, and generation, utilizing representations such as SMILES strings for molecules and amino acid sequences for proteins.
Tx-LLM demonstrates robust performance on the TDC datasets. It surpasses or matches state-of-the-art results in 43 out of 66 tasks. Additionally, it outperforms state-of-the-art models on 22 datasets and achieves near state-of-the-art performance on another 21 datasets. Notably, Tx-LLM excels in datasets that combine SMILES molecular strings with text features like disease or cell line descriptions, likely due to its pretraining on textual data. However, in datasets relying solely on SMILES strings, the model's performance is subpar, with graph-based models proving more effective.
Importantly, Tx-LLM is the first LLM trained on the diverse TDC dataset that includes molecules, proteins, cells, and diseases. Training alongside non-small molecule datasets, such as proteins, enhances performance on small molecule tasks. Although general-purpose LLMs face challenges in specialized chemical tasks, Tx-LLM excels in regression and, in some cases, even surpasses state-of-the-art models.
The model exhibits end-to-end drug development potential, ranging from gene identification to clinical trials. However, Tx-LLM remains in the research phase, with limitations in natural language instructions and prediction accuracy. To further expand its applicability, researchers need to refine and validate the model. As technology continues to advance, Tx-LLM is expected to bring more innovation and breakthroughs to the field of drug development in the future.
