Protein Language Model Architecture
Model Architecture
Transformer-based model with 36 layers, 2560 hidden dimensions, and 32 attention heads.
Parameters:
3.1B
Training Tokens:
25B
Vocabulary:
33 (AA + special)
Performance Metrics
Training Data
Trained on UniRef50 clusters from UniProt, filtered at 50% sequence identity.
Proteins:
24M
Sequences:
138M
Residues:
25B
Protein Structure Prediction
Predicted Structure Details
87% pLDDT confidence
α-helix: 42%
β-sheet: 28%
Coil: 30%
Kinase (1-280)
SH2 (300-400)
SH3 (450-520)
Sequence Embedding Analysis
Protein Sequence
Attention Map (Layer 12, Head 8)
Residue Embedding Space (UMAP)
Model Performance Comparison
| Model | Params | TM-Score | Contact Precision | Perplexity | Speed (AA/s) |
|---|---|---|---|---|---|
| ESM-2 (3B) | 3.1B | 0.78 | 0.82 | 3.21 | 1,200 |
| ProtT5-XL | 3.4B | 0.75 | 0.79 | 3.45 | 950 |
| CARP (640M) | 640M | 0.72 | 0.76 | 3.89 | 2,400 |
| AlphaFold2 | - | 0.85 | 0.88 | - | 80 |
