Finally, the day has arrived: the black box becomes transparent, but can Transformers really replace manual feature engineering?

According to AIMPACT, on May 15th, X open-sourced the "Recommended for You" information flow algorithm on Github, providing an end-to-end inference pipeline. The system retrieves followed content through Thunder, searches for relevant information across the entire network with Phoenix, and then scores and sorts interactions such as likes, replies, shares, and clicks using a modified Transformer based on Grok-1. Additionally, it introduces ad injection, a content understanding classifier (for spam detection and content policy enforcement), and hydrator data augmentation, claiming to completely abandon manual features and rely solely on the Transformer to determine relevance.