In just two minutes, the Main Conference Room was emptied, leaving only Han Luyi and Jiang Songran.
In the video, Zhao Wenyuan asked, "Boss Han, is it because of the Computing Power? I just heard from Old Jiang that the progress seems a bit slow?"
Jiang Songran nodded; it really wasn't enough.
Now, even the most skilled cook can't cook without rice. No matter how good the Scheduling Architecture is, there are only so many Computing Cards in total, and it's difficult to free up more Computing Power.
Han Luyi said, "I kept you two here to talk about this. Old Jiang, could you explain the Scheduling Optimization we did last time to Wenyuan first?"
Han Luyi was referring to some new optimizations made by the Infrastructure Department in Task Scheduling before the official release of the Glutinous Rice Ball 1.0 model's API.
The problem we encountered at the time was that the Server could not determine the complexity of each inference task that came in, so it could not allocate appropriate resources. If a large task was assigned to a small Resource Pool, the task could not be completed and had to be retried; conversely, if a small task was assigned to a large Resource Pool, that much Computing Power was not actually used, which resulted in a waste of resources.
At Han Luyi's suggestion, Jiang Songran introduced the use of Glutinous Rice Ball itself for scheduling and judgment. These judgment tasks are naturally small tasks, and many failure scenarios can be avoided after the initial judgment. Therefore, under the same Computing Power, the Concurrency that the Origin Inference Cluster can support has directly increased by one-third.
However, this is one way that Jiang Songran and his team use Glutinous Rice Ball on the application side. Now, Han Luyi called Zhao Wenyuan over to see if they could make some further optimizations at the model level and improve the performance.
Unfortunately, if Origin could have the same abundant Computing Power Google or Penal, or even as much as Dingsheng, then these software-level optimizations wouldn't be urgent, or even necessary.
However, with limited Computing Power in the Hardware, we can only find solutions at the software level. Jiang Songran nodded and explained the problem that had been troubling the Infrastructure Department for a long time, "Using Glutinous Rice Ball for initial scheduling and judgment has indeed greatly improved the efficiency of resource allocation, but we have also gradually discovered two problems."
"What's the problem?" Zhao Wenyuan asked.
"It's too slow," Jiang Songran said. He had thought of this problem when Han Luyi proposed using the Tangyuan Model for routing. With tens of thousands of concurrent requests per second, if each request had to go through the Glutinous Rice Ball in advance, it would be equivalent to doubling the number of requests. Han Luyi proposed a compromise solution: for multi-step requests, only the task difficulty would be estimated for the first step.
Even so, a single estimation can take hundreds of milliseconds. Don't underestimate the short timeframe of a few hundred milliseconds; in high-concurrency scenarios, a delay of a few hundred milliseconds can trigger an avalanche effect—the delay causes request timeouts, which in turn trigger retries, increasing invalid requests and potentially rendering the entire service unavailable.
"President Zhao, Glutinous Rice Ball is a large-parameter model. Even with a sparse architecture, the time taken, which is several hundred milliseconds, is something the scheduling layer can't handle. Slowness is the first problem, and there's a second problem caused by that slowness. Although Glutinous Rice Ball has high accuracy for task prediction, its latency is too long. We only dare to add it at the beginning of the Task Chain, and afterwards we can only rely on the first prediction result. However, some user requests suddenly increase in complexity during the interaction, which means that using the first calculation result will be inaccurate," Jiang Songran said.
Han Luyi, who was standing next to him, understood that although Jiang Songran had raised two issues, the core problem was actually the slow speed.
Because of the slow speed, it is not advisable to call the function every time it is scheduled, which leads to precision drift.
"Wenyuan, this is why I specifically called you here," Han Luyi said. "We talked about this last time, about making a lighter version of Glutinous Rice Ball. Do you think that could solve the problem?"
After the performance of the professional Flagship Version has been verified, it is common practice in the industry to release a lightweight version with smaller parameters, slightly weaker capabilities, but much lower cost.
Generally, these lightweight versions are trained by Distillation the original flagship model. Distillation refers to inputting some challenging problems into the flagship model, letting it provide the answers, and then using these problems and answers as a training set to retrain the Lightweight Model. This allows the Lightweight Model to have 70% to 80% of the capabilities of the flagship model at a relatively low cost, which is a cost-effective approach.
Of course, you don't necessarily have to Distillation your own flagship model; you can use the same method to Distillation other companies' flagship models as well. Since the release of Glutinous Rice Ball API, Origin Group has discovered and captured numerous attempts to Distillation Glutinous Rice Ball 1.0.
"Yes, many of the requests are coming from the United States." After listening to Han Luyi's words, Zhao Wenyuan pondered for a moment.
Lightweight Model do help in this regard. We could even consider using a very small model, like the original Glutinous Rice Ball 0.1 model, to predict task complexity—as long as the prediction accuracy doesn't drop too drastically.
But--
Zhao Wenyuan began by asking, "Have you ever heard of System 1 and System 2?"
Han Luyi and Jiang Songran exchanged a glance and nodded simultaneously.
The concepts of System 1 and System 2 2 originated from the research of Nobel Prize in Economics Daniel Kahneman, who elaborated on them in detail in his popular science bestseller, Thinking, Fast and Slow.
The human brain has two systems when thinking. One operates extremely fast and is very good at making quick judgments, but because it does not stimulate deep thinking, it is also more easily disturbed by noise. This is called System 1. The other mode is called System 2, which requires people to actively spend energy to think carefully, takes longer, and ultimately arrives at more accurate conclusions.
This book is certainly not a work in the field of computer science, but it is popular and famous enough that both of them happened to have read it.
In fact, Zhao Wenyuan and his fellow researchers were able to propose the concept of the Chain of Thought (CT, Cain F TT) precisely because they drew inspiration from System 2 of the human brain.
I just don't know why he suddenly brought this up.
Han Luyi said, "What does this have to do with what we're discussing? Are you saying we can use System 1 to quickly estimate the complexity of a task?"
At this point, Han Luyi suddenly realized something and asked, "Isn't our discussion about using Lightweight Model for routing based on the System 1 approach?" Zhao Wenyuan shook his head and said, "Boss Han, you're right, but it's not enough. Do you know why System 1 is faster than System 2 in the human brain's decision-making system?"