The Tangyuan Model has been remotely tested on this batch of Ximing L100s manufactured by Yunyin. Operator Adaptation has been completed, and there are no problems with single cards and small clusters.
Han Luyi and Zhao Wenyuan came here this time to conduct the final acceptance test, the most crucial of which was the Stress Test.
A Stress Test involves sending a large number of requests at once to see how many people the current architecture can support simultaneously.
According to the plan, Riding the Wind's reasoning data must be migrated within two weeks. This timeframe is very tight, so there can be no mistakes.
Riding the Wind currently has over two million registered users and two hundred thousand Daily Active Users. This usage alone is almost enough to fully utilize the Computing Power of these five hundred graphics cards.
Fortunately, the two thousand cards sent by Dingsheng are almost here, giving us a direction for the next stage of capacity expansion. However, Dingsheng's cards are from another domestic manufacturer, and their Operators and instruction sets do not overlap much with L100. This means that if we want to use them, all the work done on L100 will have to be redone.
This highlights the value of Nvidia's CUDA system. Once the ecosystem is established, the migration costs become incredibly high.
Han Luyi remained calm. Problems were normal; with him there, everything could be resolved.
What is the current data?
"The current number of requests per second (QPS) is sixteen."
Han Luyi eyebrow. Before they came, they had already done the calculations. To support 200,000 Daily Active Users, each user would send more than one request per day, possibly dozens.
The minimum number of requests per second must be at least thirty.
The current architecture can theoretically reach 50, and even considering scheduling overhead, communication overhead, and framework overhead, it can conservatively reach over 40.
It's only sixteen now, half the minimum standard.
As Zhao Wenyuan spoke, he opened the background view, monitoring curves, and output logs on the desktop.
Has Jiang Songran?
"I just sent him a message, and he's still reading it."
Jiang Songran was still in Haicheng. When he joined the company, Han Luyi had promised that he would never be allowed to go on business trips, so even though it was such a big matter, Han Luyi did not bring him along.
"Let's turn on a video call and watch it together, hurry up," Han Luyi said.
Liu Huo has already made a video call on his phone.
Jiang Songran in the picture also looks busy; his hair is messy, and the background is his bedroom. He hasn't had a moment's rest since receiving Zhao Wenyuan's message that morning.
"Boss Han, we've checked the GPU utilization; it's basically at full capacity. The problem isn't there."
It seems like every card is working, but the output is slow.
Jiang Songran and Zhao Wenyuan then went to check the driver logs together.
From a software perspective, everything is normal.
Finally, Zhao Wenyuan brought up all the model configurations and looked at them with Jiang Songran.
The configuration is correct.
The model segmentation is fine.
The parallel strategy is fine; in fact, the probability of finding a problem with the naked eye is even lower.
But sometimes production accidents are just that difficult to investigate, like the blind men and the elephant.
All the tools, various monitoring data, print logs, heatmaps, and scatter plots are designed to help you understand this phenomenon more clearly.
However, humans are not computers; humans cannot see the thought process of a computer, but can only imagine it.
When all other methods fail, simply taking out the code and configuration files and imagining the results can sometimes be effective.
It might even be more effective for the most talented person. After all, geniuses cannot be measured by common sense.
It's said that John von Neumann strongly opposed the use of high-level programming languages (like English-based programming languages). He believed that using assembly language was the simplest way to communicate with computers. There's even a legend that he could directly read code consisting only of 0s and 1s.
In fact, Jiang Songran and Zhao Wenyuan had already investigated the relevant information before, and now they did it again in front of Han Luyi, but they did not find anything new.
Han Luyi opened his Sight, his gaze fixed on the computer screen, and then he glanced around the Data Center.
With the upgrade of his Sight, Han Luyi was no longer troubled by the bandwidth of receiving information. As soon as his Sight was activated, he collected all the information around him.
Then he stood up, walked to the preparation area next to him, and put on the equipment for the Data Center.
Jiang Songran stating his deduction. "Single machines are fine, small clusters are fine, but the large clusters can't reach the required capacity. It must be at the cluster level, and it's something that large clusters have but small clusters don't."
Meanwhile, Zhao Wenyuan had already noticed Han Luyi's actions.
"Boss Han." Zhao Wenyuan stood up and chased after him.
Han Luyi was already dressed. He glanced at Zhao Wenyuan and said, "You put it on too, come and take a look."
Zhao Wenyuan quickly put on shoe covers, anti-static clothing, and anti-static wrist straps, and followed Han Luyi into the Data Center.
Liu Huo and Zhang Biao stayed outside and didn't go in.
The Data Center has extremely high requirements for static electricity prevention, and mobile phones are not allowed inside. Even if you do bring one in, it's basically unusable. This is because the Data Center itself has fire and shock protection requirements, and the large number of metal Server Cabinet inside make it a natural Faraday cage, so mobile phones simply cannot be used.
At this moment, in Zhao Wenyuan's eyes, all he saw were rows of cold metal Server Cabinet, each one looking exactly the same.
But in Han Luyi's eyes, the scene before him was completely different. The connections between Server Cabinet the flow of data were presented to him in different colors, like flowing water.
GPUs, Server, switches, fiber optic cables, ports, links—all of these things form a complex map that is beyond human comprehension.
Even if this image were shown to other people, they wouldn't be able to understand it.
It's too complicated.
However, Han Luyi understood it without any difficulty.
He pinpointed the source of the data congestion almost instantly.
There were communication problems between several inference groups. The nodes were not physically connected, but were logically grouped together. Data traveled from one place to another by a long detour and then back again by a long detour.
What may seem like a small detail can be amplified repeatedly in the parallel communication of Large Model inference.
Han Luyi spoke up, "It's not a problem with the model."
Zhao Wenyuan was taken aback. "Boss Han, you've already resolved it? So quickly?"
Han Luyi glanced back at Zhao Wenyuan and realized that he didn't seem to be asking any questions.
He then used a walkie-talkie to contact the outside world.
"The problem lies in the network topology," Han Luyi said.
The sound was transmitted to Jiang Songran's ears via walkie-talkie and then via video. He immediately started searching for the original network topology diagram on his computer.
He designed this himself to maximize the efficiency of the cluster.
Zhang Biao quickly changed his clothes and came in with the printed topology diagram.
A quick comparison between the diagram and the actual product reveals the problem.
Jiang Songran's original design was to reduce communication latency between inference groups by placing several key nodes in the same low-latency domain as much as possible.
However, when Yunyin actually delivered, it was not deployed exactly according to this diagram.
They used their own internal, more standardized, and easier-to-maintain methods.
For ordinary Server that are not sensitive to communication latency, this connection method is perfectly fine. It's a small sacrifice made for maintainability, and everyone can tolerate it.
However, for Large Model applications, this small delay can be magnified many times over.
Single-point testing works fine, and so does whole-machine testing, because the problem lies in the communication between Server.
Jiang Songran's deduction was actually correct. Given more time, and with Zhao Wenyuan's on-site investigation, they should be able to find out the truth.
It's just that it wasn't as fast Han Luyi's.
Zhao Wenyuan looked at the two drawings, and his expression turned very ugly.
He spent the entire night investigating everything – GPU, drivers, configurations, Operators, services, scheduling – but the problem turned out to be outside of any software aspect he could see.
And then there's the cabling in the Data Center.
"Thank goodness Boss Han came in person," Zhao Wenyuan thought to himself.
Jiang Songran swallowed hard on the other end of the video call. How did Boss Han figure that out? Hadn't he even seen the original network topology diagram before?
Zhang Biao quickly called the on-site Operations Engineer over, and they confirmed the problem through inter-node speed tests.
After the investigation was completed, the repairs became much easier. In less than ten minutes, Liu Ran arrived with two other people.
"Boss Han, Boss Han! I'm so sorry," Liu Ran said breathlessly. "I didn't know you were already here, otherwise I would have accompanied you the whole time. I heard you've encountered some problems here?"
Zhang Biao stepped forward and explained.
Liu Ran's expression turned somewhat grim.
It's obvious that Yunyin made a mistake. She was supposed to be trying to please the client, so why did she fail at the very first thing?
"Boss Han, it's mine." Liu Ran quickly said, "I'll have someone come right now and work overtime to get it replaced for you."
Han Luyi has State Grid Power Investment behind him, a resource he can't even curry favor with.
Han Luyi didn't care about these superficial formalities; all he wanted to see was how many requests per second could be achieved after the repair.
Now is the time to race against time; if you can achieve a score of 30 or higher, that's a victory.
The people Liu Ran brought were working overtime and giving it their all. Several hours later, the cable was reconnected according to the topology diagram.
Zhao Wenyuan ran the load testing program again.
The number of requests per second is slowly increasing.
Eight.
Sixteen.
27.
Thirty-nine.
51.
It finally stopped at fifty-one, just a little short of the theoretical limit.
However, theory is one thing, but to actually achieve this level is already quite impressive.
This is the value of the link designed by Jiang Songran.
Upon seeing this result, everyone present breathed a sigh of relief, including Liu Ran, who had been waiting to see the outcome.
Han Luyi looked at the rows of Server indicator lights that were lit up in the Data Center.
Glutinous Rice Ball, the first Large Model he created, is finally running on its own cluster.
From this moment on, he has his own universal Large Model, produced by Sight from start to finish.