Skip to main content
  • Home
  • Library
  • Read
    • Popular
    • Charts
    • Browse
    • Categories
    • Wikis
    • Glossary
  • Request
  • Activity
  • Leaderboard
  • Help
Our Apps
  • DaoSearch
  • Narratorr
Social
  • Discord
  • Patreon
The AI Wave Tycoon's Rise
Chapter 119

Plateau

1,522 words 5 min read 2 views 1mo ago Source ​
Skip to the end of the chapter
Prev
Next

It was already getting cold in Haicheng in December.

Han Luyi, wrapped in a down jacket, walked into the lobby of the office building in the Qiantan Center, swiped his employee ID card on the turnstile, and with a "beep," the gate opened.

He stepped into the elevator, and the button for the twelfth floor was already lit up.

There were two other people in the elevator chatting, wearing the same name tags as him, but he didn't recognize them.

Upon seeing Han Luyi and Zhang Biao enter, the two stopped talking. One of them glanced at him, hesitated for a moment, and then greeted him, "Boss Han." The other followed suit, saying, "Good morning, Boss Han."

Han Luyi nodded, opened his Sight to check the names: "Little Niu, Little Yang, good morning."

After exchanging greetings, an awkward silence fell over the elevator, and the two stopped talking.

Han Luyi genuinely didn't recognize either of them. The company had already hired fifty people, with a round of centralized recruitment in between, and he couldn't even match the names of several new faces. The Lark group had every employee's name and profile picture, but he hadn't had time to contact any of them.

Write them down.

We arrived at the twelfth floor. As soon as the elevator doors opened, there were already people in the hallway.

The receptionist had changed; the intern from last month had left, and now a baby-faced girl was sitting there. She stood up and said, "Boss Han," when he arrived.

Three people were waiting in line for the Coffee Machine in the break room, while others chatted with their cups in hand. At the end of the corridor, in the Main Conference Room, blinds were drawn, and a row of people sat inside, their presence indistinct; it was unclear which group was having their morning meeting.

Source Code Technology now resembles a more legitimate company.

Every morning at 10:30, a reminder automatically pops up on the Lark workbench. "What is your most important task today?" Every Friday afternoon, there's a reminder for everyone to write a "time-based synchronization" document. Meeting rooms need to be booked through the system, and are automatically released if you're 15 minutes late. The attendance system is flexible; you just need to arrive before 10:00 AM, and the time you leave the company in the afternoon is also recorded. Although the data isn't currently used to calculate performance, Shen Congyun says, "Let's keep track of everything for now."

These were all built by Shen Congyun.

Han Luyi harbored an unspoken resistance to the corporate ills brought about by standardization, but he had yet to come up with a better way to replace these standards, so he proceeded with them one by one.

As I reached the door of my office, my phone vibrated.

It was Lark message from Zhao Wenyuan: "Has it arrived? Any progress reports?"

Han Luyi replied, "In the office," pushed open the door, hung the down jacket on the hanger, and opened the blinds. Outside the window was the Huangpu River. It had been cloudy for several days, but today the sun was shining, and the river surface was glistening.

Two minutes later, Zhao Wenyuan knocked on the door and came in.

He wasn't wearing a suit today, but a leather jacket, looking like a motorcyclist, carrying a laptop and tucking a notebook under his arm. His current workstation is on the thirteenth floor; the model team occupies half the floor, and after hiring more than a dozen people, they couldn't fit everyone in, so they just expanded last month.

"Close the door?" Zhao Wenyuan asked.

Han Luyi.

Zhao Wenyuan closed the door with his heel, sat down on the sofa, opened his laptop, and turned the screen to show Han Luyi.

The screen displays a graph.

The horizontal axis represents the number of training rounds, and the vertical axis represents accuracy. Five points are marked on the curve; the fifth point represents 84.7%.

"The fifth round of fine-tuning training is complete," Zhao Wenyuan said. "The initial accuracy rate is 84.7."

Han Luyi looked at the number.

The latest benchmark announced by Qianyuan is 85%.

The difference is only 0.3 percentage points.

"Let me demonstrate."

Zhao Wenyuan opened several test cases, which were code generated using different programming languages in different application scenarios. After generation, the code was scanned by BugKiller to check the error rate. At the same time, Han Luyi opened the Sight and looked at the code directly generated by the model.

The quality of the generated data has indeed improved significantly.

But Zhao Wenyuan's expression didn't seem like that of someone coming to deliver good news.

He turned to the next page, which showed a more detailed curve. The improvement rates for the five rounds of training were clearly marked: the first round, which was shared at the previous meeting, was 76.8; from the second to the third round, it improved by 5 points; from the third to the fourth round, it improved by 2.8 points; and from the fourth to the fifth round, it improved by 0.1 points.

The curve is flattening out rapidly.

"84.7 is already the limit that post-training can achieve," Zhao Wenyuan said. "Going any higher is not a problem that can be solved by adding more data or graphics cards."

L

Han Luyi looked at the curve.

He recalled the same S-shaped curve that Zhao Wenyuan had projected onto the conference room screen during the last meeting. At that time, the accuracy rate was still in the first half of the Scaling Law.

The data point is during the rapid ascent phase of the curve.

Now that the data points have reached a plateau, the old trick of "great effort brings miracles" no longer works.

"In other words," Han Luyi tapped his hand on the desk, "We've reached the end of the road of using an open-source platform for fine-tuning."

"It's not that you've reached the end, it's that you've reached a plateau," Zhao Wenyuan corrected him. "It's like losing weight; you can break through a plateau, but you have to change your approach to do so."

"In what way?"

Zhao Wenyuan closed the notebook and pulled the book out from under his arm. Han Luyi glanced at it; it had a kraft paper cover with a bunch of English abbreviations and arrows written on it in black pen, like a notebook from his school days.

Zhao Wenyuan turned to one of the pages and spread the notebook out on the coffee table.

"Three roads."

He pointed to the first line.

"The first step is to continue with RLHF (Reinforcement Learning and Human Feedback)."

RLHF, or Reinforcement Learning from Human Feedback, simply put, lets real users act as judges. If users think the generated content is good, the model remembers it; if users think it's bad, the model modifies it.

Over time, the model will learn to identify "content that users consider good".

Han Luyi remained silent, waiting for him to continue.

"The idea is simple. In the few months since Kaiwu Open Things, we've kept all the real user data. If users thought a generated code was good, they clicked 'accept'; if they thought it wasn't good, they manually modified it or regenerated it. These actions themselves are the best feedback signals. We use this data to train a reward model, and then use Reinforcement Learning to let Heavenly Craft fit this reward function."

"Unlike our previous training based on feedback from BugKiller data, this training method allows the model to better understand the intent behind the user's input. This goes beyond bug fixing and leans more towards semantic understanding."

What about the cost?

"The annotation infrastructure, plus the reward Model Training and iteration, will cost approximately five to ten million, taking three to six months. As for the upper limit," Zhao Wenyuan thought for a moment, "I estimate we can improve it by another three to five percentage points, to around 88%."

"This improvement isn't significant."

"Every improvement at this stage is difficult, and the core advantage of this path is not at its limit," Zhao Wenyuan said. "Open Things's real user behavior data is our unique asset. Others can't buy it even if they have money."

99

Han Luyi nodded.

"Second," Zhao Wenyuan turned to the next page, "MoE, Hybrid Expert Model."

MoE, or Mixed Expert Model, doesn't mean training one model to be a jack-of-all-trades. Instead, it means training a group of smaller models, each with its own expertise, and then deciding which one to use when a problem arises. Like a company, it's better to let specialists do what they're good at than to expect every employee to be a jack-of-all-trades.

"This is mainly a change at the architectural level. The base and training method remain unchanged; the calling and inference methods are modified."

"Break down a Large Model into several specialized sub-models—one proficient in Python, one in front-end development, and one in databases. During inference, the system automatically routes the model to the most suitable expert based on the task type. This works better with the same Computing Power and should improve performance by three to five percentage points," Zhao Wenyuan continued.

"Cost?" Han Luyi asked.

"It will cost between ten and twenty million, requiring a complete rewrite of the architecture and retraining of the models, taking four to eight months," Zhao Wenyuan said. "The technical challenges aren't too great; the main cost is the large number of models that need to be tuned. However, we need to hire more people. Our current team has no experience in MoE, so we need to recruit at least two or three people who have experience with similar architectures."

"Finally, the third point." Zhao Wenyuan turned another page, his tone tinged with excitement—the excitement of a scientist.

End of Chapter
Prev
NovelSummaries
Next

Explore the wiki

Han Luyi
Zhao Wenyuan

Comments

No comments yet.
AboutStatsAPIChangelogStatusContactPrivacyTermsDMCA
© 2026 RandomTranslator·v3.26·RSS