Han Luyi made an appointment with Zhao Wenyuan as soon as she got off the plane. She originally wanted to go by herself, but Zhang Biao insisted on coming along, saying it was part of her job.
It was the same place as last time, Luckin Coffee on Nanjing West Road. When Han Lu pushed open the door, Zhao Wenyuan was already sitting by the window, holding a coconut latte.
The last time we sat here, Zhao Wenyuan said, "Large Model are too expensive; this isn't a field we can afford." Han Luyi replied, "The door to the Source Code is always open for you."
That happened less than a month ago.
Zhao Wenyuan was wearing a gray sports jacket and looked to be in good spirits. He looked up and saw Han Luyi, then saw Zhang Biao, who looked like the Terminator, following behind Han Luyi, and paused for a moment.
"What are you doing—"
"This is Biao, his professional bodyguard." Han Luyi pulled out a chair and sat down. "He was arranged by the investors."
Zhao Wenyuan sized Zhang Biao up and down. Zhang Biao sat down at the next table, expressionless, and took out his phone to order—he ordered a mocha.
How much did you raise?
Two hundred million.
Zhao Wenyuan picked up his Coconut Latte, took a sip, and showed no particular expression.
What about the valuation?
"Two billion."
"Then we can certainly talk," Zhao Wenyuan said. "How much budget have you prepared?"
Han Luyi didn't mince words. "Thirty million, the startup budget for the model team. How do you think we should spend it?"
Zhao Wenyuan didn't answer directly, but instead asked, "You want to start training from scratch?"
"I feel it's not enough, what do you think?"
"It's definitely not enough," Zhao Wenyuan said. "Far from it."
He flipped over a tissue, took out a pen from his pocket, and began to write on it.
"For general-purpose Large Model with hundreds of billions of parameters, pre-training on several thousand H100 images takes two to three months." He wrote down a figure, "The Computing Power cost starts at tens of millions of US dollars, and it's common for leading companies to burn through one or two hundred million RMB in a single round of pre-training."
Han Luyi knew these numbers, but hearing them from Zhao Wenyuan's own mouth felt different, after all, Zhao Wenyuan was someone who had handled them himself.
"Thirty million RMB," Zhao Wenyuan drew an approximate equal sign on a tissue, "A little over four million US dollars, not even a fraction of the cost of a single round of pre-training."
So what are your thoughts?
"There are other paths," Zhao Wenyuan said, tapping his pen tip on a tissue, "To do follow-up training."
He underlined the words "post-training".
"The open-source base model already has general-purpose code capabilities. The Alpaca Model and the Queen Model are excellent foundations. Post-training is about specializing on this basis."
"SFT plus DPO?"
Zhao Wenyuan glanced at him, raised an eyebrow in surprise, and then turned to look at Zhang Biao was staring blankly in their direction.
"Yes, SFT, instruction fine-tuning, teaches the model patterns for specific tasks. For example, if you give it a natural language description, it should generate a working application. DP0, direct preference optimization, is the second step, teaching the model to distinguish between good and bad output."
He drew a simple flowchart on a tissue.
"Post-training requires one to two orders of magnitude less Computing Power than pre-training; 30 million is enough to get started."
He put down his pen and looked at Zhang Biao first.
Zhang Biao felt sleepy, so he tried his best to keep his eyes open and blinked.
Zhao Wenyuan gave up on getting Zhang Biao to understand the conversation and turned to Han Luyi.
"Post-training doesn't rely on Computing Power, it relies on data, specifically high-quality Annotated Data."
Han Luyi him to continue.
"SFT requires a large number of question-and-answer pairs. The input is the user's natural language prompts, and the output is runnable application code. The quality of this data directly determines the upper limit of the model."
Zhao Wenyuan leaned forward.
"Manually labeling a single high-quality data point costs tens to hundreds of yuan, so 100,000 data points would cost tens of millions of yuan. Moreover, the skill levels of the labeling teams vary greatly, and you still have to clean the data repeatedly."
"I saw it at its Dingsheng. CodeSafe consumed a huge amount of user data, but it was dirty and noisy. They spent a lot of manpower cleaning it, but the results were never good."
"Garbage in, garbage out," Han Luyi said.
"That's exactly what I mean. If the data quality is poor, even a large model is useless." After Zhao Wenyuan finished speaking, he looked at Han Luyi with a probing gaze, waiting for Han Luyi to make his move.
"I have the data."
Zhao Wenyuan didn't speak, but he sat up straight. He had guessed before why BugKiller was so powerful; it must have either golden data or an undisclosed advanced algorithm. Now it seemed to be the former.
"Where did you get your data?" he asked, slowing his speech. "You didn't steal it, did you?"
This is no joke. Zhao Wenyuan came from Dingsheng, and he couldn't stand CodeSafe stealing user code to feed into Large Model. This was his bottom line.
Han Luyi said seriously, "All the data is shared voluntarily by users. The relevant options during registration are turned off by default. Users must actively choose to use the test results for product improvement before we can see them. The agreement was reviewed clause by clause by Lawyer, and I also have the Auditing report here."
Zhao Wenyuan stared at him.
"I will never steal data," Han Luyi said.
"Okay." Zhao Wenyuan nodded. "Is the data volume sufficient?"
Han Luyi opened the laptop in his backpack, connected it to his phone's hotspot, and logged into the Open Things backend.
A data dashboard popped up on the screen.
"Since its Open Things, Kaiwu has accumulated 230,000 prompts/codes," Han Luyi turned the screen to Zhao Wenyuan, "All from user-initiated sharing."
Zhao Wenyuan glanced at the total. "Two hundred and thirty thousand—the quantity is sufficient. But there's a problem: the code generated by AI itself has bugs. If we use it directly for training, won't it just be a case of garbage in and garbage out?"
"Therefore, there is a second step."
Han Luyi switched to another interface, the BugKiller data pipeline dashboard.
"Every line of code generated by Open Things will automatically pass through the BugKiller detection engine. Bugs will be marked, those that can be automatically fixed will be fixed, and those that cannot be fixed will be discarded."
Zhao Wenyuan leaned closer to the screen.
The data flow is clear: user prompts → original generated code → BugKiller automatic detection → fixed code → database entry.
Each piece of data has a status label: "Passed" or "Repaired"; there is no "Unrepaired".
"The original code had a pass rate of around 60%, but after BugKiller fixes, the final code integration rate reached 96%."
Zhao Wenyuan didn't say anything. He reached for the notebook and started flipping through the data himself.
He first looked at a few "fixed" entries; the original code did indeed have bugs, but the fixed code was clean and concise. Then he looked at the "passed" entries; the original code generation was fine, the code structure was clear, and the variable naming was standardized.
One, two, five, ten.
He examined the repair logic line by line.
Articles 20 and 30.
The music playing in the Luckin store changed several times, but Zhao Wenyuan didn't look up.
He stopped when he had scrolled to around fifty entries.
"Is this detection engine the core of BugKiller?"
"Yes, I wrote the core algorithm myself," Han Luyi said. "That's a technological barrier."
"Your data couldn't possibly have been stolen. The quality of customer code varies greatly; stolen code wouldn't be this high." Zhao Wenyuan sighed. "No wonder CodeSafe lost to you; I'm completely convinced."
"This accuracy, this usage," he continued, "What you have in your hands is an automated data cleaning pipeline. What others would spend tens of millions to have labeled, you can do with just one run of the algorithm."
He thought for a moment, then his speech suddenly quickened.
"Furthermore, the buggy code detected by BugKiller and the fixed versions—aren't these ready-made DPO training materials? Bugged versions are bad examples, fixed versions are good examples, naturally biased data."
Han Luyi nodded.
"SFT uses clean prompt words and code pairs, DPO uses bug and fix pairs, and the two pipelines run simultaneously." Zhao Wenyuan tapped his hand on the table twice, becoming more and more excited as he spoke, "The data flywheel, no one can steal that. Even if they steal the data, without this detection engine, they can't do the same quality of cleaning."
Han Luyi remained silent. "Wenyuan, you're now in my trap."
Zhao Wenyuan took the tissue paper covered with numbers, flipped it over, and continued writing.
"Post-training doesn't require the kind of Thousand-Card Cluster in pre-training, but it's not cheap either. A single H100 card currently costs between 200,000 and 250,000 RMB, and an 8-card Server costs between 1.6 million and 2 million RMB."
He took a picture of the tissue, put it in his phone's notes app, and then typed a string of numbers into the notes app.
"A medium-sized model with billions of parameters requires at least sixteen to thirty-two GPUs for post-training."
"Rent it first," Han Luyi, taking over the conversation.
Zhao Wenyuan looked up.
"We won't lock the Hardware until the first version is running successfully," Han Luyi said. "If we need to change the direction, we can't get a refund for the card we bought."
Zhao Wenyuan laughed. Back when he was at Google, training models had never been this meticulous.
"You're quite lucid."
"I've been poor before, thank you."
Zhao Wenyuan quickly typed numbers in his memo. "Eight million for the first batch of Computing Power rental, three to five hundred thousand for data cleaning and supplementary labeling (this can probably be saved), four million for team salaries over six months, three million for toolchain miscellaneous expenses, and ten million to be reserved."
He pushed the phone towards Han Luyi look.
"That's enough, let's get running."
The two chatted faster and faster. The Luckin store wasn't noisy at around 3 p. m., but their voices unconsciously grew louder.
A girl wearing headphones at the next table took off one earphone and glanced at them. Two pretentious guys Add One muscular guy—this combination was really strange.
A waiter came over. "Would you like another drink?"
Han Luyi then realized that he hadn't ordered anything since he sat down.
"A Coconut Latte."
"Two cups." Zhao Wenyuan didn't even look up.
Zhang Biao followed the clerk to scan the code and place his order. Han Luyi glanced at the table: several tissues covered with numbers, a laptop, and two mobile phones. It was a bit funny to be discussing a budget of tens of millions at Luckin.
I'll start work tomorrow.
Han Luyi and asked, "So rushed?"
"I can't wait any longer, I want to get started quickly."
"Oh, right," he said, "The hardest question is, what should we name the Large Model? Have you thought of a name yet?"
Han Luyi looked at him and smiled, "I've already thought it through--"
"Heavenly Craft."