The question
Korean retail investors are active on stock message boards. A teammate and I wanted to know whether those posts point the same way as retail investors’ actual trades: when a board turns bullish on a stock, do individuals really buy it?
We narrowed the scope to two memory-chip stocks, Samsung Electronics and SK hynix, over 134 trading days from January 2 to July 21, 2026. Answering the question takes two things: every post labeled as bullish, bearish, or neutral, and daily trading data to compare against.
This page covers the first half, the data and the model. The comparison with trading data is still in progress.
Collecting the posts
About 1.02M of the posts come from the Naver stock boards for the two companies, collected by a crawler I wrote. The other 352K come from the DC Inside stock gallery and were collected with a separate tool.
The Naver crawler had three properties I cared about:
- It can stop and continue. Each stock-and-day pair is saved as its own file, written to a temporary name and then renamed. If a file exists, that day is done. Rerunning the same command picks up where the last run stopped. The full crawl was interrupted and restarted several times without losing or repeating anything.
- It makes few requests. The list endpoint returns post bodies, so the crawler never opens posts one by one. The endpoint’s default page size is 20, but it accepts 100 and rejects 200. Asking for 100 meant five times fewer requests.
- It is polite. It adds random delays, rests after every 400 requests, slows down on errors, and stops entirely if the site signals it is being rate limited.
The full Naver crawl took 12,485 requests and 5.9 hours and was never blocked.
As a quality check, I crawled one day again 20 days later. The count went from 12,970 posts to 12,860, so about 0.85% of posts had been deleted in between.
Deciding what “bullish” means
Sentiment here means the direction a post expects the price to go. It does not mean the mood of the writer. A post that says “just hold on, we win in the end” sounds anxious, but it expects the price to rise, so it is bullish. Two more rules from our guideline:
- Saying you bought an inverse fund is bearish.
- Describing a move that is already happening, without a prediction, is neutral.
Our first guideline did not work. My teammate and I each labeled the same 100 posts and agreed on only 40% of them (Cohen’s kappa 0.22). We rewrote the guideline, dropped a fourth “cannot judge” label in favor of three labels plus an irony flag, and labeled again. Agreement on the final labels is 80% (kappa 0.623).
That final set has 400 posts: 100 for developing the labeling prompt and 300 held back as the test set.
Teaching a small model with a large one
Labeling 1.37M posts with a paid model would have been expensive, so the plan was to have a large model label a sample and then train a small model on that sample.
The prompt. I tuned the classification prompt over six rounds against the 100 development posts. Its score moved from 0.794 to 0.797, averaging 0.806 over three runs. Our target was 0.90 and we did not reach it.
The sample. GPT-5-mini labeled all 95,976 posts from five trading days using that prompt.
The model. I fine-tuned KLUE-RoBERTa (base) for 3 epochs with a learning rate of 2e-5, batch size 32, and a maximum length of 128 tokens. The data was split at random, keeping the label balance, into 86,018 training posts and 9,558 validation posts.
Two things went wrong here and were fixed:
- A leak. The 400 hand-labeled posts were sitting inside the 95,976 labeled posts. Training on them would have inflated the test score, so I removed all 400 from the training pool and confirmed there was no overlap by post ID.
- Hardware. My first training attempt, with a different base model, ran on an 8 GB laptop at 119 seconds per step, which worked out to about 226 hours. I moved training to a Colab T4 GPU.
Results
All three models were scored on the same 300 hand-labeled posts.
| Model | Macro-F1 | Accuracy |
|---|---|---|
| GPT-5-mini (the labeler) | 0.745 | |
| KLUE-RoBERTa (fine-tuned) | 0.739 | 0.767 |
| KcELECTRA (fine-tuned) | 0.725 | 0.753 |
The fine-tuned model lands within 0.006 of the model that labeled its training data. With 300 test posts the standard error is about 0.024, so this is a tie, not a win for either side.
Where the fine-tuned model is wrong matters more than how often. Of its 70 errors, only 6 confuse bullish with bearish. The other 64 confuse a direction with neutral. Per class, its F1 is 0.812 for neutral, 0.738 for bullish, and 0.667 for bearish.
One result changed how I evaluate models. On the validation set, which is labeled by GPT-5-mini, KcELECTRA scored higher than KLUE-RoBERTa (0.824 against 0.797). On the hand-labeled test set the order reversed. Validation scores measured agreement with the labeler, and only the hand labels measured agreement with people.
Labeling every post
The fine-tuned model then labeled all 1,371,257 posts on a Colab T4 with no API cost. I sorted posts by length before batching, so each batch holds posts of similar length and wastes less computation on padding. In a benchmark on my laptop, throughput went from 46 to 102 posts per second after sorting by length and doubling the batch size.
To check that the cloud run and my local setup agree, I ran 3,072 posts through both. Every label matched.
A second opinion
One model’s labels could carry that model’s quirks into everything downstream. So I labeled all 1.37M posts a second time with the fine-tuned KcELECTRA and compared.
- The two models disagree on 14.4% of posts.
- They assign opposite directions (bullish against bearish) on 2.4%.
- Once posts are averaged into a daily index, the two versions correlate between 0.968 and 0.992.
Post by post the models differ, but the daily index barely depends on which model made it.
What I would improve
- The training days are narrow. The labeled sample comes from five trading days, and the market was mostly falling on them.
- The test set is small and reused. It has 300 posts and I have scored against it four times.
- Irony is unsolved. The irony flag in the prompt does not work reliably.
- Training may have stopped early. The validation score was still rising at the third epoch.
Where it stands
The labeled posts are now joined to daily trading data, and we are running regressions to see how board sentiment relates to retail buying and selling. I’ll add those results here once they are final.