All models performed poorly at detecting direct money extraction attempts<\/strong> – the final and most critical scam phase. Best performance was Claude at only 5.7% accuracy.<\/p>\n\n\n\n3. Language Matters: English vs Japanese Performance<\/h5>\n\n\n\n
Claude showed a remarkable 12-point accuracy improvement on English messages (78.3% vs 66.2% for Japanese), while other models showed smaller gaps. This suggests significant language-specific bias in model training and performance.<\/p>\n\n\n\n
4. The “None” Category Challenge<\/h5>\n\n\n\n
With 67.7% of messages classified as legitimate conversation, accurately distinguishing between scam tactics and normal chat proved crucial. <\/p>\n\n\n\n
GPT-4 and GPT-3.5 performed best here, while Gemini struggled, over-classifying innocent messages as scam attempts.<\/p>\n\n\n\n
5. Cultural Context Blindness: The Japanese Politeness Problem<\/h5>\n\n\n\n
LLMs demonstrated a critical weakness in understanding Japanese cultural communication norms, with Gemini suffering most severely from this issue<\/strong>.<\/p>\n\n\n\nIn Japanese culture, expressions of care, concern, and attentiveness are standard politeness markers, not necessarily signs of manipulation.<\/p>\n\n\n\n
Gemini’s 62.6% accuracy on legitimate conversation (“none” category) compared to ~75% for other models reveals systematic over-classification of polite Japanese phrases. <\/p>\n\n\n\n
The model frequently misclassified cultural politeness like “\u304a\u75b2\u308c\u69d8\u3067\u3057\u305f” as “Manipulative Care.”<\/p>\n\n\n\n
This cultural blindness explains why Gemini paradoxically achieved the highest accuracy on actual “Manipulative Care” detection (98.3%) – it was flagging both genuine manipulation and<\/em> normal Japanese politeness as the same category.<\/p>\n\n\n\nConclusion<\/h3>\n\n\n\n
Large language models (LLMs) have emerged as powerful tools in the fight against digital deception\u2014but this study reveals both their potential and their current limitations.<\/p>\n\n\n\n
Through the classification of nearly 4,000 real scam messages from 50 cryptocurrency romance scams, I found that no single model consistently outperformed across all scam tactics<\/strong>. <\/p>\n\n\n\nWhile GPT-4 achieved the highest overall accuracy<\/strong>, other models like Claude<\/strong> and Gemini<\/strong> demonstrated unique strengths in niche categories like investment pitches or fake persona detection.<\/p>\n\n\n\nHowever, the most sobering insight is this: every model performed poorly on detecting direct money extraction attempts<\/strong>, the final and most dangerous step in the scam process. <\/p>\n\n\n\nEven the best model (Claude) only achieved 5.7% accuracy in this critical category\u2014underscoring the difficulty of identifying explicit fraud when cloaked in emotionally manipulative language.<\/p>\n\n\n\n
I also uncovered significant language-specific performance gaps<\/strong>, with all models performing better in English than Japanese. This points to the need for more diverse training data and language-aware model tuning.<\/p>\n\n\n\nUltimately, this benchmark serves as both a progress report<\/strong> and a call to action<\/strong>. GenAI tools are promising allies in scam detection, but relying on them blindly is not enough<\/strong>. Human oversight, diverse training data, and continued evaluation are essential for building safer systems.<\/p>\n\n\n\nReal-World Implications<\/h4>\n\n\n\n
The findings have immediate practical applications:<\/p>\n\n\n\n
For Platforms:<\/strong><\/p>\n\n\n\n\n- Deploy hybrid detection<\/strong>: No single model works\u2014combine GPT-4’s emotional detection with Claude’s investment pitch recognition.<\/li>\n\n\n\n
- Address cultural misinterpretation<\/strong>: Current models often confuse standard Japanese politeness with manipulation, leading to false positives.<\/li>\n\n\n\n
- Rethink money extraction detection<\/strong>: 5.7% accuracy means current approaches are fundamentally broken.<\/li>\n<\/ul>\n\n\n\n
For AI Researchers:<\/strong><\/p>\n\n\n\n\n- Train on real adversarial data<\/strong>: Synthetic scam data clearly isn’t cutting it.<\/li>\n\n\n\n
- Build conversation-aware models<\/strong>: Message-level classification misses scammer progression patterns.<\/li>\n\n\n\n
- Address cross-cultural training gaps<\/strong>: English-centric training creates dangerous blind spots.<\/li>\n<\/ul>\n\n\n\n
What’s Next: Fine-Tuning for Superior Performance<\/h3>\n\n\n\n
The performance of general-purpose LLMs in this study is just the starting point. The real opportunity lies in transforming these models through fine-tuning<\/strong>\u2014training them further on a domain-specific dataset to boost their precision, context awareness, and reliability.<\/p>\n\n\n\nWith 15,913 manually reviewed messages from 175 real cryptocurrency scammers, this dataset offers an unparalleled foundation. Unlike synthetic or simulated text, these conversations capture authentic scammer psychology, cultural nuances, and evolving fraud tactics\u2014elements no model has been exposed to at scale.<\/p>\n\n\n\n
In my upcoming blog post, I\u2019ll explore how to fine-tune open-source LLMs using this dataset. <\/p>\n\n\n\n
Fine-tuning offers three key advantages:<\/p>\n\n\n\n
\n- Boost Accuracy in Critical Categories<\/strong>: Address persistent weaknesses\u2014especially in detecting \u201cMoney Extraction\u201d\u2014by training models on more representative, labeled examples.<\/li>\n\n\n\n
- Reduce Cultural Misclassification<\/strong>: Use culturally grounded Japanese data to help models distinguish between standard politeness and manipulative grooming, minimizing false positives.<\/li>\n\n\n\n
- Build Real-World-Ready Models<\/strong>: Adapt models for practical deployment, moving from generic language understanding to nuanced threat recognition in scam detection systems.<\/li>\n<\/ul>\n\n\n\n
This next phase will detail dataset preparation, architecture selection, and performance evaluation. The goal is to show how targeted fine-tuning can turn general LLMs into expert-level fraud detectors\u2014helping us build safer, smarter tools in the GenAI era.<\/p>\n\n\n\n
By moving beyond off-the-shelf models and toward culturally-aware, tactic-specific fine-tuning, we can take meaningful steps to reduce harm from AI-assisted scams worldwide.<\/p>\n\n\n\n
Stay tuned<\/strong> for the next post, where I\u2019ll share early results\u2014and whether fine-tuned open-source LLMs can finally outsmart the scammers.<\/p>\n\n\n\n<\/i><\/div>\n
This research was conducted ethically with proper security measures. No real victims were involved, and all scammer interactions were conducted safely with appropriate protections.<\/em><\/p>\n\n\n\nData Note<\/strong>: The complete dataset of 15,913 messages from 175 scammers represents one of the largest collections of real scam conversations available for research. Anonymized subsets will be made available once I complete manual review of all messages. <\/p>\n\n\n\nTechnical Note<\/strong>: All model testing was conducted with identical prompts, security measures to prevent data leakage, and consistent evaluation criteria. Full methodology and code are available here<\/a>. <\/p>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"The Growing Threat of Cryptocurrency Romance Scams Cryptocurrency romance scams have evolved into a sophisticated, multi-billion dollar criminal enterprise. These operations combine emotional manipulation with financial fraud, often lasting weeks or months before the final money extraction. The perpetrators follow detailed scripts, systematically building trust through fake personas before introducing \u201cinvestment opportunities.\u201d To combat this<\/p>\n","protected":false},"author":1,"featured_media":6883,"parent":6413,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-6486","page","type-page","status-publish","has-post-thumbnail","hentry"],"_links":{"self":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/6486","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6486"}],"version-history":[{"count":236,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/6486\/revisions"}],"predecessor-version":[{"id":6884,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/6486\/revisions\/6884"}],"up":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/pages\/6413"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=\/wp\/v2\/media\/6883"}],"wp:attachment":[{"href":"https:\/\/tomomitanaka.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6486"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}