Track_Shovel@slrpnk.net to Lemmy Shitpost@lemmy.worldEnglish · 1 day agoHexadecimalslrpnk.netexternal-linkmessage-square79fedilinkarrow-up1850arrow-down120
arrow-up1830arrow-down1external-linkHexadecimalslrpnk.netTrack_Shovel@slrpnk.net to Lemmy Shitpost@lemmy.worldEnglish · 1 day agomessage-square79fedilink
minus-squaremorrowind@lemmy.mllinkfedilinkarrow-up1·17 hours agoNot really a concern. It’s basically translation, which language models excel at. It just needs a mapping of the hex to byte
minus-squareGissaMittJobb@lemmy.mllinkfedilinkarrow-up1·15 hours agoIt is a concern. Check out https://tiktokenizer.vercel.app/?model=deepseek-ai%2FDeepSeek-R1 and try entering some freeform hexadecimal data - you’ll notice that it does not cleanly segment the hexadecimal numbers into individual tokens.
minus-squaremorrowind@lemmy.mllinkfedilinkarrow-up1·15 hours agoI’m well aware, but you don’t need to necessarily see each character to translate to bytes
minus-squareGissaMittJobb@lemmy.mllinkfedilinkarrow-up1·15 hours agoIt’s not out of the question that we get emergent behaviour where the model can connect non-optimally mapped tokens and still translate them correctly, yeah.
Not really a concern. It’s basically translation, which language models excel at. It just needs a mapping of the hex to byte
It is a concern.
Check out https://tiktokenizer.vercel.app/?model=deepseek-ai%2FDeepSeek-R1 and try entering some freeform hexadecimal data - you’ll notice that it does not cleanly segment the hexadecimal numbers into individual tokens.
I’m well aware, but you don’t need to necessarily see each character to translate to bytes
It’s not out of the question that we get emergent behaviour where the model can connect non-optimally mapped tokens and still translate them correctly, yeah.