look into some of the automated systems for ideas: http://en.wikipedia.org/wiki/Trigram_tagger they process millions of words and might have figured out some of your answers, at least from a “this is a practical methodology”. I dont’ know about the theoretical but they might have some ideas there too.
Yeah – the http://www.coli.uni-saarland.de/~thorsten/tnt/ – link at the bottom of the wikipedia has some more info… and of course when you get into chomskian linguistics and 2nd level markov models things get crazy. I’m still iffy on chomsky ultimately and the way we process language at present… [I have other ideas involving simplifying to concepts that are less linear and more “analyzing concepts expressed”… but they’re just ideas that I can picture in my head as to how it works, but I’m too lazy to take it all the way and try to write my o
I got this far: http://www.amazon.com/Out-Context-Cross…/dp/B00KFOQIG6/ – made it in 2013, sat on it for a year, finally decided to stick the mess up there. I can explain it a little but I kinda gave up making it something formal and moved on to other projects.
But I like seeing language from a different light now at least. [not worth buying; it was my first attempt at an ebook and I think I messed up the formatting… and who wants 106000 words? But at least I got it behind me which is satisfying.
If you want some fun, here’s something I did once:
I took that SAME database [I used MTHESAURUS10 as my corpus – Moby Thesaurus] and aligned it to a POS database I found – nice big one. I simplified the parts of speech (there were too many for my purposes), and made a database called SMURF.
I made SMURF, SMURFING, SMURFED, SMURFINGLY, etc.
Then I lined them up again.
THEN I ran a few wikipedia pages through it to see what happened.
and it reflects my view on reality perfectly tongue emoticon
If smurf smurf smurf smuf, here’s smurf smurf smurfed smurf. I’d get stuff like that tongue emoticon
Now there’s an online Smurf translator that retains a LITTLE more of the meaning: Here’s your comment run through a Smuritizer:
the moment i smurft the notification on your comment smurfing chomsky smurfs the moment i smurfed chomsky on another comment for the first time smurfs well. just thought i would mention that, smurfed like an smurfing coincidence for two people to smurf the same smurf smurfed by a very vague question.he smurfs the only one i smurf found alive to smurf professionally smurfing almost exclusively on smurfing close to smurf
I suspect my project went in the OPPOSITE direction that you’re going in though tongue emoticon