I'm working on a big machine learning/nlp project and I'm stuck at a small part of it. (PM me, if you want to know what I'm working on exactly.)
I try to code a program in Javascript that learns to generate valid words, only by using all letters of the alphabet.
What I have is a database of 500K different words. It's a big JS object, structured like this (the words are german):
database = {
"um": {id: 1, word: "um", freq: 10938},
"oder": {id: 2, word: "oder", freq: 10257},
"Er": {id: 3, word: "Er", freq: 9323},
...
}
"freq" means frequency obviously. (Maybe this value sometimes gets important but I currently don't use it, so just ignore it.)
The way my program currently works is:
In the first iteration, it generates a completely random word between 2 and 13 letters long and searches for it in the database. If it's there, every letter in the word gets a good rating, if it's not there, they get a bad rating. Also the word length gets rated. If the word is valid, its word length gets a good rating, if it's not, its word length gets a bad rating.
In the iterations after that first one, it doesn't generate a word with random letters and a random word length. It uses probabilities based on the ratings of the letters and the word length.
For example, let's say it found the words "the", "so" and "if" after the first 100 iterations. So the letters "t", "h", "e" and the letters "s", "o", and the letters "i", "f" are good rated, and the word length of 2 and 3 is also good rated. So the word generated in the next iteration will more likely contain these good rated letters than bad rated letters.
Of course, the program also checks if the currently generated word already was generated and if so, then this word doesn't get rated again and it generates a new one.
In theory it should learn the optimal letter frequency and the optimal word-length-frequency by its own and sometimes only generate valid words.
Yeah. Of course this doesn't work. It gets better for the first few iterations, but as soon as it has found all the 2-lettered words it gets worse. I think my whole way how I do this is wrong. I've actually tried it out and have a (not so beautiful) graph after 5000 iterations for you:
Red line: wrong words generated
Green line: right words generated
Yeah. What is the problem here? Am I doing machine learning wrong? And do you have a solution? Some algorithm or trie system?
PS: I'm aware of this, but it's not in JS, I don't understand it and I can't comment on it.
An alternative method would be to use a Markov Model.
Start by counting up the letter frequencies and also word length frequencies in your dictionary. Then, to create a word:
Pick a weighted random number (see below) between 1 and the maximum existing word length. That's how many letters you're going to generate.
For each letter in the word, pick a weighted random letter and add it to the word.
That's an order-0 Markov model. It's based on the frequency of letters that occur in the corpus. It will probably give you results that are similar to the system you have.
You'll get better results from an order-1 Markov model, where instead of computing letter frequencies, you compute bigram (two-letter permutations) frequencies. So to pick the first letter, you choose only from the bigrams that are used to begin words. For subsequent letters, you choose a letter that follows the previously generated letter. That's going to give you somewhat better results than an order-0 model.
An order-2 model is surprisingly effective. See my blog post, Shakespeare vs. Markov, for an example.
A weighted random number is a number selected "at random," but skewed to reflect some distribution. In the English language, for example, the letter 'e' occurs approximately 12.7% of the time. 't' occurs 9.06% of the time, etc. See https://en.wikipedia.org/wiki/Letter_frequency. So you'd want your weighted random number generator's output to approximate that distribution. Or, in your case, you'd want it to approximate the distribution in your corpus. See Weighted random numbers for an example of how that's done.
Related
In my application I've got a list of words that are being shown to the user one by one. The exact number may differ, but it can get as low as about 10. It's ok if the words repeat. To select a word I currently use Math.random
const select = (arr) => arr[Math.floor(Math.random() * arr.length)]
The problem is that Math.random generates, well, random numbers, that may sometimes form clusters, so the user may see the same words several times in a row. I have a simple check that new word is different than the previous one, but I want to improve it. The question is, how can one make random numbers from that range (0 - arr.length-1) more spread, like if it was a human who would choose random words (i.e. they would repeat rarer). Or, for this particular case, how to make the user see as many different words in given time as possible?
I have two strings where the first is a master string and the second is a slave string. They both contain similar values except that the slave will have characters added or removed.
I need to find the character offset from the master string in the slave string for each character of the master string.
I'm currently using a percentage as the algorithm for finding the similar offset in the slave string.
For example;
const master = 'The chicken is blue, but not really a chicken';
const slave = 'This large bird is blue, but is really a dog';
function slaveOffset(m, offset, s): number {
return Math.floor(s.length * (offset / m.length));
}
console.log(slaveOffset(master, 15, slave)); // prints 12
When translating the position 15 from the master (which reads "The chicken is ") the slave position is calculated as 12. Which reads as "This large b" because using percentage is not at all accurate (doesn't take into account added or removed characters).
The correct value should have been 18 (which reads as "The large bird is"), because the master offset ends at "is".
I need an algorithm for slaveOffset() that can handle added and removed characters and find the most likely slave offset. It does not need to be overly accurate but should solve the problem of large deviations caused by character changes.
This is a classic problem in computer science, typically called "data comparison" or simply "diff". The most common algorithms apply Longest Common Subsequence techniques, but in the general case this is a NP-hard problem so various heuristics are applied to get a "good enough" outcome, often tuned by a human in the loop.
Look up some diff algorithms to get some ideas.
In your case you probably want to start with the heuristic of "where does the slave string start to differ from the master and where does it become the same again". The strings match for the first two characters, but the next time you get a sequence of more than 3 characters matching is at the characters , i and s. The points become markers that you can use in your slaveOffset function.
i have a word from A to Z. all word should in small latter (Capital not include) and 1 to 9 (included all special word who can be used in email address (just for a test)).
how i can generate unique 1 lacs text who never repeat itself. can anyone solve this puzzle.
i want a another thing that all words should not more then 10 char and not should minimum 6 char long
Put the characters in an array. Copy the array as the source of a new line. Randomly slice words from the array and put them in the line (use Math.random() * array.length | 0). Keep going for the required number of words.
You can also just use a string and charAt(index) if you only want single characters, but you have to keep cutting out the character that you select which is likely less efficient than using array.slice.
Whatever suits though, since performance is likely irrelevant.
I've been using String Score for a lot of projects. It's great for sorting lists, like names, countries, etc.
Right now, I'm working on a project where I want to match a term against a bigger set of text, not just a few words. Like, a paragraph.
Given the following two strings:
string1 = "I want to eat.";
string2 = "I want to eat. Let's go eat. All this talk about eating is making me hungry. Ready to eat?";
I'd like the term eat to return string2 as higher than string1. However, string1 scores higher:
string1.score('eat');
> 0.5261904761904762
string2.score('eat');
> 0.4477777777777778
Maybe I'm wrong in thinking string2 should score higher, and I'd love to hear arguments for that logic, if that is your logic. Otherwise, any ideas on a more contextual javascript matching algorithm?
If the score is not taking into account repetitions then only one occurrence of "eat" in string2 adds to the score so the other occurrences of "eat" are treated as unmatched garbage which counts against in the total score.
Many string similarity metrics behave this way, e.g. in Edit distance the more non-matching characters the lower the score and repetitions are treated as non-matching.
It's not clear to me from reading the source what algo it is using, but the score variables
var total_character_score = 0,
start_of_string_bonus,
abbreviation_score,
fuzzies=1,
final_score;
don't seem to take into account multiple repetitions.
If you want multiple occurrences to count, then it sounds like what you want is not a string-similarity algo, but a fuzzy match algo so you can find the number of matches.
Maybe yeti witch will work for you.
Implement the “Word Decoder” game. This game will present the player with a series of scrambled words (up to 20 words) and challenge him/her to attempt to unscramble them. Each time a new word is displayed, and a text input is provided for the user to write the unscrambled word.
Once the player thinks the word has been properly decoded, he clicks on the “Check answer” button. If the player’s answer is correct, his score is increased by one. If his answer is not correct, he is notified and he is then given a different word.
i understood the Question , but i dont know how to generate it , or even how to start it!!
any help please?
To start, try breaking down the problem into things you'll need; think nouns and verbs. This is simply rewriting the problem in new terms. You need:
word: just a string, but it's a noun you'll need, so list it.
dictionary: a collection of words to choose from (during testing, you don't need many)
display: these become HTML elements, since you're working with JS
scrambled word
text input
submit button to check answer
score
"wrong answer" notifier
to scramble a word
to compare words: how can you compare two words to see if one is a permutation of the other? Do it right and anagrams aren't a problem.
to check an answer
to increment score
to notify user of incorrect answer
to present a new scrambled word
Any item beginning with "to" is a verb; anything else is a noun. Nouns become objects, verbs become methods/functions.
The above is mostly a top-down approach, in contrast with bottom-up (note that top-down vs bottom-up isn't an either-or proposition). Other approaches that might help with not knowing where to start are test driven development or its offshoot, behavior driven development. With these you start by defining, in code, what the program should do, then fill in the details to make it do that.
A hint on comparing words: the problem is basically defining an equivalence class—two strings are equivalent if one is a permutation of the other. The permutations of a string, taken together, form the equivalence class for that string; two strings are in the same equivalence class if the strings are equivalent. As the linked document points out, equivalence classes are well represented by picking a single element of the class as the class representative. Lastly, you can turn the equivalence class definition around: two strings are permutations of each other if they are in the same equivalence class.
Look into loading a dictionary via XHR.
there are tons of those available online [http://www.mieliestronk.com/wordlist.html NOTE: it contains some swear words, if you're going to be doing this for academic purposes, since its your homework, you should look for a "clean" list]...
For scrambling the word: make your string into a char array, then find an array shuffle function [they are simple to write, I wrote one for my implementation of Bogosort]...
function shuffle(b)
{
var a = b.concat([]); //makes a copy of B, b won't be changed...
var final = [];
while(a.length != 0)
{
//0 -> a length-1
var targIndex = Math.round((a.length-1)*(Math.random()));
var value = a[targIndex]
a.remove(targIndex);
final.push(value);
}
return final;
}
When the user is done inputting, simply compare input with the answer [case insensitive, ignore spaces] As stated in comments, there are also the possibility of anagrams, so be sure to check for those... perhaps, you could simply verify the word exists in the dictionary.