Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

markov text generation help

Discussion in 'Scripting' started by cyphem, Sep 21, 2019.

  1. cyphem

    cyphem

    Joined:
    Apr 18, 2014
    Posts:
    5
    Converted the c# from this link to work with Unity, it works mostly, but it randomly throws KeyNotFoundException at random times. at var suffix = dict[prefix];- see below

    I have tried to catch the error using ContainsKey and TryGetValue but the dictionary is using a string list and I just can't figure it out. Maybe the construction of the dictionary has some inherent flaw? Just need someone smarter to have a look so I can catch or fix the error.

    The error message is

    KeyNotFoundException: The given key was not present in the dictionary.
    System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.List`1[System.String]].get_Item (System.String key) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
    Markovv.Markov (System.String filePath, Int32 keySize, Int32 outputSize) (at Assets/Markovv.cs:57)
    Markovv.Update () (at Assets/Markovv.cs:83)

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System;
    4. using System.Collections.Generic;
    5. using System.IO;
    6. using System.Linq;
    7.  
    8.  
    9. public class Markovv : MonoBehaviour {
    10.  
    11.  
    12. public string sad;
    13.  
    14.     void Start () {
    15. sad= "said the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near here, suffix.Countsaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near heresaid the Duchess, who seemed ready to agree to everything that Alice said; 'there's a large mustard-mine near here";
    16.     }
    17.  
    18.      public string Join(string a, string b) {
    19.             return a + " " + b;
    20.         }
    21.   public string Markov(string filePath, int keySize, int outputSize) {
    22.             if (keySize < 1) throw new ArgumentException("Key size can't be less than 1");
    23.             string body;
    24.             body=filePath;
    25.                      var words = body.Split();
    26.             if (outputSize < keySize || words.Length < outputSize) {
    27.                 throw new ArgumentException("Output size is out of range");
    28.             }
    29.             Dictionary<string, List<string>> dict = new Dictionary<string, List<string>>();
    30.             for (int i = 0; i < words.Length - keySize; i++) {
    31.                 var key = words.Skip(i).Take(keySize).Aggregate(Join);
    32.                 string value;
    33.                 if (i + keySize < words.Length) {
    34.                     value = words[i + keySize];
    35.                 } else {
    36.                     value = "";
    37.                 }
    38.                 if (dict.ContainsKey(key)) {
    39.                     dict[key].Add(value);
    40.                 } else {
    41.                     dict.Add(key, new List<string>() { value });
    42.                 }
    43.             }
    44.          
    45.  
    46.             List<string> output = new List<string>();
    47.             int n = 0;
    48.                        int rn = UnityEngine.Random.Range(0,dict.Count);
    49.             string prefix = dict.Keys.Skip(rn).Take(1).Single();
    50.             output.AddRange(prefix.Split());
    51.             while (true) {
    52.                 var suffix = dict[prefix];///////KeyNotFoundException: The given key was not present in the dictionary.
    53.                 if (suffix.Count == 1) {
    54.                     if (suffix[0] == "") {
    55.                         return output.Aggregate(Join);
    56.                     }
    57.                     output.Add(suffix[0]);
    58.                 } else {
    59.                       rn = UnityEngine.Random.Range(0,suffix.Count);
    60.                     output.Add(suffix[rn]);
    61.                 }
    62.                 if (output.Count >= outputSize) {
    63.                     return output.Take(outputSize).Aggregate(Join);
    64.                 }
    65.                 n++;
    66.                 prefix = output.Skip(n).Take(keySize).Aggregate(Join);
    67.             }
    68.   }
    69.      
    70.      
    71.         void Update() {
    72.          if (Input.GetKeyUp("i"))
    73.         {
    74.      
    75.          
    76.             Debug.Log(Markov(sad, 3, 10));
    77.      
    78.      
    79.         }
    80.         }
    81.      
    82.      
    83. }
     
    Last edited: Sep 21, 2019
  2. Yoreki

    Yoreki

    Joined:
    Apr 10, 2019
    Posts:
    2,605
    Hi and welcome.
    Generally speaking, posting the full error message and /or the line the error occured at (only if not the same as the line in the error message. Which happens when the example is not the full script, or the example has an extra free line or comment somewhere) helps immensely to speed up the process of finding the error.
     
  3. cyphem

    cyphem

    Joined:
    Apr 18, 2014
    Posts:
    5
    Thanks Yoreki.

    The error message is

    KeyNotFoundException: The given key was not present in the dictionary.
    System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.List`1[System.String]].get_Item (System.String key) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
    Markovv.Markov (System.String filePath, Int32 keySize, Int32 outputSize) (at Assets/Markovv.cs:57)
    Markovv.Update () (at Assets/Markovv.cs:83)

    This is line that the console refers to
    var suffix = dict[prefix];///////KeyNotFoundException: The given key was not present in the dictionary.
     
  4. Yoreki

    Yoreki

    Joined:
    Apr 10, 2019
    Posts:
    2,605
    I believe the problem has to do with the way you replaced Random.Next(dict.Count). You are using Random.Range(0, dict.Count) instead. While this seems similar, Random.Next excludes the number given (returning a non-negative number that is smaller than the parameter), while Random.Range includes the parameter as a possible result.
    This means that your version can actually return the number dict.Count, while the original implementation cant.

    This means that, if you are very unlucky and actually get dict.Count as return for Random.Range, then when calculating "prefix", you skip all elements of the list, take the first of the (none / 0) remaining elements and try to use that as key.
    To fix this you should be able to juse use dict.Count-1 instead.
    However, you should also be able to just use Random.Next like in the original implementation.
     
  5. cyphem

    cyphem

    Joined:
    Apr 18, 2014
    Posts:
    5
    Thanks, your right I should have looked deeper into Random.Next, before assuming it worked pretty much like Random.Range.
    So when I made your changes the code runs without errors. But when I tested with a different input string the code executes for longer, but still nonetheless throws the same error after some time-

    KeyNotFoundException: The given key was not present in the dictionary.
    System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.List`1[System.String]].get_Item (System.String key) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
    Markovv.Markov (System.String filePath, Int32 keySize, Int32 outputSize) (at Assets/Markovv.cs:71)
    Markovv.Update () (at Assets/Markovv.cs:98)

    Surely the input string does not make a difference when formatted the same way?
    How can I test if prefix is going to line up with the right value?

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System;
    4. using System.Collections.Generic;
    5. using System.IO;
    6. using System.Linq;
    7.  
    8.  
    9. public class Markovv : MonoBehaviour {
    10.  
    11.  
    12. private string sad;
    13.  
    14. void Start () {
    15. sad = "One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin. He lay on his armour-like back, and if he lifted his head a little he could see his brown belly, slightly domed and divided by arches into stiff sections. The bedding was hardly able to cover it and seemed ready to slide off any moment. His many legs, pitifully thin compared with the size of the rest of him, waved about helplessly as he looked. What's happened to me he thought. It wasn't a dream. His room, a proper human room although a little too small, lay peacefully between its four familiar walls. A collection of textile samples lay spread out on the table Samsa was a travelling salesman - and above it there hung a picture that he had recently cut out of an illustrated magazine and housed in a nice, gilded frame. It showed a lady fitted out with a fur hat and fur boa who sat upright, raising a heavy fur muff that covered the whole of her lower arm towards the viewer. Gregor then turned to look out the window at the dull weather. Drops of rain could be heard hitting the pane, which made him feel quite sad. How about if I sleep a little bit longer and forget all this nonsense, he thought, but that was something he was unable to do because he was used to sleeping on his right, and in his present state couldn't get into that position. However hard he threw himself onto his right, he always rolled back to where he was. He must have tried it a hundred times, shut his eyes so that he wouldn't have to look at the floundering legs, and only stopped when he began to feel a mild, dull pain there that he had never felt before. Oh, God, he thought, what a strenuous career it is that I've chosen! Travelling day in and day out. Doing business like this takes much more effort than doing your";}
    16. public string Join(string a, string b) {
    17. return a + " " + b;
    18. }
    19. public string Markov(string filePath, int keySize, int outputSize) {
    20. if (keySize < 1) throw new ArgumentException("Key size can't be less than 1");
    21. string body;
    22. body=filePath;
    23. var words = body.Split();
    24. if (outputSize < keySize || words.Length < outputSize) {
    25. throw new ArgumentException("Output size is out of range");
    26. }
    27. Dictionary<string, List<string>> dict = new Dictionary<string, List<string>>();
    28. for (int i = 0; i < words.Length - keySize; i++) {
    29. var key = words.Skip(i).Take(keySize).Aggregate(Join);
    30. string value;
    31. if (i + keySize < words.Length) {
    32. value = words[i + keySize];
    33. } else {
    34. value = "";
    35. }
    36. if (dict.ContainsKey(key)) {
    37. dict[key].Add(value);
    38. } else {
    39. dict.Add(key, new List<string>() { value });
    40. }
    41. }
    42. List<string> output = new List<string>();
    43. int n = 0;
    44. System.Random rand = new System.Random();
    45.  
    46. //int rn = rand.Next(dict.Count);
    47. // int rn = UnityEngine.Random.Range(0,dict.Count);
    48. int rn = rand.Next(dict.Count-1);
    49. string prefix = dict.Keys.Skip(rn).Take(1).Single();
    50. output.AddRange(prefix.Split());
    51. while (true) {
    52. var suffix = dict[prefix];///////KeyNotFoundException: The given key was not present in the dictionary.
    53. if (suffix.Count == 1) {
    54. if (suffix[0] == "") {
    55. return output.Aggregate(Join);
    56. }
    57. output.Add(suffix[0]);
    58. } else {
    59. rn = rand.Next(suffix.Count-1);
    60. //rn = UnityEngine.Random.Range(0,suffix.Count);
    61. output.Add(suffix[rn]);
    62. }
    63. if (output.Count >= outputSize) {
    64. return output.Take(outputSize).Aggregate(Join);
    65. }
    66. n++;
    67. prefix = output.Skip(n).Take(keySize).Aggregate(Join);
    68. }
    69. }
    70. void Update() {
    71. if (Input.GetKeyUp("i"))
    72. {
    73. Debug.Log(Markov(sad, 3, 10));
    74. }
    75. }
    76. }
     
  6. Yoreki

    Yoreki

    Joined:
    Apr 10, 2019
    Posts:
    2,605
    Please pay attention for the formatting in the code tags. For some reason there are no spaces in the newest example.
    That said
    i dont think this is causing your problem, but you mixed both approaches here. The -1 fixes the problem if you use Random.Range, for Random.Next it is not required.

    Right now i dont see what's causing the problem. You may wanna find some more information about the problem, for example by using Debug.Log to print the prefix, or other useful information. That way you'll see what kind of state results in the exception.
    Right now i cant do tests in Unity, but i may have a look at it later today. Does the script work as a standalone and the exception gets thrown with the "sad" string?

    Edit: What does not work if you just copy the entire original C# implementation? What did you change? If it throws the exception with the original implementation you copied, then the problem is probably linked to the algorithm implementation itself., about which i dont know a whole lot.
     
  7. doctorpangloss

    doctorpangloss

    Joined:
    Feb 20, 2013
    Posts:
    270
    Yes you should just use the original implementation unaltered and see if it works.
     
  8. cyphem

    cyphem

    Joined:
    Apr 18, 2014
    Posts:
    5
    Thanks,
    Yoreki, I was getting less errors using both approaches. Nonetheless the same error is still there when I use -1 with Random.Range, or just Random.Next.
    Yes I am testing this with a bare bones project with the script dropped onto a games object. And, yes I did try to print as many variables that I could, to try to find some kind of pattern to the problem. But all of this is just a little out of my skill set.
    Doctorpangloss, sorry to say that with my limited coding experience I did not recognise some of the code in the original code and so hacked it to what I thought would work in Unity. My project is also intending to use Webgl and so I did not want to use the Streamreader operation- just using what I know. But I am looking into doing as you say and trying to use the namespace original version. Still learning.
     
  9. Yoreki

    Yoreki

    Joined:
    Apr 10, 2019
    Posts:
    2,605
    I've had a closer look at the problem and it's actually really easy to fix.

    How to find the problem:
    • When logging the prefix, after 2-3 errors it becomes clear that it always crashes on the same word: "your"
    • The next logical thing to do is take a look at where "your" appears in your string. It's at the end.
    • What is different at the end?

    How to fix the problem:
    Add sad += " "; to your start method to add a space behind whatever string you use. Or do so manually.

    Why? I'm not entirely sure, but somehow the algorithm works with the word, should add it as a key, but does not unless it's followed by a space. I'll edit if i find the culprit, but at least the problem is solved.

    For me at least, the problem does not occur anymore. And i've spammed the living sht out of my i key.
     
    Last edited: Sep 23, 2019
  10. cyphem

    cyphem

    Joined:
    Apr 18, 2014
    Posts:
    5
    Awesome- Thanks so much for your help I appreciate it!