Group Abstract Group Abstract

Message Boards Message Boards

[Cash] $100 bounty: design a chat bot of chained responses

I need a function written and don't have time to do it so I am offering $100 to whoever can write the shortest and most simple example code that accomplishes my goals.

I use regular expressions (regex) a lot in my NLP projects. There is a bastardization of regex called spintax.

It is basically just {content|other content} chained to other {alternatives|strings}.

It usually comes nested as well. so it could be {phrases|some {other|similar} content} chained to similar {alternative {synonyms|strings}|parts of speach}

In real life, it looks like this:

"{{I need|I want|I would like} |}{help|support|assistance|help support|{customer support|customer care|customer service|support service|support services|consumer support|client care|client service|service}}{ {please|if possible|if you please|bear in mind|pls}|}"

This outputs variations of the sentence such as:

  • I want help if possible
  • I need support please
  • I want help if you please
  • I need assistance

I need a function where I can just paste this spintax code into a variable and have it spit out these variations.

I imagine it could be something like:

spinTax = 
  "{{I need|I want|I would like} |}{help|support|assistance|help support|{customer support|customer care|customer service|support service|support services|consumer support|client care|client service|service}}{ {please|if possible|if you please|bear in mind|pls}|}";

variationCount = 4;

spinFunc = (*Some cool function that spits out variations*)

spinClean = Union[spinFunc]; (*Clean up the list, remove duplicates, etc. Run function again until desired count of unique variations is reached*)

Out[10] = {"I want help if possible", "I need support please", "I want help if you please", "I need assistance"}

Here is my code in PHP that outputs in json. Maybe this can help. 

  # Show results.

  if($output['success']=='true'){

class Spintax
{
    public function process($text)
    {
        return preg_replace_callback(
            '/\{(((?>[^\{\}]+)|(?R))*)\}/x',
            array($this, 'replace'),
            $text
        );
    }

    public function replace($text)
    {
        $text = $this->process($text[1]);
        $parts = explode('|', $text);
        return $parts[array_rand($parts)];
    }
}

$spintax = new Spintax();
$string = $output['output'];

$spunVariations = array();
for ($x = 0; $x <= $variationCount; $x++) {
$spunVariations[] = $spintax->process($string);
}
$spunVariations= array_keys(array_count_values($spunVariations));
$x = count($spunVariations, COUNT_RECURSIVE);

$spunVariations= array_keys(array_count_values($spunVariations));
 $arr = $output['output'];
    //echo json_encode($arr);
    echo json_encode($spunVariations);

  }
  else{
    echo json_encode("Error:". $output[error]);
  }
}
else{
  # There were errors.
    echo json_encode("Error:". $output[error]);
}

This PHP example doesn't have the function of ensuring that the resulting list has only unique values in it. It also doesn't have any checks to make sure the resulting sentences are greatly different from each other.
POSTED BY: David Johnston
17 Replies
Sort By:

@Sander Huisman

I sent you an email about getting you paid. Please check your spam folder.
POSTED BY: David Johnston

Bounty received! Enjoy your code!
POSTED BY: Sander Huisman

Very interesting. Thank you very much for contributing! You are awesome for that!

I think Sander's code does split up the string.

spinTax = 
  "{{I need|I want|I would like} |}{help|support|assistance|help support|{customer support|customer care|customer service|support service|support services|consumer support|client care|client service|service}}{ {please|if possible|if you please|bear in mind|pls}|}";

How do you think your idea would work with the nested code?

Its not simple {phrase|phrase} {phrase|phrase} {phrase|phrase}.

There is nesting inside like {{phrase|phrase}|} {phrase|{synonym|synonym}}{ {phrase|phrase}|}

It's hard to deal with spacing because, as an example, { phrase|} means phrase with a space in front of it OR nothing. So, sometimes whole sections of the sentence need to have the option of not appearing at all.

Notice in your output that it is only generating complete sentences. This is not the desired result due to the fact that user queries are often broken parts of sentences.

Thoughts?
POSTED BY: David Johnston

David,

The spaces are relatively straightforward -- you should keep the phrases as lists and at the end, your function for joining the lists into a string can add spaces.

To make the phrase optional, just add a "" as the last element in the list. For example, firstpart = {"I need", "I want", "I would like", ""}; This phrase now becomes optional.

I have not spent time on the nesting but my first inclination is to just nest the Outer function. All of the logic you might want to apply (Union, Intersection, And, Or) is all available within Mathematica if you handle everything as lists. for example, If you want {a | b} {{c | d}{e| f}} it becomes Outer[{a,b},Outer[{c , d}{e, f}]] This list has the correct nested structure -- you just need to join the strings in the end.
POSTED BY: Neil Singer

Splitting up the string is not that straight-forward, though not impossible. But the creation of your nested Outers is not that easy to do algorithmically. Furthermore this method won't work with some of the strings David gave; it will create millions and millions of entries and it will just crash; i had the same problem on my first try, see above. The problems seems easy at first, but is more tricky than I thought it would be. Outer can indeed be used quite nicely, (or in the same way StringJoin/@Tuples[...] which might actually be a bit faster, Outer is known to be slow-ish because it doesn't use any vectorisation). Spaces should not be a big issue indeed... You could even add spaces between all the items, and then later remove double-spaces...
POSTED BY: Sander Huisman

Here is a situation in which changing your data structure a bit can make things much easier. If you start with lists of strings instead of a single string (or turn your strings into lists of strings) you can do this with one function ("Outer") in Mathematica: Here are two ways to do it depending on if your inputs are separated or together.

In[7]:= firstpart = {"I need", "I want", "I would like"};
secondpart = {" help", " support", " assistance", " customer support"};
thirdpart = {" please", " if possible"};

allparts = {firstpart, secondpart, thirdpart};

In[11]:= Outer[StringJoin, firstpart, secondpart, thirdpart]

Out[11]= {{{"I need help please", 
   "I need help if possible"}, {"I need support please", 
   "I need support if possible"}, {"I need assistance please", 
   "I need assistance if possible"}, {"I need customer support \
please", "I need customer support if possible"}}, {{"I want help \
please", "I want help if possible"}, {"I want support please", 
   "I want support if possible"}, {"I want assistance please", 
   "I want assistance if possible"}, {"I want customer support \
please", "I want customer support if possible"}}, {{"I would like \
help please", 
   "I would like help if possible"}, {"I would like support please", 
   "I would like support if possible"}, {"I would like assistance \
please", "I would like assistance if possible"}, {"I would like \
customer support please", 
   "I would like customer support if possible"}}}

In[12]:= Outer @@ Join[{StringJoin}, allparts]

Out[12]= {{{"I need help please", 
   "I need help if possible"}, {"I need support please", 
   "I need support if possible"}, {"I need assistance please", 
   "I need assistance if possible"}, {"I need customer support \
please", "I need customer support if possible"}}, {{"I want help \
please", "I want help if possible"}, {"I want support please", 
   "I want support if possible"}, {"I want assistance please", 
   "I want assistance if possible"}, {"I want customer support \
please", "I want customer support if possible"}}, {{"I would like \
help please", 
   "I would like help if possible"}, {"I would like support please", 
   "I would like support if possible"}, {"I would like assistance \
please", "I would like assistance if possible"}, {"I would like \
customer support please", 
   "I would like customer support if possible"}}}

Note that the output has every combination as a nested list (which might be useful to you or you may want to Flatten it). Also, I added spaces to the second and third parts to make the final strings readable. You could easily do this programmatically or as part of your input.
POSTED BY: Neil Singer

Note that the creation of non-similar entries automatically leads to a n^2 algorithm as one has to compare a lot of strings with each other; that is, it gets more and more computer intensive to calculate many examples. However we can get better suggestion by the following procedure:

Say we want to create 50 entries total. What we do is, we create 5x that: 250 candidates:

ClearAll[CleanPhrase,SplitUp,SplitUpAll]
CleanPhrase[x_String]:=StringTrim[StringReplace[x," "..->" "]]
SplitUp[s_String]:=StringReplace[s,"{"~~x:(Except["{"|"}"]..)~~"}":>RandomChoice[StringSplit[x,"|",All]]]
SplitUpAll[x_String]:=CleanPhrase[FixedPoint[SplitUp,x]]
SplitUpAll[x_String,n_Integer?Positive]:=Table[SplitUpAll[x],n]

spinTax="{How much|What amount|Exactly how much|How much money|What amount of money|What} {would|could|will|should|does} {it cost| the price be} {for|for the|for your|for about} {{1|2|3|4|5|6|7|8|9}{0{k|}|00{k| thousand}|000|0000}|{2|3|4|5}00000}{ {monthly|month to month|per month|every month|once a month|reoccurring|}{ {account|subscription|membership}{ level|}|}{ {service|services|program|service|plan|plans|}|}| credits|}";
variationCount=50;
candidates=SplitUpAll[spinTax,5*variationCount];

candidates will now hold 250 samples. We create the so-called NearestNeighborGraph for these entries. This will look like something like this:

enter image description here

Now each point (Vertex) is connected to it's closest other strings. So a string with many connection is close to many other; we want to eliminate those, and 'disconnect' the graph. For each vertex we ask the number of connections and then pick those vertices whose connections are the smallest:

ClearAll[CleanPhrase,SplitUp,SplitUpAll]
CleanPhrase[x_String]:=StringTrim[StringReplace[x," "..->" "]]
SplitUp[s_String]:=StringReplace[s,"{"~~x:(Except["{"|"}"]..)~~"}":>RandomChoice[StringSplit[x,"|",All]]]
SplitUpAll[x_String]:=CleanPhrase[FixedPoint[SplitUp,x]]
SplitUpAll[x_String,n_Integer?Positive]:=Table[SplitUpAll[x],n]
NonSimilarSplitUpAll[x_String,n_Integer,m_Integer:5]:=Module[{candidates,nng,vl,vd},
    candidates=Union[SplitUpAll[x,n m]];
    nng=NearestNeighborGraph[candidates,3];
    vl=VertexList[nng];
    vd=VertexDegree[nng];
    MinimalBy[{vl,vd}\[Transpose],Last,n][[All,1]]
]

testing out:

spinTax="{How much|What amount|Exactly how much|How much money|What amount of money|What} {would|could|will|should|does} {it cost| the price be} {for|for the|for your|for about} {{1|2|3|4|5|6|7|8|9}{0{k|}|00{k| thousand}|000|0000}|{2|3|4|5}00000}{ {monthly|month to month|per month|every month|once a month|reoccurring|}{ {account|subscription|membership}{ level|}|}{ {service|services|program|service|plan|plans|}|}| credits|}";
variationCount=50;
NonSimilarSplitUpAll[spinTax,50]

should give more 'diverse' choices.
POSTED BY: Sander Huisman

Here is what I got so far. I am pre

spinTax = 
  "{How much|What amount|Exactly how much|How much money|What amount of money|What} {would|could|will|should|does} {it cost| the price be} {for|for the|for your|for about} {{1|2|3|4|5|6|7|8|9}{0{k|}|00{k| thousand}|000|0000}|{2|3|4|5}00000}{ {monthly|month to month|per month|every month|once a month|reoccurring|}{ {account|subscription|membership}{ level|}|}{ {service|services|program|service|plan|plans|}|}| credits|}";
variationCount = 5;
varationLoop = 0;
spunList = {};

ClearAll[SplitUp, SplitUpAll]
SplitUp[s_String] := 
 StringReplace[s, 
  "{" ~~ x : (Except["{" | "}"] ..) ~~ "}" :> 
   RandomChoice[StringSplit[x, "|", All]]]
SplitUpAll[x_String] := FixedPoint[SplitUp, x]
SplitUpAll[x_String, n_Integer?Positive] := Table[SplitUpAll[x], n]


spunList = {};
While[varationLoop < variationCount,
 AppendTo[spunList, 
  StringTrim[StringReplace[SplitUpAll[spinTax], "  " -> " "]]];
 varationLoop++;
 ]
spunList = Union[spunList];
varationLoop = 0;

Dynamic[Dataset[spunList]]

"How much does it cost for 500k once a month subscription level"
"How much does the price be for 300000 services"
"How much should the price be for your 600k"
"What amount should the price be for the 20k credits"
"What will the price be for your 200000 credits"
POSTED BY: David Johnston

The while loop is not necessary, one can give a second argument:

ClearAll[CleanPhrase,SplitUp,SplitUpAll]
CleanPhrase[x_String]:=StringTrim[StringReplace[x," "..->" "]]
SplitUp[s_String]:=StringReplace[s,"{"~~x:(Except["{"|"}"]..)~~"}":>RandomChoice[StringSplit[x,"|",All]]]
SplitUpAll[x_String]:=CleanPhrase[FixedPoint[SplitUp,x]]
SplitUpAll[x_String,n_Integer?Positive]:=Table[SplitUpAll[x],n]

spinTax="{How much|What amount|Exactly how much|How much money|What amount of money|What} {would|could|will|should|does} {it cost| the price be} {for|for the|for your|for about} {{1|2|3|4|5|6|7|8|9}{0{k|}|00{k| thousand}|000|0000}|{2|3|4|5}00000}{ {monthly|month to month|per month|every month|once a month|reoccurring|}{ {account|subscription|membership}{ level|}|}{ {service|services|program|service|plan|plans|}|}| credits|}";
variationCount=5;
Union[SplitUpAll[spinTax,variationCount]]//Column

giving:

How much does the price be for your 200000
How much money does it cost for 3000 credits
How much money should the price be for about 1000 credits
How much money would the price be for 500000 month to month service
How much should the price be for about 60000 account plans
POSTED BY: Sander Huisman

Thank you. You are amazing! :) So much appreciated!

Can you post the notebook as I was unable to get these to work via copy-paste. Were you able to get it to work with each of the samples?

The idea is not to generate "All" variations but to generate up to the amount that is in the variable "variationCount=50" or whatever number. Otherwise, it can run forever if the Spintax is complex.

Also, the idea is not to just simply generate "Random" samples but to generate the most "Unique" and high-quality samples. Maybe there should be a variable like "similarityGen="true" or "false" so we can differentiate between them.

I was thinking of generating a random sample and then generating a large number of samples and then using Nearest function in an iterative loop to get the "True" result of very similar samples. However, I was not sure on how we could do the "False" and generate variations that are not similar to each other.

"True" result:

  • I need help
  • need help
  • needs help

"False" result:

  • I need help
  • would like assistance
  • I require support please

What would be even more awesome is if it can incorporate grammar/spell checking as part of its quality check. That would be amazing. Contact me directly so I can get you paid. :)

Also, if anyone else wants to put forth a better solution, I will also pay you in addition to Sander.
POSTED BY: David Johnston

Generating all gets very tricky as the number of possibilities goes exponential as you may have noticed, to get some random choices one can do:

ClearAll[SplitUp,SplitUpAll]
SplitUp[s_String]:=StringReplace[s,"{"~~x:(Except["{"|"}"]..)~~"}":>RandomChoice[StringSplit[x,"|",All]]]
SplitUpAll[x_String]:=FixedPoint[SplitUp,x]
SplitUpAll[x_String,n_Integer?Positive]:=Table[SplitUpAll[x],n]

now the second variable n will give you n random choices (variations). If I understood your php code correctly, I think this is what it does.
POSTED BY: Sander Huisman

Ok, this must be the minimised version, can't really compress it much more without it being completely unreadable...
POSTED BY: Sander Huisman

Here is some updated code with more use of built-in functionality:

ClearAll[PreProcess, SplitUp, SplitUpAll]
PreProcess[x_String] := FixedPoint[StringReplace[#, "{}" -> ""] &, x]
SplitUp[s_List] := Join @@ (SplitUp /@ s) 
SplitUp[s_String] := Module[{pos, parts},
  pos = StringPosition[s, "{" ~~ Except["{" | "}"] .. ~~ "}"];
  If[Length[pos] >= 1,
   pos = pos[[1]];
   parts = StringSplit[StringTake[s, pos + {1, -1}], "|", All];
   StringReplacePart[s, #, pos] & /@ parts 
   ,
   {s} 
   ]
  ]
SplitUpAll[x_String] := Union[FixedPoint[SplitUp, PreProcess[x]]]
SplitUpAll[x_String, n_Integer?Positive] := RandomSample[SplitUpAll[x], n]

Bracketless is gone, can be done (and faster) with Except. Note that PreProcess could be removed. But I'm still not sure what you want with variationCount
POSTED BY: Sander Huisman

Here are two more samples of the spintax:

"{How much|What amount|Exactly how much|How much money|What amount of money|What} {would|could|will|should|does} {it cost| the price be} {for|for the|for your|for about} {{1|2|3|4|5|6|7|8|9}{0{k|}|00{k| thousand}|000|0000}|{2|3|4|5}00000}{ {monthly|month to month|per month|every month|once a month|reoccurring|}{ {account|subscription|membership}{ level|}|}{ {service|services|program|service|plan|plans|}|}| credits|}"

and

"{{I'm|I am|Now i'm|I am just|Now I am|I'm just|We are|We're|I have been|I'm}{ {doing|trying to do|certainly|definitely|undoubtedly|surely|absolutely|most certainly|actually|really|truly|completely|totally|genuinely|evidently|just|very|exceptionally}|}|{doing|trying to do|certainly|definitely|undoubtedly|surely|absolutely|most certainly|actually|really|truly|completely|totally|genuinely|evidently|just|very|exceptionally}} {fine|good|great|okay|excellent|wonderful|fantastic|alright|ok|all right|exceptional|well|superb|amazing|decent|bad|poor|terrible|awful|lousy|horrible|dreadful|less than ideal|less than perfect|below average|crappy|disastrous|not good|wonderful|splendid|marvelous|awesome|remarkable|extraordinary|incredible|fabulous|magnificent|brilliant|outstanding|exceptional|good|lovely|phenomenal|spectacular|marvellous|glorious}"

and

"{{Specifically|Precisely|Exactly|Just|Really} how|How|How} {are u|r u|r you|are you|do you {think|believe|feel|assume|believe that|suppose|presume|think that|sense|feel that} you're|do you find yourself} {{actually|really|in fact|truly|basically|essentially|in reality|genuinely|literally|realistically|in truth|seriously|honestly|fundamentally|in essence|generally}|} {doin|doing|getting along|getting by|getting through|managing|feeling|enduring|surviving|sustaining}{ {today|these days|nowadays|right now|currently|now|at this time|at present|presently|in these days|in the present day|at the moment|now a days|right this moment|at this moment|as we speak|at this point|recently|this morning|this afternoon|lately|as of late|of late|in this time}|}?"
POSTED BY: David Johnston

Wow! Very fast response! I greatly appreciate your work! Thank you very much!

I ran the code and it works. :) As I examined it, it appears that you are a very advanced coder and accomplished the task by manually building the logic steps. It is very complicated. Maybe more complicated than my PHP code. It seems like Java to me. ;)

I assumed there are more ways to simplify using some of Wolfram's built-in string manipulation functions and other Wolfram functions?

Also, since I am building this as an API, it needs the variationCount parameter. You use SplitUpAll twice so I am not sure how to manipulate the resulting list.

Again, thank you. If it can't be simplified and made easier to work with, I will still pay. Hopefully somehow its possible though. :)
POSTED BY: David Johnston
POSTED BY: Sander Huisman
POSTED BY: Sander Huisman
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard