13

One can find advice on prompt engineering telling basically the following thing: if you are seeking advice about a topic X, start your prompt to an LLM by

You are an expert in X, you have pedagogical skills, and you are very good at synthesizing information blahblahblah...

I don't understand why this should work at all. Let's simplify the matter: assume an LLM is trained only on stackoverflow questions and answers. Since no question on SO starts with such a sentence, the pre-prompt actually makes the prompt more different from training data that it was before.

More generally, why would writing prompts like asking questions to a being, and moreover telling this being what it is give us better output? Moreover, is this even measurably true?

My hand-waving understanding of LLMs is that since they basically are excellent for predicting the following words of a context, they have, as a side-effect, learnt to answer questions just because they have learnt that on the Internet, questions are usually followed by answers.

Plop
  • 233
  • 2
  • 7
  • 1
    If only it was as easy for humans ... wait ... – user121330 Mar 13 '24 at 21:05
  • 2
    One of the benefits of prompt engineering is to find ways around the 'safety' blocks put in place by LLM's like GPT. Eg: it won't tell you how to build dangerous thing if you ask directly but it might if you preface the question by saying you are writing a sci fi book and want the science to be as accurate as possible. So the success of prompt engineering might have little to do with the technicalities of LLM's themselves. – eps Mar 13 '24 at 22:05
  • 1
    Just imagine the prompt telling the model which stack exchange to use. Makes a difference whether you ask on worldbuilding or physics. Tell the model that its a worldbuilder and it will retrieve different context. – DonQuiKong Mar 15 '24 at 23:09

6 Answers6

18

LLMs are usually fine-tuned on instruction-following datasets so that they can answer user requests. For instance, ChatGPT comes from InstructGPT, which was trained from GPT-3 with reinforcement learning with human feedback (RLHF) on an instructions dataset (you can learn more about the evolution of ChatGPT/GPT-4 from the original GPT in this answer).

As for the prompts that request the LLM to impersonate a role giving better results, you can check the article In-Context Impersonation Reveals Large Language Models' Strengths and Biases, which verifies the claim. The reason for that is difficult to figure out, given the black-box nature of LLMs and deep learning in general.

noe
  • 26,410
  • 1
  • 46
  • 76
9

I believe you misunderstand how LLMs work. They are not search engines that look through a data set trying to find a best match.

An LLM essentially (very simplified, and I'm not an expert) generates the most probable answer to a given question, using a generalized large data set of input data.

The "you are" and other context-giving information helps the LLM to narrow down where the most probably answer should come from. As a silly example, if I were to ask an LLM without any prompts "what is the best lunch?" it has a huge amount of data that matches "lunch" and no guidance how to rate "best", so it will mix and match everything from nutritional value to taste to ease of preparation. If I give it context information such as "as an expert on nutrition" it can weigh the nutritional value interpretation of "best" higher than, say, the taste.

On SO, context information can inform the LLM which community is more relevant to an answer, or which nearby concepts to include. While, say, "pedagogical skills" may not appear verbatim in a possible answer, the context of pedagogy is more likely to be present in a relevant community than in, say, answers about a computer game that by pure word-matching might fit.

Tom
  • 191
  • 2
  • 5
    I'm not sure this is correct? As far as I understand LLMs (and I could be misunderstanding them entirely), and sticking to the same level of simplification as you, they don't work at a response level, but a word-by-word level. There is no weighing of information, because the LLM has no conception of meaning in anything it responds with, it's just a very complex Markov generator. Prompting with certain words might change the likelihood of what words it responds with, but it's still fundamentally just picking words one by one as what's likely to follow what words have already been generated. – Idran Mar 14 '24 at 20:50
  • 4
    @Idran Yes, the output of a language model is generated bit-by-bit, but I'm not sure what that has to do with how the prompt influences the output. There definitely is weighing involved – that's quite literally the job of a neural network. LLMs produce text based on the prompt utilizing the gargantuan amount of weight data that encodes context and tone among other things, so Tom's explanation is very plausible. You could argue that the model doesn't deal with meanings, only text, but that's also orthogonal to the question and is a hot topic over at Philosophy Stack Exchange. :) – neonblitzer Mar 15 '24 at 00:24
  • 3
    @Idran yes, a full discussion of how LLMs work is too much for an answer here. You're essentially right except that, as far as I understand it, the unit isn't words but tokens - which can be one or several words or even just a part of a word. – Tom Mar 15 '24 at 09:23
  • @Idran I'm not an expert, but I think an LLM predicts the next token based on the whole context, including newly generate tokens – Heye Mar 19 '24 at 19:26
4

There are 3 layers to the answer why:

  1. ChatGPT is fine tuned with around 11,000 examples of humans asking questions, and receiving helpful and intelligent responses.
  2. On the surface, LLMs have only one job. To look at as much of the preceding text as possible, and give the most likely (or one of the top few most likely) next words. By taking the lead, you can say things that would be statistically more likely to be followed by intelligent responses.
  3. The transformer architecture encourages the model to look beneath the surface of language, and discover deep abstractions, and layer these abstractions into complex combinations. By triggering keywords with strong connections to your desired content, the model will be more likely to respond with the vibe you are hoping for.

And yes “vibe” is probably the most accurate way to describe this!

Danny Cisco
  • 41
  • 1
3

assume an LLM is trained only on stackoverflow questions and answers.

I think that it is where you are getting things wrong. LMM are trained on pile of texts (one such dataset is even known as "the pile"). Adding a prompt will help the model focus on text of better quality... but we can't really get further enough in that explanation as it is very difficult to tell exactly what is happening inside a LLM. Some of the effects are measurable (on a benchmark) but that's it.

Lucas Morin
  • 2,196
  • 5
  • 21
  • 42
  • 3
    This doesn't seem to answer the question. Yes, the pile is broader than StackOverflow questions used as an example by the OP, but doesn't it remain true that the pile doesn't contain many "improv game"-type prompt sentences asking the respondent to act like an expert? – Oddthinking Mar 15 '24 at 00:59
  • Without talking about RLHF, adding a word 'expert' change the embedding of your text. – Lucas Morin Mar 15 '24 at 08:44
3

I think this article is one of the best I've seen which describes LLMs in an accessible way without dumbing it down to the point where it's unhelpful: "Large language models, explained with a minimum of math and jargon"

One thing that might help you is to understand that LLMs store their 'knowledge' as vectors in a 'high-dimensional' space. As an example, here's a vector representation of 'cat':

[0.007398007903248072, 0.0029612560756504536, -0.010482859797775745, 0.0741681158542633, 0.07646718621253967, -0.0011427050922065973, 0.026497453451156616, 0.010595359839498997, 0.0190864410251379, 0.0038335588760674, -0.0468108132481575, -0.021150866523385048, 0.009098375216126442, 0.0030140099115669727, -0.05626726150512695, -0.039609555155038834, -0.09978967905044556, -0.07956799119710922, 0.057768501341342926, -0.017375102266669273, 0.015590683557093143, -0.022376490756869316, 0.10152265429496765, -0.05138462409377098, 0.025884613394737244, 0.07069036364555359, 0.0009145145886577666, -0.06275367736816406, 0.03610750287771225, 0.050807688385248184, -0.06453944742679596, -0.0434986837208271, -0.1264101266860962, -0.0003191891883034259, 0.04311852902173996, -0.14792846143245697, -0.019480768591165543, 0.01992032676935196, 0.011479354463517666, 0.02979433164000511, 0.06154156103730202, -0.04609882831573486, -0.053286727517843246, -0.016268745064735413, 0.03660176321864128, -0.07168425619602203, 0.05497466400265694, -0.1446477174758911, 0.09316877275705338, -0.1279120296239853, 0.030971739441156387, 0.03677519038319588, 0.13407474756240845, -0.028527621179819107, -0.10431249439716339, 0.03328850120306015, 0.1295083463191986, 0.0412190817296505, 0.03605308011174202, 0.0599723681807518, 0.025970442220568657, -0.03521350771188736, -0.015058198012411594, 0.005818498786538839, 0.013812823221087456, 0.015064566396176815, 0.022925062105059624, 0.039051759988069534, 0.007009583059698343, -0.02910810336470604, 0.1011449322104454, 0.13727356493473053, 0.022466043010354042, -0.07582768052816391, -0.04469817131757736, -0.06026916950941086, 0.04192522168159485, 0.1612275242805481, 0.014356226660311222, -0.0647699236869812, -0.14182332158088684, 0.07568981498479843, 0.002798931673169136, 0.012406392954289913, -0.09695082157850266, -0.0014245212078094482, -0.018527435138821602, 0.009911706671118736, 0.013058848679065704, 0.048697732388973236, 0.017661960795521736, 0.036917395889759064, 0.005680330563336611, 0.024947546422481537, 8.419259393122047e-05, -0.002204198157414794, -0.007295176852494478, 0.008355203084647655, -0.015072236768901348, -0.0032011312432587147, 0.05527794361114502, 0.020942343398928642, -0.019445667043328285, -0.15129604935646057, 0.0337672121822834, 0.0019582323729991913, -0.0014046517899259925, -0.05954226478934288, -0.08176489174365997, 0.024112699553370476, -0.1015794649720192, 0.05419696122407913, 0.13000570237636566, -0.05808615684509277, 0.004180640447884798, 0.01880498044192791, 0.01923936977982521, -0.041859131306409836, 0.010098426602780819, 0.025394367054104805, -0.03678150847554207, 0.03255629166960716, -0.008087233640253544, -0.07101460546255112, 0.024909185245633125, -0.0369131900370121, 0.035895638167858124, 0.0047763800248503685, -0.01754925213754177, -0.0029735821299254894, 0.030521586537361145, 0.04243304952979088, 0.05969628319144249, -0.07855783402919769, -0.07639002054929733, -0.004820443224161863, 0.0651308000087738, 0.13445857167243958, -0.06609761714935303, 0.01714201085269451, 0.019574925303459167, -0.00021718056814279407, 0.07559319585561752, 0.05964002385735512, -0.0715465098619461, 0.04068697988986969, -0.09640928357839584, -0.07235930114984512, -0.05935797095298767, 0.009602724574506283, -0.05649569258093834, 0.0025645969435572624, -0.05413592606782913, -0.017797887325286865, 0.05755465477705002, 0.08609342575073242, 0.050908517092466354, -0.05604008585214615, -0.005856652744114399, 0.02329830639064312, 0.08168350160121918, -0.0718611553311348, -0.027544423937797546, -0.08970167487859726, 0.024058541283011436, -0.02770240046083927, -0.025339743122458458, 0.010991393588483334, 0.02215300314128399, -0.02829679660499096, -0.07363404333591461, 0.0556303896009922, 0.0002929845068138093, -0.059732820838689804, -0.04813411086797714, -0.0021529451478272676, 0.004276854917407036, 0.04970701038837433, 0.02516869269311428, -0.05129590258002281, 0.0767771303653717, -0.08236679434776306, 0.019983036443591118, -0.05183032900094986, 0.05824366584420204, 0.047829821705818176, -0.13605566322803497, 0.02234281599521637, -0.03254450857639313, 0.011368651874363422, -0.05135396867990494, -0.00048283161595463753, -0.06719424575567245, -0.018972834572196007, 0.025254448875784874, -0.03858991339802742, 0.036364443600177765, -0.025158191099762917, 0.030907975509762764, -0.08114158362150192, 0.09369450062513351, 0.09405472874641418, 0.012534121051430702, -0.01041880901902914, 0.0552687831223011, 0.07056140154600143, 0.06628888100385666, 0.06548195332288742, 0.01580229587852955, -0.038310837000608444, -0.0032484608236700296, -0.010157674551010132, 0.085805244743824, 0.010575438849627972, 0.06210837885737419, -0.0071502267383039, -0.02955375239253044, 0.0289775263518095, 0.002539787907153368, -0.07370137423276901, 0.026873936876654625, 0.02770836278796196, 0.02373671904206276, 0.04336617887020111, 0.037974126636981964, 0.061377692967653275, 0.05020896717905998, -0.1109858900308609, -0.02423020824790001, 0.03785136342048645, 0.18769624829292297, 0.10594339668750763, -0.05118405446410179, 0.06405289471149445, -0.047474540770053864, 0.04021701216697693, -0.048911526799201965, 0.041514985263347626, -0.005742703098803759, 0.0034058222081512213, 0.01214022096246481, -0.037784647196531296, 0.008946173824369907, -0.030592333525419235, 0.039058126509189606, 0.02660788968205452, 0.05596623942255974, -0.03365514427423477, 0.09071480482816696, 0.034562114626169205, 0.08310434222221375, 0.03441822528839111, 0.003703191876411438, 0.002236866159364581, -0.06042943149805069, 0.06852643936872482, 0.09876436740159988, 0.01411499921232462, -0.07860662043094635, 0.06403335183858871, -0.1592547744512558, -0.01012679934501648, -0.10094276070594788, 0.01604175567626953, 0.006357499398291111, 0.02171235904097557, 0.01998433656990528, -0.029795801267027855, 0.020991159602999687, 0.027527112513780594, 0.07752928882837296, -0.01912834122776985, -0.10472745448350906, -0.0327356792986393, -0.11220412701368332, 0.03347017243504524, -0.04368103668093681, -0.00044717983109876513, -0.029803894460201263, 0.06123579293489456, 0.039308369159698486, -0.055449601262807846, 0.07417158037424088, -0.022331053391098976, -0.11767527461051941, -0.04385286569595337, -0.019754905253648758, 0.031432103365659714, 0.03378641605377197, 0.07572634518146515, -0.04749307036399841, -0.005324371624737978, -0.08255213499069214, -0.010222465731203556, 0.021690042689442635, -0.1339070200920105, 0.007615163456648588, -0.0929502621293068, 0.05977592244744301, 0.00015643733786419034]

The distance and mathematical relationships between vectors is an essential part of how an LLM's abilities emerge.

This is likely an over-simplificiation but the way I think about it (I may have read this) is that prompts help orient the LLM context within its vector space. Prompt engineering is essentially finding ways to shift that context to a place in that vector space where it can produce better (or worse, depending on your goal) results.

JimmyJames
  • 131
  • 3
3

The short answer: engineered prompts are deliberately similar to fragments from training data, and this similarity is sufficient to elicit desired responses. Prompts do not need to literally appear in the training data.

Suppose we have a series of answers to some question, and we are training a question-answering language model. The model learns that facets of the answers are related to the question, and it also learns that facets of the answers are internally related to other facets of themselves. This latter learned behavior allows the start of an answer to causally influence the rest of that answer.

Now, suppose that some answers are written with qualifications at the start. For example, answers might start with "in my experience doing computer science," or "as a computer scientist," or "I am a computer scientist, and". These starts are similar to each other, and we can make a probabilistic generalization about their continuations: answers which start this way are likely to continue a certain way.

For the prompt engineer, this is usually enough to be able to infer that a prompted answer could start with something like, "I am a computer scientist." in order to get answers which are continued as if they were originally written with that starting prompt.

Note that this engineered prompt may not yet return decent answers. It could be the case that anybody who self-identifies as a computer scientist also writes poor answers, in which case this prompt would decrease the quality of completions. Prompt engineers have to consider word choice, just like writers in other disciplines.

Also note that part of your question is really about the usage of commands to the second-person "you". This is specific to models which have post-training (usually RLHF); they are trained to have an additional conversational behavior which doesn't necessarily exist in their training data. If I am writing on my own, then I can use a first-person perspective; I only need to talk to you in a conversational context. This is a non-trivial insight required to get conversations out of models which weren't trained for it; the model needs to be told who is "I" and "you", requiring additional prompting, framing, and parsing.

Corbin
  • 131
  • 3