This is really interesting, but as you might anticipate, it is filled with confident untrue statements. Not as many as I expected tbh, but plenty once you start looking, especially into anything that might be a bit niche!
I have to say I don't think LLMs and encyclopedias are a good fit. But some tools that might ease the tension would include (a) manual editing, unless I've just missed this? (b) an "obliterate" button where you could just tag the hallucinations quickly (c) disclaimers on every page that indicate the experimental nature and the potential for inaccurate information (d) a way to delete articles (e) cross checking article titles with Wikipedia, and if a new article does not exist in Wikipedia, then some extra check, e. g. manual approval, or a limit to the number that can be created by each account per week
As a writer, I had the exact same response. But, then I realized that this is where we are headed--learning to prompt-as-editors, versus doing the edits ourselves. (In some areas, of course, not in others.) So, I started approaching these as exercises in learning how to prompt for content instead of get content as a first draft and then editing it.
It's a paradigm shift for sure, and I think it's a pretty important one.
This is super super interesting! Thanks for doing it. It immediately made me think something analogous could exist for programming questions - like a Stack Overflow, where I publish *both* sides of my ChatGPT conversations about programming, and they're shared and rated and curated.
Some feedback so far:
* I couldn't find the edit button. The pencil wasn't very obvious to me! And I was expecting to edit the whole article, and it be a button in the top navigation bar.
* Upon editing, I couldn't see the prompt!! It just lets me add to the prompt? I can't find a way to view the existing prompt. Which is not that fun. What's the reasoning behind not letting people edit a prompt? Or even see the prompt as far as I can tell?
Thanks for the feedback. I'll make editing more obvious.
The reason you added a section at a time is basically because I can't afford the API fees to send the entire article back and forth every time at the moment.
You can edit the prompt. At the bottom in the text box. Just enter whatever you want to edit that section, for example, add more detail or make this section more concise, or add some information about X or Y. What you enter in that box is sent to GPT along with the section text and then the edited response is reinserted into the page...
Oooo, I hadn't thought of that! So the permanent item *is* the encylopedia section text - the prompts used to modify that text are thrown away? I think it could at least show them in the edit history, with a diff of the text!
The way I was thinking of it was very different - it was a prompt to get a model to write the section of page. And it would even upgrade - so if a better model was found, the whole encylopedia would be rewritten with it. And tweaks would be done by expanding / editing the prompt, there'd be a system prompt too and probably category prompts etc.
Lots of design space to play around in with this kind of thing! Great.
Basically everything that you said is correct. I think you might just be missing that. The prompts and the history and revert is all there. Just click the floating double dagger next to the article name.
Aha! I hadn't noticed the floating double dagger at all. Well maybe subconsciously, but I had no idea it was clickable. Thanks!
I still can't see any prompts though? That shows the history of the article text. And I can diff it. But where can i see a prompt that wrote the article?
I love the idea of a 'obliterate' button to tag hallucinations - that could be kind of fun
This is really interesting, but as you might anticipate, it is filled with confident untrue statements. Not as many as I expected tbh, but plenty once you start looking, especially into anything that might be a bit niche!
I have to say I don't think LLMs and encyclopedias are a good fit. But some tools that might ease the tension would include (a) manual editing, unless I've just missed this? (b) an "obliterate" button where you could just tag the hallucinations quickly (c) disclaimers on every page that indicate the experimental nature and the potential for inaccurate information (d) a way to delete articles (e) cross checking article titles with Wikipedia, and if a new article does not exist in Wikipedia, then some extra check, e. g. manual approval, or a limit to the number that can be created by each account per week
As a writer, I had the exact same response. But, then I realized that this is where we are headed--learning to prompt-as-editors, versus doing the edits ourselves. (In some areas, of course, not in others.) So, I started approaching these as exercises in learning how to prompt for content instead of get content as a first draft and then editing it.
It's a paradigm shift for sure, and I think it's a pretty important one.
This is super super interesting! Thanks for doing it. It immediately made me think something analogous could exist for programming questions - like a Stack Overflow, where I publish *both* sides of my ChatGPT conversations about programming, and they're shared and rated and curated.
Some feedback so far:
* I couldn't find the edit button. The pencil wasn't very obvious to me! And I was expecting to edit the whole article, and it be a button in the top navigation bar.
* Upon editing, I couldn't see the prompt!! It just lets me add to the prompt? I can't find a way to view the existing prompt. Which is not that fun. What's the reasoning behind not letting people edit a prompt? Or even see the prompt as far as I can tell?
Thanks!
Thanks for the feedback. I'll make editing more obvious.
The reason you added a section at a time is basically because I can't afford the API fees to send the entire article back and forth every time at the moment.
You can edit the prompt. At the bottom in the text box. Just enter whatever you want to edit that section, for example, add more detail or make this section more concise, or add some information about X or Y. What you enter in that box is sent to GPT along with the section text and then the edited response is reinserted into the page...
Hope that makes sense?
Oooo, I hadn't thought of that! So the permanent item *is* the encylopedia section text - the prompts used to modify that text are thrown away? I think it could at least show them in the edit history, with a diff of the text!
The way I was thinking of it was very different - it was a prompt to get a model to write the section of page. And it would even upgrade - so if a better model was found, the whole encylopedia would be rewritten with it. And tweaks would be done by expanding / editing the prompt, there'd be a system prompt too and probably category prompts etc.
Lots of design space to play around in with this kind of thing! Great.
Basically everything that you said is correct. I think you might just be missing that. The prompts and the history and revert is all there. Just click the floating double dagger next to the article name.
Aha! I hadn't noticed the floating double dagger at all. Well maybe subconsciously, but I had no idea it was clickable. Thanks!
I still can't see any prompts though? That shows the history of the article text. And I can diff it. But where can i see a prompt that wrote the article?