System Prompt Needed
planned
DENNIS KIPNIS
Hi, Team
I’m programming using Sonnet 3.5, but the version of Sonnet 3.5 on Merlin (getmerlin.in) doesn’t support system prompts, which are crucial and indispensable for me when programming with LLMs. System prompts are essential for ensuring that LLMs consistently follow my coding and commenting preferences—like opting for Remix instead of Next.js or avoiding certain Python libraries known for causing conflicts—without the need for repeated corrections throughout long conversations.
Additionally, the generated code isn’t as accurate or detailed as what I get from the claude.ai or claude-3.5-sonnet-20240620 API, which means spending more time on reviewing. I’m not sure if this is related to the model parameter settings (like lower temperature), but I hope Merlin can improve and provide answers that are on par with the response quality of Claude Official (claude.ai).
I chose Merlin because I trust its potential, and I hope it lives up to that trust. Thank you for all your hard work.
endu
planned
S
Siddhartha
Hey DENNIS KIPNIS, we use the same API
claude-3-5-sonnet-20240620
, with infact upto 8K generation tokens (it doesn't reach that point generally). We use the default temperature of 1.0. So ideally there should not be a difference.Can you please share the prompt you felt a different and if you could please describe it. Thanks
DENNIS KIPNIS
Hello, Siddhartha,
Apologies, but due to confidentiality requirements, I can’t provide code related to the production project. However, I can share the prompt I created for my child and the resulting conversation to illustrate the distinct differences I mentioned earlier.
From the first set of responses, you can notice the difference in their expressions. Besides the characteristic "I understand...," Claude sounds like an applicant nervously preparing for an interview, almost self-affirming as it repeats my instructions and elaborates on the task steps. This is typical of Claude, being more human-like and offering a more guided interaction. On the other hand, Merlin’s response is much more concise, sometimes so much so that it feels almost like GPT-4O.
From the second set of responses, we can see that there are differences in both obedience to the instructions and content delivery. Claude is more obedient to my instructions, providing content that’s relatively concise (likely influenced by Claude’s own guidance to offer "thorough yet concise explanations"). It strictly followed the format I requested. Merlin, on the other hand, though offering richer content, critically deviated from the format, meaning I had to reformat it myself to optimize space usage when printing.
If you need, I can also provide some additional examples after preparing them. To be concise, I feel that Merlin's response style is similar to Claude's, but it lacks a bit in terms of obedience to instructions. Additionally, the content tends to differ from Claude.ai's responses—sometimes it's more detailed, while at other times it's more concise. I hope this helps with your work. Thanks!
---
Note:
The chat log text is too long, so I am now providing it as an attachment via Filen Cloud.S
Siddhartha
Hey DENNIS KIPNIS, thanks a lot for this detailed answer! I do see your points. I have tried the same prompt now with claude (on Merlin) and this is the response I get
**indulge** /ɪnˈdʌldʒ/
> - **English meaning**: "Indulge" is a verb meaning to allow oneself or someone else to enjoy something desired, often something considered to be a luxury or not strictly necessary. It can also mean to yield to a desire or give in to temptation.
> - **Etymology information and root/affix**: The word "indulge" comes from the Latin *indulgere*, meaning "to give way to" or "to be lenient to." The prefix *in-* means "in" or "toward," while the root *-dulgere* relates to being lenient or kind.
> - **Evolution of word meaning**: Originally, "indulge" involved a sense of granting leniency or favor to someone. Over time, it evolved to encompass the notion of allowing oneself or others to partake in something pleasurable, often with a suggestion of excess or lack of restraint.
> - **Common collocations**: "Indulge in," "indulge a craving," "indulge one's passion," "indulge a whim," "indulge oneself."
> - **Cognates**: There are fewer direct cognates in English, but related terms include "indulgence" and "indulgent," which maintain the core idea of yielding to desires.
> - **Similar-looking words**: Words like "dull" or "bulge" look similar but are entirely unrelated in meaning and origin.
**Introduction to vocabulary usage**: Understanding a word's full range of usage helps capture the nuance in communication. By learning about the roots and evolution of words, such as "indulge," we not only improve our vocabulary but also gain insights into the cultural and historical contexts that shape language. This awareness enables us to articulate ideas with greater precision and empathy.
So this time, I think it does follow the outline better. So in terms of reasoning, I do think Claude on claude.ai and Merlin should be the same, but in terms of behaviours, they can be a bit different (even though we use their officical system prompt). This can be because they use a slightly different version of added prompts.
So please do let me know if there were any other prompts that caused this issue or any issue even with the response that I have sent out.
endu
DENNIS KIPNIS we are bringing in system prompt as part of PERSONALISATION soon; & thanks for trusting Merlin we will certainly build the best UX for all AI interactions possible.
Regarding the accuracy will have a chat with Tech team; ideally it should not be the case and could be an outlier but tagging in Siddhartha for more context and visibility.
Will update here more once discussed with Tech team, cheers
DENNIS KIPNIS
Hi, endu,
Thank you for the good news! I’m thrilled to hear that the introduction of system prompts is already under consideration. However, could you please let me know where I can find the relevant link to this goal, or where I should follow and stay updated on the progress? I’m really excited about this, and I deeply appreciate the team’s thoughtful consideration.
endu
DENNIS KIPNIS: there is no any public link/ document of our roadmap; will make one and share of the things we are working on (in development phase)by this weekend�