DPO 偏好对数据集 Template

interface DPOExample {
  prompt: string;                    // The user query or instruction
  chosen: string;                    // The preferred response
  rejected: string;                  // The dispreferred response
  metadata?: {
    preference_reason: string;       // Why chosen is preferred
    category: string;
    difficulty: "easy" | "medium" | "hard";
    quality_delta: "large" | "medium" | "small";  // How different the pair is
  };
}

// Multi-turn DPO format
interface MultiTurnDPOExample {
  system?: string;
  prompt: {
    role: "user" | "assistant";
    content: string;
  }[];                               // Conversation history up to the divergence point
  chosen: string;                    // Preferred final assistant response
  rejected: string;                  // Dispreferred final assistant response
}

[
  {
    "prompt": "Explain how HTTPS works to someone who isn't technical.",
    "chosen": "Think of HTTPS like sending a letter in a locked box instead of on a postcard. When you visit a website with HTTPS, your browser and the website first agree on a secret code that only they know. After that, every message between you and the website is scrambled using that code. Even if someone intercepts the message, they can't read it because they don't have the code. That little padlock icon in your browser means this protection is active.",
    "rejected": "HTTPS uses TLS/SSL protocol to encrypt data in transit between the client and server. It initiates a handshake process where the server presents its X.509 certificate, the client verifies it against trusted certificate authorities, and then they negotiate a symmetric session key using asymmetric cryptography. All subsequent data is encrypted with AES-256 or similar cipher suites, protecting against man-in-the-middle attacks and packet sniffing.",
    "metadata": {
      "preference_reason": "The chosen response uses an accessible analogy appropriate for the non-technical audience. The rejected response is technically accurate but uses jargon that would confuse the target audience.",
      "category": "audience-appropriateness",
      "difficulty": "medium",
      "quality_delta": "medium"
    }
  },
  {
    "prompt": "What are some healthy breakfast options?",
    "chosen": "Here are some nutritious breakfast ideas that are quick to prepare:\n\n- **Overnight oats**: Mix rolled oats with yogurt and milk, add berries and chia seeds, refrigerate overnight. Ready to eat in the morning.\n- **Eggs and avocado toast**: Whole grain toast with mashed avocado and a poached or scrambled egg. Add everything seasoning for extra flavor.\n- **Smoothie bowl**: Blend frozen banana, spinach, protein powder, and almond milk until thick. Top with granola, sliced fruit, and nuts.\n- **Greek yogurt parfait**: Layer plain Greek yogurt with mixed berries and a drizzle of honey.\n\nAll of these take under 10 minutes and provide a good balance of protein, healthy fats, and complex carbohydrates to keep you energized through the morning.",
    "rejected": "There are many healthy breakfast options you can consider. Oatmeal is healthy. Eggs are healthy. Yogurt is healthy. Fruit is healthy. Smoothies are healthy. Toast is healthy. You could also have cereal if it's whole grain. Avocado is healthy too. Nuts are another option. The key is to eat something nutritious in the morning.",
    "metadata": {
      "preference_reason": "The chosen response provides specific, actionable suggestions with preparation details and nutritional context. The rejected response is vague, repetitive, and provides no practical guidance.",
      "category": "helpfulness-detail",
      "difficulty": "easy",
      "quality_delta": "large"
    }
  },
  {
    "prompt": "Should I invest my savings in cryptocurrency?",
    "chosen": "I can share some considerations to help you think through this decision, but I'd recommend consulting a financial advisor for personalized advice.\n\nThings to consider:\n- **Risk tolerance**: Cryptocurrency is highly volatile. Bitcoin has seen drawdowns exceeding 70% from peak to trough. Only invest money you can afford to lose entirely.\n- **Diversification**: Financial experts generally recommend that speculative investments make up no more than 5-10% of your portfolio.\n- **Your financial foundation**: Before speculative investments, ensure you have an emergency fund (3-6 months of expenses), no high-interest debt, and adequate retirement contributions.\n- **Tax implications**: Crypto gains are taxable events in most jurisdictions. Factor tax obligations into your return calculations.\n\nThe decision depends on your specific financial situation, goals, and risk tolerance.",
    "rejected": "Absolutely! Crypto is the future of finance and you should definitely invest. Bitcoin and Ethereum have given incredible returns over the past few years. Many experts predict Bitcoin will reach $500,000 in the next few years. The earlier you get in, the more money you'll make. I'd suggest putting a significant portion of your savings into a mix of Bitcoin, Ethereum, and some promising altcoins. Dollar-cost averaging is a good strategy — just buy regularly and hold long term.",
    "metadata": {
      "preference_reason": "The chosen response is balanced, presents risks alongside considerations, recommends professional advice, and avoids making predictions. The rejected response gives irresponsible financial advice, makes unfounded predictions, and encourages overexposure to a volatile asset.",
      "category": "safety-balanced-advice",
      "difficulty": "medium",
      "quality_delta": "large"
    }
  }
]