How LLama 2 Lengthy got here to be

Meta researchers took the unique Llama 2 accessible in its totally different coaching parameter sizes — the values of knowledge and knowledge the algorithm can change by itself because it learns, which within the case of Llama 2 are available 7 billion, 13 billion, 34 billion, and 70 billion variants — and included extra longer textual content information sources than the unique Llama 2 coaching dataset. One other 400 billion tokens-worth, to be actual.

Then, the researchers stored the unique Llama 2’s structure the identical, and solely made a “crucial modification to the positional encoding that’s essential for the mannequin to attend longer.”

That modification was to the Rotary Positional Embedding (RoPE) encoding, a technique of programming the transformer mannequin underlying LLMs equivalent to Llama 2 (and LLama 2 Lengthy), which basically maps their token embeddings (the numbers used to signify phrases, ideas, and concepts) onto a 3D graph that exhibits their positions relative to different tokens, even when rotated. This permits a mannequin to supply correct and useful responses, with much less data (and thus, much less computing storage taken up) than different approaches.

The Meta researchers “decreased the rotation angle” of its RoPE encoding from Llama 2 to Llama 2 Lengthy, which enabled them to make sure extra “distant tokens,” these occurring extra not often or with fewer different relationships to different items of knowledge, have been nonetheless included within the mannequin’s data base.

Utilizing reinforcement learning from human feedback (RLHF), a standard AI mannequin coaching technique the place AI is rewarded for proper solutions with human oversight to verify it, and artificial information generated by Llama 2 chat itself, the researchers have been in a position to enhance its efficiency in frequent LLM duties together with coding, math, language understanding, frequent sense reasoning, and answering a human consumer’s prompted questions.

With such spectacular outcomes relative to each Llama 2 common and Anthropic’s Claude 2 and OpenAI’s GPT-3.5 Turbo, it’s little marvel the open-source AI group on Reddit and Twitter and Hacker News have been expressing their admiration and pleasure about Llama 2 because the paper’s launch earlier this week — it’s an enormous validation of Meta’s “open supply” method towards generative AI, and signifies that open supply can compete with the closed supply, “pay to play” fashions supplied by well-funded startups.