What the authors built
The paper formalizes generative engines as systems that gather and summarize information with generative models to answer queries.1 It introduces GEO as a "flexible black-box optimization framework for optimizing and defining visibility metrics."1 The authors also release GEO-bench, "a benchmark consisting of 10000 queries from diverse domains and sources".1 It is split 8K, 1K and 1K into train, validation and test sets.1
The benchmark contains 80% informational queries and 10% each transactional and navigational.1 Each query is tagged with one of seven categories using GPT-4, with manual verification of high recall and precision on the test split.1 The appendix says that, owing to the automated system, the tags can be noisy.1
The test engine
The main engine is a two-step setup. Only the top 5 sources are fetched from Google for every query.1 The answer is generated by gpt3.5-turbo.1 The authors sample 5 responses at temperature 0.7.1 This is a research stand-in, not a production system.
The GEO methods are applied to website content using a large language model prompted to make specific stylistic and content changes.1 For each query, one source is chosen at random for optimization, and it stays constant across all methods.1
The methods
The authors evaluate 9 GEO methods.1 They compare them with a baseline that measures the impression metric of unmodified website sources.1 Apart from methods 3, 4 and 5, the methods enhance the presentation of existing content without requiring extra content.1
What was measured
Visibility is not a rank position here. The first metric "is the normalized word count of sentences related to a citation".1 The second is a "Subjective Impression" metric covering seven aspects, the first being relevance of the cited sentence to the user query.1 Each sub-metric is scored with GPT-3.5, following G-Eval.1
The authors say G-Eval scores are poorly calibrated, so they normalize them to the same mean and variance as Position-Adjusted Word Count.1 All impression metrics are also scaled so the impressions of all citations in a response sum to 1.1 The metric is therefore a share of one answer, not traffic or clicks.
The size of the effects
The abstract claims GEO can boost visibility by up to 40%.1 The top methods (Cite Sources, Quotation Addition, Statistics Addition) achieved a relative improvement of 30-40% on Position-Adjusted Word Count and 15-30% on Subjective Impression.1 Fluency Optimization and Easy-to-Understand gave a visibility boost of 15-30%.1
Two results run against common SEO habits. The authors found no significant improvement from a more authoritative tone.1 Keyword stuffing offered little to no improvement.1
Percentages are relative to a baseline share. In Table 1, "The best methods improve upon baseline by 41% and 28% on Position-Adjusted Word Count and Subjective Impression respectively."1 The authors run five random seeds and report the average.1
Rank, domain and combinations
When all sources were optimized at once, lower-ranked sites gained more. The Cite Sources method led to a 115.1% increase in visibility for websites ranked fifth in the search results, while the top-ranked website lost 30.3% on average.1 The authors present this as GEO's potential to democratize the digital space.1 It is a result about relative shares inside a five-source answer. It does not show that a fifth-ranked page gets retrieved in the first place.
The authors also report that the efficacy of these strategies varies across domains, which they say underscores the need for domain-specific optimization methods.1 Combining methods helped: Fluency Optimization with Statistics Addition outperformed any single strategy by more than 5.5%.1 That combination analysis was run on a subset of 200 examples from the test split.1
Perplexity check
The authors also ran the methods on Perplexity.ai.1 Because Perplexity.ai does not let the user specify source URLs, they provided source text as file uploads.1 They evaluated a subset of 200 samples of the test set.1 Quotation Addition performed best on Position-Adjusted Word Count with a 22% improvement over the baseline.1 Keyword Stuffing performed 10% worse than the baseline.1
The authors report that Cite Sources and Statistics Addition show improvements of up to 9% and 37% on the two metrics.1 This is a test of Perplexity's answer generation on supplied files, not of its live retrieval.
Limits for real sites
The authors list several themselves. Methods may need to adapt as generative engines evolve, and the nature of queries can change over time.1 They did not evaluate how GEO methods affect search rankings.1 They also note there may be discrepancies in query labeling.1
Other limits follow from the design:
- The answer "is then generated by the gpt3.5-turbo model".1
- Impressions are normalized so the citations in a response sum to 1.1
- The content changes are made by a language model prompted to perform specific stylistic and content changes.1
- In the main comparison, one randomly chosen source per query is optimized.1
What the later papers add
Two later papers test a different threat model. Pfrommer et al. study "the impact of prompt injections on the ranking order of sources referenced by conversational search engines".2 They present a tree-of-attacks-based jailbreaking technique that reliably promotes low-ranked products.2 They report that these attacks transfer effectively to perplexity.ai.2 They also show that different LLMs vary significantly in prioritizing product name, document content and context position.2 That last point is a warning against assuming one engine's behavior holds for another.
Kumar and Lakkaraju show that adding a strategic text sequence to a product's information page can significantly increase its likelihood of being listed as the LLM's top recommendation.3 Their catalog consists of fictitious coffee machines.3 The GEO paper describes that approach as adversarial text and its own strategies as non-adversarial.1 These papers add evidence that page text can shift LLM ranking. They do not validate the GEO methods, and they are not a recommended tactic.
How to use the paper
Treat it as a hypothesis list. Statistics, quotations and cited sources are worth testing on your own pages, against the engines your audience uses. Measure inclusion and citation share in real answers rather than relying on the paper's percentages.
Comments 0
No comments yet.
Sign in with GitHub or Google to comment. Reading needs no account.