In 2018, Google launched 3,234 improvements to Search. That is an average of 8.86 changes a day. Yowza! Google engineers are busy.
For the curious, I’ve compiled some videos and quotes from Google engineers that explain very well how an update goes from new idea, to approved, to launched.
If you’re short on time and just want a quick summary, watch Video 1 below. It’s just under 4 minutes.
But for those of you who are SEO geeks like me and have more time, I present to you a few more videos. Below the videos, I have pulled out snippets from the videos and an article from Google, where I attempt to come up with a more in depth flow of the process behind how SERP and algorithm changes get approved at Google.
Video 1: How Google makes improvements to its search algorithm
Video 2: SMX West 2016 – How Google Works: A Google Ranking Engineer’s Story
Video 3: The Evolution of Search
Video 4: Search Quality Meeting: Spelling for Long Queries
Goal
Video 3 at 00:08-00:26 — “Our goal is actually to make improvements to Search that just answer the user’s information need. Get them to their answer faster and faster. So that there’s almost a seamless connection between their thoughts and their information needs and the search results that they find.”
Ideas
Video 1 at 00:34-00:37 — “The first step in improving Google Search is coming up with an idea.”
Video 1 at 00:43-00:50 — “Ranking engineers then come up with a hypothesis about what signal, what data, could we integrate into our algorithm.”
Video 2 at 25:30-25:42 — “[Google has] a team of a few hundred software engineers. They are focused on [the] metrics and signals. They run lots and lots of experiments and they make a lot of changes.” And they come up with a lot of ideas.
Search Quality Tests
Video 1 at 00:50-00:58 — “We test all these reasonable ideas through rigorous scientific testing. The first is with raters.”
Rigorous Testing Article — “We work with external Search Quality Raters to measure the quality of search results on an ongoing basis. Raters assess how well a website gives people who click on it what they are looking for, and evaluate the quality of results based on the expertise, authoritativeness and trustworthiness of the content.”
Video 2 at 16:34-16:56 — “We published our human rater guidelines last year (in 2015). There are about 160 pages (168 pages in 2019) of great detail about how we think about what’s good for users. If you’re wondering why Google is doing something, often the answer is to make it look more like what the rater guidelines say.
[You can find the rater guidelines here.]
Rigorous Testing Article — 595,429 Search quality tests with raters (2018)
Side-by-Side Experiments
Video 1 at 01:01-01:07 — “We [show] these raters side-by-side [experiments] for queries that the engineer’s experiment might be affecting.
Rigorous Testing Article — In a side-by-side experiment, we show Raters two different sets of search results: one with the proposed change already implemented and one without.
Rigorous Testing Article — 44,155 Side-by-side experiments (2018)
Live Traffic Experiments
Video 1 at 01:11-1:25 — “We also confirm these changes with live experiments on real users. And we do this in something that’s called a Sandbox. We send a very small fraction of actual Google traffic to the Sandbox. We compute lots of different metrics.”
Rigorous Testing Article — “We enable the feature in question to just a small percentage of people, usually starting at 0.1%.”
Video 2 at 12:48-13:26 — “We do A/B experiments on real traffic. And then we look for changes in click patterns. And I should just mention that we run a lot of experiments. It is very rare, if you do a search on Google and you’re not in at least one experiment. Now not all of these are ranking experiments. Famously Google tested the color blue that we use for links and other blue highlighting with 41 different blues and came to the conclusion what was the perfect blue.”
Video 2 at 28:36-28:48 — “By and large, [Google engineers] try to move results with good human ratings and live experiment ratings up. And move results with bad ratings down.”
Rigorous Testing Article — 15,096 Live traffic experiments (2018)
Launch Reports by Quantitative Analysts
Video 2 at 26:31-26:42 — “A launch report is written by a quantitative analyst who is basically a statistician and someone who’s an expert at analyzing our experiments.”
Video 1 at 01:34-01:41 — “For each project, it’s usually one analyst assigned from the moment that we’re talking to the engineers trying to learn about their change.
Video 1 at 01:51-01:58 — “Ultimately, the goal of the search eval analyst team is just to provide an informed, data-driven decision and present an unbiased view.”
Launch Reviews
Video 2 at 27:54-28:00 — He mentions a video Google created of one of their weekly Search Quality meetings. I’m pretty sure Video 4 above is the one he is referencing.
Video 4 in video description — “We hold the meeting almost every Thursday to discuss possible algorithmic improvements and make decisions about what to launch.”
Just look how many people are at that meeting. After looking at all the angles, I count at least 37 people crammed into that conference room but I think I’m missing a few more that are behind the cameras angle of vision. That is one expensive meeting!
Video 2 at 27:29-27:41 — “We try to debate… is this good for the users, is this good for the system architecture, are we going to be able to keep improving the system after one of the these changes is made.”
Rigorous Testing Article — “Of the proposed changes this past year (2018), many never went live, because unless we can show a change actually makes things better for people, we don’t launch it.”
Launch
Video 1 at 2:03-2:10 — “If our scientific testing says this is a good idea for Google users, we will launch it on Google.”
Video 5 at 0:31-0:48 — “We might batch [algorithm changes] up and go to a meeting once a week where we talk about 8 or 10 or 12 or 6 different things that we would want to launch, but then after those get approved … those will roll out as we can get them into production.”
Video 2 at 28:07-28:36 — “[Once] approved, getting into production can be an easy thing. Some teams will ship it the same week. Sometimes you have to rewrite your code to actually make it suitable for our production architecture. Making things fast enough, making things clean enough. That can take awhile. I’ve known it to take months. In the worst case I know about, it took just shy of two years between approval and actually launching.”
Rigorous Testing Article — 3,234 Launches (2018)
Issues With The Process and How Google Engineers Fix It
Video 2 at 28:51-32:35 — This is a great 4 minute segment showing two cases that illustrate where the raters and the metrics weren’t cutting it and how the engineers fixed the problem.
Two kinds of problems at the core of everything engineers see:
- Systematically bad ratings
- Metrics don’t capture things they care about
Video 2 at 32:28-32:35 — “[W]hen the metrics miss something, what ranking engineers need to do is fix the guidelines or develop new metrics.”
Summary
Video 3 at 04:24-04:40 — “Our goal is to make it so that the improvement we make is so much what you wanted and fits so cleanly into the flow of what you are looking for that you almost don’t notice that it’s happened. And looking back at it, it seems obvious that that’s the way it should have always been in the first place.”
Video 1 at 03:24-03:29 — “We’ve put a huge investment into understanding what works for users.”
Video 1 at 03:36-03:41 — “I think we get excited when we feel like we’ve hit on an idea that really helps a lot of users.”
As you can see, Google engineers are modern day superheroes. They work very hard to improve Search for all of us. And it’s obviously working since Google has been the number one search engine for just under two decades now.
Appendix A: Videos and Content Quoted
Video 1: https://www.youtube.com/watch?v=J5RZOU6vK4Q
Video 2: https://www.youtube.com/watch?v=iJPu4vHETXw
Video 3: https://www.youtube.com/watch?v=mTBShTwCnD4
Video 4: https://www.youtube.com/watch?v=JtRJXnXgE-A
Video 5: https://www.youtube.com/watch?v=1_jm_isupFY
Rigorous Testing Article: https://www.google.com/search/howsearchworks/mission/users/
Other interesting and related articles you’ll enjoy: