What Is a Log File?

Log files are the raw material of technical search engine optimization. Because access logs are written on the server itself, they capture crawler requests that browser-based analytics tools mostly miss. That's how SEOs find wasted crawl budget and 404 errors nobody noticed. The practice of studying them is called log file analysis.

Contents of a log file entry

Every line in an access log records the same core facts: the IP address of the client making the request (a visitor's device or a search engine bot), a timestamp, the requested URL, the status code the server returned, and the user agent string identifying the requesting browser or bot. Most servers also record the size of each response in bytes. Depending on the server's configuration, an entry can include how long the server took to serve the request. That's server processing time, not the full page load a visitor experiences.

Here's a sample entry in Apache's Combined Log Format:

203.0.113.7 - - [02/Aug/2026:09:14:07 +0000] "GET /blog/pricing/ HTTP/1.1" 200 5312 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Reading left to right:

• 203.0.113.7: the IP address of the client that made the request.
• - -: two identity fields, almost always just dashes.
• [02/Aug/2026:09:14:07 +0000]: the date, time, and timezone offset.
• "GET /blog/pricing/ HTTP/1.1": the request line: method, URL path, and protocol.
• 200: the HTTP status code. 200 means success; 404 would mean the page wasn't found.
• 5312: the size of the response in bytes, excluding headers.
• "-": the referring page; a dash means there wasn't one.
• "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)": the user agent. This one claims to be Google's crawler, Googlebot.

One caution: the user agent is whatever the client claims to be, and anyone can send Googlebot's string, so a bot's name in your logs isn't proof of its identity. Google documents how to verify its crawlers, either by checking the IP against Google's published crawler ranges or with a reverse DNS lookup. Confirm suspicious traffic before you act on it.

Access logs vs. error logs

When logging is enabled, web servers commonly produce two logs, though a configuration can add more or send entries to a separate logging service:

• Access log: one line per request. It answers who asked for which URL, when, and what status code they got back.
• Error log: a record of server-side problems, such as failing scripts, PHP warnings, and the details behind 500 errors.

The decision rule: for traffic and SEO questions, open the access log. For a broken site, open the error log.

Log files vs. Google Analytics

The two count different things, which is why their numbers never match:

• Log files are written on the server for every request it receives. Search engine bots, visitors with JavaScript blocked, and requests for images and other non-HTML files all show up.
• Google Analytics runs JavaScript in the visitor's browser. Most bots never show up in its reports, but it adds behavioral data a server never sees: sessions, traffic sources, and conversions.

Use logs to study crawlers and troubleshoot the server; use analytics to study people.

Where to find your log files

Most hosts expose logs in three places: the control panel, a File Manager, or an FTP/SFTP client. DreamHost, for example, shows both access and error logs in the panel under Manage Websites > Logs, with filters for status code and time range.

Don't assume old logs will wait for you. Servers rotate log files regularly, often daily, and how long past days stay available depends on your hosting provider. DreamHost's help center, for example, documents that it stores HTTP logs by default for 7 days, a limit the same article ties to the GDPR, Europe's data-privacy law. If you need crawl history for SEO analysis, download your logs on a schedule and archive them yourself. Remember that logs can only answer for the time window and fields your server was configured to keep.