The short version. On August 28, 2026 every pretty URL on this site returned 404 and not one log recorded an error. I found an unescaped quote in the virtual host’s logFormat line, escaped it, reloaded, and the site came back. I wrote that lesson into three internal documents and into my own operating notes.
It was wrong. Another site on this same server runs that exact malformed line today and serves its pretty URLs fine. And the server’s own logs show the rewrite engine was already working 32 minutes before I touched it. This note is the post-mortem of my post-mortem, and it ends with the defect that is still there.
The site had been created that afternoon with cyberpanel createWebsite, without letting the panel install WordPress; WordPress went on afterward by hand. Permalinks were set to /%postname%/. The homepage answered. Nothing else did.
A site that answers and a site that 404s
The symptom was total and clean. / returned 200. Every other path returned 404, including the login path this site keeps behind wps-hide-login. The 404 was 711 bytes on the wire, which is a number worth remembering, and I will come back to it.
Four things ruled themselves out within the first ten minutes, and all four were misleading in the same direction — they said WordPress was healthy, so I stopped suspecting the layer above it:
/?page_id=6returned a 301 to the pretty URL. WordPress knew the page and could generate its address./wp-login.phpreturned WordPress’s own 404 page, not the server’s. PHP was running, the database was reachable, the theme was rendering.- The
.htaccessin the document root was present and byte-for-byte identical to the one on a sibling site that worked. - The virtual host file was being applied: the security headers declared in its
context /block were on every response, and that block sits at the very bottom of the file.
So the conclusion wrote itself. Everything in the virtual host worked except the one block that mattered, and nothing anywhere had complained. That is the part that costs hours: a failure with no string to search for.
What OpenLiteSpeed actually does with that line
The line I eventually blamed sits inside the accesslog block, seven lines from the top of the file, and has nothing to do with routing:
logFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i""Eight double quotes, none of them escaped. It looks broken because it is a copy of Apache’s combined log format with the escaping filed off. The theory I formed — and published internally — was that the parser choked there and silently discarded everything after it, the rewrite block included. It is a tidy story. Here is what the parser actually does.
The configuration parser has no idea what a quote is
OpenLiteSpeed is GPLv3, so this is checkable rather than arguable. In plainconf.cpp at tag v1.9.1, parseLine splits a line into exactly two pieces: the first token is the name, and every remaining byte is the value. It skips whitespace only while the value is still empty:
else if ((*p == ' ' || *p == '\t') && value[0] == 0)
{
bNameSet = true;
continue;
}Once the opening quote lands in the value, everything after it is copied character by character, spaces included. Search the whole file for a branch that compares a character against " and you get nothing. There is no quote stripping, no escape handling, no string state.
The braces in %{Referer}i are equally harmless, and the code says so in a comment: an opening brace starts a block only if it is the last character on the line, and a closing brace ends one only if it is the first. Neither is true here, so the block structure of the file never goes out of alignment.
The result is that logFormat is accepted as a known keyword with a value that happens to contain eight literal quotes, and the parser moves to the next line. It has nothing to report, which is exactly why nothing was reported.
Where the escaping convention comes from, and what Apache would have done
The format string is Apache’s. mod_log_config states the rule plainly: “Literal quotes and backslashes should be escaped with backslashes.” The reason is not in the logging module at all but in how Apache reads configuration files: “Inside a quoted string, only two escape sequences are recognized: \\ produces a literal backslash, and \" … produces a literal quote character without ending the string.” Without the backslash, the quote ends the string.
The rule is not buried, either. The documented default value of Apache’s own LogFormat directive is "%h %l %u %t \"%r\" %>s %b" — the same string CyberPanel ships, with the backslashes still in it.
Apache also treats the arity as strict: the directive is registered to accept one or two arguments (AP_INIT_TAKE12, mod_log_config.c:1461), and Apache ships apachectl configtest to read the files before you restart. I have not run this particular line through Apache, so that is what its parser is registered to accept, not output I have seen. What matters here is that OpenLiteSpeed has no working equivalent of configtest for a virtual host, and that difference is most of why this was expensive. More on that below.
Why escaping it changed nothing at all
This is the part I did not check at the time, and it is checkable in under a minute. The access log format is compiled by CustomFormat::parseFormat in accesslog.cpp. When it meets a backslash it handles \n, \r and \t, and then, for anything else:
memmove(p, p + 1, pEnd - p);
--pEnd;It deletes the backslash and keeps the character that followed. So \" and " arrive at the same place. The escaped form and the malformed form compile to the identical log format, and you can see it on this server without reading any source at all. Two sites, one with the malformed line and one with the escaped line, produce log records shaped exactly the same way:
python3 - <<'PY'
for dom in ("activeforgepro.com", "visibleorigin.com"):
p = f"/home/{dom}/logs/{dom}.access_log"
ln = [l for l in open(p, errors="replace") if l.strip()][-1].rstrip("\n")
print(dom, "first char:", repr(ln[0]), "last three:", repr(ln[-3:]))
PY
# activeforgepro.com first char: '"' last three: '1""'
# visibleorigin.com first char: '"' last three: '1""'Both start with a stray quote and end with two. The escaping I applied was a no-op on the only thing that line controls.
The counterexample was on the same machine the whole time
This server runs five virtual hosts. Three carry the escaped form and two carry the malformed one:
for d in /usr/local/lsws/conf/vhosts/*/vhost.conf; do
printf "%-26s %s\n" "$(basename $(dirname $d))" "$(grep -m1 logFormat $d | sed 's/^ *//')"
doneOne of the two carrying it is activeforgepro.com, a WordPress site I also run. Its pretty URLs answer, from the public internet, right now:
IP=$(dig +short @1.1.1.1 A activeforgepro.com | head -1)
for u in / /training/ /nutrition/ /equipment/; do
printf "%-14s %s\n" "$u" "$(curl -s -o /dev/null -w '%{http_code}' \
--resolve activeforgepro.com:443:$IP https://activeforgepro.com$u)"
done
# / 200
# /training/ 200
# /nutrition/ 200
# /equipment/ 200Dated backups of that file from August 3, August 5 and August 23 all carry the same malformed line, so it predates the incident by weeks and cost that site nothing. A theory that a line kills the rewrite engine does not survive a site on the same binary running that line and serving every URL. I had that counterexample in my own hands for four days and never ran the comparison, because I was not looking for one.
There is a second thing in that output worth pausing on. CyberPanel’s template can only produce the malformed line — I will show that in a moment — so the three escaped ones were edited by hand. Two of those edits are mine, from that night and the days after. I have not traced the third, and I am not going to pretend I have.
What this looks like as a service. Everything above is one habit applied to a server instead of to a page: state what you believe, then run the command that could contradict it. Most of the checks in this note are one line long and could have been run at any point in those four days.
The same discipline is what the free check is made of. You send a domain, I request its homepage as five named crawlers, one at a time, and send back a written report of what each one actually received — with the command that produced each answer, so you can disagree with me using your own terminal.
The line does break something. Just not the rewrite engine
Having cleared the line of the crime, I checked whether it was guilty of anything at all. It is. It breaks the access log it configures, and it breaks it in both forms.
Because plainconf never strips the quotes wrapping a directive value, those wrapping quotes end up inside every log record: one before the client address, one after the user agent. The result is no longer the NCSA combined format it is trying to be. Any parser built on the standard pattern rejects it:
python3 - <<'PY'
import re
COMBINED = re.compile(
r'^(\S+) (\S+) (\S+) \[([^\]]+)\] "([^"]*)" (\d{3}) (\S+) "([^"]*)" "([^"]*)"$')
casos = [("server log, no logFormat set", "/usr/local/lsws/logs/access.log"),
("vhost, malformed logFormat", "/home/activeforgepro.com/logs/activeforgepro.com.access_log"),
("vhost, escaped logFormat", "/home/visibleorigin.com/logs/visibleorigin.com.access_log")]
for nombre, p in casos:
ln = [l for l in open(p, errors="replace") if l.strip()][-1].strip()
print(f"{nombre:30} first char {ln[0]!r:5} valid combined: {bool(COMBINED.match(ln))}")
PY| Access log | Where its format comes from | First character of a record | Parses as combined format |
|---|---|---|---|
Server-wide logs/access.log | No logFormat directive at all — the server’s built-in default | 1 (the address) | yes |
| Virtual host with the malformed line | CyberPanel’s template, quotes unescaped | " | no |
| Virtual host with the escaped line | CyberPanel’s template, quotes escaped by me | " | no |
Run on this server on September 4, 2026, against the last record in each file, using the regular expression that log parsers build on. The control is the first row: the server-wide log has no logFormat line, falls back to the built-in default, and produces a conforming record. The two virtual host logs do not, and escaping makes no difference — which is the point. I have not tested which form of the directive would produce a conforming record, because finding out means restarting a server that hosts four live sites. That is an open defect in this note, not a finished investigation.
The defect is in CyberPanel, and it is a Python detail rather than a configuration one. Its OpenLiteSpeed templates are plain triple-quoted strings rather than raw strings, so Python resolves the \" sequences into bare quotes before the file is ever written. You can watch it happen on the installed copy, without touching GitHub:
python3 - <<'PY'
import ast
mod = ast.parse(open("/usr/local/CyberCP/plogical/vhostConfs.py").read())
for node in ast.walk(mod):
if isinstance(node, ast.Assign):
for t in node.targets:
if getattr(t, "id", "") in ("olsMasterConf", "olsChildConf", "OLSLBConf"):
for line in node.value.value.splitlines():
if "logFormat" in line:
print(f"{t.id:16} backslash present: {chr(92) in line}")
PY
# olsMasterConf backslash present: False
# olsChildConf backslash present: False
# OLSLBConf backslash present: FalseAll three OpenLiteSpeed templates are affected. The line has been emitted this way since February 3, 2020, when CyberPanel moved its access logs to the combined format, and it is still there: the copy installed on this machine is byte-identical to the template in CyberPanel’s repository, so the run above is a reproduction of the shipping code and not of a local edit. I searched the issue trackers of both projects, both official forums, the CyberPanel community, Stack Overflow and Server Fault, and found nobody connecting it to anything. That is not proof no one has; it is what I got.
Two different 404s, and the one log line that tells them apart
Back to the 711 bytes. That number is not a symptom of a broken virtual host — it is OpenLiteSpeed’s built-in error page, and a perfectly healthy site serves it whenever no rewrite rule fires and the file does not exist. You can produce it on demand today, because the ACME challenge context has rewriting switched off by design:
IP=$(dig +short @1.1.1.1 A visibleorigin.com | head -1)
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
--resolve visibleorigin.com:443:$IP \
https://visibleorigin.com/.well-known/acme-challenge/does-not-exist-12345
# 404 1249
curl -s -o /dev/null -H 'Accept-Encoding: gzip' -w '%{http_code} %{size_download}\n' \
--resolve visibleorigin.com:443:$IP \
https://visibleorigin.com/.well-known/acme-challenge/does-not-exist-12345
# 404 7111,249 bytes uncompressed, 711 with gzip, which is why the access log records 711 and a plain curl shows 1,249. Meanwhile a request that does reach WordPress gets WordPress’s own 404 page, which on this site is two orders of magnitude larger. The response size tells you which engine answered, and it is recorded for every request you never thought to make.
That is how the incident can be dated at all, and it is why the log had the answer. But there is a second line, in the error log rather than the access log, that is the real discriminator — and misreading it is what sent me to the wrong file.
| Line in the server error log | What it actually tells you | What I read it as |
|---|---|---|
RewriteFile [/path/.htaccess] parsed, return 0. | The file was opened and read without error. It says nothing about what was in it. An empty file returns 0 too. | “The rewrite rules loaded, so the rules are not the problem.” |
RewriteBase [/] parsed, set to HttpContext 0x… | RewriteBase / exists in exactly one place on a WordPress site: inside the # BEGIN WordPress block. If this line is absent, that block is absent. | I never looked for it. |
Both lines are logged at INFO, which is why they are easy to scroll past — and why the server-level logLevel matters. This machine was running at DEBUG throughout, so nothing was being suppressed. The other three virtual hosts on the box drew 132 genuine [ERROR] lines from the configuration parser that same night, over an unrelated unsupported directive. The broken one drew zero. The silence was real, and it was informative.
Reading the reload history for that night through the second line, rather than the first, dates the repair precisely:
python3 - <<'PY'
p = "/usr/local/lsws/logs/error.log.2026_09_04"
rows = [l.rstrip() for l in open(p, errors="replace")
if "2026-08-28" in l and ("RewriteBase" in l or "RewriteFile" in l)]
for i, ln in enumerate(rows):
if "RewriteFile" in ln and "visibleorigin.com/public_html/.htaccess" in ln:
print(ln.split()[0], ln.split()[1][:8],
"RewriteBase present" if "RewriteBase" in rows[i-1] else "RewriteBase ABSENT")
PY
# 2026-08-28 19:42:01 RewriteBase ABSENT
# 2026-08-28 20:57:01 RewriteBase present
# 2026-08-28 20:57:32 RewriteBase present
# 2026-08-28 21:00:20 RewriteBase present
# ...
# 2026-08-28 21:36:07 RewriteBase presentThe .htaccess was being read from the first reload of the evening. It simply had nothing in it to read. The discontinuity is at 20:57:01, and the reload that loaded my escaped logFormat is the one at 21:36:07 — thirty-nine minutes later.
Where this stops being a blog post. Reading a site from outside will tell you that a crawler gets a 404 where a human gets a page. It will not tell you which of the four layers under that answer produced it, and no scan can: the evidence is in files only the server operator can open.
Closing that gap is the work I do for money — on OpenLiteSpeed, Apache, Nginx and Cloudflare, on sites I did not build. Every finding arrives the way the ones above did: with the command, the output, and the part I could not establish.
Ask the server what it read, not what you wrote
The tool that would have ended this in ninety seconds was sitting next to the file the whole time, and I did not know it existed.
Every time OpenLiteSpeed parses a configuration file it writes the parsed tree back out beside it, as <filename>.txt. That file is not your configuration — it is the server’s understanding of your configuration, serialized, and regenerated on every start. On the site still carrying the malformed line it reads:
grep -nE 'logformat|^rewrite|enable 1|autoloadhtaccess' \
/usr/local/lsws/conf/vhosts/activeforgepro.com/vhost.conf.txt
# 18:rewrite {
# 57: enable 1
# 58: autoloadhtaccess 1
# 86: logformat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i""The malformed line and an intact rewrite block, in the same parsed tree. That single output falsifies the theory I had been operating on, and it was available at any moment without restarting anything. It is worth confirming that the dump is current rather than stale, which takes one comparison:
stat -c '%y %n' /usr/local/lsws/conf/vhosts/*/vhost.conf.txt
ps -eo pid,lstart,comm | grep -m1 litespeed
# all five dumps: 2026-09-02 15:03:01
# 79277 Wed Sep 2 15:03:01 2026 litespeedSame second as the running process started. The dump describes the configuration in memory.
Correction, added September 5, 2026, the day after publishing this. The dump is not a complete round trip, and I should not have implied it was. Comparing the 141 top-level directives in this server’s main configuration file against its dump, eight do not appear in the dump at all — the ones the parser does not recognize at that level, which on a CyberPanel box means things like enableHSTS, autoSSL and modsecurity. So the dump is sound evidence for a positive claim, which is how it is used above: this block was loaded, here it is. It is not evidence for a negative one. Something missing from the dump may still have been read. I nearly drew exactly that wrong conclusion about a different directive a day later, which is the only reason this paragraph exists.
The obvious alternative does not work here, and it is worth knowing why. OpenLiteSpeed does ship a configuration test, documented as openlitespeed -t. I tested it against a copy of this server’s configuration — a separate root reached through LSWS_HOME, with every listener moved to a high port and every socket path redirected, so that nothing could touch the live server — and injected the malformed line into it. The output was byte-identical with and without the malformed line, apart from the process ID. The reason is structural: the test mode regenerates the dump for the main configuration file and for no virtual host at all. Virtual hosts are loaded lazily, and the test never gets to them.
So on this platform the answer to “how do I check a virtual host before I restart” is: you read the dump from the last time it was loaded, and you compare it against what you wrote. There is no equivalent of configtest for the file that actually broke. The dump has a second limit worth stating plainly: it reports what the server parsed, never what a request will do with it — which is why, when the question is about PHP rather than routing, the only answer I ended up trusting came from an HTTP request.
What I got wrong, and for how long
Laid end to end, the failure of the diagnosis is more instructive than the failure of the server.
I made four changes that evening: I added the WordPress block to the .htaccess at 20:55, added a context / block to the virtual host at 21:04, added inline rewrite rules to it at 21:08, and escaped the logFormat line at 21:36. I measured before the first one and after the last one, and not once in between. The last failing request in the log is at 20:55:04. The first succeeding one is at 21:04:01. The change I credited is thirty-two minutes after that.
Then I wrote it down. The wrong cause went into three internal documents, into a diagnostic rule that told my future self to compare virtual host files whenever every pretty URL 404s, and into the summary I keep for exactly these situations. It survived because it explained the symptom, was cheap to remember, and was never tested again. Two other numbers went in with it and were also wrong: there were not four restarts that night, there were ten reloads, and the “bare 1,249-byte 404 with no WordPress in it” that I recorded as the signature of a broken virtual host is the error page every healthy OpenLiteSpeed serves.
The cause I now believe is the missing # BEGIN WordPress block. It fits every observation: the rewrite engine was enabled and loading an .htaccess that had no rules in it, the homepage answered through the index file, /wp-login.php answered because it is a real file on disk, and everything else fell through to the built-in 404. The archived record of that evening even names it — it is written down in the table of hypotheses I had discarded, in the row that says the block was added and was identical to a working site’s.
What I cannot claim is that it was the only cause. I have no copy of the .htaccess between 20:55:26 and the 20:57:01 reload, and no request in that window asks for a URL that depends on rewriting. I can date the discontinuity. I cannot rule out a second factor inside it, and I would rather publish that hole than round it off. The experiment that would close it — create a disposable site, which is born with the malformed line, measure, escape it, restart once, measure again — requires restarting a server that four live sites share, and I have not judged that worth doing to win an argument I have already won by counterexample.
If any of this feels familiar, it is the same failure mode as the one in the note about publishing a DNS record for an unfinished draft, where the rule that saved me was that every negative result needs a control. A control is precisely what was missing here. I had one, running on the same machine, four days old, and I never queried it.
OpenLiteSpeed 404s: frequently asked questions
Every URL on my OpenLiteSpeed site returns 404 except the homepage. Where should I look first?
At the .htaccess in your document root, for the # BEGIN WordPress block, and then at the server error log for RewriteBase [/] parsed on the last reload. If the block is missing, the rewrite engine is running with nothing to run, and the symptom is exactly this: the homepage answers through the index file, real files on disk answer, and every generated URL falls through to the built-in 404. Re-saving the permalink settings in WordPress rewrites the block. On OpenLiteSpeed the file is read when the virtual host is loaded, so a reload is required afterward: the manual shipped with the server says that with auto-load from .htaccess, “Once initially loaded, a graceful restart must be performed for any further changes to that .htaccess file to take effect.”
Does an unescaped quote in logFormat break rewrite rules?
No. I believed it did for a week and it is not true. The configuration parser has no quote handling to break, the access log and the rewrite block are configured in two independent steps with no early exit between them, and a log format that fails to compile only produces one error line and is otherwise ignored. The decisive evidence is not the source code but a second site on the same server running the malformed line with every pretty URL returning 200.
What does RewriteFile ... parsed, return 0 mean?
That the file was opened and read without an error. It is not a statement about the contents, and an empty file produces it too. This is the line I misread as confirmation that my rules were loaded, which is what sent me looking for the fault in the virtual host instead of in the file the message names.
Does OpenLiteSpeed have anything like apachectl configtest?
There is openlitespeed -t, and it is documented, but it did not detect this class of problem when I tested it against an isolated copy: it regenerates the parsed dump for the main configuration and for no virtual host. Since virtual hosts are where CyberPanel writes and where most breakage lives, the practical answer for a vhost is to read the parsed dump from the last load instead.
Where can I see the configuration OpenLiteSpeed actually loaded?
Next to each configuration file, as <filename>.txt. For a virtual host that is /usr/local/lsws/conf/vhosts/<domain>/vhost.conf.txt. It is the parsed tree serialized back out, rewritten on every start, and it will disagree with your file whenever the running server disagrees with you. Check its modification time against the start time of the litespeed process before trusting it.
Is the CyberPanel logFormat line worth fixing?
Not for the reason I originally gave, and possibly not at all. It does not affect routing, performance or security. What it does is put a stray quote at both ends of every access log record, so the file is not the combined format it claims to be and standard parsers reject it. Escaping the inner quotes does not fix that, because the format compiler discards the backslashes. If you feed those logs to anything, that is the reason to care; if you do not, it is cosmetic.
The general shape of this
A cause you can name is not the same thing as a cause you have tested, and the gap between them closes only when something could have contradicted you and did not. I had the counterexample on the same machine, the parsed configuration dump next to the file, and the repair timestamped in the access log. All three were free, all three were available that night, and I reached for none of them because I already had an explanation that fit.
The transferable part is not OpenLiteSpeed. It is that a change made right before something starts working is the weakest kind of evidence there is, and it is the kind almost every server post-mortem is built on. The defense is boring and mechanical: measure between changes, not around them, and write down what you could not establish next to what you could. The same shape turns up wherever a decision is made in one layer and stored in another — it is why a cache can hand a browser the Markdown an agent asked for without a line appearing in any log either.
Testing whether a rewrite rule is alive has its own trap: my probe returned the same status code whether the htaccess rule was working or not.
Every command above runs against this server, and the ones that reach the public internet run against domains you can reach too. If one of them returns something other than what I said it does, I want the email. And if it is your own site that a crawler is getting the wrong answer from, that is what the free check is for: you send the domain, I make the requests, and you get back what each crawler received along with the command that asked.
