Default page

Edit File: bugs.html

<!DOCTYPE html>

<title>Reporting bugs — Varnish version 7.5.0 documentation</title>
    <link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
    <link rel="stylesheet" type="text/css" href="../_static/classic.css?v=e2714048" />
    
    <script src="../_static/documentation_options.js?v=8214db13"></script>
    <script src="../_static/doctools.js?v=888ff710"></script>
    <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
    
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Platform specific notes" href="platformnotes.html" />
    <link rel="prev" title="Getting help" href="help.html" /> 
  </head><body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="platformnotes.html" title="Platform specific notes"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="help.html" title="Getting help"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="../index.html">Varnish version 7.5.0 documentation</a> »</li>
          <li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Varnish Installation</a> »</li>
        <li class="nav-item nav-item-this"><a href="">Reporting bugs</a></li> 
      </ul>
    </div>

<div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <section id="reporting-bugs">
<h1>Reporting bugs<a class="headerlink" href="#reporting-bugs" title="Link to this heading">¶</a></h1>
<p>Varnish can be a tricky beast to debug, having potentially thousands
of threads crowding into a few data structures makes for <em>interesting</em>
core dumps.</p>
<p>Actually, let me rephrase that without irony:  You tire of the “no,
not thread 438 either, lets look at 439 then…” routine really fast.</p>
<p>So if you run into a bug, it is important that you spend a little bit
of time collecting the right information, to help us fix the bug.</p>
<p>The most valuable information you can give us, is <strong>always</strong> how
to trigger and reproduce the problem. If you can tell us that, we
rarely need anything else to solve it.The caveat being, that we
do not have a way to simulate high levels of real-life web-traffic,
so telling us to “have 10.000 clients hit at once” does not really
allow us to reproduce.</p>
<p>To report a bug please follow the suggested procedure described in the “Trouble Tickets”
section of the documentation (above).</p>
<p>Roughly we categorize bugs in to three kinds of bugs (described below) with Varnish. The information
we need to debug them depends on what kind of bug we are facing.</p>
<section id="varnish-crashes">
<h2>Varnish crashes<a class="headerlink" href="#varnish-crashes" title="Link to this heading">¶</a></h2>
<p>Plain and simple: <strong>boom</strong></p>
<p>Varnish is split over two processes, the manager and the child.  The child
does all the work, and the manager hangs around to resurrect it if it
crashes.</p>
<p>Therefore, the first thing to do if you see a Varnish crash, is to examine
your syslogs to see if it has happened before. (One site is rumoured
to have had Varnish restarting every 10 minutes and <em>still</em> provide better
service than their CMS system.)</p>
<p>When it crashes, which is highly unlikely to begin with, Varnish will spew out a crash dump
that looks something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Child (32619) died signal=6 (core dumped)
Child (32619) Panic message: Assert error in ccf_panic(), cache_cli.c line 153:
  Condition(!strcmp("", "You asked for it")) not true.
errno = 9 (Bad file descriptor)
thread = (cache-main)
ident = FreeBSD,9.0-CURRENT,amd64,-sfile,-hcritbit,kqueue
Backtrace:
  0x42bce1: pan_ic+171
  0x4196af: ccf_panic+4f
  0x8006b3ef2: _end+80013339a
  0x8006b4307: _end+8001337af
  0x8006b8b76: _end+80013801e
  0x8006b8d84: _end+80013822c
  0x8006b51c1: _end+800134669
  0x4193f6: CLI_Run+86
  0x429f8b: child_main+14b
  0x43ef68: start_child+3f8
[...]
</pre></div>
</div>
<p>If you can get that information to us, we are usually able to
see exactly where things went haywire, and that speeds up bugfixing
a lot.</p>
<p>There will be a lot more information in the crash dump besides this, and before sending
it all to us, you should obscure any sensitive/secret
data/cookies/passwords/ip# etc.  Please make sure to keep context
when you do so, ie: do not change all the IP# to “X.X.X.X”, but
change each IP# to something unique, otherwise we are likely to be
more confused than informed.</p>
<p>The most important line is the “Panic Message”, which comes in two
general forms:</p>
<dl>
<dt>“Missing errorhandling code in …”</dt><dd><p>This is a situation where we can conceive Varnish ending up, which we have not
(yet) written the padded-box error handling code for.</p>
<p>The most likely cause here, is that you need a larger workspace
for HTTP headers and Cookies.</p>
<p>Please try that before reporting a bug.</p>
</dd>
<dt>“Assert error in …”</dt><dd><p>This is something bad that should never happen, and a bug
report is almost certainly in order. As always, if in doubt
ask us on IRC before opening the ticket.</p>
</dd>
</dl>
<p>In your syslog it may all be joined into one single line, but if you
can reproduce the crash, do so while running <a class="reference internal" href="../reference/varnishd.html#varnishd-1"><span class="std std-ref">varnishd</span></a> manually:</p>
<blockquote>
<div><p><code class="docutils literal notranslate"><span class="pre">varnishd</span> <span class="pre">-d</span> <span class="pre"><your</span> <span class="pre">other</span> <span class="pre">arguments></span> <span class="pre">|&</span> <span class="pre">tee</span> <span class="pre">/tmp/_catch_bug</span></code></p>
</div></blockquote>
<p>That will get you the entire panic message into a file.</p>
<p>(Remember to type <code class="docutils literal notranslate"><span class="pre">start</span></code> to launch the worker process, that is not
automatic when <code class="docutils literal notranslate"><span class="pre">-d</span></code> is used.)</p>
</section>
<section id="varnish-goes-on-vacation">
<h2>Varnish goes on vacation<a class="headerlink" href="#varnish-goes-on-vacation" title="Link to this heading">¶</a></h2>
<p>This kind of bug is nasty to debug, because usually people tend to
kill the process and send us an email saying “Varnish hung, I
restarted it” which gives us only about 1.01 bit of usable debug
information to work with.</p>
<p>What we need here is all the information you can squeeze out of
your operating system <strong>before</strong> you kill the Varnish process.</p>
<p>One of the most valuable bits of information, is if all Varnish’
threads are waiting for something or if one of them is spinning
furiously on some futile condition.</p>
<p>Commands like <code class="docutils literal notranslate"><span class="pre">top</span> <span class="pre">-H</span></code> or <code class="docutils literal notranslate"><span class="pre">ps</span> <span class="pre">-Haxlw</span></code> or <code class="docutils literal notranslate"><span class="pre">ps</span> <span class="pre">-efH</span></code> should be
able to figure that out.</p>
<p>If one or more threads are spinning, use <code class="docutils literal notranslate"><span class="pre">strace</span></code> or <code class="docutils literal notranslate"><span class="pre">ktrace</span></code> or <code class="docutils literal notranslate"><span class="pre">truss</span></code>
(or whatever else your OS provides) to get a trace of which system calls
the Varnish process issues. Be aware that this may generate a lot
of very repetitive data, usually one second worth of data is more than enough.</p>
<p>Also, run <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a> for a second, and collect the output
for us, and if <a class="reference internal" href="../reference/varnishstat.html#varnishstat-1"><span class="std std-ref">varnishstat</span></a> shows any activity, capture that
also.</p>
<p>When you have done this, kill the Varnish <em>child</em> process, and let
the <em>master</em> process restart it.  Remember to tell us if that does
or does not work. If it does not, kill all Varnish processes, and
start from scratch. If that does not work either, tell us, that
means that we have wedged your kernel.</p>
</section>
<section id="varnish-does-something-wrong">
<h2>Varnish does something wrong<a class="headerlink" href="#varnish-does-something-wrong" title="Link to this heading">¶</a></h2>
<p>These are the easy bugs: usually all we need from you is the relevant
transactions recorded with <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a> and your explanation
of what is wrong about what Varnish does.</p>
<p>Be aware, that often Varnish does exactly what you asked it to, rather
than what you intended it to do. If it sounds like a bug that would
have tripped up everybody else, take a moment to read through your
VCL and see if it really does what you think it does.</p>
<p>You can also try setting the <code class="docutils literal notranslate"><span class="pre">vsl_mask=+VCL_trace</span></code> parameter (or use
<code class="docutils literal notranslate"><span class="pre">varnishadm</span> <span class="pre">param.set</span> <span class="pre">vsl_mask</span> <span class="pre">+VCL_trace</span></code> on a running instance),
that will generate log records with like and character number for each
statement executed in your VCL program.</p>
</section>
</section>

<div class="clearer"></div>
          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <div>
    <h3><a href="../index.html">Table of Contents</a></h3>
    <ul>
<li><a class="reference internal" href="#">Reporting bugs</a><ul>
<li><a class="reference internal" href="#varnish-crashes">Varnish crashes</a></li>
<li><a class="reference internal" href="#varnish-goes-on-vacation">Varnish goes on vacation</a></li>
<li><a class="reference internal" href="#varnish-does-something-wrong">Varnish does something wrong</a></li>
</ul>
</li>
</ul>

</div>
  <div>
    <h4>Previous topic</h4>
    <p class="topless"><a href="help.html"
                          title="previous chapter">Getting help</a></p>
  </div>
  <div>
    <h4>Next topic</h4>
    <p class="topless"><a href="platformnotes.html"
                          title="next chapter">Platform specific notes</a></p>
  </div>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="../_sources/installation/bugs.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3 id="searchlabel">Quick search</h3>
    <div class="searchformwrapper">
    <form class="search" action="../search.html" method="get">
      <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
      <input type="submit" value="Go" />
    </form>
    </div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="platformnotes.html" title="Platform specific notes"
             >next</a> |</li>
        <li class="right" >
          <a href="help.html" title="Getting help"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="../index.html">Varnish version 7.5.0 documentation</a> »</li>
          <li class="nav-item nav-item-1"><a href="index.html" >Varnish Installation</a> »</li>
        <li class="nav-item nav-item-this"><a href="">Reporting bugs</a></li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
    © Copyright 2010-2014, Varnish Software AS.
      Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 7.2.6.
    </div>
  </body>
</html>