<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Problem-Solving on blog.iankulin.com</title><link>https://blog.iankulin.com/tags/problem-solving/</link><description>Recent content in Problem-Solving on blog.iankulin.com</description><generator>Hugo</generator><language>en-AU</language><lastBuildDate>Mon, 30 Oct 2023 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.iankulin.com/tags/problem-solving/index.xml" rel="self" type="application/rss+xml"/><item><title>apt update - BADSIG 871920D1991BC93C</title><link>https://blog.iankulin.com/apt-update-badsig-871920d1991bc93c/</link><pubDate>Mon, 30 Oct 2023 00:00:00 +0000</pubDate><guid>https://blog.iankulin.com/apt-update-badsig-871920d1991bc93c/</guid><description>&lt;p&gt;I have an ansible script that runs each weekend which basically does an &lt;code&gt;apt update &amp;amp;&amp;amp; apt upgrade -Y&lt;/code&gt; on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#d8dee9;background-color:#2e3440;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-gdscript3" data-lang="gdscript3"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Hit&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;1&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy InRelease
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Hit&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;2&lt;/span&gt; https&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;download&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;docker&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;linux&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy InRelease 
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Hit&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;3&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;backports InRelease 
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Hit&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;4&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;security InRelease 
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Get&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;5&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;updates InRelease &lt;span style="color:#eceff4"&gt;[&lt;/span&gt;&lt;span style="color:#b48ead"&gt;119&lt;/span&gt; kB&lt;span style="color:#eceff4"&gt;]&lt;/span&gt; 
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Err&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;5&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;updates InRelease 
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; The following signatures were invalid&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; BADSIG &lt;span style="color:#b48ead"&gt;871920&lt;/span&gt;D1991BC93C Ubuntu Archive Automatic Signing Key &lt;span style="color:#eceff4"&gt;(&lt;/span&gt;&lt;span style="color:#b48ead"&gt;2018&lt;/span&gt;&lt;span style="color:#eceff4"&gt;)&lt;/span&gt; &lt;span style="color:#81a1c1"&gt;&amp;lt;&lt;/span&gt;ftpmaster&lt;span style="color:#bf616a"&gt;@&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Get&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#b48ead"&gt;6&lt;/span&gt; https&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;pkgs&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;tailscale&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;stable&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy InRelease
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Fetched &lt;span style="color:#b48ead"&gt;125&lt;/span&gt; kB &lt;span style="color:#81a1c1;font-weight:bold"&gt;in&lt;/span&gt; &lt;span style="color:#b48ead"&gt;1&lt;/span&gt;s &lt;span style="color:#eceff4"&gt;(&lt;/span&gt;&lt;span style="color:#b48ead"&gt;125&lt;/span&gt; kB&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;s&lt;span style="color:#eceff4"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Reading package lists&lt;span style="color:#81a1c1"&gt;...&lt;/span&gt; Done
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Building dependency tree&lt;span style="color:#81a1c1"&gt;...&lt;/span&gt; Done
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Reading state information&lt;span style="color:#81a1c1"&gt;...&lt;/span&gt; Done
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&lt;span style="color:#b48ead"&gt;11&lt;/span&gt; packages can be upgraded&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt; Run &lt;span style="color:#a3be8c"&gt;&amp;#39;apt list --upgradable&amp;#39;&lt;/span&gt; to see them&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;W&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; An error occurred during the signature verification&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt; The repository is &lt;span style="color:#81a1c1;font-weight:bold"&gt;not&lt;/span&gt; updated &lt;span style="color:#81a1c1;font-weight:bold"&gt;and&lt;/span&gt; the previous index files will be used&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt; GPG error&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;updates InRelease&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; The following signatures were invalid&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; BADSIG &lt;span style="color:#b48ead"&gt;871920&lt;/span&gt;D1991BC93C Ubuntu Archive Automatic Signing Key &lt;span style="color:#eceff4"&gt;(&lt;/span&gt;&lt;span style="color:#b48ead"&gt;2018&lt;/span&gt;&lt;span style="color:#eceff4"&gt;)&lt;/span&gt; &lt;span style="color:#81a1c1"&gt;&amp;lt;&lt;/span&gt;ftpmaster&lt;span style="color:#bf616a"&gt;@&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;W&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; Failed to fetch http&lt;span style="color:#eceff4"&gt;:&lt;/span&gt;&lt;span style="color:#81a1c1"&gt;//&lt;/span&gt;au&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;archive&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;dists&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;jammy&lt;span style="color:#81a1c1"&gt;-&lt;/span&gt;updates&lt;span style="color:#81a1c1"&gt;/&lt;/span&gt;InRelease The following signatures were invalid&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; BADSIG &lt;span style="color:#b48ead"&gt;871920&lt;/span&gt;D1991BC93C Ubuntu Archive Automatic Signing Key &lt;span style="color:#eceff4"&gt;(&lt;/span&gt;&lt;span style="color:#b48ead"&gt;2018&lt;/span&gt;&lt;span style="color:#eceff4"&gt;)&lt;/span&gt; &lt;span style="color:#81a1c1"&gt;&amp;lt;&lt;/span&gt;ftpmaster&lt;span style="color:#bf616a"&gt;@&lt;/span&gt;ubuntu&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;com&lt;span style="color:#81a1c1"&gt;&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;W&lt;span style="color:#eceff4"&gt;:&lt;/span&gt; Some index files failed to download&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt; They have been ignored&lt;span style="color:#eceff4"&gt;,&lt;/span&gt; &lt;span style="color:#81a1c1;font-weight:bold"&gt;or&lt;/span&gt; old ones used instead&lt;span style="color:#81a1c1"&gt;.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id="solved"&gt;Solved&lt;/h3&gt;
&lt;p&gt;The first &lt;a href="https://ubuntuforums.org/showthread.php?t=2484710"&gt;google result&lt;/a&gt; mentions apt-cache - which &lt;a href="https://blog.iankulin.com/caching-apt-updates/"&gt;I also run&lt;/a&gt;, so a first level debug step is to delete the &lt;code&gt;/etc/apt/apt.conf.d/00aptproxy&lt;/code&gt; file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the &lt;code&gt;apt update&lt;/code&gt; it works perfectly. Seems like a problem with the cache then. I&amp;rsquo;m not sure why it would only affect this host though - I have other Ubuntu VM&amp;rsquo;s in the fleet that are not getting the original error.&lt;/p&gt;
&lt;p&gt;In any case, adding the conf back to force the server to use the cache made the error reappear - so it&amp;rsquo;s definitely related to the cache. With any type of cache, when there&amp;rsquo;s a problem related to it, deleting the contents is usually a &amp;ldquo;plan A&amp;rdquo; response. Assuming there&amp;rsquo;s some mechanism in &lt;a href="https://wiki.debian.org/AptCacherNg"&gt;Apt Cacher NG&lt;/a&gt; to do this, I went to the little stats/config webpage it serves up.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://blog.iankulin.com/images/screen-shot-2023-10-08-at-8.41.44-am.jpg" alt=""&gt;&lt;/p&gt;
&lt;p&gt;Well &amp;ldquo;Force the download of index files&amp;rdquo; sounds promising, let&amp;rsquo;s try that.&lt;/p&gt;
&lt;p&gt;I ticked the box for Force the download of index files (even having fresh ones), but it wasn&amp;rsquo;t clear to me how to make that change stick. The first button I could click further down the page was &amp;ldquo;Start Scan&amp;rdquo; which was related to some different checkboxes. I tried it anyway, but it didn&amp;rsquo;t force the downloading of index files. Time for some command line comandoing.&lt;/p&gt;
&lt;p&gt;The cache files for &lt;code&gt;aptcacher-ng&lt;/code&gt; are in &lt;code&gt;/var/cache/apt-cacher-ng/&lt;/code&gt; each distro has a directory in there.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://blog.iankulin.com/images/screen-shot-2023-10-08-at-9.08.48-am.jpg" alt=""&gt;&lt;/p&gt;
&lt;p&gt;Guessing the Ubuntu repository cache is probably stored in &lt;code&gt;uburep&lt;/code&gt;, I deleted that with &lt;code&gt;rm -R /var/cache/apt-cacher-ng/uburep&lt;/code&gt;. When I retried the &lt;code&gt;apt update&lt;/code&gt;, it worked perfectly, and I could see that the &lt;code&gt;/var/cache/apt-cacher-ng/uburep&lt;/code&gt; directory had re-appeared.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s the immediate problem fixed. The cause of this problem is unclear. Presumably it related to a package running on this Ubuntu machine (runs docker with a couple of small services) that is not running on my other Ubuntu hosts. It probably falls into the category of &amp;ldquo;don&amp;rsquo;t worry about unless it crops up again&amp;rdquo;.&lt;/p&gt;</description></item></channel></rss>