November 2021 - metrics-alerts - lists.torproject.org

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 30 Nov '21

30 Nov '21

2021-11-29 08:40:02,185 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-29 08:40:02,219 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:40:02,436 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-29 08:40:02,516 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:40:05,576 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-29 08:40:08,106 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-29 08:42:25,416 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:42:25,783 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-29 08:42:25,783 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:42:25,784 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-29 08:42:44,952 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-29 09:02:41,475 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-29 09:03:33,494 INFO o.t.m.s.o.Main:402 Writing 59299 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-29 09:03:35,370 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-29 09:03:42,993 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-29 09:03:43,018 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-29 09:04:39,300 INFO o.t.m.s.o.Main:402 Writing 47800 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-29 09:04:39,604 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-29 09:05:05,430 INFO o.t.m.s.o.Main:402 Writing 18804 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-29 09:05:05,434 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-29 09:05:17,112 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-29 09:05:17,115 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-29 09:05:17,116 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-29 09:27:47,805 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:27:52,827 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-29 09:29:47,966 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:29:49,652 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-29 09:29:52,008 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-29 09:29:52,011 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,062 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-29 09:31:24,538 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-29 09:33:29,068 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-29 09:33:29,744 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-29 09:40:09,701 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-29 09:41:55,611 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-29 09:41:55,903 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-29 09:41:55,903 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:41:55,904 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-29 09:41:55,920 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-29 09:41:55,921 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-29 09:41:56,072 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-29 09:43:34,054 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-29 09:44:14,279 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-29 10:50:44,845 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-29 10:50:59,615 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-29 10:50:59,755 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-29 10:51:33,157 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-29 10:51:33,158 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-29 10:51:33,159 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-29 11:07:33,468 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-29 11:21:25,834 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-29 11:21:25,937 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-29 11:40:14,737 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-29 11:40:14,948 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-29 11:40:14,949 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-29 11:40:16,975 INFO o.t.m.s.w.Main:113 Found 286017 previously imported log files. 2021-11-29 11:49:38,828 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-29 12:06:39,357 INFO o.t.m.s.w.Main:298 Writing 1423685 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-29 12:06:40,273 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-29 12:06:40,274 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-29 12:08:27,822 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-29 12:08:28,010 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-29 12:08:30,973 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-29 12:08:31,197 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <onionoo@onionoo-backend-01> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-01.torproject.org 30 Nov '21

30 Nov '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <onionoo@onionoo-backend-02> /srv/onionoo.torproject.org/onionoo/bin/service/onionoo
by root＠onionoo-backend-02.torproject.org 30 Nov '21

30 Nov '21

Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <module> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 23, in <listcomp> list_of_errors = Counter([(x.split(' ')[2], x.split(' ')[3].split(':')[0]) for x in str(out).split('\\n') if x != '"']) IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/onionoo.torproject.org/onionoo/bin/service/onionoo", line 29, in <module> g_warnings.labels(host, "Problem running exporter - {}".format(e)).set(1) File "/usr/lib/python3/dist-packages/prometheus_client/metrics.py", line 146, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 29 Nov '21

29 Nov '21

2021-11-29 08:40:02,185 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-29 08:40:02,219 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:40:02,436 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-29 08:40:02,516 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:40:05,576 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-29 08:40:08,106 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-29 08:42:25,416 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:42:25,783 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-29 08:42:25,783 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:42:25,784 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-29 08:42:44,952 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-29 09:02:41,475 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-29 09:03:33,494 INFO o.t.m.s.o.Main:402 Writing 59299 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-29 09:03:35,370 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-29 09:03:42,993 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-29 09:03:43,018 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-29 09:04:39,300 INFO o.t.m.s.o.Main:402 Writing 47800 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-29 09:04:39,604 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-29 09:05:05,430 INFO o.t.m.s.o.Main:402 Writing 18804 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-29 09:05:05,434 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-29 09:05:17,112 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-29 09:05:17,115 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-29 09:05:17,116 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-29 09:27:47,805 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:27:52,827 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-29 09:29:47,966 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:29:49,652 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-29 09:29:52,008 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-29 09:29:52,011 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,062 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-29 09:31:24,538 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-29 09:33:29,068 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-29 09:33:29,744 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-29 09:40:09,701 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-29 09:41:55,611 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-29 09:41:55,903 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-29 09:41:55,903 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:41:55,904 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-29 09:41:55,920 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-29 09:41:55,921 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-29 09:41:56,072 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-29 09:43:34,054 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-29 09:44:14,279 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-29 10:50:44,845 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-29 10:50:59,615 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-29 10:50:59,755 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-29 10:51:33,157 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-29 10:51:33,158 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-29 10:51:33,159 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-29 11:07:33,468 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-29 11:21:25,834 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-29 11:21:25,937 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-29 11:40:14,737 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-29 11:40:14,948 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-29 11:40:14,949 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-29 11:40:16,975 INFO o.t.m.s.w.Main:113 Found 286017 previously imported log files. 2021-11-29 11:49:38,828 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-29 12:06:39,357 INFO o.t.m.s.w.Main:298 Writing 1423685 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-29 12:06:40,273 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-29 12:06:40,274 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-29 12:08:27,822 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-29 12:08:28,010 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-29 12:08:30,973 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-29 12:08:31,197 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 29 Nov '21

29 Nov '21

2021-11-29 00:07:44,662 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.73834170135458). 2021-11-29 00:37:49,783 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.17284003800489). 2021-11-29 01:07:34,494 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.44209405716267). 2021-11-29 01:37:11,649 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.56604028215966). 2021-11-29 02:07:33,762 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (15.036678284518356). 2021-11-29 02:37:07,489 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.565547680229878). 2021-11-29 03:07:37,185 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.24516754119397). 2021-11-29 03:38:40,174 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.85103960570622). 2021-11-29 04:12:05,595 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (37.56521144104953). 2021-11-29 04:39:15,481 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.48916192743105). 2021-11-29 05:10:48,679 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (42.61823351169134). 2021-11-29 05:38:04,666 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.67107870077258). 2021-11-29 06:08:48,890 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.429767795962995). 2021-11-29 06:40:12,777 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.97200059134554). 2021-11-29 07:09:01,852 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (48.11476717338221). 2021-11-29 07:38:31,838 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.876196851106155). 2021-11-29 08:08:13,047 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.0410958465314). 2021-11-29 08:40:37,755 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.589965209803722). 2021-11-29 09:15:41,659 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.111055530971623). 2021-11-29 09:40:16,868 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.426370813323754). 2021-11-29 10:05:13,782 WARN o.t.m.c.r.RelayDescriptorParser:175 Could not parse m line 'm 28,29 sha256=cCqF+uyAvCbWYHohA1eBxxYRRK40FbBiN18L2pUQbK' in descriptor. Skipping. 2021-11-29 10:08:44,793 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.85932694036491). 2021-11-29 10:38:28,950 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.509326940365). 2021-11-29 11:07:59,706 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.70874121868096). 2021-11-29 11:39:47,625 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.813063005220034). 2021-11-29 12:08:06,768 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.97119887438669). 2021-11-29 12:40:24,479 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (23.910314699576524). 2021-11-29 13:07:50,055 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.82608646779379). 2021-11-29 13:38:49,281 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.79374200142158). 2021-11-29 14:09:13,517 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (28.3122418844933). 2021-11-29 14:38:40,679 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.255407218392474). 2021-11-29 15:07:58,561 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.376960077879232). 2021-11-29 15:37:39,027 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.687701685396856). 2021-11-29 16:07:27,713 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (31.854630974550386). 2021-11-29 16:37:12,319 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (21.899261423873174). 2021-11-29 17:07:16,626 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.77393370269112). 2021-11-29 17:37:18,210 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (18.59835029622982). 2021-11-29 18:07:28,344 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (27.90774295379938).

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/ && src/main/resources/create-tarballs.sh
by root＠colchicifolium.torproject.org 29 Nov '21

29 Nov '21

Mon Nov 29 03:25:01 UTC 2021 Starting Mon Nov 29 03:25:01 UTC 2021 Touching symlink and directories for exit-list-2021-11 Mon Nov 29 03:25:01 UTC 2021 Creating symlink for torperf-2021-11 ln: failed to create symbolic link 'torperf-2021-11': File exists Mon Nov 29 03:25:01 UTC 2021 Touching symlink and directories for onionperf-2021-11 Mon Nov 29 03:25:01 UTC 2021 Touching symlink and directories for certs Mon Nov 29 03:25:02 UTC 2021 Touching symlink and directories for microdescs-2021-11 Mon Nov 29 03:25:02 UTC 2021 Touching symlink and directories for consensuses-2021-11 Mon Nov 29 03:25:02 UTC 2021 Touching symlink and directories for votes-2021-11 Mon Nov 29 03:25:02 UTC 2021 Touching symlink and directories for bandwidths-2021-11 Mon Nov 29 03:25:02 UTC 2021 Touching symlink and directories for server-descriptors-2021-11 Mon Nov 29 03:25:05 UTC 2021 Touching symlink and directories for extra-infos-2021-11 Mon Nov 29 03:25:07 UTC 2021 Touching symlink and directories for bridge-statuses-2021-11 Mon Nov 29 03:25:07 UTC 2021 Touching symlink and directories for bridge-server-descriptors-2021-11 Mon Nov 29 03:25:08 UTC 2021 Touching symlink and directories for bridge-extra-infos-2021-11 Mon Nov 29 03:25:08 UTC 2021 Touching symlink and directories for snowflakes-2021-11 Mon Nov 29 03:25:08 UTC 2021 Touching symlink and directories for bridge-pool-assignments-2021-11 Mon Nov 29 03:25:08 UTC 2021 Touching symlink and directories for bridgedb-metrics-2021-11 Mon Nov 29 03:25:08 UTC 2021 Touching symlink and directories for bridgestrap-2021-11 Mon Nov 29 03:25:09 UTC 2021 Creating exit-list-2021-11.tar Mon Nov 29 03:25:44 UTC 2021 Compressing exit-list-2021-11.tar Mon Nov 29 03:27:10 UTC 2021 Creating torperf-2021-11.tar tar: torperf-2021-11: Cannot stat: No such file or directory tar: Exiting with failure status due to previous errors Mon Nov 29 03:27:10 UTC 2021 Compressing torperf-2021-11.tar Mon Nov 29 03:27:10 UTC 2021 Creating onionperf-2021-11.tar Mon Nov 29 03:27:17 UTC 2021 Compressing onionperf-2021-11.tar Mon Nov 29 03:27:53 UTC 2021 Creating certs.tar Mon Nov 29 03:27:54 UTC 2021 Compressing certs.tar Mon Nov 29 03:27:55 UTC 2021 Creating microdescs-2021-11.tar Mon Nov 29 03:33:13 UTC 2021 Compressing microdescs-2021-11.tar Mon Nov 29 03:40:16 UTC 2021 Creating consensuses-2021-11.tar Mon Nov 29 03:42:24 UTC 2021 Compressing consensuses-2021-11.tar Mon Nov 29 03:51:11 UTC 2021 Creating votes-2021-11.tar Mon Nov 29 05:07:41 UTC 2021 Compressing votes-2021-11.tar Mon Nov 29 08:20:04 UTC 2021 Creating bandwidths-2021-11.tar Mon Nov 29 09:20:52 UTC 2021 Compressing bandwidths-2021-11.tar Mon Nov 29 11:05:06 UTC 2021 Creating server-descriptors-2021-11.tar tar: server-descriptors-2021-11/b/5: file changed as we read it tar: server-descriptors-2021-11/6/d: file changed as we read it Mon Nov 29 11:53:32 UTC 2021 Compressing server-descriptors-2021-11.tar Mon Nov 29 12:10:59 UTC 2021 Creating extra-infos-2021-11.tar Mon Nov 29 12:41:58 UTC 2021 Compressing extra-infos-2021-11.tar Mon Nov 29 13:01:13 UTC 2021 Creating bridge-statuses-2021-11.tar Mon Nov 29 13:01:36 UTC 2021 Compressing bridge-statuses-2021-11.tar Mon Nov 29 13:03:17 UTC 2021 Creating bridge-server-descriptors-2021-11.tar tar: bridge-server-descriptors-2021-11/2/c: file changed as we read it Mon Nov 29 13:21:11 UTC 2021 Compressing bridge-server-descriptors-2021-11.tar Mon Nov 29 13:26:59 UTC 2021 Creating bridge-extra-infos-2021-11.tar Mon Nov 29 13:43:13 UTC 2021 Compressing bridge-extra-infos-2021-11.tar Mon Nov 29 13:54:31 UTC 2021 Creating snowflakes-2021-11.tar Mon Nov 29 13:54:31 UTC 2021 Compressing snowflakes-2021-11.tar Mon Nov 29 13:54:31 UTC 2021 Creating bridge-pool-assignments-2021-11.tar Mon Nov 29 13:54:46 UTC 2021 Compressing bridge-pool-assignments-2021-11.tar Mon Nov 29 13:55:25 UTC 2021 Creating bridgedb-metrics-2021-11.tar Mon Nov 29 13:55:25 UTC 2021 Compressing bridgedb-metrics-2021-11.tar Mon Nov 29 13:55:25 UTC 2021 Creating bridgestrap-2021-11.tar Mon Nov 29 13:55:26 UTC 2021 Compressing bridgestrap-2021-11.tar Mon Nov 29 13:55:28 UTC 2021 Creating webstats-2021-11.tar Mon Nov 29 13:55:41 UTC 2021 Creating webstats-2021-11.tar Mon Nov 29 13:55:45 UTC 2021 Moving tarballs into place Mon Nov 29 13:55:45 UTC 2021 Finished tarball creation. Starting symlink-update ... Mon Nov 29 13:55:45 UTC 2021 Finished.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 29 Nov '21

29 Nov '21

2021-11-29 00:07:44,662 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.73834170135458). 2021-11-29 00:37:49,783 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.17284003800489). 2021-11-29 01:07:34,494 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.44209405716267). 2021-11-29 01:37:11,649 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.56604028215966). 2021-11-29 02:07:33,762 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (15.036678284518356). 2021-11-29 02:37:07,489 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.565547680229878). 2021-11-29 03:07:37,185 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.24516754119397). 2021-11-29 03:38:40,174 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.85103960570622). 2021-11-29 04:12:05,595 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (37.56521144104953). 2021-11-29 04:39:15,481 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.48916192743105). 2021-11-29 05:10:48,679 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (42.61823351169134). 2021-11-29 05:38:04,666 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.67107870077258). 2021-11-29 06:08:48,890 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.429767795962995). 2021-11-29 06:40:12,777 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.97200059134554). 2021-11-29 07:09:01,852 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (48.11476717338221). 2021-11-29 07:38:31,838 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (24.876196851106155). 2021-11-29 08:08:13,047 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.0410958465314). 2021-11-29 08:40:37,755 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (20.589965209803722). 2021-11-29 09:15:41,659 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (29.111055530971623). 2021-11-29 09:40:16,868 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.426370813323754). 2021-11-29 10:05:13,782 WARN o.t.m.c.r.RelayDescriptorParser:175 Could not parse m line 'm 28,29 sha256=cCqF+uyAvCbWYHohA1eBxxYRRK40FbBiN18L2pUQbK' in descriptor. Skipping. 2021-11-29 10:08:44,793 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.85932694036491). 2021-11-29 10:38:28,950 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.509326940365). 2021-11-29 11:07:59,706 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (32.70874121868096). 2021-11-29 11:39:47,625 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.813063005220034). 2021-11-29 12:08:06,768 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.97119887438669).

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 29 Nov '21

29 Nov '21

2021-11-29 08:40:02,185 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-29 08:40:02,219 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:40:02,436 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-29 08:40:02,516 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:40:05,576 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-29 08:40:08,106 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-29 08:42:25,416 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-29 08:42:25,783 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-29 08:42:25,783 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-29 08:42:25,784 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-29 08:42:43,886 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-29 08:42:43,889 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-29 08:42:44,952 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-29 09:02:41,475 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-29 09:03:33,494 INFO o.t.m.s.o.Main:402 Writing 59299 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-29 09:03:35,370 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-29 09:03:42,993 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-29 09:03:43,018 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-29 09:04:39,300 INFO o.t.m.s.o.Main:402 Writing 47800 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-29 09:04:39,604 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-29 09:05:05,430 INFO o.t.m.s.o.Main:402 Writing 18804 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-29 09:05:05,434 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-29 09:05:17,112 INFO o.t.m.s.o.Main:402 Writing 19771 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-29 09:05:17,115 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-29 09:05:17,116 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-29 09:05:17,116 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-29 09:05:17,117 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-29 09:27:47,805 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:27:52,827 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-29 09:29:47,966 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153609 network status entries and 113481 bandwidth history elements 2021-11-29 09:29:49,652 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-29 09:29:52,008 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-29 09:29:52,011 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-29 09:29:52,012 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,062 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:31:24,112 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-29 09:31:24,538 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-29 09:33:29,068 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-29 09:33:29,744 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-29 09:40:09,701 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-29 09:41:55,611 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-29 09:41:55,903 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-29 09:41:55,903 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-29 09:41:55,904 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-29 09:41:55,920 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-29 09:41:55,921 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-29 09:41:56,072 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-29 09:43:34,054 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-29 09:44:14,279 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-29 10:50:44,845 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-29 10:50:59,615 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-29 10:50:59,755 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-29 10:51:33,157 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-29 10:51:33,157 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-29 10:51:33,158 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-29 10:51:33,159 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-29 11:07:33,468 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-29 11:21:25,834 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-29 11:21:25,937 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-29 11:40:14,737 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-29 11:40:14,920 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-29 11:40:14,948 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-29 11:40:14,949 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-29 11:40:15,002 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-29 11:40:16,975 INFO o.t.m.s.w.Main:113 Found 286017 previously imported log files. 2021-11-29 11:49:38,828 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-29 12:06:39,357 INFO o.t.m.s.w.Main:298 Writing 1423685 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-29 12:06:40,273 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-29 12:06:40,274 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-29 12:06:40,274 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-29 12:06:40,275 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-29 12:08:27,822 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-29 12:08:28,010 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-29 12:08:30,973 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-29 12:08:31,197 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-29 12:08:31,198 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-29 12:08:31,458 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-29 12:08:34,596 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <metrics@meronense> cd /srv/metrics.torproject.org/metrics/work/modules/logs && cat metrics.log | grep -v "Could not insert bandwidth "
by root＠meronense.torproject.org 29 Nov '21

29 Nov '21

2021-11-28 08:40:02,474 INFO o.t.m.s.main.Main:32 Starting metrics update run. 2021-11-28 08:40:02,759 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.collectdescs.Main module. 2021-11-28 08:40:02,762 INFO o.t.d.i.DescriptorIndexCollector:47 Starting descriptor collection. 2021-11-28 08:40:02,833 INFO o.t.d.i.DescriptorIndexCollector:63 Indexing local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-28 08:40:06,820 INFO o.t.d.i.DescriptorIndexCollector:75 Fetching remote index file https://collector.torproject.org/index/index.json. 2021-11-28 08:40:08,387 INFO o.t.d.i.DescriptorIndexCollector:84 Fetching remote files from https://collector.torproject.org. 2021-11-28 08:40:44,747 INFO o.t.d.i.DescriptorIndexCollector:90 Deleting extraneous files from local directory /srv/metrics.torproject.org/metrics/work/shared/in. 2021-11-28 08:40:45,595 INFO o.t.d.i.DescriptorIndexCollector:95 Finished descriptor collection. 2021-11-28 08:40:45,595 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.collectdescs.Main module. 2021-11-28 08:40:45,595 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.connbidirect.Main module. 2021-11-28 08:41:00,721 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.connbidirect.Main module. 2021-11-28 08:41:00,722 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.onionperf.Main module. 2021-11-28 08:41:00,727 INFO o.t.m.s.o.Main:46 Starting onionperf module. 2021-11-28 08:41:00,727 INFO o.t.m.s.o.Main:67 Connecting to database. 2021-11-28 08:41:01,234 INFO o.t.m.s.o.Main:70 Successfully connected to database. 2021-11-28 08:55:43,349 INFO o.t.m.s.o.Main:261 Querying statistics including partials from database. 2021-11-28 08:56:30,927 INFO o.t.m.s.o.Main:402 Writing 59272 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-including-partials.csv. 2021-11-28 08:56:31,227 INFO o.t.m.s.o.Main:286 Querying timeout and failure statistics from database. 2021-11-28 08:56:45,034 INFO o.t.m.s.o.Main:402 Writing 19761 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-failures.csv. 2021-11-28 08:56:45,052 INFO o.t.m.s.o.Main:309 Querying buildtime statistics from database. 2021-11-28 08:57:29,802 INFO o.t.m.s.o.Main:402 Writing 47782 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/buildtimes.csv. 2021-11-28 08:57:30,310 INFO o.t.m.s.o.Main:332 Querying latency statistics from database. 2021-11-28 08:57:42,439 INFO o.t.m.s.o.Main:402 Writing 18794 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/latencies.csv. 2021-11-28 08:57:42,443 INFO o.t.m.s.o.Main:357 Querying throughput statistics from database. 2021-11-28 08:57:54,040 INFO o.t.m.s.o.Main:402 Writing 19761 lines to /srv/metrics.torproject.org/metrics/work/modules/onionperf/stats/onionperf-throughput.csv. 2021-11-28 08:57:54,045 INFO o.t.m.s.o.Main:409 Disconnecting from database. 2021-11-28 08:57:54,046 INFO o.t.m.s.o.Main:62 Terminated onionperf module. 2021-11-28 08:57:54,046 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.onionperf.Main module. 2021-11-28 08:57:54,046 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bwhist.Main module. 2021-11-28 08:57:54,047 INFO o.t.m.s.bwhist.Main:37 Starting bwhist module. 2021-11-28 08:57:54,047 INFO o.t.m.s.bwhist.Main:39 Reading descriptors and inserting relevant parts into the database. 2021-11-28 09:10:25,764 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153524 network status entries and 126511 bandwidth history elements 2021-11-28 09:10:26,528 INFO o.t.m.s.bwhist.Main:50 Aggregating database entries. 2021-11-28 09:11:49,358 INFO o.t.m.s.b.RelayDescriptorDatabaseImporter:543 Finished importing relay descriptors: 153524 network status entries and 126511 bandwidth history elements 2021-11-28 09:11:50,058 INFO o.t.m.s.bwhist.Main:53 Querying aggregated statistics from the database. 2021-11-28 09:11:50,426 INFO o.t.m.s.bwhist.Main:57 Closing database connection. 2021-11-28 09:11:50,427 INFO o.t.m.s.bwhist.Main:60 Terminating bwhist module. 2021-11-28 09:11:50,427 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bwhist.Main module. 2021-11-28 09:11:50,427 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.advbwdist.Main module. 2021-11-28 09:12:43,688 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.advbwdist.Main module. 2021-11-28 09:12:43,688 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.hidserv.Main module. 2021-11-28 09:12:43,688 INFO o.t.m.s.hidserv.Main:37 Initializing parser and reading parse history... 2021-11-28 09:12:44,060 INFO o.t.m.s.hidserv.Main:52 Parsing descriptors... 2021-11-28 09:14:27,668 INFO o.t.m.s.hidserv.Main:59 Writing parse history... 2021-11-28 09:14:27,799 INFO o.t.m.s.hidserv.Main:66 Extrapolating statistics... 2021-11-28 09:19:29,181 INFO o.t.m.s.hidserv.Main:82 Aggregating v2 statistics... 2021-11-28 09:21:04,300 INFO o.t.m.s.hidserv.Main:98 Aggregating v3 statistics... 2021-11-28 09:21:04,773 INFO o.t.m.s.hidserv.Main:108 Terminating. 2021-11-28 09:21:04,773 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.hidserv.Main module. 2021-11-28 09:21:04,773 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.clients.Main module. 2021-11-28 09:21:04,841 INFO o.t.m.s.clients.Main:36 Starting clients module. 2021-11-28 09:21:04,841 INFO o.t.m.s.clients.Main:38 Connecting to database. 2021-11-28 09:21:05,086 INFO o.t.m.s.clients.Main:41 Reading relay descriptors and importing relevant parts into the database. 2021-11-28 09:22:36,656 INFO o.t.m.s.clients.Main:45 Reading bridge descriptors and importing relevant parts into the database. 2021-11-28 09:23:20,530 INFO o.t.m.s.clients.Main:49 Processing newly imported data. 2021-11-28 10:32:08,714 INFO o.t.m.s.clients.Main:53 Querying aggregated statistics from the database. 2021-11-28 10:32:27,195 INFO o.t.m.s.clients.Main:59 Disconnecting from database. 2021-11-28 10:32:27,215 INFO o.t.m.s.clients.Main:62 Running detector. 2021-11-28 10:33:01,875 INFO o.t.m.s.clients.Main:65 Terminating clients module. 2021-11-28 10:33:01,979 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.clients.Main module. 2021-11-28 10:33:01,979 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.servers.Main module. 2021-11-28 10:33:01,980 INFO o.t.m.s.servers.Main:43 Starting servers module. 2021-11-28 10:33:01,981 INFO o.t.m.s.servers.Main:45 Reading descriptors and inserting relevant parts into the database. 2021-11-28 10:47:04,856 INFO o.t.m.s.servers.Main:77 Aggregating database entries. 2021-11-28 11:20:03,456 INFO o.t.m.s.servers.Main:80 Committing all updated parts in the database. 2021-11-28 11:20:03,711 INFO o.t.m.s.servers.Main:90 Querying aggregated statistics from the database. 2021-11-28 11:38:15,783 INFO o.t.m.s.servers.Main:105 Terminating servers module. 2021-11-28 11:38:16,094 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.servers.Main module. 2021-11-28 11:38:16,094 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.webstats.Main module. 2021-11-28 11:38:16,173 INFO o.t.m.s.w.Main:75 Starting webstats module. 2021-11-28 11:38:16,173 INFO o.t.m.s.w.Main:92 Connecting to database. 2021-11-28 11:38:16,250 INFO o.t.m.s.w.Main:95 Successfully connected to database. 2021-11-28 11:38:16,250 INFO o.t.m.s.w.Main:101 Querying previously imported log files. 2021-11-28 11:38:18,489 INFO o.t.m.s.w.Main:113 Found 285835 previously imported log files. 2021-11-28 11:46:57,241 INFO o.t.m.s.w.Main:268 Querying statistics from database. 2021-11-28 12:03:53,531 INFO o.t.m.s.w.Main:298 Writing 1422843 lines to /srv/metrics.torproject.org/metrics/work/modules/webstats/stats/webstats.csv. 2021-11-28 12:03:55,214 INFO o.t.m.s.w.Main:305 Disconnecting from database. 2021-11-28 12:03:55,216 INFO o.t.m.s.w.Main:87 Terminated webstats module. 2021-11-28 12:03:55,216 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.webstats.Main module. 2021-11-28 12:03:55,216 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.totalcw.Main module. 2021-11-28 12:03:55,217 INFO o.t.m.s.totalcw.Main:38 Starting totalcw module. 2021-11-28 12:03:55,217 INFO o.t.m.s.totalcw.Main:40 Reading consensuses and votes and inserting relevant parts into the database. 2021-11-28 12:05:56,243 INFO o.t.m.s.totalcw.Main:64 Committing all updated parts in the database. 2021-11-28 12:05:56,372 INFO o.t.m.s.totalcw.Main:74 Querying aggregated statistics from the database. 2021-11-28 12:06:03,457 INFO o.t.m.s.totalcw.Main:77 Writing aggregated statistics to /srv/metrics.torproject.org/metrics/work/modules/totalcw/stats/totalcw.csv. 2021-11-28 12:06:03,682 INFO o.t.m.s.totalcw.Main:82 Terminating totalcw module. 2021-11-28 12:06:03,683 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.totalcw.Main module. 2021-11-28 12:06:03,683 INFO o.t.m.s.main.Main:70 Starting org.torproject.metrics.stats.bridgedb.Main module. 2021-11-28 12:06:03,948 INFO o.t.m.s.main.Main:73 Completed org.torproject.metrics.stats.bridgedb.Main module. 2021-11-28 12:06:03,949 INFO o.t.m.s.main.Main:82 Making module data available. 2021-11-28 12:06:12,619 INFO o.t.m.s.main.Main:126 Successfully copied 22 files to stats output directory: [connbidirect2.csv, onionperf-failures.csv, buildtimes.csv, latencies.csv, torperf-1.1.csv, onionperf-throughput.csv, onionperf-including-partials.csv, bandwidth.csv, advbwdist.csv, v3hidserv.csv, hidserv.csv, clients.csv, userstats-combined.csv, platforms.csv, relayflags.csv, versions.csv, advbw.csv, networksize.csv, ipv6servers.csv, webstats.csv, totalcw.csv, bridgedb-stats.csv] 2021-11-28 12:06:12,620 INFO o.t.m.s.main.Main:130 Completed metrics update run.

1 0

Cron <collector@colchicifolium> cd /srv/collector.torproject.org/collector/logs/ && cat metrics.log | grep -E "WARN|ERROR|FATAL" | grep -v "Overwriting existing descriptor file"
by root＠colchicifolium.torproject.org 29 Nov '21

29 Nov '21

2021-11-29 00:07:44,662 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (45.73834170135458). 2021-11-29 00:37:49,783 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (36.17284003800489). 2021-11-29 01:07:34,494 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (33.44209405716267). 2021-11-29 01:37:11,649 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (26.56604028215966). 2021-11-29 02:07:33,762 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (15.036678284518356). 2021-11-29 02:37:07,489 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (14.565547680229878). 2021-11-29 03:07:37,185 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (25.24516754119397). 2021-11-29 03:38:40,174 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (22.85103960570622). 2021-11-29 04:12:05,595 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (37.56521144104953). 2021-11-29 04:39:15,481 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.48916192743105). 2021-11-29 05:10:48,679 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (42.61823351169134). 2021-11-29 05:38:04,666 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (35.67107870077258). 2021-11-29 06:08:48,890 WARN o.t.m.c.r.ReferenceChecker:314 Missing too many referenced descriptors (39.429767795962995).

1 0

2024

2023

2022

2021

2020

2019

2018

metrics-alerts November 2021