Log Validator for New Relic

Which of an account's logs anything actually asks for, and what the rest costs a month.

Every alert condition and every dashboard widget that reads a log table is collected, and a day of log ingest is measured stream by stream. A stream nothing names is one nobody has asked a question of: it is still paid for, still stored, and nothing on any screen would notice if it stopped arriving.

Choosing an account

The account list is the profiler's, largest by this month's ingest first, because log spend is concentrated in a handful of accounts. The screen opens on whatever was stored for that account by the nightly job. Validate now is an admin's button, and it asks before it starts. It reads every alert condition and every dashboard in the account and then measures every partition, which is thousands of calls to New Relic on the shared key: seconds on a small account, around six minutes on the largest. It is there for when an answer is needed sooner than tonight, and rarely for anything else, because the nightly job caches every account it covers and what is on screen is never more than 23 hours old.

What counts as required

A stream is required when at least one alert condition or dashboard widget names it, exactly or through a specific wildcard. Naming means a filter on the attribute that identifies the stream, not on what is inside it.

Most log queries filter on message text, and those read every stream in their table: counting them as asking for a stream would make everything required and the answer worth nothing. The screen says how many of an account's log queries are that kind. A wildcard that reduces to match-anything is ignored for the same reason. Two of them, entity.name LIKE '%' and service.name LIKE '%%', were enough to mark all 322 streams in Aurora Retail Prod required before they were excluded.

Widgets on the account's dashboards that read a different account are left out, since they say nothing about whether these logs are wanted.

Partitions, and why they change the answer

A query written FROM Log reads the default partition only. Where a rule routes an account's logs elsewhere, a query naming the app but reading the wrong table sees nothing, however precisely it names it. Each partition is therefore matched only against the queries that read that partition.

This is what the wrong partition verdict means, and it is not a saving. In Digital Production, 841 GB a day belongs to services named in log queries that read FROM Log while a rule sends their data to Log_EMEA: checkout-options-primary is named in 42 queries and none of them can see its EL region logs. Those are queries to fix. Dropping the data would make the blind spot permanent.

Keyed on says which attribute identified the streams in that partition. It is chosen per partition from what the records carry and what the account's own queries name, not assumed: a Lambda partition carries aws.logGroup and no appName, and keying the default partition on entity.name because more records carry it would report an estate as unused while every alert sits on appName.

A partition marked read wholesale is one queries read constantly without ever naming a stream in it, so nothing in it can be called unused and it is left out of the drop rules.

Two cards lead that table's title, each giving the records a month, the ingest a month and the money a month behind it.

The green one is the saving: the streams nothing mentions anywhere. It counts exactly what the drop rules cover, so the headline and the rules under it cannot disagree. The amber one is the question: streams named only in queries on other event types, where the service is watched through its metrics or its transactions and nobody reads its logs. Whether that is waste is a judgement about what someone would reach for after an incident, which is why it is not drawn in the color of money already found. Both leave out the partitions read wholesale.

Only untargeted, at the right of the same row, is on when the screen opens: the table lists the streams nothing asks for, largest first, each with the reason it is there. Turn it off for every stream the account ingests. Only overlapping, beside it, keeps just the streams that arrive by more than one route, with the same overlap marks the table already shows. It combines with Only untargeted, so both on lists the streams nothing asks for that are also shipped twice. It is switched off when the account has no overlapping streams.

Note that turning it off changes little above the fold. Streams sort largest first and the large ones are nearly always required, so what the switch adds is a thousand rows below what is already on screen; the count beside the title is the quicker read.

The three kinds of untargeted

Named nowhere is the candidate to stop collecting: no condition and no widget mentions it at all.

Wrong partition is a monitoring gap, as above.

Non-log queries only means the name appears in queries on other event types, so the service is watched through its metrics or transactions while its logs are read by nobody. Whether that is waste is a judgement about what someone would reach for after an incident.

Who else reads a stream

A dashboard or an alert condition is not the only thing that reads a stream. People search in the Logs screen, write queries by hand, and scripts ask through the API. The validator reads New Relic's query log for the last 30 days to find them: which streams were named, how often, by whom and from where.

A busy account runs millions of log queries a month, so they are not read one at a time. For each attribute a stream is named by (appName, entity.name, service.name, hostname, logtype, aws.logGroup), New Relic is asked to lift the value out of every log query that names one, and to count by value, person and place. One consequence: only the first value a query gives for an attribute is counted, so a query listing five applications is credited to the first.

A stream counts as queried by name when a query says appName = 'orders-api', and by wildcard when only a LIKE pattern matched it, which is weaker evidence that anybody needs it. Queries made with this application's own API key are left out.

Most log queries name no stream at all: they search by message text, or filter on something else such as a namespace or a pod. Those are counted against the partition they read, under Runs in the log and People in the Partitions table. That is the caution on every low risk rating here: a stream nobody names, in a partition hundreds of people search, will stop turning up in those searches once it is dropped.

Overlapping log streams

A stream arriving by two routes at once is the same application's logs being shipped twice, for example by an agent's log forwarding and by a log forwarder reading the same files. Each stream is broken down by what shipped it, and one where a second route carries a tenth or more is listed, grouped by partition and by the pair of routes.

A possible overlap is one where some of a stream's records do not say what shipped them. They may be a second copy, or may have come the same way as the rest, so a possible overlap is shown but does not change a stream's rating.

+ Create task, in the heading of this section, makes one task for every overlapping set. It names each set's partition and routes, how much arrives by each, and every stream in the set with its volume, and recommends keeping one route and stopping the other at its source. The saving it quotes is an upper bound: the volume carried by the routes other than the largest in each set. The button appears only for people who can reach Workstreams, and only when the account has overlapping streams.

Risk of dropping

• In use. A dashboard or a live alert condition names it, or ten or more people queried it.
• Medium risk. No dashboard or alert names it, but fewer than ten people queried it. They are named, with how often and from where, so they can be asked first.
• Low risk. Nothing in the account names it and nobody queried it in the 30 days. Low risk, overlapping also arrives by more than one route.
• Wrong partition. Queries name it but read a partition it is not in. A query to fix, never data to drop.
• Partition read wholesale. Queries read the partition without ever naming a stream, so nothing in it can be called unused.

Watched another way beside a rating marks a stream whose name appears in queries on other event types: the service is watched through its metrics or transactions.

What to do about it

Drop rules - Low risk: unused and overlapping and Drop rules - Low risk: unused, no overlap are NRQL ready to paste into New Relic as Drop data rules, one per partition, chunked to stay inside New Relic's limits. These are the ones to act on first.

Medium risk - Queried by fewer than 10 people lists each such stream with everyone who queried it. No drop rule is offered: the task it creates is to ask them.

Remediation - Wrong partition is not a drop rule and deliberately never becomes one. These queries name the stream correctly and read a partition that cannot hold it, so they return nothing and any alert built on them cannot fire. It gives the pair of partitions involved, the streams stranded between them, the conditions and dashboards doing it, and the FROM clause that fixes it. Dropping this data would make the blind spot permanent.

X-RAMP creates nothing here, and a drop rule discards at ingest: what it drops is not recoverable.

The nightly job

Configured under Admin, Log validator config, and the accounts are the Log analysis column of the table under Cache scheduler. The Metrics Validator keeps its own job and its own column beside it. With that column left empty it validates the busiest 15 accounts by this month's ingest, which is where log spend actually is. It runs at 5am by default, an hour after the alert audit, because it reads every dashboard in every chosen account and has no reason to compete with the other three jobs for the same key.