Drop Rule Effectiveness for New Relic

Whether an account's drop rules do what they say. New Relic does not report how much each drop rule removed, so each one is read from what is left in storage.

Running a test

Choose an account and press Test rules. X-RAMP reads the account's drop rules from New Relic and checks each one against the last 24 hours of stored data, with read-only queries. Click the Created heading to sort the rules by when they were made: once for newest first, again for oldest first, a third time to return to the test's own order, which groups them by verdict. Rules with no date stay at the bottom either way. A hundred rules take about two minutes. The result is kept for a day, and the Search box and the buttons above the table narrow what is shown.

How a rule is checked

• A drop data rule should leave no matching record. X-RAMP counts the records that still match the rule's query.
• A drop attributes rule should leave the records but take the listed attributes off them. X-RAMP counts the matching records, and for each attribute, how many still carry it.

The verdicts

• Holding. What the rule targets is not being stored. For an attribute rule the records are arriving and none carries a dropped attribute.
• Dormant. An attribute rule that nothing matched in the window, so the data it was written for has stopped arriving. A candidate for retirement, once the source is known not to be periodic.
• Cannot verify. The rule's own condition reads an attribute it drops. Once it works, nothing can match it, so a working rule and a quiet source look the same. These need another check, such as sending a test record.
• Not effective. What the rule targets is still stored. An attribute that New Relic does not allow to be dropped shows up here.
• Could not test. New Relic did not answer that rule's query. Test again.

A rule made within the last day is checked only from when it took effect, ten minutes after it was made, because records stored before it existed are not the rule's to remove. If nothing matching has been stored since, and records matching the rule arrived in the day before it, the verdict is Holding and the evidence says Dropping. If the source sent nothing in the day before either, it is still Holding, but X-RAMP does not claim the rule is dropping, because a working rule and a quiet source look the same. A rule made less than ten minutes ago is Could not test and is tested again at the next run.

A rule that is an exact copy of an earlier one is also marked, with the number of the rule it copies. The oldest of the set is the one to keep.

New rules are measured

The checks above say whether a rule holds. For a rule made recently they can also say what it saved. When a rule was made within a day of a test, X-RAMP measures the volume it targets for the day before the rule and for the time since, as daily rates, and shows the two on the rule's row with the saving in gigabytes a day, as a percentage and, when a price is set, in dollars a month.

• A drop data rule's saving is the volume that went.
• A drop attributes rule keeps its records and makes each one smaller, so the saving is the bytes removed from a record times the records now arriving.

A rule is measured again at each test until it is a week old, so the "since" side grows from a few hours into a settled figure. The "before" side does not move. After a week the measurement is kept as it was, because the data it came from ages out of New Relic. A rule less than an hour old is marked too new. A rule whose condition reads the attribute it drops cannot be measured, and a rule that matched nothing in the day before it was made had nothing to save.

It is an estimate. One ordinary day stands in for "before", so a source that grew or shrank since the rule was made moves it, and a saving is never shown as negative.

What it cannot show

A rule that works and a source that has gone quiet look the same for a drop data rule: both read zero. For a rule that is not new, the test cannot say how many gigabytes it saved, because New Relic does not publish that and the data from before the rule has aged out. Only a rule measured while new has a figure. The window is 24 hours, so a source that sends weekly shows as dormant. Treat dormant and cannot verify as prompts to check, not as proof.

The nightly job

A result is kept for a day, and a job keeps it current so the screen opens already answered. It is configured under Admin, Drop rule test config, with its accounts in the Drop rules column of the table under Cache scheduler. Left empty it takes the busiest 15 accounts by ingest. It runs at 7am in the application's time zone, an hour after the metrics validator, because the jobs share one New Relic key and queue rather than compete. Run now starts it at once. An account with no drop rules is recorded as having none.

Using the result

Download CSV saves every rule and its result. Print gives the table without the controls. Each row's The rule opens the rule's NRQL. Deleting a rule is done in New Relic. This screen only reads.