The Postal Service is proposing to change the way it measures the on-time service performance of First Class Mail. Instead of contracting a third-party to evaluate how long it takes for the mail to be delivered, the Postal Service wants to count the days itself. The change requires the approval of the Postal Regulatory Commission, and yesterday several stakeholders and postal watchdogs filed comments to PRC Docket PI2015-1.
The current system is called External First-Class Measurement (EXFC). The Postal Service has been using this system since 1990. As the Postal Service explains on one of its quarterly performance reports:
“EXFC is a rigorous external sampling system measuring the time it takes from deposit of mail into a collection box or lobby chute until its delivery to a home or business. EXFC measures the transit time for single-piece rate First- Class cards, letters, and flat envelopes and compares this actual service against service standards.”
The EXFC system is conducted by an external independent third-party — IBM — and it measures the end-to-end length of time it takes for mail to be delivered. The participants, known as droppers and reporters, are supposed to be kept confidential, and the whole process is supposed to be conducted without managers and workers knowing which pieces are being tested. The test mail is statistically analyzed based on sample volume, mail characteristics, and the location where the mail was entered and delivered.
The results of the EXFC tests are published quarterly on the USPS website here. (For previous quarters, just change the dates in the URL.)
The results typically show that the Postal Service is meeting its targets for First Class mail, with about 95 percent being delivered within the service standard for overnight and 2-day mail and about 85 percent for 3-5-day mail.
It should be noted that this high level of performance may be declining as a result of the new service standards that were introduced at the beginning of this year, which allowed the Postal Service to make significant changes in how mail is processed. There have been many anecdotal reports of delays, and it’s likely that less mail is meeting the service standards.
In a motion filed yesterday, the APWU says just that: “At this time the new degraded service standards that went into effect on January 5, 2015, have not been met by most of the mail processing facilities across the country, including the losing and gaining facilities on the list for Consolidations or Closures. The EXFC scores show after 12 weeks that mail is being delayed.”
The service performance results for the second quarter of the fiscal year (January-March) have not been released yet, but they're likely to show exactly what the APWU alleges.
The new measurement system
For various reasons, the Postal Service wants to change the measuring system from EXFC to what it calls Service Performance Measurement (SPM). Under the proposed system, the Postal Service would do the measuring itself, rather than using an external third-party, and it would take advantage of the fact that much of the mail is now barcoded.
Letter carriers would scan barcoded mailpieces from randomly selected collection points (collection boxes and office building chutes). Instead of having a reporter mark down when a piece of mail is received, the carrier would scan the barcode to mark the delivery time. The collection and delivery points would be selected based on a statistical design to make sure they’re representative of the population being measured. (The details of the new system are described in the plan submitted by the USPS to the PRC here.)
The new system is essentially an idea that was recommended by a USPS OIG report in September 2012 (FF-AR-12-006: “Evaluation of the External First-Class Measurement System Audit Report”). The OIG concluded that thanks to barcoding and hand-held scanners, the Postal Service could automate the service performance process, eliminate the need for costly manual recording and reporting, and do the work internally instead of paying a third-party.
The OIG also suggested that a new system might be less vulnerable to the problem of “gaming.” That's when management and staffers are able to identify which pieces of mail are possibly being tested and then giving these pieces preferential treatment in order to raise performance scores. There has been plenty of incentive to do so, since bonuses and pay increases are said to hinge on these scores.
Concerns about the plan
In order to implement the new measuring system, the Postal Service needs the approval of the Postal Regulatory Commission, so for the past few weeks, the PRC, stakeholders, and a couple of postal watchdogs have been reviewing the details of the new system. A number of issues have arisen, and comments have been filed pointing to potential problems.
In comments submitted by the PRC’s Public Representative, the PR suggests that the new system is “a step in the right direction” because it’s technology driven (it uses barcodes and scanners), but “many aspects regarding the underlying methodology and the measurement results are unclear.”
The PR is concerned that the proposed SPM system may not measure the same end-to-end delivery time as EXFC. In EXFC, the start time is when the mail is dropped into a collection box. (The “dropper” puts the mail in the box and records the time.) In the SPM system, the Start-the-Clock is unknown so it needs to be determined, using the pick-up date and time and an “adjustment methodology” described in the Postal Service’s proposal. But whatever method is used, there will still be uncertainty about the date and time when the mailpiece entered the mailstream.
The PR also expresses concern about the way the new system moves from a real mailpiece to a “virtual” mailpiece. EXFC measures the end-to-end performance of actual pieces of mail sent by the droppers and received by reporters. The new SPM system creates what the PR calls a “virtual” piece of mail.
That’s because the SPM doesn't track the delivery time of a particular piece of mail. Insead, it measures a statistical composite that averages, weights, and combines mailpieces on three different stages of the End-to-End delivery process — First Mile, Processing Operation, and Last Mile. This method puts much more significance on the quality of the statistical design and the accuracy of sampling.
Another issue covered by the Public Representative is costs. The current measurement system costs about $40 million a year (not all of it for EXFC per se). The Postal Service says the new system will cost about $12 million for external vendor support, but the PR points out that there are also the costs for implementing the SPM (about $11 million) and internal costs for which the Postal Service won't offer an estimate. Overall, it’s possible the new system would cost more than the current one, which would completely undermine one of the main reasons for switching over to the new system.
Other comments filed with the PRC address the fact that under the new system the Postal Service will no longer be using an independent third party. That’s one of the main concerns in comments filed by the APWU.
The APWU observes that gaming the system was always a problem with EXFC, as discussed in that 2012 OIG report. But it could become an even bigger problem with SPM. A related problem is that employees can be blamed and disciplined when EXFC goals aren’t met, even when they may have had nothing to do with the delays. The union is concerned that these problems could get worse with SPM.
The gaming problem is also discussed in the comments filed by postal watchdogs David Popkin and Douglas Carlson.
In his comments, Popkin recalls instances where postal employees were caught gaming the system, and he suggests that "there were probably many more instances that this took place." He questions whether the new system will really solve the problem.
In his comments, Carlson suggests that one way of addressing the issue would be to scan a larger proportion of the collection mail than currently proposed. Supervisors and employees would then need to assume that test mail could be in all the containers of collection mail, which (if I’m understanding the idea correctly) would presumably reduce the potential for gaming.
Another issue with the proposed SPM involves which mail is being tested. As it turns out, the system will not encompass mail that is left for the carrier in a mailbox, which is the most common method of sending single-piece First Class Mail. The EXFC system doesn’t test carrier-collected mail either, but the Postal Service says that mail entered at collection points like blue boxes and post offices “serve as reasonable proxies for the mail left at customer mail receptacles.”
Carlson notes that there are often cases where the carrier does not return to the post office from his or her route before the final dispatch truck departs. That’s a problem under the current system, and it will continue to be one in the new system. As a solution, he suggests that carriers could scan collection mail from their routes every hour to measure the first mile time.
One way of addressing some of these various concerns would be to run both the EXFC and SMP systems simultaneously for a period of time so that the results could be compared. Carlson makes this recommendation, and so do the comments submitted by a group of mailers that includes the Association for Postal Commerce, IDEAlliance and the National Association of Presort Mailers.
There are still more issues to be addressed. Yesterday the APWU filed a motion asking that the PRC pose a number of additional questions to the Postal Service. The APWU wants to know how much the Postal Service is paying to IBM for the sampling software program the SPM will use, what guarantees the Postal Service will put in place to identify errors and cover-ups by management, and if management will receive financial incentives for high scores.
The PRC will continue its review of the new performance service system over the coming weeks. A final ruling should come some time this summer.
It seems likely at this point that the Postal Service will eventually switch over to the new system, but the Commission may want these concerns (and others) addressed, one way or another.