Blog

Demystifying nDCG and ERR

Welcome back, dear reader! In this post, we unwrap the mystery behind two popular search relevance metrics through visualization, and discuss their pros and cons. Our subjects for this exercise are Normalized Discounted Cumulative Gain, and Expected Reciprocal Rank, commonly acronymified as nDCG and ERR. We’ll start with some refresher background, visualize what these metrics actually look like, and paint a picture of how each can be either helpful or misleading, depending on the situation. Afterwards, you’ll have a better understanding of their behavior and which ones to use when (and why).

Assumptions

Note that, while some basic things are explained, this is not an introduction to these metrics – so I’ll assume you’ve at least heard of nDCG and what it’s used for! So if you’re new to relevance measurement, you probably want to start with something like the book “Relevant Search”, or at the very least the Wikipedia article on search metrics.

As a formality, we’ll stick with the relevance grading scale of poor=1, fair=2, good=3, and perfect=4. We’re also only going to look at the grades of the top 4 result positions, and assume each of those results has a grade.

We’re going to also assume that your results are listed one at a time on each row, and not on a grid. There are varying opinions on how best to measure grid results, but that’s beyond the scope of this post.

OK! No more assumptions, let’s get to it…

Background

nDCG has been the go-to metric for measuring relevance in a typical search results list since its first introduction in SIGIR 2002. In the original ERR paper, improvements over its predecessor were mightily touted. However, though being around for about a decade, ERR is surprisingly underused.

The similarities shared between the two metrics are, as noted in the title, the discount and cumulative functions of relevance. These respectively mean that a result is ‘worth’ less the lower on the result list appears, and the grades of all the documents in the list contribute to the score for a query’s relevance. When I spoke at Haystack in April, I showed a simple breakdown of DCG while diagraming the components of the formula:

DCG-formula-annotated.png

To get a DCG score, just follow these steps!

  1. For all the results
  2. Award a result by its relevance
  3. Punish a result by its rank
  4. Add all the result scores

Step 3 in the above diagram is the ‘discount’, and step 4 is the ‘cumulative’. Together they provide the motivation to get relevant documents to the top (and the irrelevant documents to the bottom) to acheive a higher overall score. nDCG has an additional trick up its sleeve – it uses the ideal list of documents to normalize the score between 0 and 1:

nDCG-formula.png

While this is helpful (or not, depending on your math chops), it’s hard to tease out the full picture. For one thing, it doesn’t use any sort of maximum grade. Grades can be anything, and this can be troublesome when performing apples-to-apples comparissons between results. We’ll see why later, but first let’s dive into Expected Reciprocal Rank, and its purported improvements over nDCG.

The first thing ERR incorporated as a fix, was that you must outright declare what the maximum possible grade is. This helps by not needing to calculate an ‘ideal’ score, but it also can be misleading because it assumes there are always relevant documents available. Another addition is the cascading model of relevance. When a user sees a search result they like, they are satisfied. Starting with the understanding that a user will lose trust in your search engine the longer it takes them to be satisfied is the cascade. It works by keeping a running tab of this ‘trust’ a user has, and punishes a query’s score when it’s not satisfying a user quickly.

ERR-formula-annotated.png

Showdown

Whew! With that out of the way, time for the fun part. Let’s look at how these two actually differ in the real practical world. To do that, we’re going to map out every possible combination of grades for the top 4 positions, get their scores, and plot them. The Y axis is the resulting score based on its respective X axis grades of the top 4 results. For example, given the first four results with grades 2, 4, 3, 2, that gives us a standard nDCG of 0.7697. The Y axis is 0.7697, and the X axis is “2,4,3,2”. To make things more interesting, we’ll also look at different discount models. We can change the way lower scores are punished, and it is useful to see how this impacts the metric. The standard for nDCG is 1/log2(r+1), and the standard for ERR is 1/r (where ‘r’ is the position rank).

nDCG-visualized.png

Noting specifics, for the standard discount model of nDCG, right away we start off at about 0.45. This means that for the top four results, you’ll never have a score lower than that. You can also see that lots of the possible combinations for nDCG yields a perfect score of 1.0. Why is this? Well, If your top four results are 1,1,1,1 nDCG will say that’s a perfect score because the ideal sort is the same. We’re going to actually list out the full table below, and you’ll see more of that harsh truth there.

But first, let’s show the visuals for ERR and compare:

ERR-visualized.png

Fascinating. You can clearly see the juxtaposition of nDCG favoring higher scores, and ERR with a more balanced growth. You can also clearly see the dropoff cliff in the ERR scores as soon as the top result becomes a 4 (perfectly relevant). For many of the discounts, ERR heavily favors queries that give a perfect first result. This is not surprising – because the user will be satisfied immediately, making the other results inconsequential.

You may have also noticed something interesting when examining the discount functions. Did you happen to catch 1/(r^0.18)? I arrived at that function through human learning (trial and error), looking for a good discount that provided a more gradual dropoff when the first result was not perfect. While this makes for a more balanced metric however, it can be seen as going against the cascading model’s purpose. With the far more drastic cliff of 1/r (the green line), there will be a much clearer signal for an irrelevant top result.

Pros and Cons

The visualizations above (and the data tables below), gave us an interesting glimpse into the behavior of these two formidable metrics.

nDCG

nDCG is a great metric when you’ve done your best job at grading, and don’t mind a high score when you have nothing better to offer. Remember, nDCG will return a perfect 1.0 for any result set that has the grades in order of highest to lowest in the resultset. When using nDCG, I always recommend using the global ideal rather than the local ideal. This means that when you know a better document exists is out of your measurement scope (like the 10th document in an [email protected] measurement), use that as part of your ideal and avoid just sorting the top 4. Also, for learning-to-rank, consider just using DCG without the normalization! If the goal is a higher number, nobody cares that it’s between 0 and 1. nDCG also has no way to indicate what the maximum score is. To get around this, sometimes it might suit well for the ideal to be all perfect scores for the positions as a best theoretically possible relevant set, as the ideal denominator.

ERR

The default ERR is a great metric for providing a good signal whether the top result is relevant. Some practitioners will argue this is all it’s good for, but If you are serving content that needs more flexibility, you can also tune the discount function for when this is not the case and you want more forgiveness. One interesting thing about ERR is that it never returns a perfect 1.0 score, and it will always assume that the score can be better, which is a main contrast with nDCG and one I happen to like.

Conclusions

In this authors opinion, I prefer using ERR and modifying it to your needs for most cases. It is more advanced than nDCG and may be more complex to explain, but it’s more closely aligned with how people actually behave and react – people do get frustrated with search engines that don’t show relevant documents at the top, so it’s a good idea to use a metric that models that frustration. There are those that argue for information needs with multiple good results (such as exploratory search), that ERR doesn’t accurately reflect this, but there are ways to customize ERR to build the desired measurement – the paper itself has a section on diversity for such occasions, which usually goes overlooked.

Thanks for reading, and see you next time!

Papers

Code

The code used to create the above plots and the data tables below can be found at https://github.com/o19s/metric-plots

Data Tables

Here are the colorized data tables for nDCG and ERR, as visualized above.

nDCG

Position GradesDiscounted Scores
12341/r^0.181/r^0.51/log2(r+1)1/r2/2^r
111111111
11120.9190.7910.750.6330.548
11130.860.6580.6010.4430.333
11140.8230.5830.5190.3470.228
11210.9350.8230.7810.6730.613
11220.9220.7970.760.6390.538
11230.8710.680.6260.4680.352
11240.8330.6010.5380.3660.244
11310.8860.7110.650.5050.429
11320.8860.7110.6540.5050.408
11330.8850.710.660.5040.379
11340.8480.630.570.3970.272
11410.8560.6470.5770.420.339
11420.8590.6520.5850.4240.333
11430.8620.6590.5970.4320.325
11440.8670.670.6140.4440.311
12110.9570.8780.8380.7550.742
12120.9390.8390.8040.7050.641
12130.8820.7060.6520.5050.408
12140.8390.6150.5530.3850.274
12210.950.8640.8280.7380.692
12220.950.8640.8330.7390.674
12230.8970.7360.6840.5380.44
12240.8520.6380.5770.4080.295
12310.9040.7510.6950.560.493
12320.910.7630.710.5730.493
12330.9040.7520.7050.560.451
12340.8630.6610.6020.4350.316
12410.8690.6750.6070.4540.378
12420.8760.6860.6210.4650.381
12430.8770.6890.6280.4680.368
12440.8780.6930.6380.4740.348
13110.9260.80.7410.6290.619
13120.9190.7870.7340.6150.577
13130.910.770.7230.5940.517
13140.8640.6670.6080.450.351
13210.9260.8020.7480.6330.606
13220.9290.8090.7580.6410.6
13230.9190.7880.7440.6170.538
13240.8730.6840.6270.4680.368
13310.9270.8050.7570.6390.586
13320.9290.810.7650.6450.582
13330.9330.8180.7770.6560.576
13340.8880.7140.6590.5020.399
13410.890.7190.6560.5110.444
13420.8940.7260.6660.5190.445
13430.90.7390.6820.5340.448
13440.8970.7330.6820.5280.415
14110.9060.7560.6880.5650.559
14120.9040.7520.6880.5610.541
14130.9010.7460.6870.5550.51
14140.8960.7380.6860.5450.464
14210.9080.760.6950.5710.556
14220.9110.7670.7040.5770.554
14230.9070.7590.7020.570.523
14240.9020.7490.6980.5580.476
14310.9120.7680.7070.5810.55
14320.9140.7730.7150.5860.548
14330.9180.7820.7270.5970.546
14340.9110.7680.7190.5810.497
14410.9160.7780.7240.5960.541
14420.9180.7820.730.60.54
14430.9210.7880.7390.6080.538
14440.9250.7990.7540.6220.536
211111111
21120.9710.9330.9310.9020.846
21130.9030.7630.7270.6150.521
21140.8520.6480.5940.4440.333
21210.9830.9580.9550.9340.897
21220.9770.9440.9410.9130.86
21230.9150.7870.7530.6410.547
21240.8630.6690.6160.4650.353
21310.9250.8090.770.670.606
21320.9280.8150.7780.6750.6
21330.9180.7930.760.6450.538
21340.8730.6870.6370.4850.368
21410.8820.7070.6490.5120.437
21420.8870.7170.660.5210.439
21430.8860.7150.6630.5190.419
21440.8860.7140.6660.5160.39
221111111
22120.9910.9790.980.9710.953
22130.9250.810.7770.6750.6
22140.8690.6820.630.4840.381
222111111
222211111
22230.9390.8380.8050.7070.636
22240.8820.7050.6530.5070.404
22310.9440.8510.8150.7260.68
22320.950.8630.8290.740.688
22330.9370.8340.8030.7010.613
22340.8880.7190.6690.5230.414
22410.8970.7380.6810.5490.482
22420.9040.7510.6950.5620.489
22430.9010.7450.6940.5560.465
22440.8970.7380.6920.5460.429
23110.9580.8850.850.780.775
23120.9570.8830.8510.7780.76
23130.9410.8480.8190.730.67
23140.8880.7230.6730.5360.445
23210.9640.8970.8640.7950.787
23220.9680.9050.8740.8050.792
23230.9510.8680.840.7550.699
23240.8980.7410.6920.5560.465
23310.9560.8810.850.7730.736
23320.960.8890.8590.7820.742
23330.9580.8860.860.7790.723
23340.9090.7650.7180.5830.491
23410.9120.7710.7190.5950.535
23420.9170.7810.730.6050.541
23430.9210.7890.7410.6140.539
23440.9130.7740.7310.5960.492
24110.9270.8070.7520.6490.644
24120.9290.810.7570.6530.64
24130.9220.7970.7490.6370.6
24140.9130.7790.7360.6140.54
24210.9330.8180.7650.6620.655
24220.9370.8270.7750.6710.66
24230.930.8130.7650.6540.618
24240.920.7920.7490.6290.556
24310.9320.8180.7680.6620.639
24320.9360.8260.7770.6710.643
24330.9380.830.7840.6760.636
24340.9270.8070.7660.6480.574
24410.9320.8180.7730.6630.615
24420.9350.8240.780.670.619
24430.9360.8270.7850.6740.614
24440.9380.8320.7950.6810.606
311111111
31120.9820.9590.9590.9450.915
31130.9580.9050.9030.8650.793
31140.8950.7510.7170.6070.51
31210.9890.9750.9730.9630.944
31220.9840.9640.9630.9490.92
31230.9620.9120.910.8720.802
31240.9020.7640.730.620.523
31310.9750.940.9360.910.862
31320.9720.9340.9310.9010.846
31330.9680.9250.9220.8850.818
31340.9140.7860.7530.6440.546
31410.9210.8020.7650.6680.603
31420.9230.8050.7690.6710.6
31430.9260.8110.7770.6760.595
31440.9170.7920.760.6480.538
321111111
32120.9940.9860.9870.9830.973
32130.970.9310.930.9010.846
32140.9070.7750.7430.6370.548
322111111
322211111
32230.9770.9450.9440.9180.871
32240.9160.7920.7590.6540.567
32310.9850.9630.9610.9430.912
32320.9860.9660.9630.9460.914
32330.9810.9530.9510.9260.881
32340.9250.8110.7790.6760.588
32410.9310.8240.7880.6960.639
32420.9350.8320.7960.7040.643
32430.9370.8350.8020.7070.636
32440.9260.8120.7810.6740.574
331111111
33120.9950.9890.990.9860.978
33130.9870.9720.9730.9620.939
33140.9270.8180.7870.6920.62
332111111
332211111
33230.9920.9820.9830.9750.96
33240.9340.8310.80.7070.636
333111111
333211111
333311111
33340.9450.8540.8240.7340.669
33410.9480.8620.8290.7470.706
33420.9510.8670.8350.7530.709
33430.9560.8770.8460.7640.716
33440.9430.8480.8190.7240.642
34110.9590.8890.8550.790.788
34120.9580.8870.8550.7890.781
34130.9570.8850.8560.7870.767
34140.9420.8520.8250.7410.682
34210.9620.8940.8620.7970.794
34220.9640.8990.8670.8020.796
34230.9620.8960.8670.7990.782
34240.9470.8620.8350.7520.695
34310.9660.9040.8740.810.804
34320.9680.9080.8780.8150.806
34330.9710.9150.8860.8230.811
34340.9550.8790.8530.7740.721
34410.9590.8880.8590.7870.754
34420.960.8910.8630.7920.756
34430.9640.8980.8710.7990.761
34440.9620.8940.870.7950.742
411111111
41120.9890.9770.9780.9710.956
41130.9730.9410.9410.9210.881
41140.9510.8910.890.8480.77
41210.9940.9860.9850.980.97
41220.990.9790.9790.9720.957
41230.9750.9440.9440.9240.884
41240.9540.8960.8940.8530.775
41310.9840.9630.9610.9480.921
41320.9810.9580.9570.9410.91
41330.9780.9490.9490.9290.89
41340.9580.9040.9010.860.785
41410.9710.9320.9280.8990.847
41420.970.9290.9250.8950.84
41430.9680.9240.9210.8870.826
41440.9650.9170.9130.8740.801
421111111
42120.9960.9920.9930.9910.986
42130.980.9560.9560.9410.91
42140.9580.9050.9040.8670.797
422111111
422211111
42230.9840.9640.9640.9510.924
42240.9620.9140.9120.8760.81
42310.990.9770.9750.9660.948
42320.990.9780.9760.9670.949
42330.9860.9680.9670.9540.927
42340.9660.920.9180.8830.817
42410.9760.9440.940.9160.872
42420.9770.9460.9420.9180.873
42430.9750.940.9370.9090.858
42440.9710.9310.9280.8940.831
431111111
43120.9970.9930.9940.9920.987
43130.9910.9810.9820.9760.963
43140.9690.930.9290.90.846
432111111
432211111
43230.9940.9880.9890.9850.976
43240.9730.9370.9360.9090.858
433111111
433211111
433311111
43340.9790.950.9480.9250.881
43410.9860.9660.9630.9470.918
43420.9860.9670.9640.9480.919
43430.9870.9690.9660.950.92
43440.9820.9560.9540.9320.889
441111111
44120.9970.9940.9950.9930.989
44130.9930.9850.9850.980.969
44140.9860.9690.970.9580.934
442111111
442211111
44230.9950.990.9910.9870.98
44240.9880.9740.9750.9650.944
443111111
443211111
443311111
44340.9930.9840.9840.9770.963
444111111
444211111
444311111
444411111

ERR

Position GradesDiscounted Scores
12341/r^0.181/r^0.51/log2(r+1)1/r2/2^r
11110.1590.1360.1270.110.106
11120.1590.1360.1270.110.106
11130.1590.1360.1270.110.106
11140.1590.1360.1270.110.106
11210.2490.1990.1820.1470.133
11220.2490.1990.1820.1470.133
11230.2490.1990.1820.1470.133
11240.2490.1990.1820.1470.133
11310.430.3260.2920.220.188
11320.430.3260.2920.220.188
11330.430.3260.2920.220.188
11340.430.3260.2920.220.188
11410.790.580.5110.3660.298
11420.790.580.5110.3660.298
11430.790.580.5110.3660.298
11440.790.580.5110.3660.298
12110.2570.2140.1970.1660.162
12120.2570.2140.1970.1660.162
12130.2570.2140.1970.1660.162
12140.2570.2140.1970.1660.162
12210.3350.2690.2450.1980.186
12220.3350.2690.2450.1980.186
12230.3350.2690.2450.1980.186
12240.3350.2690.2450.1980.186
12310.4910.3790.340.2610.234
12320.4910.3790.340.2610.234
12330.4910.3790.340.2610.234
12340.4910.3790.340.2610.234
12410.8040.5990.530.3880.329
12420.8040.5990.530.3880.329
12430.8040.5990.530.3880.329
12440.8040.5990.530.3880.329
13110.4520.3720.3380.2790.276
13120.4520.3720.3380.2790.276
13130.4520.3720.3380.2790.276
13140.4520.3720.3380.2790.276
13210.5060.410.3710.3010.292
13220.5060.410.3710.3010.292
13230.5060.410.3710.3010.292
13240.5060.410.3710.3010.292
13310.6140.4860.4370.3440.325
13320.6140.4860.4370.3440.325
13330.6140.4860.4370.3440.325
13340.6140.4860.4370.3440.325
13410.830.6380.5680.4320.391
13420.830.6380.5680.4320.391
13430.830.6380.5680.4320.391
13440.830.6380.5680.4320.391
14110.8410.6860.6190.5030.503
14120.8410.6860.6190.5030.503
14130.8410.6860.6190.5030.503
14140.8410.6860.6190.5030.503
14210.8470.690.6230.5060.505
14220.8470.690.6230.5060.505
14230.8470.690.6230.5060.505
14240.8470.690.6230.5060.505
14310.8590.6990.630.510.508
14320.8590.6990.630.510.508
14330.8590.6990.630.510.508
14340.8590.6990.630.510.508
14410.8830.7160.6440.520.516
14420.8830.7160.6440.520.516
14430.8830.7160.6440.520.516
14440.8830.7160.6440.520.516
21110.2710.2510.2430.2290.225
21120.2710.2510.2430.2290.225
21130.2710.2510.2430.2290.225
21140.2710.2510.2430.2290.225
21210.350.3060.2910.260.249
21220.350.3060.2910.260.249
21230.350.3060.2910.260.249
21240.350.3060.2910.260.249
21310.5060.4160.3860.3240.296
21320.5060.4160.3860.3240.296
21330.5060.4160.3860.3240.296
21340.5060.4160.3860.3240.296
21410.8180.6360.5770.4510.391
21420.8180.6360.5770.4510.391
21430.8180.6360.5770.4510.391
21440.8180.6360.5770.4510.391
22110.3560.3190.3040.2770.274
22120.3560.3190.3040.2770.274
22130.3560.3190.3040.2770.274
22140.3560.3190.3040.2770.274
22210.4240.3670.3460.3050.295
22220.4240.3670.3460.3050.295
22230.4240.3670.3460.3050.295
22240.4240.3670.3460.3050.295
22310.5590.4620.4280.360.336
22320.5590.4620.4280.360.336
22330.5590.4620.4280.360.336
22340.5590.4620.4280.360.336
22410.830.6530.5930.470.418
22420.830.6530.5930.470.418
22430.830.6530.5930.470.418
22440.830.6530.5930.470.418
23110.5250.4550.4260.3750.372
23120.5250.4550.4260.3750.372
23130.5250.4550.4260.3750.372
23140.5250.4550.4260.3750.372
23210.5720.4880.4550.3940.387
23220.5720.4880.4550.3940.387
23230.5720.4880.4550.3940.387
23240.5720.4880.4550.3940.387
23310.6650.5540.5120.4320.415
23320.6650.5540.5120.4320.415
23330.6650.5540.5120.4320.415
23340.6650.5540.5120.4320.415
23410.8530.6860.6260.5080.472
23420.8530.6860.6260.5080.472
23430.8530.6860.6260.5080.472
23440.8530.6860.6260.5080.472
24110.8620.7280.670.5690.569
24120.8620.7280.670.5690.569
24130.8620.7280.670.5690.569
24140.8620.7280.670.5690.569
24210.8680.7320.6730.5720.571
24220.8680.7320.6730.5720.571
24230.8680.7320.6730.5720.571
24240.8680.7320.6730.5720.571
24310.8780.7390.6790.5760.574
24320.8780.7390.6790.5760.574
24330.8780.7390.6790.5760.574
24340.8780.7390.6790.5760.574
24410.8990.7540.6920.5840.58
24420.8990.7540.6920.5840.58
24430.8990.7540.6920.5840.58
24440.8990.7540.6920.5840.58
31110.4960.4810.4760.4660.463
31120.4960.4810.4760.4660.463
31130.4960.4810.4760.4660.463
31140.4960.4810.4760.4660.463
31210.550.5190.5090.4880.48
31220.550.5190.5090.4880.48
31230.550.5190.5090.4880.48
31240.550.5190.5090.4880.48
31310.6580.5960.5750.5320.513
31320.6580.5960.5750.5320.513
31330.6580.5960.5750.5320.513
31340.6580.5960.5750.5320.513
31410.8740.7480.7070.620.579
31420.8740.7480.7070.620.579
31430.8740.7480.7070.620.579
31440.8740.7480.7070.620.579
32110.5540.5290.5180.50.497
32120.5540.5290.5180.50.497
32130.5540.5290.5180.50.497
32140.5540.5290.5180.50.497
32210.6010.5620.5470.5190.512
32220.6010.5620.5470.5190.512
32230.6010.5620.5470.5190.512
32240.6010.5620.5470.5190.512
32310.6950.6280.6040.5570.54
32320.6950.6280.6040.5570.54
32330.6950.6280.6040.5570.54
32340.6950.6280.6040.5570.54
32410.8820.7590.7180.6330.597
32420.8820.7590.7180.6330.597
32430.8820.7590.7180.6330.597
32440.8820.7590.7180.6330.597
33110.6710.6230.6030.5670.565
33120.6710.6230.6030.5670.565
33130.6710.6230.6030.5670.565
33140.6710.6230.6030.5670.565
33210.7030.6460.6220.580.575
33220.7030.6460.6220.580.575
33230.7030.6460.6220.580.575
33240.7030.6460.6220.580.575
33310.7680.6910.6620.6070.595
33320.7680.6910.6620.6070.595
33330.7680.6910.6620.6070.595
33340.7680.6910.6620.6070.595
33410.8980.7830.7410.6590.635
33420.8980.7830.7410.6590.635
33430.8980.7830.7410.6590.635
33440.8980.7830.7410.6590.635
34110.9050.8120.7710.7020.702
34120.9050.8120.7710.7020.702
34130.9050.8120.7710.7020.702
34140.9050.8120.7710.7020.702
34210.9080.8140.7740.7030.703
34220.9080.8140.7740.7030.703
34230.9080.8140.7740.7030.703
34240.9080.8140.7740.7030.703
34310.9160.8190.7780.7060.705
34320.9160.8190.7780.7060.705
34330.9160.8190.7780.7060.705
34340.9160.8190.7780.7060.705
34410.930.8290.7870.7120.709
34420.930.8290.7870.7120.709
34430.930.8290.7870.7120.709
34440.930.8290.7870.7120.709
41110.9440.9420.9420.9410.94
41120.9440.9420.9420.9410.94
41130.9440.9420.9420.9410.94
41140.9440.9420.9420.9410.94
41210.950.9470.9450.9430.942
41220.950.9470.9450.9430.942
41230.950.9470.9450.9430.942
41240.950.9470.9450.9430.942
41310.9620.9550.9530.9480.946
41320.9620.9550.9530.9480.946
41330.9620.9550.9530.9480.946
41340.9620.9550.9530.9480.946
41410.9860.9720.9670.9580.953
41420.9860.9720.9670.9580.953
41430.9860.9720.9670.9580.953
41440.9860.9720.9670.9580.953
42110.950.9480.9460.9440.944
42120.950.9480.9460.9440.944
42130.950.9480.9460.9440.944
42140.950.9480.9460.9440.944
42210.9560.9510.950.9470.946
42220.9560.9510.950.9470.946
42230.9560.9510.950.9470.946
42240.9560.9510.950.9470.946
42310.9660.9590.9560.9510.949
42320.9660.9590.9560.9510.949
42330.9660.9590.9560.9510.949
42340.9660.9590.9560.9510.949
42410.9870.9730.9690.9590.955
42420.9870.9730.9690.9590.955
42430.9870.9730.9690.9590.955
42440.9870.9730.9690.9590.955
43110.9630.9580.9560.9520.952
43120.9630.9580.9560.9520.952
43130.9630.9580.9560.9520.952
43140.9630.9580.9560.9520.952
43210.9670.9610.9580.9530.953
43220.9670.9610.9580.9530.953
43230.9670.9610.9580.9530.953
43240.9670.9610.9580.9530.953
43310.9740.9660.9620.9560.955
43320.9740.9660.9620.9560.955
43330.9740.9660.9620.9560.955
43340.9740.9660.9620.9560.955
43410.9890.9760.9710.9620.959
43420.9890.9760.9710.9620.959
43430.9890.9760.9710.9620.959
43440.9890.9760.9710.9620.959
44110.9890.9790.9750.9670.967
44120.9890.9790.9750.9670.967
44130.9890.9790.9750.9670.967
44140.9890.9790.9750.9670.967
44210.990.9790.9750.9670.967
44220.990.9790.9750.9670.967
44230.990.9790.9750.9670.967
44240.990.9790.9750.9670.967
44310.9910.980.9750.9670.967
44320.9910.980.9750.9670.967
44330.9910.980.9750.9670.967
44340.9910.980.9750.9670.967
44410.9920.9810.9760.9680.968
44420.9920.9810.9760.9680.968
44430.9920.9810.9760.9680.968
44440.9920.9810.9760.9680.968