,'

Interpretability models​

Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ?

Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations.

Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ?

This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11].

1. The main idea

The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ?

In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance:

The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction:

To achieve that, most of approaches iterate between 2 steps:

1. Set a prohibition to some part of the input
2. Observe the change in the output (fitted answer)

Repeat step 1 and step 2 for different prohibitions of the input.

2. Existing approaches

The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability.

2.1. Cooperative game theory based

Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium.

A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution.

2.2. Architecture specific: Deep Neural Network

Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model.

For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject.

Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models:

Random Forest:

Deep Neural Network:

2.3. A unified approach

The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph.

Cooperative Game theory-based:

An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model.

3.1. The SHAP model

Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that:

where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows.

3.2. The Natural properties

(1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained:

g(x') = f(x)

(2) Missingness: put the output to 0 corresponds to turning the feature off:

x'i = 0 ⇒ Φi = 0

(3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one.

Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if:

fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i)

then for all input z' ∈ {0,1}M :

Φi (f 1, x) ≥ Φi (f 2, x)

3.3 Computing SHAP values

3.3.1. Back to the Shapley values

The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values.

In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below:

where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]).

3.3.2. The SHAP values

In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that:

where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...).

4. Practical example with SHAP library

Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers).

I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap .

By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0.

The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%.

Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below:

This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset.

For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score.

We can apply this explainer model to all correct predicted examples in the test set, as below:

The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature.

It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample):

In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score.

The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure.

Conclusion

There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context.

The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap.

Bibliography

[0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953).

[1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001).

[2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011).

[3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/

[4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014).

[5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015).

[6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016).

[7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017).

[8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018).

[9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017).

[10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017).

[11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017).

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

], ['\\\$$','\\\$$']]}\n });\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;script type=\"text\/javascript\" async src=\"\/\/cdn.mathjax.org\/mathjax\/latest\/MathJax.js?config=TeX-AMS_CHTML\"\u0026gt;\u0026lt;\/script\u0026gt;\n\n\n\u0026lt;script\u0026gt;\n!function(){\/*\n\n Copyright (C) 2013 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n Copyright (C) 2006 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n*\/\n(function(){function ba(g){function k(){try{M.doScroll(\"left\")}catch(g){t.setTimeout(k,50);return}z(\"poll\")}function z(k){if(\"readystatechange\"!=k.type||\"complete\"==A.readyState)(\"load\"==k.type?t:A)[B](p+k.type,z,!1),!q\u0026amp;\u0026amp;(q=!0)\u0026amp;\u0026amp;g.call(t,k.type||k)}var Y=A.addEventListener,q=!1,C=!0,x=Y?\"addEventListener\":\"attachEvent\",B=Y?\"removeEventListener\":\"detachEvent\",p=Y?\"\":\"on\";if(\"complete\"==A.readyState)g.call(t,\"lazy\");else{if(A.createEventObject\u0026amp;\u0026amp;M.doScroll){try{C=!t.frameElement}catch(da){}C\u0026amp;\u0026amp;k()}A[x](p+\n\"DOMContentLoaded\",z,!1);A[x](p+\"readystatechange\",z,!1);t[x](p+\"load\",z,!1)}}function U(){V\u0026amp;\u0026amp;ba(function(){var g=N.length;ca(g?function(){for(var k=0;k\u0026lt;g;++k)(function(g){t.setTimeout(function(){t.exports[N[g]].apply(t,arguments)},0)})(k)}:void 0)})}for(var t=window,A=document,M=A.documentElement,O=A.head||A.getElementsByTagName(\"head\")[0]||M,B=\"\",F=A.getElementsByTagName(\"script\"),q=F.length;0\u0026lt;=--q;){var P=F[q],Z=P.src.match(\/^[^?#]*\\\/run_prettify\\.js(\\?[^#]*)?(?:#.*)?\/);if(Z){B=Z[1]||\"\";P.parentNode.removeChild(P);\nbreak}}var V=!0,H=[],Q=[],N=[];B.replace(\/[?\u0026amp;]([^\u0026amp;=]+)=([^\u0026amp;]+)\/g,function(g,k,z){z=decodeURIComponent(z);k=decodeURIComponent(k);\"autorun\"==k?V=!\/^[0fn]\/i.test(z):\"lang\"==k?H.push(z):\"skin\"==k?Q.push(z):\"callback\"==k\u0026amp;\u0026amp;N.push(z)});q=0;for(B=H.length;q\u0026lt;B;++q)(function(){var g=A.createElement(\"script\");g.onload=g.onerror=g.onreadystatechange=function(){!g||g.readyState\u0026amp;\u0026amp;!\/loaded|complete\/.test(g.readyState)||(g.onerror=g.onload=g.onreadystatechange=null,--T,T||t.setTimeout(U,0),g.parentNode\u0026amp;\u0026amp;g.parentNode.removeChild(g),\ng=null)};g.type=\"text\/javascript\";g.src=\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/lang-\"+encodeURIComponent(H[q])+\".js\";O.insertBefore(g,O.firstChild)})(H[q]);for(var T=H.length,F=[],q=0,B=Q.length;q\u0026lt;B;++q)F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/skins\/\"+encodeURIComponent(Q[q])+\".css\");F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/prettify.css\");(function(g){function k(q){if(q!==z){var t=A.createElement(\"link\");t.rel=\"stylesheet\";t.type=\n\"text\/css\";q+1\u0026lt;z\u0026amp;\u0026amp;(t.error=t.onerror=function(){k(q+1)});t.href=g[q];O.appendChild(t)}}var z=g.length;k(0)})(F);var ca=function(){window.PR_SHOULD_USE_CONTINUATION=!0;var g;(function(){function k(a){function d(e){var b=e.charCodeAt(0);if(92!==b)return b;var a=e.charAt(1);return(b=W[a])?b:\"0\"\u0026lt;=a\u0026amp;\u0026amp;\"7\"\u0026gt;=a?parseInt(e.substring(1),8):\"u\"===a||\"x\"===a?parseInt(e.substring(2),16):e.charCodeAt(1)}function f(e){if(32\u0026gt;e)return(16\u0026gt;e?\"\\\\x0\":\"\\\\x\")+e.toString(16);e=String.fromCharCode(e);return\"\\\\\"===e||\"-\"===\ne||\"]\"===e||\"^\"===e?\"\\\\\"+e:e}function b(e){var b=e.substring(1,e.length-1).match(\/\\\\u[0-9A-Fa-f]{4}|\\\\x[0-9A-Fa-f]{2}|\\\$0-3][0-7]{0,2}|\\\\[0-7]{1,2}|\\\\[\\s\\S]|-|[^-\\\$\/g);e=[];var a=\"^\"===b[0],c=[\"[\"];a\u0026amp;\u0026amp;c.push(\"^\");for(var a=a?1:0,h=b.length;a\u0026lt;h;++a){var l=b[a];if(\/\\\$bdsw]\/i.test(l))c.push(l);else{var l=d(l),n;a+2\u0026lt;h\u0026amp;\u0026amp;\"-\"===b[a+1]?(n=d(b[a+2]),a+=2):n=l;e.push([l,n]);65\u0026gt;n||122\u0026lt;l||(65\u0026gt;n||90\u0026lt;l||e.push([Math.max(65,l)|32,Math.min(n,90)|32]),97\u0026gt;n||122\u0026lt;l||e.push([Math.max(97,l)\u0026amp;-33,Math.min(n,122)\u0026amp;-33]))}}e.sort(function(e,\na){return e[0]-a[0]||a[1]-e[1]});b=[];h=[];for(a=0;a\u0026lt;e.length;++a)l=e[a],l[0]\u0026lt;=h[1]+1?h[1]=Math.max(h[1],l[1]):b.push(h=l);for(a=0;a\u0026lt;b.length;++a)l=b[a],c.push(f(l[0])),l[1]\u0026gt;l[0]\u0026amp;\u0026amp;(l[1]+1\u0026gt;l[0]\u0026amp;\u0026amp;c.push(\"-\"),c.push(f(l[1])));c.push(\"]\");return c.join(\"\")}function g(e){for(var a=e.source.match(\/(?:\\[(?:[^\\x5C\\x5D]|\\\\[\\s\\S])*\$|\\\\u[A-Fa-f0-9]{4}|\\\\x[A-Fa-f0-9]{2}|\\\$0-9]+|\\\\[^ux0-9]|\$$\\?[:!=]|[\\(\$$\\^]|[^\\x5B\\x5C\$$\$$\\^]+)\/g),c=a.length,d=[],h=0,l=0;h\u0026lt;c;++h){var n=a[h];\"(\"===n?++l:\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=\n+n.substring(1))\u0026amp;\u0026amp;(n\u0026lt;=l?d[n]=-1:a[h]=f(n))}for(h=1;h\u0026lt;d.length;++h)-1===d[h]\u0026amp;\u0026amp;(d[h]=++k);for(l=h=0;h\u0026lt;c;++h)n=a[h],\"(\"===n?(++l,d[l]||(a[h]=\"(?:\")):\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=+n.substring(1))\u0026amp;\u0026amp;n\u0026lt;=l\u0026amp;\u0026amp;(a[h]=\"\\\\\"+d[n]);for(h=0;h\u0026lt;c;++h)\"^\"===a[h]\u0026amp;\u0026amp;\"^\"!==a[h+1]\u0026amp;\u0026amp;(a[h]=\"\");if(e.ignoreCase\u0026amp;\u0026amp;I)for(h=0;h\u0026lt;c;++h)n=a[h],e=n.charAt(0),2\u0026lt;=n.length\u0026amp;\u0026amp;\"[\"===e?a[h]=b(n):\"\\\\\"!==e\u0026amp;\u0026amp;(a[h]=n.replace(\/[a-zA-Z]\/g,function(a){a=a.charCodeAt(0);return\"[\"+String.fromCharCode(a\u0026amp;-33,a|32)+\"]\"}));return a.join(\"\")}for(var k=0,I=!1,\nm=!1,J=0,c=a.length;J\u0026lt;c;++J){var r=a[J];if(r.ignoreCase)m=!0;else if(\/[a-z]\/i.test(r.source.replace(\/\\\\u[0-9a-f]{4}|\\\\x[0-9a-f]{2}|\\\\[^ux]\/gi,\"\"))){I=!0;m=!1;break}}for(var W={b:8,t:9,n:10,v:11,f:12,r:13},u=[],J=0,c=a.length;J\u0026lt;c;++J){r=a[J];if(r.global||r.multiline)throw Error(\"\"+r);u.push(\"(?:\"+g(r)+\")\")}return new RegExp(u.join(\"|\"),m?\"gi\":\"g\")}function q(a,d){function f(a){var c=a.nodeType;if(1==c){if(!b.test(a.className)){for(c=a.firstChild;c;c=c.nextSibling)f(c);c=a.nodeName.toLowerCase();if(\"br\"===\nc||\"li\"===c)g[m]=\"\\n\",I[m\u0026lt;\u0026lt;1]=k++,I[m++\u0026lt;\u0026lt;1|1]=a}}else if(3==c||4==c)c=a.nodeValue,c.length\u0026amp;\u0026amp;(c=d?c.replace(\/\\r\\n?\/g,\"\\n\"):c.replace(\/[ \\t\\r\\n]+\/g,\" \"),g[m]=c,I[m\u0026lt;\u0026lt;1]=k,k+=c.length,I[m++\u0026lt;\u0026lt;1|1]=a)}var b=\/(?:^|\\s)nocode(?:\\s|)\/,g=[],k=0,I=[],m=0;f(a);return{a:g.join(\"\").replace(\/\\n\/,\"\"),c:I}}function t(a,d,f,b,g){f\u0026amp;\u0026amp;(a={h:a,l:1,j:null,m:null,a:f,c:null,i:d,g:null},b(a),g.push.apply(g,a.g))}function A(a){for(var d=void 0,f=a.firstChild;f;f=f.nextSibling)var b=f.nodeType,d=1===b?d?a:f:3===b?T.test(f.nodeValue)?\na:d:d;return d===a?void 0:d}function C(a,d){function f(a){for(var m=a.i,k=a.h,c=[m,\"pln\"],r=0,W=a.a.match(g)||[],u={},e=0,q=W.length;e\u0026lt;q;++e){var D=W[e],w=u[D],h=void 0,l;if(\"string\"===typeof w)l=!1;else{var n=b[D.charAt(0)];if(n)h=D.match(n[1]),w=n[0];else{for(l=0;l\u0026lt;p;++l)if(n=d[l],h=D.match(n[1])){w=n[0];break}h||(w=\"pln\")}!(l=5\u0026lt;=w.length\u0026amp;\u0026amp;\"lang-\"===w.substring(0,5))||h\u0026amp;\u0026amp;\"string\"===typeof h[1]||(l=!1,w=\"src\");l||(u[D]=w)}n=r;r+=D.length;if(l){l=h[1];var E=D.indexOf(l),G=E+l.length;h[2]\u0026amp;\u0026amp;(G=D.length-\nh[2].length,E=G-l.length);w=w.substring(5);t(k,m+n,D.substring(0,E),f,c);t(k,m+n+E,l,F(w,l),c);t(k,m+n+G,D.substring(G),f,c)}else c.push(m+n,w)}a.g=c}var b={},g;(function(){for(var f=a.concat(d),m=[],p={},c=0,r=f.length;c\u0026lt;r;++c){var q=f[c],u=q[3];if(u)for(var e=u.length;0\u0026lt;=--e;)b[u.charAt(e)]=q;q=q[1];u=\"\"+q;p.hasOwnProperty(u)||(m.push(q),p[u]=null)}m.push(\/[\\0-\\uffff]\/);g=k(m)})();var p=d.length;return f}function x(a){var d=[],f=[];a.tripleQuotedStrings?d.push([\"str\",\/^(?:\\'\\'\\'(?:[^\\'\\\$|\\\$\\s\\S]|\\'{1,2}(?=[^\\']))*(?:\\'\\'\\'|)|\\\"\\\"\\\"(?:[^\\\"\\\$|\\\$\\s\\S]|\\\"{1,2}(?=[^\\\"]))*(?:\\\"\\\"\\\"|)|\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|))\/,\nnull,\"'\\\"\"]):a.multiLineStrings?d.push([\"str\",\/^(?:\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|)|\\(?:[^\\\\\\]|\\\\[\\s\\S])*(?:\\|))\/,null,\"'\\\"\"]):d.push([\"str\",\/^(?:\\'(?:[^\\\\\\'\\r\\n]|\\\\.)*(?:\\'|)|\\\"(?:[^\\\\\\\"\\r\\n]|\\\\.)*(?:\\\"|))\/,null,\"\\\"'\"]);a.verbatimStrings\u0026amp;\u0026amp;f.push([\"str\",\/^@\\\"(?:[^\\\"]|\\\"\\\")*(?:\\\"|)\/,null]);var b=a.hashComments;b\u0026amp;\u0026amp;(a.cStyleComments?(1\u0026lt;b?d.push([\"com\",\/^#(?:##(?:[^#]|#(?!##))*(?:###|)|.*)\/,null,\"#\"]):d.push([\"com\",\/^#(?:(?:define|e(?:l|nd)if|else|error|ifn?def|include|line|pragma|undef|warning)\\b|[^\\r\\n]*)\/,\nnull,\"#\"]),f.push([\"str\",\/^\u0026lt;(?:(?:(?:\\.\\.\\\/)*|\\\/?)(?:[\\w-]+(?:\\\/[\\w-]+)+)?[\\w-]+\\.h(?:h|pp|\\+\\+)?|[a-z]\\w*)\u0026gt;\/,null])):d.push([\"com\",\/^#[^\\r\\n]*\/,null,\"#\"]));a.cStyleComments\u0026amp;\u0026amp;(f.push([\"com\",\/^\\\/\\\/[^\\r\\n]*\/,null]),f.push([\"com\",\/^\\\/\\*[\\s\\S]*?(?:\\*\\\/|)\/,null]));if(b=a.regexLiterals){var g=(b=1\u0026lt;b?\"\":\"\\n\\r\")?\".\":\"[\\\\S\\\\s]\";f.push([\"lang-regex\",RegExp(\"^(?:^^\\\\.?|[+-]|[!=]=?=?|\\\\#|%=?|\u0026amp;\u0026amp;?=?|\\\|\\\\*=?|[+\\\\-]=|-\u0026gt;|\\\\\/=?|::?|\u0026lt;\u0026lt;?=?|\u0026gt;\u0026gt;?\u0026gt;?=?|,|;|\\\\?|@|\\\\[|~|{|\\\\^\\\\^?=?|\\\\|\\\\|?=?|break|case|continue|delete|do|else|finally|instanceof|return|throw|try|typeof)\\\\s*(\"+\n(\"\/(?=[^\/*\"+b+\"])(?:[^\/\\\\x5B\\\\x5C\"+b+\"]|\\\\x5C\"+g+\"|\\\\x5B(?:[^\\\\x5C\\\\x5D\"+b+\"]|\\\\x5C\"+g+\")*(?:\\\\x5D|))+\/\")+\")\")])}(b=a.types)\u0026amp;\u0026amp;f.push([\"typ\",b]);b=(\"\"+a.keywords).replace(\/^ | \/g,\"\");b.length\u0026amp;\u0026amp;f.push([\"kwd\",new RegExp(\"^(?:\"+b.replace(\/[\\s,]+\/g,\"|\")+\")\\\\b\"),null]);d.push([\"pln\",\/^\\s+\/,null,\" \\r\\n\\t\\u00a0\"]);b=\"^.[^\\\\s\\\\w.@'\\\"\/\\\\\\\$*\";a.regexLiterals\u0026amp;\u0026amp;(b+=\"(?!s*\/)\");f.push([\"lit\",\/^@[a-z_][a-z_@0-9]*\/i,null],[\"typ\",\/^(?:[@_]?[A-Z]+[a-z][A-Za-z_@0-9]*|\\w+_t\\b)\/,null],[\"pln\",\/^[a-z_][a-z_@0-9]*\/i,\nnull],[\"lit\",\/^(?:0x[a-f0-9]+|(?:\\d(?:_\\d+)*\\d*(?:\\.\\d*)?|\\.\\d\\+)(?:e[+\\-]?\\d+)?)[a-z]*\/i,null,\"0123456789\"],[\"pln\",\/^\\\\\s\\S]?\/,null],[\"pun\",new RegExp(b),null]);return C(d,f)}function B(a,d,f){function b(a){var c=a.nodeType;if(1==c\u0026amp;\u0026amp;!k.test(a.className))if(\"br\"===a.nodeName)g(a),a.parentNode\u0026amp;\u0026amp;a.parentNode.removeChild(a);else for(a=a.firstChild;a;a=a.nextSibling)b(a);else if((3==c||4==c)\u0026amp;\u0026amp;f){var d=a.nodeValue,p=d.match(q);p\u0026amp;\u0026amp;(c=d.substring(0,p.index),a.nodeValue=c,(d=d.substring(p.index+p[0].length))\u0026amp;\u0026amp;\na.parentNode.insertBefore(m.createTextNode(d),a.nextSibling),g(a),c||a.parentNode.removeChild(a))}}function g(a){function b(a,c){var d=c?a.cloneNode(!1):a,n=a.parentNode;if(n){var n=b(n,1),e=a.nextSibling;n.appendChild(d);for(var f=e;f;f=e)e=f.nextSibling,n.appendChild(f)}return d}for(;!a.nextSibling;)if(a=a.parentNode,!a)return;a=b(a.nextSibling,0);for(var d;(d=a.parentNode)\u0026amp;\u0026amp;1===d.nodeType;)a=d;c.push(a)}for(var k=\/(?:^|\\s)nocode(?:\\s|)\/,q=\/\\r\\n?|\\n\/,m=a.ownerDocument,p=m.createElement(\"li\");a.firstChild;)p.appendChild(a.firstChild);\nfor(var c=[p],r=0;r\u0026lt;c.length;++r)b(c[r]);d===(d|0)\u0026amp;\u0026amp;c[0].setAttribute(\"value\",d);var t=m.createElement(\"ol\");t.className=\"linenums\";d=Math.max(0,d-1|0)||0;for(var r=0,u=c.length;r\u0026lt;u;++r)p=c[r],p.className=\"L\"+(r+d)%10,p.firstChild||p.appendChild(m.createTextNode(\"\\u00a0\")),t.appendChild(p);a.appendChild(t)}function p(a,d){for(var f=d.length;0\u0026lt;=--f;){var b=d[f];X.hasOwnProperty(b)?R.console\u0026amp;\u0026amp;console.warn(\"cannot override language handler %s\",b):X[b]=a}}function F(a,d){a\u0026amp;\u0026amp;X.hasOwnProperty(a)||(a=\/^\\s*\u0026lt;\/.test(d)?\n\"default-markup\":\"default-code\");return X[a]}function H(a){var d=a.j;try{var f=q(a.h,a.l),b=f.a;a.a=b;a.c=f.c;a.i=0;F(d,b)(a);var g=\/\\bMSIE\\s(\\d+)\/.exec(navigator.userAgent),g=g\u0026amp;\u0026amp;8\u0026gt;=+g[1],d=\/\\n\/g,p=a.a,k=p.length,f=0,m=a.c,t=m.length,b=0,c=a.g,r=c.length,x=0;c[r]=k;var u,e;for(e=u=0;e\u0026lt;r;)c[e]!==c[e+2]?(c[u++]=c[e++],c[u++]=c[e++]):e+=2;r=u;for(e=u=0;e\u0026lt;r;){for(var A=c[e],D=c[e+1],w=e+2;w+2\u0026lt;=r\u0026amp;\u0026amp;c[w+1]===D;)w+=2;c[u++]=A;c[u++]=D;e=w}c.length=u;var h=a.h;a=\"\";h\u0026amp;\u0026amp;(a=h.style.display,h.style.display=\"none\");\ntry{for(;b\u0026lt;t;){var l=m[b+2]||k,n=c[x+2]||k,w=Math.min(l,n),E=m[b+1],G;if(1!==E.nodeType\u0026amp;\u0026amp;(G=p.substring(f,w))){g\u0026amp;\u0026amp;(G=G.replace(d,\"\\r\"));E.nodeValue=G;var aa=E.ownerDocument,v=aa.createElement(\"span\");v.className=c[x+1];var B=E.parentNode;B.replaceChild(v,E);v.appendChild(E);f\u0026lt;l\u0026amp;\u0026amp;(m[b+1]=E=aa.createTextNode(p.substring(w,l)),B.insertBefore(E,v.nextSibling))}f=w;f\u0026gt;=l\u0026amp;\u0026amp;(b+=2);f\u0026gt;=n\u0026amp;\u0026amp;(x+=2)}}finally{h\u0026amp;\u0026amp;(h.style.display=a)}}catch(y){R.console\u0026amp;\u0026amp;console.log(y\u0026amp;\u0026amp;y.stack||y)}}var R=window,K=[\"break,continue,do,else,for,if,return,while\"],\nL=[[K,\"auto,case,char,const,default,double,enum,extern,float,goto,inline,int,long,register,short,signed,sizeof,static,struct,switch,typedef,union,unsigned,void,volatile\"],\"catch,class,delete,false,import,new,operator,private,protected,public,this,throw,true,try,typeof\"],S=[L,\"alignof,align_union,asm,axiom,bool,concept,concept_map,const_cast,constexpr,decltype,delegate,dynamic_cast,explicit,export,friend,generic,late_check,mutable,namespace,nullptr,property,reinterpret_cast,static_assert,static_cast,template,typeid,typename,using,virtual,where\"],\nM=[L,\"abstract,assert,boolean,byte,extends,finally,final,implements,import,instanceof,interface,null,native,package,strictfp,super,synchronized,throws,transient\"],N=[L,\"abstract,as,base,bool,by,byte,checked,decimal,delegate,descending,dynamic,event,finally,fixed,foreach,from,group,implicit,in,interface,internal,into,is,let,lock,null,object,out,override,orderby,params,partial,readonly,ref,sbyte,sealed,stackalloc,string,select,uint,ulong,unchecked,unsafe,ushort,var,virtual,where\"],L=[L,\"debugger,eval,export,function,get,instanceof,null,set,undefined,var,with,Infinity,NaN\"],\nO=[K,\"and,as,assert,class,def,del,elif,except,exec,finally,from,global,import,in,is,lambda,nonlocal,not,or,pass,print,raise,try,with,yield,False,True,None\"],P=[K,\"alias,and,begin,case,class,def,defined,elsif,end,ensure,false,in,module,next,nil,not,or,redo,rescue,retry,self,super,then,true,undef,unless,until,when,yield,BEGIN,END\"],K=[K,\"case,done,elif,esac,eval,fi,function,in,local,set,then,until\"],Q=\/^(DIR|FILE|vector|(de|priority_)?queue|list|stack|(const_)?iterator|(multi)?(set|map)|bitset|u?(int|float)\\d*)\\b\/,\nT=\/\\S\/,U=x({keywords:[S,N,M,L,\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",O,P,K],hashComments:!0,cStyleComments:!0,multiLineStrings:!0,regexLiterals:!0}),X={};p(U,[\"default-code\"]);p(C([],[[\"pln\",\/^[^\u0026lt;?]+\/],[\"dec\",\/^\u0026lt;!\\w[^\u0026gt;]*(?:\u0026gt;|)\/],[\"com\",\/^\u0026lt;\\!--[\\s\\S]*?(?:-\\-\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;\\?([\\s\\S]+?)(?:\\?\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;%([\\s\\S]+?)(?:%\u0026gt;|)\/],[\"pun\",\/^(?:\u0026lt;[%?]|[%?]\u0026gt;)\/],[\"lang-\",\n\/^\u0026lt;xmp\\b[^\u0026gt;]*\u0026gt;([\\s\\S]+?)\u0026lt;\\\/xmp\\b[^\u0026gt;]*\u0026gt;\/i],[\"lang-js\",\/^\u0026lt;script\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/script\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-css\",\/^\u0026lt;style\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/style\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-in.tag\",\/^(\u0026lt;\\\/?[a-z][^\u0026lt;\u0026gt;]*\u0026gt;)\/i]]),\"default-markup htm html mxml xhtml xml xsl\".split(\" \"));p(C([[\"pln\",\/^[\\s]+\/,null,\" \\t\\r\\n\"],[\"atv\",\/^(?:\\\"[^\\\"]*\\\"?|\\'[^\\']*\\'?)\/,null,\"\\\"'\"]],[[\"tag\",\/^^\u0026lt;\\\/?[a-z](?:[\\w.:-]*\\w)?|\\\/?\u0026gt;\/i],[\"atn\",\/^(?!style[\\s=]|on)[a-z](?:[\\w:-]*\\w)?\/i],[\"lang-uq.val\",\/^=\\s*([^\u0026gt;\\'\\\"\\s]*(?:[^\u0026gt;\\'\\\"\\s\\\/]|\\\/(?=\\s)))\/],\n[\"pun\",\/^[=\u0026lt;\u0026gt;\\\/]+\/],[\"lang-js\",\/^on\\w+\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i],[\"lang-css\",\/^style\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-css\",\/^style\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-css\",\/^style\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i]]),[\"in.tag\"]);p(C([],[[\"atv\",\/^[\\s\\S]+\/]]),[\"uq.val\"]);p(x({keywords:S,hashComments:!0,cStyleComments:!0,types:Q}),\"c cc cpp cxx cyc m\".split(\" \"));p(x({keywords:\"null,true,false\"}),[\"json\"]);p(x({keywords:N,hashComments:!0,cStyleComments:!0,\nverbatimStrings:!0,types:Q}),[\"cs\"]);p(x({keywords:M,cStyleComments:!0}),[\"java\"]);p(x({keywords:K,hashComments:!0,multiLineStrings:!0}),[\"bash\",\"bsh\",\"csh\",\"sh\"]);p(x({keywords:O,hashComments:!0,multiLineStrings:!0,tripleQuotedStrings:!0}),[\"cv\",\"py\",\"python\"]);p(x({keywords:\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",hashComments:!0,multiLineStrings:!0,regexLiterals:2}),\n[\"perl\",\"pl\",\"pm\"]);p(x({keywords:P,hashComments:!0,multiLineStrings:!0,regexLiterals:!0}),[\"rb\",\"ruby\"]);p(x({keywords:L,cStyleComments:!0,regexLiterals:!0}),[\"javascript\",\"js\"]);p(x({keywords:\"all,and,by,catch,class,else,extends,false,finally,for,if,in,is,isnt,loop,new,no,not,null,of,off,on,or,return,super,then,throw,true,try,unless,until,when,while,yes\",hashComments:3,cStyleComments:!0,multilineStrings:!0,tripleQuotedStrings:!0,regexLiterals:!0}),[\"coffee\"]);p(C([],[[\"str\",\/^[\\s\\S]+\/]]),[\"regex\"]);\nvar V=R.PR={createSimpleLexer:C,registerLangHandler:p,sourceDecorator:x,PR_ATTRIB_NAME:\"atn\",PR_ATTRIB_VALUE:\"atv\",PR_COMMENT:\"com\",PR_DECLARATION:\"dec\",PR_KEYWORD:\"kwd\",PR_LITERAL:\"lit\",PR_NOCODE:\"nocode\",PR_PLAIN:\"pln\",PR_PUNCTUATION:\"pun\",PR_SOURCE:\"src\",PR_STRING:\"str\",PR_TAG:\"tag\",PR_TYPE:\"typ\",prettyPrintOne:function(a,d,f){f=f||!1;d=d||null;var b=document.createElement(\"div\");b.innerHTML=\"\u0026lt;pre\u0026gt;\"+a+\"\u0026lt;\/pre\u0026gt;\";b=b.firstChild;f\u0026amp;\u0026amp;B(b,f,!0);H({j:d,m:f,h:b,l:1,a:null,i:null,c:null,g:null});return b.innerHTML},\nprettyPrint:g=g=function(a,d){function f(){for(var b=R.PR_SHOULD_USE_CONTINUATION?c.now()+250:Infinity;r\u0026lt;p.length\u0026amp;\u0026amp;c.now()\u0026lt;b;r++){for(var d=p[r],k=h,q=d;q=q.previousSibling;){var m=q.nodeType,v=(7===m||8===m)\u0026amp;\u0026amp;q.nodeValue;if(v?!\/^\\??prettify\\b\/.test(v):3!==m||\/\\S\/.test(q.nodeValue))break;if(v){k={};v.replace(\/\\b(\\w+)=([\\w:.%+-]+)\/g,function(a,b,c){k[b]=c});break}}q=d.className;if((k!==h||u.test(q))\u0026amp;\u0026amp;!e.test(q)){m=!1;for(v=d.parentNode;v;v=v.parentNode)if(w.test(v.tagName)\u0026amp;\u0026amp;v.className\u0026amp;\u0026amp;u.test(v.className)){m=\n!0;break}if(!m){d.className+=\" prettyprinted\";m=k.lang;if(!m){var m=q.match(t),C;!m\u0026amp;\u0026amp;(C=A(d))\u0026amp;\u0026amp;z.test(C.tagName)\u0026amp;\u0026amp;(m=C.className.match(t));m\u0026amp;\u0026amp;(m=m[1])}if(x.test(d.tagName))v=1;else var v=d.currentStyle,y=g.defaultView,v=(v=v?v.whiteSpace:y\u0026amp;\u0026amp;y.getComputedStyle?y.getComputedStyle(d,null).getPropertyValue(\"white-space\"):0)\u0026amp;\u0026amp;\"pre\"===v.substring(0,3);y=k.linenums;(y=\"true\"===y||+y)||(y=(y=q.match(\/\\blinenums\\b(?::(\\d+))?\/))?y[1]\u0026amp;\u0026amp;y[1].length?+y[1]:!0:!1);y\u0026amp;\u0026amp;B(d,y,v);H({j:m,h:d,m:y,l:v,a:null,i:null,c:null,\ng:null})}}}r\u0026lt;p.length?R.setTimeout(f,250):\"function\"===typeof a\u0026amp;\u0026amp;a()}for(var b=d||document.body,g=b.ownerDocument||document,b=[b.getElementsByTagName(\"pre\"),b.getElementsByTagName(\"code\"),b.getElementsByTagName(\"xmp\")],p=[],k=0;k\u0026lt;b.length;++k)for(var m=0,q=b[k].length;m\u0026lt;q;++m)p.push(b[k][m]);var b=null,c=Date;c.now||(c={now:function(){return+new Date}});var r=0,t=\/\\blang(?:uage)?-([\\w.]+)(?!\\S)\/,u=\/\\bprettyprint\\b\/,e=\/\\bprettyprinted\\b\/,x=\/pre|xmp\/i,z=\/^code\/i,w=\/^(?:pre|code|xmp)\/i,h={};f()}},\nS=R.define;\"function\"===typeof S\u0026amp;\u0026amp;S.amd\u0026amp;\u0026amp;S(\"google-code-prettify\",[],function(){return V})})();return g}();T||t.setTimeout(U,0)})();}()\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;div\u0026gt;\n\u0026lt;pre class=\"prettyprint\"\u0026gt;\n\n# import dataset\n\n\u0026gt;\u0026gt;\u0026gt; import pandas as pd\n\u0026gt;\u0026gt;\u0026gt; data = pd.read_csv('heart.csv')\n\u0026gt;\u0026gt;\u0026gt; data.describe()\n\n# split the label and explaining data\n\n\u0026gt;\u0026gt;\u0026gt; y = data['target']\n\u0026gt;\u0026gt;\u0026gt; cols = ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']\n\u0026gt;\u0026gt;\u0026gt; X = data[cols]\n\n# Split the dataset into train and test sets\n\u0026gt;\u0026gt;\u0026gt; from sklearn.model_selection import train_test_split\n\u0026gt;\u0026gt;\u0026gt; from sklearn import preprocessing\n\n\u0026gt;\u0026gt;\u0026gt; X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.25, random_state=40) # for reproducibility results\n\n# scaling the data\n\u0026gt;\u0026gt;\u0026gt; scaler = preprocessing.StandardScaler().fit(X_train)\n\u0026gt;\u0026gt;\u0026gt; X_train_sc = scaler.transform(X_train)\n\u0026gt;\u0026gt;\u0026gt; X_test_sc = scaler.transform(X_test)\n\n# Learn a classifier\n\u0026gt;\u0026gt;\u0026gt; from sklearn.svm import SVC\n\u0026gt;\u0026gt;\u0026gt; from sklearn.metrics import accuracy_score\n\n\n# classification step\n\u0026gt;\u0026gt;\u0026gt; classifiers = SVC(kernel='linear', probability=True)\n\u0026gt;\u0026gt;\u0026gt; classifiers.fit(X_train_sc,y_train)\n\n# prediction:\n\u0026gt;\u0026gt;\u0026gt; pred = classifiers.predict(X_test_sc)\n\u0026gt;\u0026gt;\u0026gt; print(accuracy_score(y_test,pred)) \n\nclassification accuracy on the test set: 0.9078947368421053\n\n\u0026lt;\/pre\u0026gt;\n\u0026lt;\/div\u0026gt;\n\u0026lt;hr\u0026gt;","render_as_iframe":true,"selected_app_name":"HtmlApp","app_list":"{\"HtmlApp\":556065}"}},{"type":"Blog.Section","id":"f_bce38bd8-3b43-4aba-9216-02ded616842f","defaultValue":null,"component":{"type":"RichText","id":"f_b357bdaa-8c84-4eeb-9b65-6f5af8e591ec","defaultValue":false,"value":"\u003cp style=\"text-align: justify;\"\u003eOnce the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (\u003ca target=\"_blank\" href=\"https:\/\/github.com\/slundberg\/shap\"\u003ehttps:\/\/github.com\/slundberg\/shap\u003c\/a\u003e) before running the code below:\u00a0\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_28edd409-fe05-42a9-b61b-a9005db21a58","defaultValue":null,"component":{"type":"HtmlComponent","id":1583642,"defaultValue":false,"value":"\u0026lt;script type=\"text\/x-mathjax-config\"\u0026gt;\n MathJax.Hub.Config({\n tex2jax: {inlineMath: [[' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ,' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ], ['\\\\(','\\\']]}\n });\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;script type=\"text\/javascript\" async src=\"\/\/cdn.mathjax.org\/mathjax\/latest\/MathJax.js?config=TeX-AMS_CHTML\"\u0026gt;\u0026lt;\/script\u0026gt;\n\n\n\u0026lt;script\u0026gt;\n!function(){\/*\n\n Copyright (C) 2013 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n Copyright (C) 2006 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n*\/\n(function(){function ba(g){function k(){try{M.doScroll(\"left\")}catch(g){t.setTimeout(k,50);return}z(\"poll\")}function z(k){if(\"readystatechange\"!=k.type||\"complete\"==A.readyState)(\"load\"==k.type?t:A)[B](p+k.type,z,!1),!q\u0026amp;\u0026amp;(q=!0)\u0026amp;\u0026amp;g.call(t,k.type||k)}var Y=A.addEventListener,q=!1,C=!0,x=Y?\"addEventListener\":\"attachEvent\",B=Y?\"removeEventListener\":\"detachEvent\",p=Y?\"\":\"on\";if(\"complete\"==A.readyState)g.call(t,\"lazy\");else{if(A.createEventObject\u0026amp;\u0026amp;M.doScroll){try{C=!t.frameElement}catch(da){}C\u0026amp;\u0026amp;k()}A[x](p+\n\"DOMContentLoaded\",z,!1);A[x](p+\"readystatechange\",z,!1);t[x](p+\"load\",z,!1)}}function U(){V\u0026amp;\u0026amp;ba(function(){var g=N.length;ca(g?function(){for(var k=0;k\u0026lt;g;++k)(function(g){t.setTimeout(function(){t.exports[N[g]].apply(t,arguments)},0)})(k)}:void 0)})}for(var t=window,A=document,M=A.documentElement,O=A.head||A.getElementsByTagName(\"head\")[0]||M,B=\"\",F=A.getElementsByTagName(\"script\"),q=F.length;0\u0026lt;=--q;){var P=F[q],Z=P.src.match(\/^[^?#]*\\\/run_prettify\\.js(\\?[^#]*)?(?:#.*)?\/);if(Z){B=Z[1]||\"\";P.parentNode.removeChild(P);\nbreak}}var V=!0,H=[],Q=[],N=[];B.replace(\/[?\u0026amp;]([^\u0026amp;=]+)=([^\u0026amp;]+)\/g,function(g,k,z){z=decodeURIComponent(z);k=decodeURIComponent(k);\"autorun\"==k?V=!\/^[0fn]\/i.test(z):\"lang\"==k?H.push(z):\"skin\"==k?Q.push(z):\"callback\"==k\u0026amp;\u0026amp;N.push(z)});q=0;for(B=H.length;q\u0026lt;B;++q)(function(){var g=A.createElement(\"script\");g.onload=g.onerror=g.onreadystatechange=function(){!g||g.readyState\u0026amp;\u0026amp;!\/loaded|complete\/.test(g.readyState)||(g.onerror=g.onload=g.onreadystatechange=null,--T,T||t.setTimeout(U,0),g.parentNode\u0026amp;\u0026amp;g.parentNode.removeChild(g),\ng=null)};g.type=\"text\/javascript\";g.src=\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/lang-\"+encodeURIComponent(H[q])+\".js\";O.insertBefore(g,O.firstChild)})(H[q]);for(var T=H.length,F=[],q=0,B=Q.length;q\u0026lt;B;++q)F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/skins\/\"+encodeURIComponent(Q[q])+\".css\");F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/prettify.css\");(function(g){function k(q){if(q!==z){var t=A.createElement(\"link\");t.rel=\"stylesheet\";t.type=\n\"text\/css\";q+1\u0026lt;z\u0026amp;\u0026amp;(t.error=t.onerror=function(){k(q+1)});t.href=g[q];O.appendChild(t)}}var z=g.length;k(0)})(F);var ca=function(){window.PR_SHOULD_USE_CONTINUATION=!0;var g;(function(){function k(a){function d(e){var b=e.charCodeAt(0);if(92!==b)return b;var a=e.charAt(1);return(b=W[a])?b:\"0\"\u0026lt;=a\u0026amp;\u0026amp;\"7\"\u0026gt;=a?parseInt(e.substring(1),8):\"u\"===a||\"x\"===a?parseInt(e.substring(2),16):e.charCodeAt(1)}function f(e){if(32\u0026gt;e)return(16\u0026gt;e?\"\\\\x0\":\"\\\\x\")+e.toString(16);e=String.fromCharCode(e);return\"\\\\\"===e||\"-\"===\ne||\"]\"===e||\"^\"===e?\"\\\\\"+e:e}function b(e){var b=e.substring(1,e.length-1).match(\/\\\\u[0-9A-Fa-f]{4}|\\\\x[0-9A-Fa-f]{2}|\\\\[0-3][0-7]{0,2}|\\\\[0-7]{1,2}|\\\\[\\s\\S]|-|[^-\\\\/g);e=[];var a=\"^\"===b[0],c=[\"[\"];a\u0026amp;\u0026amp;c.push(\"^\");for(var a=a?1:0,h=b.length;a\u0026lt;h;++a){var l=b[a];if(\/\\\$bdsw]\/i.test(l))c.push(l);else{var l=d(l),n;a+2\u0026lt;h\u0026amp;\u0026amp;\"-\"===b[a+1]?(n=d(b[a+2]),a+=2):n=l;e.push([l,n]);65\u0026gt;n||122\u0026lt;l||(65\u0026gt;n||90\u0026lt;l||e.push([Math.max(65,l)|32,Math.min(n,90)|32]),97\u0026gt;n||122\u0026lt;l||e.push([Math.max(97,l)\u0026amp;-33,Math.min(n,122)\u0026amp;-33]))}}e.sort(function(e,\na){return e[0]-a[0]||a[1]-e[1]});b=[];h=[];for(a=0;a\u0026lt;e.length;++a)l=e[a],l[0]\u0026lt;=h[1]+1?h[1]=Math.max(h[1],l[1]):b.push(h=l);for(a=0;a\u0026lt;b.length;++a)l=b[a],c.push(f(l[0])),l[1]\u0026gt;l[0]\u0026amp;\u0026amp;(l[1]+1\u0026gt;l[0]\u0026amp;\u0026amp;c.push(\"-\"),c.push(f(l[1])));c.push(\"]\");return c.join(\"\")}function g(e){for(var a=e.source.match(\/(?:\\[(?:[^\\x5C\\x5D]|\\\\[\\s\\S])*\$|\\\\u[A-Fa-f0-9]{4}|\\\\x[A-Fa-f0-9]{2}|\\\$0-9]+|\\\\[^ux0-9]|\$$\\?[:!=]|[\\(\$$\\^]|[^\\x5B\\x5C\$$\$$\\^]+)\/g),c=a.length,d=[],h=0,l=0;h\u0026lt;c;++h){var n=a[h];\"(\"===n?++l:\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=\n+n.substring(1))\u0026amp;\u0026amp;(n\u0026lt;=l?d[n]=-1:a[h]=f(n))}for(h=1;h\u0026lt;d.length;++h)-1===d[h]\u0026amp;\u0026amp;(d[h]=++k);for(l=h=0;h\u0026lt;c;++h)n=a[h],\"(\"===n?(++l,d[l]||(a[h]=\"(?:\")):\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=+n.substring(1))\u0026amp;\u0026amp;n\u0026lt;=l\u0026amp;\u0026amp;(a[h]=\"\\\\\"+d[n]);for(h=0;h\u0026lt;c;++h)\"^\"===a[h]\u0026amp;\u0026amp;\"^\"!==a[h+1]\u0026amp;\u0026amp;(a[h]=\"\");if(e.ignoreCase\u0026amp;\u0026amp;I)for(h=0;h\u0026lt;c;++h)n=a[h],e=n.charAt(0),2\u0026lt;=n.length\u0026amp;\u0026amp;\"[\"===e?a[h]=b(n):\"\\\\\"!==e\u0026amp;\u0026amp;(a[h]=n.replace(\/[a-zA-Z]\/g,function(a){a=a.charCodeAt(0);return\"[\"+String.fromCharCode(a\u0026amp;-33,a|32)+\"]\"}));return a.join(\"\")}for(var k=0,I=!1,\nm=!1,J=0,c=a.length;J\u0026lt;c;++J){var r=a[J];if(r.ignoreCase)m=!0;else if(\/[a-z]\/i.test(r.source.replace(\/\\\\u[0-9a-f]{4}|\\\\x[0-9a-f]{2}|\\\\[^ux]\/gi,\"\"))){I=!0;m=!1;break}}for(var W={b:8,t:9,n:10,v:11,f:12,r:13},u=[],J=0,c=a.length;J\u0026lt;c;++J){r=a[J];if(r.global||r.multiline)throw Error(\"\"+r);u.push(\"(?:\"+g(r)+\")\")}return new RegExp(u.join(\"|\"),m?\"gi\":\"g\")}function q(a,d){function f(a){var c=a.nodeType;if(1==c){if(!b.test(a.className)){for(c=a.firstChild;c;c=c.nextSibling)f(c);c=a.nodeName.toLowerCase();if(\"br\"===\nc||\"li\"===c)g[m]=\"\\n\",I[m\u0026lt;\u0026lt;1]=k++,I[m++\u0026lt;\u0026lt;1|1]=a}}else if(3==c||4==c)c=a.nodeValue,c.length\u0026amp;\u0026amp;(c=d?c.replace(\/\\r\\n?\/g,\"\\n\"):c.replace(\/[ \\t\\r\\n]+\/g,\" \"),g[m]=c,I[m\u0026lt;\u0026lt;1]=k,k+=c.length,I[m++\u0026lt;\u0026lt;1|1]=a)}var b=\/(?:^|\\s)nocode(?:\\s|)\/,g=[],k=0,I=[],m=0;f(a);return{a:g.join(\"\").replace(\/\\n\/,\"\"),c:I}}function t(a,d,f,b,g){f\u0026amp;\u0026amp;(a={h:a,l:1,j:null,m:null,a:f,c:null,i:d,g:null},b(a),g.push.apply(g,a.g))}function A(a){for(var d=void 0,f=a.firstChild;f;f=f.nextSibling)var b=f.nodeType,d=1===b?d?a:f:3===b?T.test(f.nodeValue)?\na:d:d;return d===a?void 0:d}function C(a,d){function f(a){for(var m=a.i,k=a.h,c=[m,\"pln\"],r=0,W=a.a.match(g)||[],u={},e=0,q=W.length;e\u0026lt;q;++e){var D=W[e],w=u[D],h=void 0,l;if(\"string\"===typeof w)l=!1;else{var n=b[D.charAt(0)];if(n)h=D.match(n[1]),w=n[0];else{for(l=0;l\u0026lt;p;++l)if(n=d[l],h=D.match(n[1])){w=n[0];break}h||(w=\"pln\")}!(l=5\u0026lt;=w.length\u0026amp;\u0026amp;\"lang-\"===w.substring(0,5))||h\u0026amp;\u0026amp;\"string\"===typeof h[1]||(l=!1,w=\"src\");l||(u[D]=w)}n=r;r+=D.length;if(l){l=h[1];var E=D.indexOf(l),G=E+l.length;h[2]\u0026amp;\u0026amp;(G=D.length-\nh[2].length,E=G-l.length);w=w.substring(5);t(k,m+n,D.substring(0,E),f,c);t(k,m+n+E,l,F(w,l),c);t(k,m+n+G,D.substring(G),f,c)}else c.push(m+n,w)}a.g=c}var b={},g;(function(){for(var f=a.concat(d),m=[],p={},c=0,r=f.length;c\u0026lt;r;++c){var q=f[c],u=q[3];if(u)for(var e=u.length;0\u0026lt;=--e;)b[u.charAt(e)]=q;q=q[1];u=\"\"+q;p.hasOwnProperty(u)||(m.push(q),p[u]=null)}m.push(\/[\\0-\\uffff]\/);g=k(m)})();var p=d.length;return f}function x(a){var d=[],f=[];a.tripleQuotedStrings?d.push([\"str\",\/^(?:\\'\\'\\'(?:[^\\'\\\$|\\\$\\s\\S]|\\'{1,2}(?=[^\\']))*(?:\\'\\'\\'|)|\\\"\\\"\\\"(?:[^\\\"\\\$|\\\$\\s\\S]|\\\"{1,2}(?=[^\\\"]))*(?:\\\"\\\"\\\"|)|\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|))\/,\nnull,\"'\\\"\"]):a.multiLineStrings?d.push([\"str\",\/^(?:\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|)|\\(?:[^\\\\\\]|\\\\[\\s\\S])*(?:\\|))\/,null,\"'\\\"\"]):d.push([\"str\",\/^(?:\\'(?:[^\\\\\\'\\r\\n]|\\\\.)*(?:\\'|)|\\\"(?:[^\\\\\\\"\\r\\n]|\\\\.)*(?:\\\"|))\/,null,\"\\\"'\"]);a.verbatimStrings\u0026amp;\u0026amp;f.push([\"str\",\/^@\\\"(?:[^\\\"]|\\\"\\\")*(?:\\\"|)\/,null]);var b=a.hashComments;b\u0026amp;\u0026amp;(a.cStyleComments?(1\u0026lt;b?d.push([\"com\",\/^#(?:##(?:[^#]|#(?!##))*(?:###|)|.*)\/,null,\"#\"]):d.push([\"com\",\/^#(?:(?:define|e(?:l|nd)if|else|error|ifn?def|include|line|pragma|undef|warning)\\b|[^\\r\\n]*)\/,\nnull,\"#\"]),f.push([\"str\",\/^\u0026lt;(?:(?:(?:\\.\\.\\\/)*|\\\/?)(?:[\\w-]+(?:\\\/[\\w-]+)+)?[\\w-]+\\.h(?:h|pp|\\+\\+)?|[a-z]\\w*)\u0026gt;\/,null])):d.push([\"com\",\/^#[^\\r\\n]*\/,null,\"#\"]));a.cStyleComments\u0026amp;\u0026amp;(f.push([\"com\",\/^\\\/\\\/[^\\r\\n]*\/,null]),f.push([\"com\",\/^\\\/\\*[\\s\\S]*?(?:\\*\\\/|)\/,null]));if(b=a.regexLiterals){var g=(b=1\u0026lt;b?\"\":\"\\n\\r\")?\".\":\"[\\\\S\\\\s]\";f.push([\"lang-regex\",RegExp(\"^(?:^^\\\\.?|[+-]|[!=]=?=?|\\\\#|%=?|\u0026amp;\u0026amp;?=?|\\\|\\\\*=?|[+\\\\-]=|-\u0026gt;|\\\\\/=?|::?|\u0026lt;\u0026lt;?=?|\u0026gt;\u0026gt;?\u0026gt;?=?|,|;|\\\\?|@|\\\\[|~|{|\\\\^\\\\^?=?|\\\\|\\\\|?=?|break|case|continue|delete|do|else|finally|instanceof|return|throw|try|typeof)\\\\s*(\"+\n(\"\/(?=[^\/*\"+b+\"])(?:[^\/\\\\x5B\\\\x5C\"+b+\"]|\\\\x5C\"+g+\"|\\\\x5B(?:[^\\\\x5C\\\\x5D\"+b+\"]|\\\\x5C\"+g+\")*(?:\\\\x5D|))+\/\")+\")\")])}(b=a.types)\u0026amp;\u0026amp;f.push([\"typ\",b]);b=(\"\"+a.keywords).replace(\/^ | \/g,\"\");b.length\u0026amp;\u0026amp;f.push([\"kwd\",new RegExp(\"^(?:\"+b.replace(\/[\\s,]+\/g,\"|\")+\")\\\\b\"),null]);d.push([\"pln\",\/^\\s+\/,null,\" \\r\\n\\t\\u00a0\"]);b=\"^.[^\\\\s\\\\w.@'\\\"\/\\\\\\\$*\";a.regexLiterals\u0026amp;\u0026amp;(b+=\"(?!s*\/)\");f.push([\"lit\",\/^@[a-z_][a-z_@0-9]*\/i,null],[\"typ\",\/^(?:[@_]?[A-Z]+[a-z][A-Za-z_@0-9]*|\\w+_t\\b)\/,null],[\"pln\",\/^[a-z_][a-z_@0-9]*\/i,\nnull],[\"lit\",\/^(?:0x[a-f0-9]+|(?:\\d(?:_\\d+)*\\d*(?:\\.\\d*)?|\\.\\d\\+)(?:e[+\\-]?\\d+)?)[a-z]*\/i,null,\"0123456789\"],[\"pln\",\/^\\\\\s\\S]?\/,null],[\"pun\",new RegExp(b),null]);return C(d,f)}function B(a,d,f){function b(a){var c=a.nodeType;if(1==c\u0026amp;\u0026amp;!k.test(a.className))if(\"br\"===a.nodeName)g(a),a.parentNode\u0026amp;\u0026amp;a.parentNode.removeChild(a);else for(a=a.firstChild;a;a=a.nextSibling)b(a);else if((3==c||4==c)\u0026amp;\u0026amp;f){var d=a.nodeValue,p=d.match(q);p\u0026amp;\u0026amp;(c=d.substring(0,p.index),a.nodeValue=c,(d=d.substring(p.index+p[0].length))\u0026amp;\u0026amp;\na.parentNode.insertBefore(m.createTextNode(d),a.nextSibling),g(a),c||a.parentNode.removeChild(a))}}function g(a){function b(a,c){var d=c?a.cloneNode(!1):a,n=a.parentNode;if(n){var n=b(n,1),e=a.nextSibling;n.appendChild(d);for(var f=e;f;f=e)e=f.nextSibling,n.appendChild(f)}return d}for(;!a.nextSibling;)if(a=a.parentNode,!a)return;a=b(a.nextSibling,0);for(var d;(d=a.parentNode)\u0026amp;\u0026amp;1===d.nodeType;)a=d;c.push(a)}for(var k=\/(?:^|\\s)nocode(?:\\s|)\/,q=\/\\r\\n?|\\n\/,m=a.ownerDocument,p=m.createElement(\"li\");a.firstChild;)p.appendChild(a.firstChild);\nfor(var c=[p],r=0;r\u0026lt;c.length;++r)b(c[r]);d===(d|0)\u0026amp;\u0026amp;c[0].setAttribute(\"value\",d);var t=m.createElement(\"ol\");t.className=\"linenums\";d=Math.max(0,d-1|0)||0;for(var r=0,u=c.length;r\u0026lt;u;++r)p=c[r],p.className=\"L\"+(r+d)%10,p.firstChild||p.appendChild(m.createTextNode(\"\\u00a0\")),t.appendChild(p);a.appendChild(t)}function p(a,d){for(var f=d.length;0\u0026lt;=--f;){var b=d[f];X.hasOwnProperty(b)?R.console\u0026amp;\u0026amp;console.warn(\"cannot override language handler %s\",b):X[b]=a}}function F(a,d){a\u0026amp;\u0026amp;X.hasOwnProperty(a)||(a=\/^\\s*\u0026lt;\/.test(d)?\n\"default-markup\":\"default-code\");return X[a]}function H(a){var d=a.j;try{var f=q(a.h,a.l),b=f.a;a.a=b;a.c=f.c;a.i=0;F(d,b)(a);var g=\/\\bMSIE\\s(\\d+)\/.exec(navigator.userAgent),g=g\u0026amp;\u0026amp;8\u0026gt;=+g[1],d=\/\\n\/g,p=a.a,k=p.length,f=0,m=a.c,t=m.length,b=0,c=a.g,r=c.length,x=0;c[r]=k;var u,e;for(e=u=0;e\u0026lt;r;)c[e]!==c[e+2]?(c[u++]=c[e++],c[u++]=c[e++]):e+=2;r=u;for(e=u=0;e\u0026lt;r;){for(var A=c[e],D=c[e+1],w=e+2;w+2\u0026lt;=r\u0026amp;\u0026amp;c[w+1]===D;)w+=2;c[u++]=A;c[u++]=D;e=w}c.length=u;var h=a.h;a=\"\";h\u0026amp;\u0026amp;(a=h.style.display,h.style.display=\"none\");\ntry{for(;b\u0026lt;t;){var l=m[b+2]||k,n=c[x+2]||k,w=Math.min(l,n),E=m[b+1],G;if(1!==E.nodeType\u0026amp;\u0026amp;(G=p.substring(f,w))){g\u0026amp;\u0026amp;(G=G.replace(d,\"\\r\"));E.nodeValue=G;var aa=E.ownerDocument,v=aa.createElement(\"span\");v.className=c[x+1];var B=E.parentNode;B.replaceChild(v,E);v.appendChild(E);f\u0026lt;l\u0026amp;\u0026amp;(m[b+1]=E=aa.createTextNode(p.substring(w,l)),B.insertBefore(E,v.nextSibling))}f=w;f\u0026gt;=l\u0026amp;\u0026amp;(b+=2);f\u0026gt;=n\u0026amp;\u0026amp;(x+=2)}}finally{h\u0026amp;\u0026amp;(h.style.display=a)}}catch(y){R.console\u0026amp;\u0026amp;console.log(y\u0026amp;\u0026amp;y.stack||y)}}var R=window,K=[\"break,continue,do,else,for,if,return,while\"],\nL=[[K,\"auto,case,char,const,default,double,enum,extern,float,goto,inline,int,long,register,short,signed,sizeof,static,struct,switch,typedef,union,unsigned,void,volatile\"],\"catch,class,delete,false,import,new,operator,private,protected,public,this,throw,true,try,typeof\"],S=[L,\"alignof,align_union,asm,axiom,bool,concept,concept_map,const_cast,constexpr,decltype,delegate,dynamic_cast,explicit,export,friend,generic,late_check,mutable,namespace,nullptr,property,reinterpret_cast,static_assert,static_cast,template,typeid,typename,using,virtual,where\"],\nM=[L,\"abstract,assert,boolean,byte,extends,finally,final,implements,import,instanceof,interface,null,native,package,strictfp,super,synchronized,throws,transient\"],N=[L,\"abstract,as,base,bool,by,byte,checked,decimal,delegate,descending,dynamic,event,finally,fixed,foreach,from,group,implicit,in,interface,internal,into,is,let,lock,null,object,out,override,orderby,params,partial,readonly,ref,sbyte,sealed,stackalloc,string,select,uint,ulong,unchecked,unsafe,ushort,var,virtual,where\"],L=[L,\"debugger,eval,export,function,get,instanceof,null,set,undefined,var,with,Infinity,NaN\"],\nO=[K,\"and,as,assert,class,def,del,elif,except,exec,finally,from,global,import,in,is,lambda,nonlocal,not,or,pass,print,raise,try,with,yield,False,True,None\"],P=[K,\"alias,and,begin,case,class,def,defined,elsif,end,ensure,false,in,module,next,nil,not,or,redo,rescue,retry,self,super,then,true,undef,unless,until,when,yield,BEGIN,END\"],K=[K,\"case,done,elif,esac,eval,fi,function,in,local,set,then,until\"],Q=\/^(DIR|FILE|vector|(de|priority_)?queue|list|stack|(const_)?iterator|(multi)?(set|map)|bitset|u?(int|float)\\d*)\\b\/,\nT=\/\\S\/,U=x({keywords:[S,N,M,L,\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",O,P,K],hashComments:!0,cStyleComments:!0,multiLineStrings:!0,regexLiterals:!0}),X={};p(U,[\"default-code\"]);p(C([],[[\"pln\",\/^[^\u0026lt;?]+\/],[\"dec\",\/^\u0026lt;!\\w[^\u0026gt;]*(?:\u0026gt;|)\/],[\"com\",\/^\u0026lt;\\!--[\\s\\S]*?(?:-\\-\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;\\?([\\s\\S]+?)(?:\\?\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;%([\\s\\S]+?)(?:%\u0026gt;|)\/],[\"pun\",\/^(?:\u0026lt;[%?]|[%?]\u0026gt;)\/],[\"lang-\",\n\/^\u0026lt;xmp\\b[^\u0026gt;]*\u0026gt;([\\s\\S]+?)\u0026lt;\\\/xmp\\b[^\u0026gt;]*\u0026gt;\/i],[\"lang-js\",\/^\u0026lt;script\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/script\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-css\",\/^\u0026lt;style\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/style\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-in.tag\",\/^(\u0026lt;\\\/?[a-z][^\u0026lt;\u0026gt;]*\u0026gt;)\/i]]),\"default-markup htm html mxml xhtml xml xsl\".split(\" \"));p(C([[\"pln\",\/^[\\s]+\/,null,\" \\t\\r\\n\"],[\"atv\",\/^(?:\\\"[^\\\"]*\\\"?|\\'[^\\']*\\'?)\/,null,\"\\\"'\"]],[[\"tag\",\/^^\u0026lt;\\\/?[a-z](?:[\\w.:-]*\\w)?|\\\/?\u0026gt;\/i],[\"atn\",\/^(?!style[\\s=]|on)[a-z](?:[\\w:-]*\\w)?\/i],[\"lang-uq.val\",\/^=\\s*([^\u0026gt;\\'\\\"\\s]*(?:[^\u0026gt;\\'\\\"\\s\\\/]|\\\/(?=\\s)))\/],\n[\"pun\",\/^[=\u0026lt;\u0026gt;\\\/]+\/],[\"lang-js\",\/^on\\w+\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i],[\"lang-css\",\/^style\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-css\",\/^style\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-css\",\/^style\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i]]),[\"in.tag\"]);p(C([],[[\"atv\",\/^[\\s\\S]+\/]]),[\"uq.val\"]);p(x({keywords:S,hashComments:!0,cStyleComments:!0,types:Q}),\"c cc cpp cxx cyc m\".split(\" \"));p(x({keywords:\"null,true,false\"}),[\"json\"]);p(x({keywords:N,hashComments:!0,cStyleComments:!0,\nverbatimStrings:!0,types:Q}),[\"cs\"]);p(x({keywords:M,cStyleComments:!0}),[\"java\"]);p(x({keywords:K,hashComments:!0,multiLineStrings:!0}),[\"bash\",\"bsh\",\"csh\",\"sh\"]);p(x({keywords:O,hashComments:!0,multiLineStrings:!0,tripleQuotedStrings:!0}),[\"cv\",\"py\",\"python\"]);p(x({keywords:\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",hashComments:!0,multiLineStrings:!0,regexLiterals:2}),\n[\"perl\",\"pl\",\"pm\"]);p(x({keywords:P,hashComments:!0,multiLineStrings:!0,regexLiterals:!0}),[\"rb\",\"ruby\"]);p(x({keywords:L,cStyleComments:!0,regexLiterals:!0}),[\"javascript\",\"js\"]);p(x({keywords:\"all,and,by,catch,class,else,extends,false,finally,for,if,in,is,isnt,loop,new,no,not,null,of,off,on,or,return,super,then,throw,true,try,unless,until,when,while,yes\",hashComments:3,cStyleComments:!0,multilineStrings:!0,tripleQuotedStrings:!0,regexLiterals:!0}),[\"coffee\"]);p(C([],[[\"str\",\/^[\\s\\S]+\/]]),[\"regex\"]);\nvar V=R.PR={createSimpleLexer:C,registerLangHandler:p,sourceDecorator:x,PR_ATTRIB_NAME:\"atn\",PR_ATTRIB_VALUE:\"atv\",PR_COMMENT:\"com\",PR_DECLARATION:\"dec\",PR_KEYWORD:\"kwd\",PR_LITERAL:\"lit\",PR_NOCODE:\"nocode\",PR_PLAIN:\"pln\",PR_PUNCTUATION:\"pun\",PR_SOURCE:\"src\",PR_STRING:\"str\",PR_TAG:\"tag\",PR_TYPE:\"typ\",prettyPrintOne:function(a,d,f){f=f||!1;d=d||null;var b=document.createElement(\"div\");b.innerHTML=\"\u0026lt;pre\u0026gt;\"+a+\"\u0026lt;\/pre\u0026gt;\";b=b.firstChild;f\u0026amp;\u0026amp;B(b,f,!0);H({j:d,m:f,h:b,l:1,a:null,i:null,c:null,g:null});return b.innerHTML},\nprettyPrint:g=g=function(a,d){function f(){for(var b=R.PR_SHOULD_USE_CONTINUATION?c.now()+250:Infinity;r\u0026lt;p.length\u0026amp;\u0026amp;c.now()\u0026lt;b;r++){for(var d=p[r],k=h,q=d;q=q.previousSibling;){var m=q.nodeType,v=(7===m||8===m)\u0026amp;\u0026amp;q.nodeValue;if(v?!\/^\\??prettify\\b\/.test(v):3!==m||\/\\S\/.test(q.nodeValue))break;if(v){k={};v.replace(\/\\b(\\w+)=([\\w:.%+-]+)\/g,function(a,b,c){k[b]=c});break}}q=d.className;if((k!==h||u.test(q))\u0026amp;\u0026amp;!e.test(q)){m=!1;for(v=d.parentNode;v;v=v.parentNode)if(w.test(v.tagName)\u0026amp;\u0026amp;v.className\u0026amp;\u0026amp;u.test(v.className)){m=\n!0;break}if(!m){d.className+=\" prettyprinted\";m=k.lang;if(!m){var m=q.match(t),C;!m\u0026amp;\u0026amp;(C=A(d))\u0026amp;\u0026amp;z.test(C.tagName)\u0026amp;\u0026amp;(m=C.className.match(t));m\u0026amp;\u0026amp;(m=m[1])}if(x.test(d.tagName))v=1;else var v=d.currentStyle,y=g.defaultView,v=(v=v?v.whiteSpace:y\u0026amp;\u0026amp;y.getComputedStyle?y.getComputedStyle(d,null).getPropertyValue(\"white-space\"):0)\u0026amp;\u0026amp;\"pre\"===v.substring(0,3);y=k.linenums;(y=\"true\"===y||+y)||(y=(y=q.match(\/\\blinenums\\b(?::(\\d+))?\/))?y[1]\u0026amp;\u0026amp;y[1].length?+y[1]:!0:!1);y\u0026amp;\u0026amp;B(d,y,v);H({j:m,h:d,m:y,l:v,a:null,i:null,c:null,\ng:null})}}}r\u0026lt;p.length?R.setTimeout(f,250):\"function\"===typeof a\u0026amp;\u0026amp;a()}for(var b=d||document.body,g=b.ownerDocument||document,b=[b.getElementsByTagName(\"pre\"),b.getElementsByTagName(\"code\"),b.getElementsByTagName(\"xmp\")],p=[],k=0;k\u0026lt;b.length;++k)for(var m=0,q=b[k].length;m\u0026lt;q;++m)p.push(b[k][m]);var b=null,c=Date;c.now||(c={now:function(){return+new Date}});var r=0,t=\/\\blang(?:uage)?-([\\w.]+)(?!\\S)\/,u=\/\\bprettyprint\\b\/,e=\/\\bprettyprinted\\b\/,x=\/pre|xmp\/i,z=\/^code\/i,w=\/^(?:pre|code|xmp)\/i,h={};f()}},\nS=R.define;\"function\"===typeof S\u0026amp;\u0026amp;S.amd\u0026amp;\u0026amp;S(\"google-code-prettify\",[],function(){return V})})();return g}();T||t.setTimeout(U,0)})();}()\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;div\u0026gt;\n\u0026lt;pre class=\"prettyprint\"\u0026gt;\n\n\u0026gt;\u0026gt;\u0026gt; import shap\n\n# load JS visualization code to notebook\n\u0026gt;\u0026gt;\u0026gt; shap.initjs()\n\n# True Positive and False Positive for label 1:\n\u0026gt;\u0026gt;\u0026gt; TP = [i for i in range(len(pred)) if pred[i] == y_test.values[i] == 1] \n\u0026gt;\u0026gt;\u0026gt; FP = [i for i in range(len(pred)) if pred[i] != y_test.values[i] and y_test.values[i] == 0] \n\n# Explaining a True Positive example:\n\u0026gt;\u0026gt;\u0026gt; x = X_test_sc[TP[2],:]\n\u0026gt;\u0026gt;\u0026gt; dx = pd.DataFrame(x.reshape(1,-1))\n\u0026gt;\u0026gt;\u0026gt; dx.columns = X_train.columns\n\n\u0026gt;\u0026gt;\u0026gt; explainer = shap.KernelExplainer(classifiers.predict, X_train_sc)\n\u0026gt;\u0026gt;\u0026gt; shap_values = explainer.shap_values(dx, nsamples=100)\n\u0026gt;\u0026gt;\u0026gt; shap.force_plot(explainer.expected_value, shap_values,feature_names=X_train.columns)\n\n\u0026lt;\/pre\u0026gt;\n\u0026lt;\/div\u0026gt;\n\u0026lt;hr\u0026gt;","render_as_iframe":true,"selected_app_name":"HtmlApp","app_list":"{\"HtmlApp\":556066}"}},{"type":"Blog.Section","id":"f_292bee13-e820-41fe-978e-9b2b3fcee147","defaultValue":null,"component":{"type":"Image","id":"f_0a101465-6fe1-4bc5-9074-d9442ab3ebe9","defaultValue":null,"link_url":"","thumb_url":"!","url":"!","caption":"","description":"","storageKey":"1225579\/Screen_Shot_2019-02-14_at_10.28.20_AM_wius0q","storage":"c","storagePrefix":null,"format":"png","h":141,"w":1200,"s":15572,"new_target":true,"noCompression":null,"cropMode":null}},{"type":"Blog.Section","id":"f_faf8f992-28cd-489e-8a95-3f2a7091f48b","defaultValue":null,"component":{"type":"RichText","id":"f_665c8ff8-af4d-4641-b5d3-5c1a6d8c688f","defaultValue":false,"value":"\u003cp style=\"text-align: justify;\"\u003eThis figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on\u00a0the average model output on the training dataset.\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003eFor that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score.\u00a0\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003eWe can apply this explainer model to all correct predicted examples in the test set, as below:\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_c0957d46-aedb-4a67-b962-7baef3aba0dc","defaultValue":null,"component":{"type":"HtmlComponent","id":1583726,"defaultValue":false,"value":"\u0026lt;script type=\"text\/x-mathjax-config\"\u0026gt;\n MathJax.Hub.Config({\n tex2jax: {inlineMath: [[' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ,' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ], ['\\\\(','\\\']]}\n });\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;script type=\"text\/javascript\" async src=\"\/\/cdn.mathjax.org\/mathjax\/latest\/MathJax.js?config=TeX-AMS_CHTML\"\u0026gt;\u0026lt;\/script\u0026gt;\n\n\n\u0026lt;script\u0026gt;\n!function(){\/*\n\n Copyright (C) 2013 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n Copyright (C) 2006 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n*\/\n(function(){function ba(g){function k(){try{M.doScroll(\"left\")}catch(g){t.setTimeout(k,50);return}z(\"poll\")}function z(k){if(\"readystatechange\"!=k.type||\"complete\"==A.readyState)(\"load\"==k.type?t:A)[B](p+k.type,z,!1),!q\u0026amp;\u0026amp;(q=!0)\u0026amp;\u0026amp;g.call(t,k.type||k)}var Y=A.addEventListener,q=!1,C=!0,x=Y?\"addEventListener\":\"attachEvent\",B=Y?\"removeEventListener\":\"detachEvent\",p=Y?\"\":\"on\";if(\"complete\"==A.readyState)g.call(t,\"lazy\");else{if(A.createEventObject\u0026amp;\u0026amp;M.doScroll){try{C=!t.frameElement}catch(da){}C\u0026amp;\u0026amp;k()}A[x](p+\n\"DOMContentLoaded\",z,!1);A[x](p+\"readystatechange\",z,!1);t[x](p+\"load\",z,!1)}}function U(){V\u0026amp;\u0026amp;ba(function(){var g=N.length;ca(g?function(){for(var k=0;k\u0026lt;g;++k)(function(g){t.setTimeout(function(){t.exports[N[g]].apply(t,arguments)},0)})(k)}:void 0)})}for(var t=window,A=document,M=A.documentElement,O=A.head||A.getElementsByTagName(\"head\")[0]||M,B=\"\",F=A.getElementsByTagName(\"script\"),q=F.length;0\u0026lt;=--q;){var P=F[q],Z=P.src.match(\/^[^?#]*\\\/run_prettify\\.js(\\?[^#]*)?(?:#.*)?\/);if(Z){B=Z[1]||\"\";P.parentNode.removeChild(P);\nbreak}}var V=!0,H=[],Q=[],N=[];B.replace(\/[?\u0026amp;]([^\u0026amp;=]+)=([^\u0026amp;]+)\/g,function(g,k,z){z=decodeURIComponent(z);k=decodeURIComponent(k);\"autorun\"==k?V=!\/^[0fn]\/i.test(z):\"lang\"==k?H.push(z):\"skin\"==k?Q.push(z):\"callback\"==k\u0026amp;\u0026amp;N.push(z)});q=0;for(B=H.length;q\u0026lt;B;++q)(function(){var g=A.createElement(\"script\");g.onload=g.onerror=g.onreadystatechange=function(){!g||g.readyState\u0026amp;\u0026amp;!\/loaded|complete\/.test(g.readyState)||(g.onerror=g.onload=g.onreadystatechange=null,--T,T||t.setTimeout(U,0),g.parentNode\u0026amp;\u0026amp;g.parentNode.removeChild(g),\ng=null)};g.type=\"text\/javascript\";g.src=\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/lang-\"+encodeURIComponent(H[q])+\".js\";O.insertBefore(g,O.firstChild)})(H[q]);for(var T=H.length,F=[],q=0,B=Q.length;q\u0026lt;B;++q)F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/skins\/\"+encodeURIComponent(Q[q])+\".css\");F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/prettify.css\");(function(g){function k(q){if(q!==z){var t=A.createElement(\"link\");t.rel=\"stylesheet\";t.type=\n\"text\/css\";q+1\u0026lt;z\u0026amp;\u0026amp;(t.error=t.onerror=function(){k(q+1)});t.href=g[q];O.appendChild(t)}}var z=g.length;k(0)})(F);var ca=function(){window.PR_SHOULD_USE_CONTINUATION=!0;var g;(function(){function k(a){function d(e){var b=e.charCodeAt(0);if(92!==b)return b;var a=e.charAt(1);return(b=W[a])?b:\"0\"\u0026lt;=a\u0026amp;\u0026amp;\"7\"\u0026gt;=a?parseInt(e.substring(1),8):\"u\"===a||\"x\"===a?parseInt(e.substring(2),16):e.charCodeAt(1)}function f(e){if(32\u0026gt;e)return(16\u0026gt;e?\"\\\\x0\":\"\\\\x\")+e.toString(16);e=String.fromCharCode(e);return\"\\\\\"===e||\"-\"===\ne||\"]\"===e||\"^\"===e?\"\\\\\"+e:e}function b(e){var b=e.substring(1,e.length-1).match(\/\\\\u[0-9A-Fa-f]{4}|\\\\x[0-9A-Fa-f]{2}|\\\\[0-3][0-7]{0,2}|\\\\[0-7]{1,2}|\\\\[\\s\\S]|-|[^-\\\\/g);e=[];var a=\"^\"===b[0],c=[\"[\"];a\u0026amp;\u0026amp;c.push(\"^\");for(var a=a?1:0,h=b.length;a\u0026lt;h;++a){var l=b[a];if(\/\\\$bdsw]\/i.test(l))c.push(l);else{var l=d(l),n;a+2\u0026lt;h\u0026amp;\u0026amp;\"-\"===b[a+1]?(n=d(b[a+2]),a+=2):n=l;e.push([l,n]);65\u0026gt;n||122\u0026lt;l||(65\u0026gt;n||90\u0026lt;l||e.push([Math.max(65,l)|32,Math.min(n,90)|32]),97\u0026gt;n||122\u0026lt;l||e.push([Math.max(97,l)\u0026amp;-33,Math.min(n,122)\u0026amp;-33]))}}e.sort(function(e,\na){return e[0]-a[0]||a[1]-e[1]});b=[];h=[];for(a=0;a\u0026lt;e.length;++a)l=e[a],l[0]\u0026lt;=h[1]+1?h[1]=Math.max(h[1],l[1]):b.push(h=l);for(a=0;a\u0026lt;b.length;++a)l=b[a],c.push(f(l[0])),l[1]\u0026gt;l[0]\u0026amp;\u0026amp;(l[1]+1\u0026gt;l[0]\u0026amp;\u0026amp;c.push(\"-\"),c.push(f(l[1])));c.push(\"]\");return c.join(\"\")}function g(e){for(var a=e.source.match(\/(?:\\[(?:[^\\x5C\\x5D]|\\\\[\\s\\S])*\$|\\\\u[A-Fa-f0-9]{4}|\\\\x[A-Fa-f0-9]{2}|\\\$0-9]+|\\\\[^ux0-9]|\$$\\?[:!=]|[\\(\$$\\^]|[^\\x5B\\x5C\$$\$$\\^]+)\/g),c=a.length,d=[],h=0,l=0;h\u0026lt;c;++h){var n=a[h];\"(\"===n?++l:\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=\n+n.substring(1))\u0026amp;\u0026amp;(n\u0026lt;=l?d[n]=-1:a[h]=f(n))}for(h=1;h\u0026lt;d.length;++h)-1===d[h]\u0026amp;\u0026amp;(d[h]=++k);for(l=h=0;h\u0026lt;c;++h)n=a[h],\"(\"===n?(++l,d[l]||(a[h]=\"(?:\")):\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=+n.substring(1))\u0026amp;\u0026amp;n\u0026lt;=l\u0026amp;\u0026amp;(a[h]=\"\\\\\"+d[n]);for(h=0;h\u0026lt;c;++h)\"^\"===a[h]\u0026amp;\u0026amp;\"^\"!==a[h+1]\u0026amp;\u0026amp;(a[h]=\"\");if(e.ignoreCase\u0026amp;\u0026amp;I)for(h=0;h\u0026lt;c;++h)n=a[h],e=n.charAt(0),2\u0026lt;=n.length\u0026amp;\u0026amp;\"[\"===e?a[h]=b(n):\"\\\\\"!==e\u0026amp;\u0026amp;(a[h]=n.replace(\/[a-zA-Z]\/g,function(a){a=a.charCodeAt(0);return\"[\"+String.fromCharCode(a\u0026amp;-33,a|32)+\"]\"}));return a.join(\"\")}for(var k=0,I=!1,\nm=!1,J=0,c=a.length;J\u0026lt;c;++J){var r=a[J];if(r.ignoreCase)m=!0;else if(\/[a-z]\/i.test(r.source.replace(\/\\\\u[0-9a-f]{4}|\\\\x[0-9a-f]{2}|\\\\[^ux]\/gi,\"\"))){I=!0;m=!1;break}}for(var W={b:8,t:9,n:10,v:11,f:12,r:13},u=[],J=0,c=a.length;J\u0026lt;c;++J){r=a[J];if(r.global||r.multiline)throw Error(\"\"+r);u.push(\"(?:\"+g(r)+\")\")}return new RegExp(u.join(\"|\"),m?\"gi\":\"g\")}function q(a,d){function f(a){var c=a.nodeType;if(1==c){if(!b.test(a.className)){for(c=a.firstChild;c;c=c.nextSibling)f(c);c=a.nodeName.toLowerCase();if(\"br\"===\nc||\"li\"===c)g[m]=\"\\n\",I[m\u0026lt;\u0026lt;1]=k++,I[m++\u0026lt;\u0026lt;1|1]=a}}else if(3==c||4==c)c=a.nodeValue,c.length\u0026amp;\u0026amp;(c=d?c.replace(\/\\r\\n?\/g,\"\\n\"):c.replace(\/[ \\t\\r\\n]+\/g,\" \"),g[m]=c,I[m\u0026lt;\u0026lt;1]=k,k+=c.length,I[m++\u0026lt;\u0026lt;1|1]=a)}var b=\/(?:^|\\s)nocode(?:\\s|)\/,g=[],k=0,I=[],m=0;f(a);return{a:g.join(\"\").replace(\/\\n\/,\"\"),c:I}}function t(a,d,f,b,g){f\u0026amp;\u0026amp;(a={h:a,l:1,j:null,m:null,a:f,c:null,i:d,g:null},b(a),g.push.apply(g,a.g))}function A(a){for(var d=void 0,f=a.firstChild;f;f=f.nextSibling)var b=f.nodeType,d=1===b?d?a:f:3===b?T.test(f.nodeValue)?\na:d:d;return d===a?void 0:d}function C(a,d){function f(a){for(var m=a.i,k=a.h,c=[m,\"pln\"],r=0,W=a.a.match(g)||[],u={},e=0,q=W.length;e\u0026lt;q;++e){var D=W[e],w=u[D],h=void 0,l;if(\"string\"===typeof w)l=!1;else{var n=b[D.charAt(0)];if(n)h=D.match(n[1]),w=n[0];else{for(l=0;l\u0026lt;p;++l)if(n=d[l],h=D.match(n[1])){w=n[0];break}h||(w=\"pln\")}!(l=5\u0026lt;=w.length\u0026amp;\u0026amp;\"lang-\"===w.substring(0,5))||h\u0026amp;\u0026amp;\"string\"===typeof h[1]||(l=!1,w=\"src\");l||(u[D]=w)}n=r;r+=D.length;if(l){l=h[1];var E=D.indexOf(l),G=E+l.length;h[2]\u0026amp;\u0026amp;(G=D.length-\nh[2].length,E=G-l.length);w=w.substring(5);t(k,m+n,D.substring(0,E),f,c);t(k,m+n+E,l,F(w,l),c);t(k,m+n+G,D.substring(G),f,c)}else c.push(m+n,w)}a.g=c}var b={},g;(function(){for(var f=a.concat(d),m=[],p={},c=0,r=f.length;c\u0026lt;r;++c){var q=f[c],u=q[3];if(u)for(var e=u.length;0\u0026lt;=--e;)b[u.charAt(e)]=q;q=q[1];u=\"\"+q;p.hasOwnProperty(u)||(m.push(q),p[u]=null)}m.push(\/[\\0-\\uffff]\/);g=k(m)})();var p=d.length;return f}function x(a){var d=[],f=[];a.tripleQuotedStrings?d.push([\"str\",\/^(?:\\'\\'\\'(?:[^\\'\\\$|\\\$\\s\\S]|\\'{1,2}(?=[^\\']))*(?:\\'\\'\\'|)|\\\"\\\"\\\"(?:[^\\\"\\\$|\\\$\\s\\S]|\\\"{1,2}(?=[^\\\"]))*(?:\\\"\\\"\\\"|)|\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|))\/,\nnull,\"'\\\"\"]):a.multiLineStrings?d.push([\"str\",\/^(?:\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|)|\\(?:[^\\\\\\]|\\\\[\\s\\S])*(?:\\|))\/,null,\"'\\\"\"]):d.push([\"str\",\/^(?:\\'(?:[^\\\\\\'\\r\\n]|\\\\.)*(?:\\'|)|\\\"(?:[^\\\\\\\"\\r\\n]|\\\\.)*(?:\\\"|))\/,null,\"\\\"'\"]);a.verbatimStrings\u0026amp;\u0026amp;f.push([\"str\",\/^@\\\"(?:[^\\\"]|\\\"\\\")*(?:\\\"|)\/,null]);var b=a.hashComments;b\u0026amp;\u0026amp;(a.cStyleComments?(1\u0026lt;b?d.push([\"com\",\/^#(?:##(?:[^#]|#(?!##))*(?:###|)|.*)\/,null,\"#\"]):d.push([\"com\",\/^#(?:(?:define|e(?:l|nd)if|else|error|ifn?def|include|line|pragma|undef|warning)\\b|[^\\r\\n]*)\/,\nnull,\"#\"]),f.push([\"str\",\/^\u0026lt;(?:(?:(?:\\.\\.\\\/)*|\\\/?)(?:[\\w-]+(?:\\\/[\\w-]+)+)?[\\w-]+\\.h(?:h|pp|\\+\\+)?|[a-z]\\w*)\u0026gt;\/,null])):d.push([\"com\",\/^#[^\\r\\n]*\/,null,\"#\"]));a.cStyleComments\u0026amp;\u0026amp;(f.push([\"com\",\/^\\\/\\\/[^\\r\\n]*\/,null]),f.push([\"com\",\/^\\\/\\*[\\s\\S]*?(?:\\*\\\/|)\/,null]));if(b=a.regexLiterals){var g=(b=1\u0026lt;b?\"\":\"\\n\\r\")?\".\":\"[\\\\S\\\\s]\";f.push([\"lang-regex\",RegExp(\"^(?:^^\\\\.?|[+-]|[!=]=?=?|\\\\#|%=?|\u0026amp;\u0026amp;?=?|\\\|\\\\*=?|[+\\\\-]=|-\u0026gt;|\\\\\/=?|::?|\u0026lt;\u0026lt;?=?|\u0026gt;\u0026gt;?\u0026gt;?=?|,|;|\\\\?|@|\\\\[|~|{|\\\\^\\\\^?=?|\\\\|\\\\|?=?|break|case|continue|delete|do|else|finally|instanceof|return|throw|try|typeof)\\\\s*(\"+\n(\"\/(?=[^\/*\"+b+\"])(?:[^\/\\\\x5B\\\\x5C\"+b+\"]|\\\\x5C\"+g+\"|\\\\x5B(?:[^\\\\x5C\\\\x5D\"+b+\"]|\\\\x5C\"+g+\")*(?:\\\\x5D|))+\/\")+\")\")])}(b=a.types)\u0026amp;\u0026amp;f.push([\"typ\",b]);b=(\"\"+a.keywords).replace(\/^ | \/g,\"\");b.length\u0026amp;\u0026amp;f.push([\"kwd\",new RegExp(\"^(?:\"+b.replace(\/[\\s,]+\/g,\"|\")+\")\\\\b\"),null]);d.push([\"pln\",\/^\\s+\/,null,\" \\r\\n\\t\\u00a0\"]);b=\"^.[^\\\\s\\\\w.@'\\\"\/\\\\\\\$*\";a.regexLiterals\u0026amp;\u0026amp;(b+=\"(?!s*\/)\");f.push([\"lit\",\/^@[a-z_][a-z_@0-9]*\/i,null],[\"typ\",\/^(?:[@_]?[A-Z]+[a-z][A-Za-z_@0-9]*|\\w+_t\\b)\/,null],[\"pln\",\/^[a-z_][a-z_@0-9]*\/i,\nnull],[\"lit\",\/^(?:0x[a-f0-9]+|(?:\\d(?:_\\d+)*\\d*(?:\\.\\d*)?|\\.\\d\\+)(?:e[+\\-]?\\d+)?)[a-z]*\/i,null,\"0123456789\"],[\"pln\",\/^\\\\\s\\S]?\/,null],[\"pun\",new RegExp(b),null]);return C(d,f)}function B(a,d,f){function b(a){var c=a.nodeType;if(1==c\u0026amp;\u0026amp;!k.test(a.className))if(\"br\"===a.nodeName)g(a),a.parentNode\u0026amp;\u0026amp;a.parentNode.removeChild(a);else for(a=a.firstChild;a;a=a.nextSibling)b(a);else if((3==c||4==c)\u0026amp;\u0026amp;f){var d=a.nodeValue,p=d.match(q);p\u0026amp;\u0026amp;(c=d.substring(0,p.index),a.nodeValue=c,(d=d.substring(p.index+p[0].length))\u0026amp;\u0026amp;\na.parentNode.insertBefore(m.createTextNode(d),a.nextSibling),g(a),c||a.parentNode.removeChild(a))}}function g(a){function b(a,c){var d=c?a.cloneNode(!1):a,n=a.parentNode;if(n){var n=b(n,1),e=a.nextSibling;n.appendChild(d);for(var f=e;f;f=e)e=f.nextSibling,n.appendChild(f)}return d}for(;!a.nextSibling;)if(a=a.parentNode,!a)return;a=b(a.nextSibling,0);for(var d;(d=a.parentNode)\u0026amp;\u0026amp;1===d.nodeType;)a=d;c.push(a)}for(var k=\/(?:^|\\s)nocode(?:\\s|)\/,q=\/\\r\\n?|\\n\/,m=a.ownerDocument,p=m.createElement(\"li\");a.firstChild;)p.appendChild(a.firstChild);\nfor(var c=[p],r=0;r\u0026lt;c.length;++r)b(c[r]);d===(d|0)\u0026amp;\u0026amp;c[0].setAttribute(\"value\",d);var t=m.createElement(\"ol\");t.className=\"linenums\";d=Math.max(0,d-1|0)||0;for(var r=0,u=c.length;r\u0026lt;u;++r)p=c[r],p.className=\"L\"+(r+d)%10,p.firstChild||p.appendChild(m.createTextNode(\"\\u00a0\")),t.appendChild(p);a.appendChild(t)}function p(a,d){for(var f=d.length;0\u0026lt;=--f;){var b=d[f];X.hasOwnProperty(b)?R.console\u0026amp;\u0026amp;console.warn(\"cannot override language handler %s\",b):X[b]=a}}function F(a,d){a\u0026amp;\u0026amp;X.hasOwnProperty(a)||(a=\/^\\s*\u0026lt;\/.test(d)?\n\"default-markup\":\"default-code\");return X[a]}function H(a){var d=a.j;try{var f=q(a.h,a.l),b=f.a;a.a=b;a.c=f.c;a.i=0;F(d,b)(a);var g=\/\\bMSIE\\s(\\d+)\/.exec(navigator.userAgent),g=g\u0026amp;\u0026amp;8\u0026gt;=+g[1],d=\/\\n\/g,p=a.a,k=p.length,f=0,m=a.c,t=m.length,b=0,c=a.g,r=c.length,x=0;c[r]=k;var u,e;for(e=u=0;e\u0026lt;r;)c[e]!==c[e+2]?(c[u++]=c[e++],c[u++]=c[e++]):e+=2;r=u;for(e=u=0;e\u0026lt;r;){for(var A=c[e],D=c[e+1],w=e+2;w+2\u0026lt;=r\u0026amp;\u0026amp;c[w+1]===D;)w+=2;c[u++]=A;c[u++]=D;e=w}c.length=u;var h=a.h;a=\"\";h\u0026amp;\u0026amp;(a=h.style.display,h.style.display=\"none\");\ntry{for(;b\u0026lt;t;){var l=m[b+2]||k,n=c[x+2]||k,w=Math.min(l,n),E=m[b+1],G;if(1!==E.nodeType\u0026amp;\u0026amp;(G=p.substring(f,w))){g\u0026amp;\u0026amp;(G=G.replace(d,\"\\r\"));E.nodeValue=G;var aa=E.ownerDocument,v=aa.createElement(\"span\");v.className=c[x+1];var B=E.parentNode;B.replaceChild(v,E);v.appendChild(E);f\u0026lt;l\u0026amp;\u0026amp;(m[b+1]=E=aa.createTextNode(p.substring(w,l)),B.insertBefore(E,v.nextSibling))}f=w;f\u0026gt;=l\u0026amp;\u0026amp;(b+=2);f\u0026gt;=n\u0026amp;\u0026amp;(x+=2)}}finally{h\u0026amp;\u0026amp;(h.style.display=a)}}catch(y){R.console\u0026amp;\u0026amp;console.log(y\u0026amp;\u0026amp;y.stack||y)}}var R=window,K=[\"break,continue,do,else,for,if,return,while\"],\nL=[[K,\"auto,case,char,const,default,double,enum,extern,float,goto,inline,int,long,register,short,signed,sizeof,static,struct,switch,typedef,union,unsigned,void,volatile\"],\"catch,class,delete,false,import,new,operator,private,protected,public,this,throw,true,try,typeof\"],S=[L,\"alignof,align_union,asm,axiom,bool,concept,concept_map,const_cast,constexpr,decltype,delegate,dynamic_cast,explicit,export,friend,generic,late_check,mutable,namespace,nullptr,property,reinterpret_cast,static_assert,static_cast,template,typeid,typename,using,virtual,where\"],\nM=[L,\"abstract,assert,boolean,byte,extends,finally,final,implements,import,instanceof,interface,null,native,package,strictfp,super,synchronized,throws,transient\"],N=[L,\"abstract,as,base,bool,by,byte,checked,decimal,delegate,descending,dynamic,event,finally,fixed,foreach,from,group,implicit,in,interface,internal,into,is,let,lock,null,object,out,override,orderby,params,partial,readonly,ref,sbyte,sealed,stackalloc,string,select,uint,ulong,unchecked,unsafe,ushort,var,virtual,where\"],L=[L,\"debugger,eval,export,function,get,instanceof,null,set,undefined,var,with,Infinity,NaN\"],\nO=[K,\"and,as,assert,class,def,del,elif,except,exec,finally,from,global,import,in,is,lambda,nonlocal,not,or,pass,print,raise,try,with,yield,False,True,None\"],P=[K,\"alias,and,begin,case,class,def,defined,elsif,end,ensure,false,in,module,next,nil,not,or,redo,rescue,retry,self,super,then,true,undef,unless,until,when,yield,BEGIN,END\"],K=[K,\"case,done,elif,esac,eval,fi,function,in,local,set,then,until\"],Q=\/^(DIR|FILE|vector|(de|priority_)?queue|list|stack|(const_)?iterator|(multi)?(set|map)|bitset|u?(int|float)\\d*)\\b\/,\nT=\/\\S\/,U=x({keywords:[S,N,M,L,\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",O,P,K],hashComments:!0,cStyleComments:!0,multiLineStrings:!0,regexLiterals:!0}),X={};p(U,[\"default-code\"]);p(C([],[[\"pln\",\/^[^\u0026lt;?]+\/],[\"dec\",\/^\u0026lt;!\\w[^\u0026gt;]*(?:\u0026gt;|)\/],[\"com\",\/^\u0026lt;\\!--[\\s\\S]*?(?:-\\-\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;\\?([\\s\\S]+?)(?:\\?\u0026gt;|)\/],[\"lang-\",\/^\u0026lt;%([\\s\\S]+?)(?:%\u0026gt;|)\/],[\"pun\",\/^(?:\u0026lt;[%?]|[%?]\u0026gt;)\/],[\"lang-\",\n\/^\u0026lt;xmp\\b[^\u0026gt;]*\u0026gt;([\\s\\S]+?)\u0026lt;\\\/xmp\\b[^\u0026gt;]*\u0026gt;\/i],[\"lang-js\",\/^\u0026lt;script\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/script\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-css\",\/^\u0026lt;style\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/style\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-in.tag\",\/^(\u0026lt;\\\/?[a-z][^\u0026lt;\u0026gt;]*\u0026gt;)\/i]]),\"default-markup htm html mxml xhtml xml xsl\".split(\" \"));p(C([[\"pln\",\/^[\\s]+\/,null,\" \\t\\r\\n\"],[\"atv\",\/^(?:\\\"[^\\\"]*\\\"?|\\'[^\\']*\\'?)\/,null,\"\\\"'\"]],[[\"tag\",\/^^\u0026lt;\\\/?[a-z](?:[\\w.:-]*\\w)?|\\\/?\u0026gt;\/i],[\"atn\",\/^(?!style[\\s=]|on)[a-z](?:[\\w:-]*\\w)?\/i],[\"lang-uq.val\",\/^=\\s*([^\u0026gt;\\'\\\"\\s]*(?:[^\u0026gt;\\'\\\"\\s\\\/]|\\\/(?=\\s)))\/],\n[\"pun\",\/^[=\u0026lt;\u0026gt;\\\/]+\/],[\"lang-js\",\/^on\\w+\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i],[\"lang-css\",\/^style\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-css\",\/^style\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-css\",\/^style\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i]]),[\"in.tag\"]);p(C([],[[\"atv\",\/^[\\s\\S]+\/]]),[\"uq.val\"]);p(x({keywords:S,hashComments:!0,cStyleComments:!0,types:Q}),\"c cc cpp cxx cyc m\".split(\" \"));p(x({keywords:\"null,true,false\"}),[\"json\"]);p(x({keywords:N,hashComments:!0,cStyleComments:!0,\nverbatimStrings:!0,types:Q}),[\"cs\"]);p(x({keywords:M,cStyleComments:!0}),[\"java\"]);p(x({keywords:K,hashComments:!0,multiLineStrings:!0}),[\"bash\",\"bsh\",\"csh\",\"sh\"]);p(x({keywords:O,hashComments:!0,multiLineStrings:!0,tripleQuotedStrings:!0}),[\"cv\",\"py\",\"python\"]);p(x({keywords:\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",hashComments:!0,multiLineStrings:!0,regexLiterals:2}),\n[\"perl\",\"pl\",\"pm\"]);p(x({keywords:P,hashComments:!0,multiLineStrings:!0,regexLiterals:!0}),[\"rb\",\"ruby\"]);p(x({keywords:L,cStyleComments:!0,regexLiterals:!0}),[\"javascript\",\"js\"]);p(x({keywords:\"all,and,by,catch,class,else,extends,false,finally,for,if,in,is,isnt,loop,new,no,not,null,of,off,on,or,return,super,then,throw,true,try,unless,until,when,while,yes\",hashComments:3,cStyleComments:!0,multilineStrings:!0,tripleQuotedStrings:!0,regexLiterals:!0}),[\"coffee\"]);p(C([],[[\"str\",\/^[\\s\\S]+\/]]),[\"regex\"]);\nvar V=R.PR={createSimpleLexer:C,registerLangHandler:p,sourceDecorator:x,PR_ATTRIB_NAME:\"atn\",PR_ATTRIB_VALUE:\"atv\",PR_COMMENT:\"com\",PR_DECLARATION:\"dec\",PR_KEYWORD:\"kwd\",PR_LITERAL:\"lit\",PR_NOCODE:\"nocode\",PR_PLAIN:\"pln\",PR_PUNCTUATION:\"pun\",PR_SOURCE:\"src\",PR_STRING:\"str\",PR_TAG:\"tag\",PR_TYPE:\"typ\",prettyPrintOne:function(a,d,f){f=f||!1;d=d||null;var b=document.createElement(\"div\");b.innerHTML=\"\u0026lt;pre\u0026gt;\"+a+\"\u0026lt;\/pre\u0026gt;\";b=b.firstChild;f\u0026amp;\u0026amp;B(b,f,!0);H({j:d,m:f,h:b,l:1,a:null,i:null,c:null,g:null});return b.innerHTML},\nprettyPrint:g=g=function(a,d){function f(){for(var b=R.PR_SHOULD_USE_CONTINUATION?c.now()+250:Infinity;r\u0026lt;p.length\u0026amp;\u0026amp;c.now()\u0026lt;b;r++){for(var d=p[r],k=h,q=d;q=q.previousSibling;){var m=q.nodeType,v=(7===m||8===m)\u0026amp;\u0026amp;q.nodeValue;if(v?!\/^\\??prettify\\b\/.test(v):3!==m||\/\\S\/.test(q.nodeValue))break;if(v){k={};v.replace(\/\\b(\\w+)=([\\w:.%+-]+)\/g,function(a,b,c){k[b]=c});break}}q=d.className;if((k!==h||u.test(q))\u0026amp;\u0026amp;!e.test(q)){m=!1;for(v=d.parentNode;v;v=v.parentNode)if(w.test(v.tagName)\u0026amp;\u0026amp;v.className\u0026amp;\u0026amp;u.test(v.className)){m=\n!0;break}if(!m){d.className+=\" prettyprinted\";m=k.lang;if(!m){var m=q.match(t),C;!m\u0026amp;\u0026amp;(C=A(d))\u0026amp;\u0026amp;z.test(C.tagName)\u0026amp;\u0026amp;(m=C.className.match(t));m\u0026amp;\u0026amp;(m=m[1])}if(x.test(d.tagName))v=1;else var v=d.currentStyle,y=g.defaultView,v=(v=v?v.whiteSpace:y\u0026amp;\u0026amp;y.getComputedStyle?y.getComputedStyle(d,null).getPropertyValue(\"white-space\"):0)\u0026amp;\u0026amp;\"pre\"===v.substring(0,3);y=k.linenums;(y=\"true\"===y||+y)||(y=(y=q.match(\/\\blinenums\\b(?::(\\d+))?\/))?y[1]\u0026amp;\u0026amp;y[1].length?+y[1]:!0:!1);y\u0026amp;\u0026amp;B(d,y,v);H({j:m,h:d,m:y,l:v,a:null,i:null,c:null,\ng:null})}}}r\u0026lt;p.length?R.setTimeout(f,250):\"function\"===typeof a\u0026amp;\u0026amp;a()}for(var b=d||document.body,g=b.ownerDocument||document,b=[b.getElementsByTagName(\"pre\"),b.getElementsByTagName(\"code\"),b.getElementsByTagName(\"xmp\")],p=[],k=0;k\u0026lt;b.length;++k)for(var m=0,q=b[k].length;m\u0026lt;q;++m)p.push(b[k][m]);var b=null,c=Date;c.now||(c={now:function(){return+new Date}});var r=0,t=\/\\blang(?:uage)?-([\\w.]+)(?!\\S)\/,u=\/\\bprettyprint\\b\/,e=\/\\bprettyprinted\\b\/,x=\/pre|xmp\/i,z=\/^code\/i,w=\/^(?:pre|code|xmp)\/i,h={};f()}},\nS=R.define;\"function\"===typeof S\u0026amp;\u0026amp;S.amd\u0026amp;\u0026amp;S(\"google-code-prettify\",[],function(){return V})})();return g}();T||t.setTimeout(U,0)})();}()\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;div\u0026gt;\n\u0026lt;pre class=\"prettyprint\"\u0026gt;\n\u0026gt;\u0026gt;\u0026gt; TP_TN = [i for i in range(len(pred)) if pred[i] == y_test.values[i]] \n\u0026gt;\u0026gt;\u0026gt; shap_values = explainer.shap_values(X_test_sc[TP_TN,:], nsamples=100)\n\u0026gt;\u0026gt;\u0026gt; shap.force_plot(explainer.expected_value, shap_values, \\ \n X_test_sc[TP_TN,:],feature_names=X_train.columns)\n\n\u0026lt;\/pre\u0026gt;\n\u0026lt;\/div\u0026gt;\n\u0026lt;hr\u0026gt;","render_as_iframe":true,"selected_app_name":"HtmlApp","app_list":"{\"HtmlApp\":556073}"}},{"type":"Blog.Section","id":"f_218767d0-d624-4b11-9246-70f4fc7ff95b","defaultValue":null,"component":{"type":"Image","id":"f_fd8dcd2b-e94e-4368-af25-3286b8e9c150","defaultValue":null,"link_url":"","thumb_url":"!","url":"!","caption":"","description":"","storageKey":"1225579\/Screen_Shot_2019-02-14_at_10.31.09_AM_ugv7ek","storage":"c","storagePrefix":null,"format":"png","h":449,"w":1200,"s":97644,"new_target":true,"noCompression":null,"cropMode":null}},{"type":"Blog.Section","id":"f_7d0d8381-62db-4d7b-9895-bfe6dafc61bb","defaultValue":null,"component":{"type":"RichText","id":"f_48293fc9-7d71-4273-b0de-6ea849fdff43","defaultValue":null,"value":"\u003cp style=\"text-align: justify;\"\u003eThe figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature.\u003c\/p\u003e","backupValue":null,"version":null}},{"type":"Blog.Section","id":"f_e176b612-6fd9-4063-8da0-384b0e4189f4","defaultValue":null,"component":{"type":"RichText","id":"f_a1896bcb-39b1-423b-aef2-305d02678204","defaultValue":false,"value":"\u003cp style=\"text-align: justify;\"\u003eIt can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample):\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_1e1af1d3-d228-4bb6-bcf0-b07c080546a6","defaultValue":null,"component":{"type":"HtmlComponent","id":1607235,"defaultValue":false,"value":"\u0026lt;script type=\"text\/x-mathjax-config\"\u0026gt;\n MathJax.Hub.Config({\n tex2jax: {inlineMath: [[' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ,' Interpretability models​ Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ? Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations. Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ? This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11]. 1. The main idea The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ? In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance: The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction: To achieve that, most of approaches iterate between 2 steps: 1. Set a prohibition to some part of the input 2. Observe the change in the output (fitted answer) Repeat step 1 and step 2 for different prohibitions of the input. 2. Existing approaches The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability. 2.1. Cooperative game theory based Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium. A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution. 2.2. Architecture specific: Deep Neural Network Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model. For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject. Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models: Random Forest: Deep Neural Network: 2.3. A unified approach The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph. Cooperative Game theory-based: 3. SHAP: Additive feature attribution methods An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model. 3.1. The SHAP model Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that: where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows. 3.2. The Natural properties (1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained: g(x') = f(x) (2) Missingness: put the output to 0 corresponds to turning the feature off: x'i = 0 ⇒ Φi = 0 (3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one. Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if: fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i) then for all input z' ∈ {0,1}M : Φi (f 1, x) ≥ Φi (f 2, x) 3.3 Computing SHAP values 3.3.1. Back to the Shapley values The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values. In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below: where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]). 3.3.2. The SHAP values In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that: where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...). 4. Practical example with SHAP library Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers). I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap . By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0. The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%. Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below: This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset. For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score. We can apply this explainer model to all correct predicted examples in the test set, as below: The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature. It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample): In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score. The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure. Conclusion There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context. The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap. Bibliography [0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953). [1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001). [2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011). [3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/ [4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014). [5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015). [6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016). [7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017). [8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018). [9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017). [10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017). [11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017). All Posts × Almost done… We just sent you an email. Please click the link in the email to confirm your subscription! ], ['\\\\(','\\\']]}\n });\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;script type=\"text\/javascript\" async src=\"\/\/cdn.mathjax.org\/mathjax\/latest\/MathJax.js?config=TeX-AMS_CHTML\"\u0026gt;\u0026lt;\/script\u0026gt;\n\n\n\u0026lt;script\u0026gt;\n!function(){\/*\n\n Copyright (C) 2013 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n Copyright (C) 2006 Google Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n*\/\n(function(){function ba(g){function k(){try{M.doScroll(\"left\")}catch(g){t.setTimeout(k,50);return}z(\"poll\")}function z(k){if(\"readystatechange\"!=k.type||\"complete\"==A.readyState)(\"load\"==k.type?t:A)[B](p+k.type,z,!1),!q\u0026amp;\u0026amp;(q=!0)\u0026amp;\u0026amp;g.call(t,k.type||k)}var Y=A.addEventListener,q=!1,C=!0,x=Y?\"addEventListener\":\"attachEvent\",B=Y?\"removeEventListener\":\"detachEvent\",p=Y?\"\":\"on\";if(\"complete\"==A.readyState)g.call(t,\"lazy\");else{if(A.createEventObject\u0026amp;\u0026amp;M.doScroll){try{C=!t.frameElement}catch(da){}C\u0026amp;\u0026amp;k()}A[x](p+\n\"DOMContentLoaded\",z,!1);A[x](p+\"readystatechange\",z,!1);t[x](p+\"load\",z,!1)}}function U(){V\u0026amp;\u0026amp;ba(function(){var g=N.length;ca(g?function(){for(var k=0;k\u0026lt;g;++k)(function(g){t.setTimeout(function(){t.exports[N[g]].apply(t,arguments)},0)})(k)}:void 0)})}for(var t=window,A=document,M=A.documentElement,O=A.head||A.getElementsByTagName(\"head\")[0]||M,B=\"\",F=A.getElementsByTagName(\"script\"),q=F.length;0\u0026lt;=--q;){var P=F[q],Z=P.src.match(\/^[^?#]*\\\/run_prettify\\.js(\\?[^#]*)?(?:#.*)?\/);if(Z){B=Z[1]||\"\";P.parentNode.removeChild(P);\nbreak}}var V=!0,H=[],Q=[],N=[];B.replace(\/[?\u0026amp;]([^\u0026amp;=]+)=([^\u0026amp;]+)\/g,function(g,k,z){z=decodeURIComponent(z);k=decodeURIComponent(k);\"autorun\"==k?V=!\/^[0fn]\/i.test(z):\"lang\"==k?H.push(z):\"skin\"==k?Q.push(z):\"callback\"==k\u0026amp;\u0026amp;N.push(z)});q=0;for(B=H.length;q\u0026lt;B;++q)(function(){var g=A.createElement(\"script\");g.onload=g.onerror=g.onreadystatechange=function(){!g||g.readyState\u0026amp;\u0026amp;!\/loaded|complete\/.test(g.readyState)||(g.onerror=g.onload=g.onreadystatechange=null,--T,T||t.setTimeout(U,0),g.parentNode\u0026amp;\u0026amp;g.parentNode.removeChild(g),\ng=null)};g.type=\"text\/javascript\";g.src=\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/lang-\"+encodeURIComponent(H[q])+\".js\";O.insertBefore(g,O.firstChild)})(H[q]);for(var T=H.length,F=[],q=0,B=Q.length;q\u0026lt;B;++q)F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/skins\/\"+encodeURIComponent(Q[q])+\".css\");F.push(\"https:\/\/cdn.rawgit.com\/google\/code-prettify\/master\/loader\/prettify.css\");(function(g){function k(q){if(q!==z){var t=A.createElement(\"link\");t.rel=\"stylesheet\";t.type=\n\"text\/css\";q+1\u0026lt;z\u0026amp;\u0026amp;(t.error=t.onerror=function(){k(q+1)});t.href=g[q];O.appendChild(t)}}var z=g.length;k(0)})(F);var ca=function(){window.PR_SHOULD_USE_CONTINUATION=!0;var g;(function(){function k(a){function d(e){var b=e.charCodeAt(0);if(92!==b)return b;var a=e.charAt(1);return(b=W[a])?b:\"0\"\u0026lt;=a\u0026amp;\u0026amp;\"7\"\u0026gt;=a?parseInt(e.substring(1),8):\"u\"===a||\"x\"===a?parseInt(e.substring(2),16):e.charCodeAt(1)}function f(e){if(32\u0026gt;e)return(16\u0026gt;e?\"\\\\x0\":\"\\\\x\")+e.toString(16);e=String.fromCharCode(e);return\"\\\\\"===e||\"-\"===\ne||\"]\"===e||\"^\"===e?\"\\\\\"+e:e}function b(e){var b=e.substring(1,e.length-1).match(\/\\\\u[0-9A-Fa-f]{4}|\\\\x[0-9A-Fa-f]{2}|\\\\[0-3][0-7]{0,2}|\\\\[0-7]{1,2}|\\\\[\\s\\S]|-|[^-\\\\/g);e=[];var a=\"^\"===b[0],c=[\"[\"];a\u0026amp;\u0026amp;c.push(\"^\");for(var a=a?1:0,h=b.length;a\u0026lt;h;++a){var l=b[a];if(\/\\\$bdsw]\/i.test(l))c.push(l);else{var l=d(l),n;a+2\u0026lt;h\u0026amp;\u0026amp;\"-\"===b[a+1]?(n=d(b[a+2]),a+=2):n=l;e.push([l,n]);65\u0026gt;n||122\u0026lt;l||(65\u0026gt;n||90\u0026lt;l||e.push([Math.max(65,l)|32,Math.min(n,90)|32]),97\u0026gt;n||122\u0026lt;l||e.push([Math.max(97,l)\u0026amp;-33,Math.min(n,122)\u0026amp;-33]))}}e.sort(function(e,\na){return e[0]-a[0]||a[1]-e[1]});b=[];h=[];for(a=0;a\u0026lt;e.length;++a)l=e[a],l[0]\u0026lt;=h[1]+1?h[1]=Math.max(h[1],l[1]):b.push(h=l);for(a=0;a\u0026lt;b.length;++a)l=b[a],c.push(f(l[0])),l[1]\u0026gt;l[0]\u0026amp;\u0026amp;(l[1]+1\u0026gt;l[0]\u0026amp;\u0026amp;c.push(\"-\"),c.push(f(l[1])));c.push(\"]\");return c.join(\"\")}function g(e){for(var a=e.source.match(\/(?:\\[(?:[^\\x5C\\x5D]|\\\\[\\s\\S])*\$|\\\\u[A-Fa-f0-9]{4}|\\\\x[A-Fa-f0-9]{2}|\\\$0-9]+|\\\\[^ux0-9]|\$$\\?[:!=]|[\\(\$$\\^]|[^\\x5B\\x5C\$$\$$\\^]+)\/g),c=a.length,d=[],h=0,l=0;h\u0026lt;c;++h){var n=a[h];\"(\"===n?++l:\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=\n+n.substring(1))\u0026amp;\u0026amp;(n\u0026lt;=l?d[n]=-1:a[h]=f(n))}for(h=1;h\u0026lt;d.length;++h)-1===d[h]\u0026amp;\u0026amp;(d[h]=++k);for(l=h=0;h\u0026lt;c;++h)n=a[h],\"(\"===n?(++l,d[l]||(a[h]=\"(?:\")):\"\\\\\"===n.charAt(0)\u0026amp;\u0026amp;(n=+n.substring(1))\u0026amp;\u0026amp;n\u0026lt;=l\u0026amp;\u0026amp;(a[h]=\"\\\\\"+d[n]);for(h=0;h\u0026lt;c;++h)\"^\"===a[h]\u0026amp;\u0026amp;\"^\"!==a[h+1]\u0026amp;\u0026amp;(a[h]=\"\");if(e.ignoreCase\u0026amp;\u0026amp;I)for(h=0;h\u0026lt;c;++h)n=a[h],e=n.charAt(0),2\u0026lt;=n.length\u0026amp;\u0026amp;\"[\"===e?a[h]=b(n):\"\\\\\"!==e\u0026amp;\u0026amp;(a[h]=n.replace(\/[a-zA-Z]\/g,function(a){a=a.charCodeAt(0);return\"[\"+String.fromCharCode(a\u0026amp;-33,a|32)+\"]\"}));return a.join(\"\")}for(var k=0,I=!1,\nm=!1,J=0,c=a.length;J\u0026lt;c;++J){var r=a[J];if(r.ignoreCase)m=!0;else if(\/[a-z]\/i.test(r.source.replace(\/\\\\u[0-9a-f]{4}|\\\\x[0-9a-f]{2}|\\\\[^ux]\/gi,\"\"))){I=!0;m=!1;break}}for(var W={b:8,t:9,n:10,v:11,f:12,r:13},u=[],J=0,c=a.length;J\u0026lt;c;++J){r=a[J];if(r.global||r.multiline)throw Error(\"\"+r);u.push(\"(?:\"+g(r)+\")\")}return new RegExp(u.join(\"|\"),m?\"gi\":\"g\")}function q(a,d){function f(a){var c=a.nodeType;if(1==c){if(!b.test(a.className)){for(c=a.firstChild;c;c=c.nextSibling)f(c);c=a.nodeName.toLowerCase();if(\"br\"===\nc||\"li\"===c)g[m]=\"\\n\",I[m\u0026lt;\u0026lt;1]=k++,I[m++\u0026lt;\u0026lt;1|1]=a}}else if(3==c||4==c)c=a.nodeValue,c.length\u0026amp;\u0026amp;(c=d?c.replace(\/\\r\\n?\/g,\"\\n\"):c.replace(\/[ \\t\\r\\n]+\/g,\" \"),g[m]=c,I[m\u0026lt;\u0026lt;1]=k,k+=c.length,I[m++\u0026lt;\u0026lt;1|1]=a)}var b=\/(?:^|\\s)nocode(?:\\s|)\/,g=[],k=0,I=[],m=0;f(a);return{a:g.join(\"\").replace(\/\\n\/,\"\"),c:I}}function t(a,d,f,b,g){f\u0026amp;\u0026amp;(a={h:a,l:1,j:null,m:null,a:f,c:null,i:d,g:null},b(a),g.push.apply(g,a.g))}function A(a){for(var d=void 0,f=a.firstChild;f;f=f.nextSibling)var b=f.nodeType,d=1===b?d?a:f:3===b?T.test(f.nodeValue)?\na:d:d;return d===a?void 0:d}function C(a,d){function f(a){for(var m=a.i,k=a.h,c=[m,\"pln\"],r=0,W=a.a.match(g)||[],u={},e=0,q=W.length;e\u0026lt;q;++e){var D=W[e],w=u[D],h=void 0,l;if(\"string\"===typeof w)l=!1;else{var n=b[D.charAt(0)];if(n)h=D.match(n[1]),w=n[0];else{for(l=0;l\u0026lt;p;++l)if(n=d[l],h=D.match(n[1])){w=n[0];break}h||(w=\"pln\")}!(l=5\u0026lt;=w.length\u0026amp;\u0026amp;\"lang-\"===w.substring(0,5))||h\u0026amp;\u0026amp;\"string\"===typeof h[1]||(l=!1,w=\"src\");l||(u[D]=w)}n=r;r+=D.length;if(l){l=h[1];var E=D.indexOf(l),G=E+l.length;h[2]\u0026amp;\u0026amp;(G=D.length-\nh[2].length,E=G-l.length);w=w.substring(5);t(k,m+n,D.substring(0,E),f,c);t(k,m+n+E,l,F(w,l),c);t(k,m+n+G,D.substring(G),f,c)}else c.push(m+n,w)}a.g=c}var b={},g;(function(){for(var f=a.concat(d),m=[],p={},c=0,r=f.length;c\u0026lt;r;++c){var q=f[c],u=q[3];if(u)for(var e=u.length;0\u0026lt;=--e;)b[u.charAt(e)]=q;q=q[1];u=\"\"+q;p.hasOwnProperty(u)||(m.push(q),p[u]=null)}m.push(\/[\\0-\\uffff]\/);g=k(m)})();var p=d.length;return f}function x(a){var d=[],f=[];a.tripleQuotedStrings?d.push([\"str\",\/^(?:\\'\\'\\'(?:[^\\'\\\$|\\\$\\s\\S]|\\'{1,2}(?=[^\\']))*(?:\\'\\'\\'|)|\\\"\\\"\\\"(?:[^\\\"\\\$|\\\$\\s\\S]|\\\"{1,2}(?=[^\\\"]))*(?:\\\"\\\"\\\"|)|\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|))\/,\nnull,\"'\\\"\"]):a.multiLineStrings?d.push([\"str\",\/^(?:\\'(?:[^\\\\\\']|\\\\[\\s\\S])*(?:\\'|)|\\\"(?:[^\\\\\\\"]|\\\\[\\s\\S])*(?:\\\"|)|\\(?:[^\\\\\\]|\\\\[\\s\\S])*(?:\\|))\/,null,\"'\\\"\"]):d.push([\"str\",\/^(?:\\'(?:[^\\\\\\'\\r\\n]|\\\\.)*(?:\\'|)|\\\"(?:[^\\\\\\\"\\r\\n]|\\\\.)*(?:\\\"|))\/,null,\"\\\"'\"]);a.verbatimStrings\u0026amp;\u0026amp;f.push([\"str\",\/^@\\\"(?:[^\\\"]|\\\"\\\")*(?:\\\"|)\/,null]);var b=a.hashComments;b\u0026amp;\u0026amp;(a.cStyleComments?(1\u0026lt;b?d.push([\"com\",\/^#(?:##(?:[^#]|#(?!##))*(?:###|)|.*)\/,null,\"#\"]):d.push([\"com\",\/^#(?:(?:define|e(?:l|nd)if|else|error|ifn?def|include|line|pragma|undef|warning)\\b|[^\\r\\n]*)\/,\nnull,\"#\"]),f.push([\"str\",\/^\u0026lt;(?:(?:(?:\\.\\.\\\/)*|\\\/?)(?:[\\w-]+(?:\\\/[\\w-]+)+)?[\\w-]+\\.h(?:h|pp|\\+\\+)?|[a-z]\\w*)\u0026gt;\/,null])):d.push([\"com\",\/^#[^\\r\\n]*\/,null,\"#\"]));a.cStyleComments\u0026amp;\u0026amp;(f.push([\"com\",\/^\\\/\\\/[^\\r\\n]*\/,null]),f.push([\"com\",\/^\\\/\\*[\\s\\S]*?(?:\\*\\\/|)\/,null]));if(b=a.regexLiterals){var g=(b=1\u0026lt;b?\"\":\"\\n\\r\")?\".\":\"[\\\\S\\\\s]\";f.push([\"lang-regex\",RegExp(\"^(?:^^\\\\.?|[+-]|[!=]=?=?|\\\\#|%=?|\u0026amp;\u0026amp;?=?|\\\\(|\\\\*=?|[+\\\\-]=|-\u0026gt;|\\\\\/=?|::?|\u0026lt;\u0026lt;?=?|\u0026gt;\u0026gt;?\u0026gt;?=?|,|;|\\\\?|@|\\\\[|~|{|\\\\^\\\\^?=?|\\\\|\\\\|?=?|break|case|continue|delete|do|else|finally|instanceof|return|throw|try|typeof)\\\\s*(\"+\n(\"\/(?=[^\/*\"+b+\"])(?:[^\/\\\\x5B\\\\x5C\"+b+\"]|\\\\x5C\"+g+\"|\\\\x5B(?:[^\\\\x5C\\\\x5D\"+b+\"]|\\\\x5C\"+g+\")*(?:\\\\x5D|))+\/\")+\")\")])}(b=a.types)\u0026amp;\u0026amp;f.push([\"typ\",b]);b=(\"\"+a.keywords).replace(\/^ | \/g,\"\");b.length\u0026amp;\u0026amp;f.push([\"kwd\",new RegExp(\"^(?:\"+b.replace(\/[\\s,]+\/g,\"|\")+\")\\\\b\"),null]);d.push([\"pln\",\/^\\s+\/,null,\" \\r\\n\\t\\u00a0\"]);b=\"^.[^\\\\s\\\\w.@'\\\"\/\\\\\\\$*\";a.regexLiterals\u0026amp;\u0026amp;(b+=\"(?!s*\/)\");f.push([\"lit\",\/^@[a-z_][a-z_$@0-9]*\/i,null],[\"typ\",\/^(?:[@_]?[A-Z]+[a-z][A-Za-z_$@0-9]*|\\w+_t\\b)\/,null],[\"pln\",\/^[a-z_$][a-z_$@0-9]*\/i,\nnull],[\"lit\",\/^(?:0x[a-f0-9]+|(?:\\d(?:_\\d+)*\\d*(?:\\.\\d*)?|\\.\\d\\+)(?:e[+\\-]?\\d+)?)[a-z]*\/i,null,\"0123456789\"],[\"pln\",\/^\\\\[\\s\\S]?\/,null],[\"pun\",new RegExp(b),null]);return C(d,f)}function B(a,d,f){function b(a){var c=a.nodeType;if(1==c\u0026amp;\u0026amp;!k.test(a.className))if(\"br\"===a.nodeName)g(a),a.parentNode\u0026amp;\u0026amp;a.parentNode.removeChild(a);else for(a=a.firstChild;a;a=a.nextSibling)b(a);else if((3==c||4==c)\u0026amp;\u0026amp;f){var d=a.nodeValue,p=d.match(q);p\u0026amp;\u0026amp;(c=d.substring(0,p.index),a.nodeValue=c,(d=d.substring(p.index+p[0].length))\u0026amp;\u0026amp;\na.parentNode.insertBefore(m.createTextNode(d),a.nextSibling),g(a),c||a.parentNode.removeChild(a))}}function g(a){function b(a,c){var d=c?a.cloneNode(!1):a,n=a.parentNode;if(n){var n=b(n,1),e=a.nextSibling;n.appendChild(d);for(var f=e;f;f=e)e=f.nextSibling,n.appendChild(f)}return d}for(;!a.nextSibling;)if(a=a.parentNode,!a)return;a=b(a.nextSibling,0);for(var d;(d=a.parentNode)\u0026amp;\u0026amp;1===d.nodeType;)a=d;c.push(a)}for(var k=\/(?:^|\\s)nocode(?:\\s|)\/,q=\/\\r\\n?|\\n\/,m=a.ownerDocument,p=m.createElement(\"li\");a.firstChild;)p.appendChild(a.firstChild);\nfor(var c=[p],r=0;r\u0026lt;c.length;++r)b(c[r]);d===(d|0)\u0026amp;\u0026amp;c[0].setAttribute(\"value\",d);var t=m.createElement(\"ol\");t.className=\"linenums\";d=Math.max(0,d-1|0)||0;for(var r=0,u=c.length;r\u0026lt;u;++r)p=c[r],p.className=\"L\"+(r+d)%10,p.firstChild||p.appendChild(m.createTextNode(\"\\u00a0\")),t.appendChild(p);a.appendChild(t)}function p(a,d){for(var f=d.length;0\u0026lt;=--f;){var b=d[f];X.hasOwnProperty(b)?R.console\u0026amp;\u0026amp;console.warn(\"cannot override language handler %s\",b):X[b]=a}}function F(a,d){a\u0026amp;\u0026amp;X.hasOwnProperty(a)||(a=\/^\\s*\u0026lt;\/.test(d)?\n\"default-markup\":\"default-code\");return X[a]}function H(a){var d=a.j;try{var f=q(a.h,a.l),b=f.a;a.a=b;a.c=f.c;a.i=0;F(d,b)(a);var g=\/\\bMSIE\\s(\\d+)\/.exec(navigator.userAgent),g=g\u0026amp;\u0026amp;8\u0026gt;=+g[1],d=\/\\n\/g,p=a.a,k=p.length,f=0,m=a.c,t=m.length,b=0,c=a.g,r=c.length,x=0;c[r]=k;var u,e;for(e=u=0;e\u0026lt;r;)c[e]!==c[e+2]?(c[u++]=c[e++],c[u++]=c[e++]):e+=2;r=u;for(e=u=0;e\u0026lt;r;){for(var A=c[e],D=c[e+1],w=e+2;w+2\u0026lt;=r\u0026amp;\u0026amp;c[w+1]===D;)w+=2;c[u++]=A;c[u++]=D;e=w}c.length=u;var h=a.h;a=\"\";h\u0026amp;\u0026amp;(a=h.style.display,h.style.display=\"none\");\ntry{for(;b\u0026lt;t;){var l=m[b+2]||k,n=c[x+2]||k,w=Math.min(l,n),E=m[b+1],G;if(1!==E.nodeType\u0026amp;\u0026amp;(G=p.substring(f,w))){g\u0026amp;\u0026amp;(G=G.replace(d,\"\\r\"));E.nodeValue=G;var aa=E.ownerDocument,v=aa.createElement(\"span\");v.className=c[x+1];var B=E.parentNode;B.replaceChild(v,E);v.appendChild(E);f\u0026lt;l\u0026amp;\u0026amp;(m[b+1]=E=aa.createTextNode(p.substring(w,l)),B.insertBefore(E,v.nextSibling))}f=w;f\u0026gt;=l\u0026amp;\u0026amp;(b+=2);f\u0026gt;=n\u0026amp;\u0026amp;(x+=2)}}finally{h\u0026amp;\u0026amp;(h.style.display=a)}}catch(y){R.console\u0026amp;\u0026amp;console.log(y\u0026amp;\u0026amp;y.stack||y)}}var R=window,K=[\"break,continue,do,else,for,if,return,while\"],\nL=[[K,\"auto,case,char,const,default,double,enum,extern,float,goto,inline,int,long,register,short,signed,sizeof,static,struct,switch,typedef,union,unsigned,void,volatile\"],\"catch,class,delete,false,import,new,operator,private,protected,public,this,throw,true,try,typeof\"],S=[L,\"alignof,align_union,asm,axiom,bool,concept,concept_map,const_cast,constexpr,decltype,delegate,dynamic_cast,explicit,export,friend,generic,late_check,mutable,namespace,nullptr,property,reinterpret_cast,static_assert,static_cast,template,typeid,typename,using,virtual,where\"],\nM=[L,\"abstract,assert,boolean,byte,extends,finally,final,implements,import,instanceof,interface,null,native,package,strictfp,super,synchronized,throws,transient\"],N=[L,\"abstract,as,base,bool,by,byte,checked,decimal,delegate,descending,dynamic,event,finally,fixed,foreach,from,group,implicit,in,interface,internal,into,is,let,lock,null,object,out,override,orderby,params,partial,readonly,ref,sbyte,sealed,stackalloc,string,select,uint,ulong,unchecked,unsafe,ushort,var,virtual,where\"],L=[L,\"debugger,eval,export,function,get,instanceof,null,set,undefined,var,with,Infinity,NaN\"],\nO=[K,\"and,as,assert,class,def,del,elif,except,exec,finally,from,global,import,in,is,lambda,nonlocal,not,or,pass,print,raise,try,with,yield,False,True,None\"],P=[K,\"alias,and,begin,case,class,def,defined,elsif,end,ensure,false,in,module,next,nil,not,or,redo,rescue,retry,self,super,then,true,undef,unless,until,when,yield,BEGIN,END\"],K=[K,\"case,done,elif,esac,eval,fi,function,in,local,set,then,until\"],Q=\/^(DIR|FILE|vector|(de|priority_)?queue|list|stack|(const_)?iterator|(multi)?(set|map)|bitset|u?(int|float)\\d*)\\b\/,\nT=\/\\S\/,U=x({keywords:[S,N,M,L,\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",O,P,K],hashComments:!0,cStyleComments:!0,multiLineStrings:!0,regexLiterals:!0}),X={};p(U,[\"default-code\"]);p(C([],[[\"pln\",\/^[^\u0026lt;?]+\/],[\"dec\",\/^\u0026lt;!\\w[^\u0026gt;]*(?:\u0026gt;|)\/],[\"com\",\/^\u0026lt;\\!--[\\s\\S]*?(?:-\\-\u0026gt;|$)\/],[\"lang-\",\/^\u0026lt;\\?([\\s\\S]+?)(?:\\?\u0026gt;|$)\/],[\"lang-\",\/^\u0026lt;%([\\s\\S]+?)(?:%\u0026gt;|$)\/],[\"pun\",\/^(?:\u0026lt;[%?]|[%?]\u0026gt;)\/],[\"lang-\",\n\/^\u0026lt;xmp\\b[^\u0026gt;]*\u0026gt;([\\s\\S]+?)\u0026lt;\\\/xmp\\b[^\u0026gt;]*\u0026gt;\/i],[\"lang-js\",\/^\u0026lt;script\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/script\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-css\",\/^\u0026lt;style\\b[^\u0026gt;]*\u0026gt;([\\s\\S]*?)(\u0026lt;\\\/style\\b[^\u0026gt;]*\u0026gt;)\/i],[\"lang-in.tag\",\/^(\u0026lt;\\\/?[a-z][^\u0026lt;\u0026gt;]*\u0026gt;)\/i]]),\"default-markup htm html mxml xhtml xml xsl\".split(\" \"));p(C([[\"pln\",\/^[\\s]+\/,null,\" \\t\\r\\n\"],[\"atv\",\/^(?:\\\"[^\\\"]*\\\"?|\\'[^\\']*\\'?)\/,null,\"\\\"'\"]],[[\"tag\",\/^^\u0026lt;\\\/?[a-z](?:[\\w.:-]*\\w)?|\\\/?\u0026gt;$\/i],[\"atn\",\/^(?!style[\\s=]|on)[a-z](?:[\\w:-]*\\w)?\/i],[\"lang-uq.val\",\/^=\\s*([^\u0026gt;\\'\\\"\\s]*(?:[^\u0026gt;\\'\\\"\\s\\\/]|\\\/(?=\\s)))\/],\n[\"pun\",\/^[=\u0026lt;\u0026gt;\\\/]+\/],[\"lang-js\",\/^on\\w+\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-js\",\/^on\\w+\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i],[\"lang-css\",\/^style\\s*=\\s*\\\"([^\\\"]+)\\\"\/i],[\"lang-css\",\/^style\\s*=\\s*\\'([^\\']+)\\'\/i],[\"lang-css\",\/^style\\s*=\\s*([^\\\"\\'\u0026gt;\\s]+)\/i]]),[\"in.tag\"]);p(C([],[[\"atv\",\/^[\\s\\S]+\/]]),[\"uq.val\"]);p(x({keywords:S,hashComments:!0,cStyleComments:!0,types:Q}),\"c cc cpp cxx cyc m\".split(\" \"));p(x({keywords:\"null,true,false\"}),[\"json\"]);p(x({keywords:N,hashComments:!0,cStyleComments:!0,\nverbatimStrings:!0,types:Q}),[\"cs\"]);p(x({keywords:M,cStyleComments:!0}),[\"java\"]);p(x({keywords:K,hashComments:!0,multiLineStrings:!0}),[\"bash\",\"bsh\",\"csh\",\"sh\"]);p(x({keywords:O,hashComments:!0,multiLineStrings:!0,tripleQuotedStrings:!0}),[\"cv\",\"py\",\"python\"]);p(x({keywords:\"caller,delete,die,do,dump,elsif,eval,exit,foreach,for,goto,if,import,last,local,my,next,no,our,print,package,redo,require,sub,undef,unless,until,use,wantarray,while,BEGIN,END\",hashComments:!0,multiLineStrings:!0,regexLiterals:2}),\n[\"perl\",\"pl\",\"pm\"]);p(x({keywords:P,hashComments:!0,multiLineStrings:!0,regexLiterals:!0}),[\"rb\",\"ruby\"]);p(x({keywords:L,cStyleComments:!0,regexLiterals:!0}),[\"javascript\",\"js\"]);p(x({keywords:\"all,and,by,catch,class,else,extends,false,finally,for,if,in,is,isnt,loop,new,no,not,null,of,off,on,or,return,super,then,throw,true,try,unless,until,when,while,yes\",hashComments:3,cStyleComments:!0,multilineStrings:!0,tripleQuotedStrings:!0,regexLiterals:!0}),[\"coffee\"]);p(C([],[[\"str\",\/^[\\s\\S]+\/]]),[\"regex\"]);\nvar V=R.PR={createSimpleLexer:C,registerLangHandler:p,sourceDecorator:x,PR_ATTRIB_NAME:\"atn\",PR_ATTRIB_VALUE:\"atv\",PR_COMMENT:\"com\",PR_DECLARATION:\"dec\",PR_KEYWORD:\"kwd\",PR_LITERAL:\"lit\",PR_NOCODE:\"nocode\",PR_PLAIN:\"pln\",PR_PUNCTUATION:\"pun\",PR_SOURCE:\"src\",PR_STRING:\"str\",PR_TAG:\"tag\",PR_TYPE:\"typ\",prettyPrintOne:function(a,d,f){f=f||!1;d=d||null;var b=document.createElement(\"div\");b.innerHTML=\"\u0026lt;pre\u0026gt;\"+a+\"\u0026lt;\/pre\u0026gt;\";b=b.firstChild;f\u0026amp;\u0026amp;B(b,f,!0);H({j:d,m:f,h:b,l:1,a:null,i:null,c:null,g:null});return b.innerHTML},\nprettyPrint:g=g=function(a,d){function f(){for(var b=R.PR_SHOULD_USE_CONTINUATION?c.now()+250:Infinity;r\u0026lt;p.length\u0026amp;\u0026amp;c.now()\u0026lt;b;r++){for(var d=p[r],k=h,q=d;q=q.previousSibling;){var m=q.nodeType,v=(7===m||8===m)\u0026amp;\u0026amp;q.nodeValue;if(v?!\/^\\??prettify\\b\/.test(v):3!==m||\/\\S\/.test(q.nodeValue))break;if(v){k={};v.replace(\/\\b(\\w+)=([\\w:.%+-]+)\/g,function(a,b,c){k[b]=c});break}}q=d.className;if((k!==h||u.test(q))\u0026amp;\u0026amp;!e.test(q)){m=!1;for(v=d.parentNode;v;v=v.parentNode)if(w.test(v.tagName)\u0026amp;\u0026amp;v.className\u0026amp;\u0026amp;u.test(v.className)){m=\n!0;break}if(!m){d.className+=\" prettyprinted\";m=k.lang;if(!m){var m=q.match(t),C;!m\u0026amp;\u0026amp;(C=A(d))\u0026amp;\u0026amp;z.test(C.tagName)\u0026amp;\u0026amp;(m=C.className.match(t));m\u0026amp;\u0026amp;(m=m[1])}if(x.test(d.tagName))v=1;else var v=d.currentStyle,y=g.defaultView,v=(v=v?v.whiteSpace:y\u0026amp;\u0026amp;y.getComputedStyle?y.getComputedStyle(d,null).getPropertyValue(\"white-space\"):0)\u0026amp;\u0026amp;\"pre\"===v.substring(0,3);y=k.linenums;(y=\"true\"===y||+y)||(y=(y=q.match(\/\\blinenums\\b(?::(\\d+))?\/))?y[1]\u0026amp;\u0026amp;y[1].length?+y[1]:!0:!1);y\u0026amp;\u0026amp;B(d,y,v);H({j:m,h:d,m:y,l:v,a:null,i:null,c:null,\ng:null})}}}r\u0026lt;p.length?R.setTimeout(f,250):\"function\"===typeof a\u0026amp;\u0026amp;a()}for(var b=d||document.body,g=b.ownerDocument||document,b=[b.getElementsByTagName(\"pre\"),b.getElementsByTagName(\"code\"),b.getElementsByTagName(\"xmp\")],p=[],k=0;k\u0026lt;b.length;++k)for(var m=0,q=b[k].length;m\u0026lt;q;++m)p.push(b[k][m]);var b=null,c=Date;c.now||(c={now:function(){return+new Date}});var r=0,t=\/\\blang(?:uage)?-([\\w.]+)(?!\\S)\/,u=\/\\bprettyprint\\b\/,e=\/\\bprettyprinted\\b\/,x=\/pre|xmp\/i,z=\/^code$\/i,w=\/^(?:pre|code|xmp)$\/i,h={};f()}},\nS=R.define;\"function\"===typeof S\u0026amp;\u0026amp;S.amd\u0026amp;\u0026amp;S(\"google-code-prettify\",[],function(){return V})})();return g}();T||t.setTimeout(U,0)})();}()\n\u0026lt;\/script\u0026gt;\n\n\u0026lt;div\u0026gt;\n\u0026lt;pre class=\"prettyprint\"\u0026gt;\n\u0026gt;\u0026gt;\u0026gt; shap.summary_plot(shap_values, X_test_sc[TP_TN,:], feature_names=X_train.columns )\n\n## for plot bar\n\u0026gt;\u0026gt;\u0026gt; shap.summary_plot(shap_values, X_test_sc[TP_TN,:],feature_names=X_train.columns, plot_type=\"bar\")\n\n\u0026lt;\/pre\u0026gt;\n\u0026lt;\/div\u0026gt;\n\u0026lt;hr\u0026gt;","render_as_iframe":true,"selected_app_name":"HtmlApp","app_list":"{\"HtmlApp\":576539}"}},{"type":"Blog.Section","id":"f_093adfa6-4364-4440-be2d-50841db88e71","defaultValue":null,"component":{"type":"Image","id":"f_97179954-56ce-47e5-b4fd-32778ca753d3","defaultValue":null,"link_url":"","thumb_url":"!","url":"!","caption":"","description":"","storageKey":"1225579\/Screen_Shot_2019-02-14_at_11.30.58_AM_kywoxa","storage":"c","storagePrefix":null,"format":"png","h":352,"w":1200,"s":61329,"new_target":true,"noCompression":null,"cropMode":null}},{"type":"Blog.Section","id":"f_7b26aded-ec0a-49b6-82ad-a037774c33b0","defaultValue":null,"component":{"type":"RichText","id":"f_748e7486-8a99-4f00-95d7-ecbee3f71f7e","defaultValue":false,"value":"\u003cp style=\"text-align: justify;\"\u003eIn the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of\u003cu\u003e SHAP values\u003c\/u\u003e computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the \u003cu\u003efeature value\u003c\/u\u003e (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score.\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003eThe second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure.\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_92b259c5-394d-4621-8b7d-eea6fe035562","defaultValue":null,"component":{"type":"Blog.Title","id":"f_c91b0f9b-a13c-4574-a896-9dc83fae5ef1","defaultValue":false,"value":"\u003cp\u003eConclusion\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_3fa6ef79-5705-4502-a435-d56e533e71b3","defaultValue":null,"component":{"type":"RichText","id":"f_d26b1b42-6417-44b1-ab8e-c7b4a12b596a","defaultValue":false,"value":"\u003cp style=\"text-align: justify;\"\u003eThere are lots of approaches proposed in the literature to deal with interpretability\/explainable models in the supervised context.\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003eThe main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: \u003ca target=\"_blank\" href=\"https:\/\/github.com\/slundberg\/shap\"\u003ehttps:\/\/github.com\/slundberg\/shap\u003c\/a\u003e.\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_e14e977f-909f-449e-8949-a0a1cd801896","defaultValue":null,"component":{"type":"Separator","id":"f_1a677746-633c-4d6e-8165-d3a15cae3e5b","defaultValue":null,"value":null}},{"type":"Blog.Section","id":"f_977a40d3-0cf9-4cb1-bb15-9e63242d372a","defaultValue":null,"component":{"type":"Blog.Title","id":"f_5adb6f39-17ab-4688-96e2-c3d92b212d03","defaultValue":false,"value":"\u003cp\u003e\u003cspan class=\"s-text-color-gray\"\u003eBibliography\u003c\/span\u003e\u003c\/p\u003e","backupValue":null,"version":1}},{"type":"Blog.Section","id":"f_7b2fb3a8-c705-4ed2-8f20-20389af56b9e","defaultValue":null,"component":{"type":"RichText","id":"f_93d9bfaf-adfb-4841-b02f-389fb0b450b9","defaultValue":false,"value":"\u003cp style=\"font-size: 100%; text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[0] Shapley, Lloyd S. \u201cA Value for N-Person Games.\u201d Contributions to the Theory of Games 2 (28): 307\u201317, (1953).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"font-size: 100%; text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[1] Lipovetsky, S. and Conklin, M. \"Analysis of regression in game theory approach.\" Applied Stochastic Models in business and industry (17-4):319-330, (2001).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[2] Strumbelj et al., \"A General Method for Visualizing and Explaining Black-Box Regression Models.\" Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[3] Saabas et al., \"Interpreting random forests\", \u003c\/span\u003e\u003ca target=\"_blank\" href=\"http:\/\/Interpreting%20random%20forests%20(https:\/\/blog.datadive.net\/interpreting-random-forests\/)\"\u003e\u003cspan class=\"s-text-color-black\"\u003ehttps:\/\/blog.datadive.net\/interpreting-random-forests\/\u003c\/span\u003e\u003c\/a\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[4] Springenberg et al., \"Striving for simplicity: The all convolutional net\", arXiv:1412.6806 (2014).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[5] Bach et al. \"On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation\", PLOS ONE: (10-7): 130-140, (2015).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[6] Ribeiro et al. \"Why should I Trust You ? Explaining the predictions of any classifier\", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[7] Shrikumar et al. \"Learning Important Features Through Propagating Activation Difference\", Proceedings in ICML (2017).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e[8] \"Explainable and Interpretable Models in Computer Vision and Machine Learning\", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018).\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[9] Sunderarajan et al., \"Axiomatic Attribution for Deep Networks\", Proceedings in ICML (2017).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[10] Montavon et al. \"Explaining nonlinear classification decision with deep Taylor decomposition\", Pattern Recognition (65):211-222 (2017).\u003c\/span\u003e\u003c\/p\u003e\u003cp style=\"text-align: justify;\"\u003e\u003cspan class=\"s-text-color-black\"\u003e[11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017).\u003c\/span\u003e\u003c\/p\u003e","backupValue":null,"version":1}}]},"settings":{"hideBlogDate":false},"pageMode":null};$S.siteData={"terms_text":null,"privacy_policy_text":null,"show_terms_and_conditions":false,"show_privacy_policy":false,"gdpr_html":null,"live_chat":false};$S.stores={"fonts_v2":[{"name":"roboto condensed","fontType":"google","displayName":"Roboto Condensed","cssValue":"\"roboto condensed\"","settings":{"weight":"300,700"},"hidden":false,"cssFallback":"sans-serif","disableBody":null,"isSuggested":true}],"features":{"allFeatures":[{"name":"analytics","canBeUsed":true,"hidden":false},{"name":"fb_image","canBeUsed":true,"hidden":false},{"name":"twitter_card","canBeUsed":true,"hidden":false},{"name":"favicon","canBeUsed":true,"hidden":false},{"name":"style_panel","canBeUsed":true,"hidden":false},{"name":"google_analytics","canBeUsed":true,"hidden":false},{"name":"blog_custom_url","canBeUsed":true,"hidden":false},{"name":"page_collaboration","canBeUsed":true,"hidden":false},{"name":"premium_templates","canBeUsed":true,"hidden":false},{"name":"custom_domain","canBeUsed":true,"hidden":false},{"name":"premium_support","canBeUsed":true,"hidden":false},{"name":"remove_branding_title","canBeUsed":true,"hidden":false},{"name":"full_analytics","canBeUsed":true,"hidden":false},{"name":"ecommerce_layout","canBeUsed":true,"hidden":false},{"name":"portfolio_layout","canBeUsed":true,"hidden":false},{"name":"password_protection","canBeUsed":true,"hidden":false},{"name":"remove_logo","canBeUsed":true,"hidden":false},{"name":"optimizely","canBeUsed":true,"hidden":false},{"name":"custom_code","canBeUsed":true,"hidden":false},{"name":"blog_custom_code","canBeUsed":true,"hidden":false},{"name":"mobile_actions","canBeUsed":true,"hidden":false},{"name":"premium_assets","canBeUsed":true,"hidden":false},{"name":"premium_apps","canBeUsed":true,"hidden":false},{"name":"premium_sections","canBeUsed":true,"hidden":false},{"name":"blog_mailchimp_integration","canBeUsed":true,"hidden":false},{"name":"ecommerce_coupon","canBeUsed":true,"hidden":false},{"name":"ecommerce_shipping_region","canBeUsed":true,"hidden":false},{"name":"multiple_page","canBeUsed":true,"hidden":false},{"name":"ecommerce_taxes","canBeUsed":true,"hidden":false},{"name":"ecommerce_layout","canBeUsed":true,"hidden":false},{"name":"portfolio_layout","canBeUsed":true,"hidden":false},{"name":"ecommerce_category","canBeUsed":true,"hidden":false},{"name":"facebook_pixel","canBeUsed":true,"hidden":false},{"name":"blog_category","canBeUsed":true,"hidden":false},{"name":"product_page","canBeUsed":true,"hidden":false},{"name":"font_search","canBeUsed":true,"hidden":false},{"name":"custom_font","canBeUsed":true,"hidden":false},{"name":"membership","canBeUsed":true,"hidden":false},{"name":"blog_post_amp","canBeUsed":true,"hidden":false},{"name":"site_search","canBeUsed":true,"hidden":false},{"name":"portfolio_category","canBeUsed":true,"hidden":false},{"name":"ecommerce_digital_download","canBeUsed":true,"hidden":false},{"name":"popup","canBeUsed":true,"hidden":false},{"name":"live_chat","canBeUsed":false,"hidden":false}]},"showStatic":{"isEditMode":false}};

Interpretability models​

Why interpretability is so important in machine learning ? Why can't we just trust the prediction of a supervised model ?

Several possible explanations to that: we can think about improving social acceptance for the integration of ML algorithms into our lives ; correcting a model by discovering a bias in the population of the training set; understanding the cases for which the model fails; following the law and regulations.

Nowadays, complex supervised models can be very accurate on specific tasks but remain quite uninterpretable; at the opposite, when using simple models, it is indeed easy to interpret them but are often less accurate. How can we solve such a dilemma ?

This post tends to answer to this question by going through the ML literature in interpretability models and by focusing on a class of additive feature attribution methods [11].

1. The main idea

The problem of giving an interpretation to the model prediction can be recasted as it follows: Which part of the input is particularly important to explain the output of the model ?

In order to illustrate this purpose, let's consider the example given during the ICML conference by Shrikumar. Lets suppose you have already trained a model with DNA mutations causing diseases. Now, let's consider a DNA sequence as input, as for instance:

The model is going to predict if this sequence can be linked to any known diseases the model learnt. If so, what you would like to understand is why your model gives this prediction in particular; ie which part of the input sequence leads your model to predict a specific disease. So, you would like to have higher weights for the parts of the sequence which explain the most the decision of your model and lower ones for those which do not explain the prediction:

To achieve that, most of approaches iterate between 2 steps:

1. Set a prohibition to some part of the input
2. Observe the change in the output (fitted answer)

Repeat step 1 and step 2 for different prohibitions of the input.

2. Existing approaches

The need of tools for explaining prediction models came with the development of more complex models to deal with more complex data and therefore the recent literature in Computer Vision and Machine Learning has developed a new field linked to interpretability.

2.1. Cooperative game theory based

Back to the beginning of the 21th century, Lipovetsky et al. (2001)[1] highlight the multicollinearity problem in the analysis of regressor importance in the multiple regression context: important variables can have significant coefficient because of their collinearity. To that end, they use a tool from the cooperative game theory to obtain comparative importance of predictors: the Shapley Values imputation [0] derived from an axiomatic approach and produces a unique solution satisfying general requirements of Nash equilibrium.

A decade later, Strumbelj et al. (2011)[2] generalize the use of Shapley values for black box models such as SVM and artificial neural network models in order to make models more informative, easier to understand and to use. They propose an approximation algorithm by assuming mutual independence of individual features in order to encompass the time complexity limitation of the solution.

2.2. Architecture specific: Deep Neural Network

Since then, several specific methods have been proposed in the literature and take advantages of the structure/architecture of the model.

For neural networks, we can think about back-propagation based methods such as Guided Propagation (Springenberg et al. 2014)[4] which use the relationship between the neurons and the output. The idea is to assign a score to neurons according to how much they affect the output. This is done in the single backward pass where you get the scores for all parts of the input. Other approaches propose to build a linear model to locally approximate the more complicated model based on data which affects the output (LIME [6]). Shrikumar et al. 2016 [7] introduces DeepLift which assigns contribution scores to the feature based on the difference between the activation of each neuron to its ‘reference activation’. Other explaining prediction models Deep Neural Network-specific have been proposed in the literature and the reader could read [8] for additional references on this subject.

Below, a list of methods and their available python code which summarizes the most recent approaches for specific-models:

Random Forest:

Deep Neural Network:

2.3. A unified approach

The most recent and general approach for interpretability models is the SHAP model from Lundberg et al. 2018. It proposes a class of methods called Additive Feature attribution methods that contains most of the approaches cited above. These methods use the same explanation model (ie any interpretable approximation of the original model) that we introduce in the next paragraph.

Cooperative Game theory-based:

An explanation model is a simple model which describes the behavior of the complex model. The additive attribution methods introduce a linear function of binary variables to represent such an explanation model.

3.1. The SHAP model

Let f be the original prediction model to be explained and g the explanation model. Additive feature attribution methods have an explanation model that is a linear function of binary variables such that:

where M is the number of features ; the z'i variables represent a feature being observed (zi' = 1) or unknown (zi'= 0), and the Φi ∈ ’s are the feature attribution values. There is only one solution for Φ_i satisfying general requirements of Nash equilibrium and satisfying three natural properties explained in the paragraph that follows.

3.2. The Natural properties

(1) Local Accuracy: the output of the explanation model matches the original model for the prediction being explained:

g(x') = f(x)

(2) Missingness: put the output to 0 corresponds to turning the feature off:

x'i = 0 ⇒ Φi = 0

(3) Accuracy: if turning the feature off in one model which always makes a bigger difference in another model then the importance should be higher in the first model than in the second one.

Lets consider z' \ i meaning z'i = 0, then for any 2 models f 1 and f 2, if:

fx1(z') - fx1(z' \ i) ≥ fx2(z') - fx2(z' \ i)

then for all input z' ∈ {0,1}M :

Φi (f 1, x) ≥ Φi (f 2, x)

3.3 Computing SHAP values

3.3.1. Back to the Shapley values

The computation of features importance -- the SHAP values -- comes from cooperative games theory [0] with the Shapley values.

In our context, a Shapley value can be viewed as a weighted average of all possible differences between predictions of the model without feature i, and the ones with feature i as expressed below:

where |z′| stands for the number of features different from zero, and z′ ⊆ x′ stands for all z′ vectors where the non-zero entries are a subset of entries of x′ except feature i. Since the problem is combinatorial different strategies have been proposed in the literature to approximate the solution ([0,1]).

3.3.2. The SHAP values

In the more general context the SHAP values can be viewed as Shapley values of a conditional expectation function of the original model such that:

where S is the set of non-zero entries of z'. In practice, the computation of SHAP values are challenging that is why Lundberg and al.[11] propose different approximation algorithms according to the specificities of your model or your data (tree ensembles, independent features, deep network,...).

4. Practical example with SHAP library

Lundenberg created a GitHub repository to that end with very nice and quite complete notebooks explaining different use cases for SHAP and its different approximation algorithms (Tree/ Deep / Gradient /Linear or Kernel Explainers).

I do really encourage the reader to visit the page of the author: https://github.com/slundberg/shap .

By the way, I am just going to introduce a very simple example in order to give insights of the kind of results we could obtain when looking for interpreting a prediction. Lets consider the heart dataset coming from kaggle competition (https://www.kaggle.com/ronitf/heart-disease-uci). The dataset consists in 13 variables describing 303 patients and 1 label describing the angiographic disease status (target \in {0,1}). The set is quite balanced since 165 patients have label 1 and 138 have label 0.

The data have been pre-processed a little bit such as we keep only the most informative variables which are ['sex', 'cp', 'thalach', 'exang', 'oldpeak', 'ca', 'thal']. Then we split the dataset into a random train (75% of data) and test sets (the remaining 25%) and we scale them. A svm classifier has been learnt, and we obtain a classification accuracy around 91%.

Once the classification model is learnt, we are looking for explaining a particular prediction (a true one ;) ) based on the shap library developed by Lundberg. You need to install the shap library (https://github.com/slundberg/shap) before running the code below:

This figure illustrates features that push the prediction higher (in pink) and the ones which push the prediction lower (in blue) from a base value computed on the average model output on the training dataset.

For that true positive fitted answer, we can see that the main features which tend to push the probabilities towards 1 is mainly explained by 'sex', 'oldpeak', 'thalach' and 'exang', whereas the 'ca' feature tends to push down the prediction score.

We can apply this explainer model to all correct predicted examples in the test set, as below:

The figure above stands for all individual feature contribution that have been stacked horizontally and ordered by output value. The 39 first predictions are correctly classified in class 1 and the 30 last ones are labeled and correctly classified in class 0. Note that the visualisation is interactive and we can see the effect of a particular feature by changing the y-axis in the menu of the left side of the figure. Symmetrically, you can change the x-axis menu in order to order the sample according to output values, similarities or SHAP values by feature.

It can also be very interested to have in one plot an overview of the distribution of SHAP values for each feature and an idea of their overall impact (Note that the example is still on the correct predicted sample):

In the first plot (subfigure a.), there are three kinds of information: in the x-axis you have the SHAP values of each feature described in the y-axis. Each line stands for the set of SHAP values computed for a specific feature and this is done for every features of your model. The third dimension is the color of points: it represents the feature value (pink for high value of the feature and blue for a low value). You can therefore see the dispersion of SHAP values according to features and also, their impact in the output model. For instance, high values of 'cp' feature implies high SHAP values predicted score and tends to push up the prediction whereas high values of 'thal' feature (pink points) tends to lower the predicted score.

The second plot (subfigure b.) is the Mean Absolute Value of SHAP values obtained for each feature. This can be seen as a summary of the left figure.

Conclusion

There are lots of approaches proposed in the literature to deal with interpretability/explainable models in the supervised context.

The main strength of the additive feature attribution model is its theoretical properties on one hand and on the other hand, its general framework which tends to explain most of explainable models developed in the literature. Different approximations algorithms have been proposed by Lundberg et al. in order to take advantage of the structure of the model and types of data to improve time computations. If you deal with Deep Neural Network or tree ensembles, I really encourage the reader to see more examples on the GitHub repository of the author: https://github.com/slundberg/shap.

Bibliography

[0] Shapley, Lloyd S. “A Value for N-Person Games.” Contributions to the Theory of Games 2 (28): 307–17, (1953).

[1] Lipovetsky, S. and Conklin, M. "Analysis of regression in game theory approach." Applied Stochastic Models in business and industry (17-4):319-330, (2001).

[2] Strumbelj et al., "A General Method for Visualizing and Explaining Black-Box Regression Models." Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6594. (2011).

[3] Saabas et al., "Interpreting random forests", https://blog.datadive.net/interpreting-random-forests/

[4] Springenberg et al., "Striving for simplicity: The all convolutional net", arXiv:1412.6806 (2014).

[5] Bach et al. "On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation", PLOS ONE: (10-7): 130-140, (2015).

[6] Ribeiro et al. "Why should I Trust You ? Explaining the predictions of any classifier", Proceedings of the 22nd ACM SIGKDD: 1135-1144 (2016).

[7] Shrikumar et al. "Learning Important Features Through Propagating Activation Difference", Proceedings in ICML (2017).

[8] "Explainable and Interpretable Models in Computer Vision and Machine Learning", Springer Verlag, The Springer Series on Challenges in Machine Learning, 9783319981307 (2018).

[9] Sunderarajan et al., "Axiomatic Attribution for Deep Networks", Proceedings in ICML (2017).

[10] Montavon et al. "Explaining nonlinear classification decision with deep Taylor decomposition", Pattern Recognition (65):211-222 (2017).

[11] Lundberg et al., ''A unified approach to interpreting model predictions'', NIPS (2017).

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!