| Server IP : 217.160.0.135 / Your IP : 216.73.217.85 Web Server : Apache System : Linux www 6.18.52-i1-ampere #1203 SMP Mon Sep 14 18:29:59 CEST 2026 aarch64 User : sws1074145052 ( 1074145052) PHP Version : 8.3.32 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : /lib/python3/dist-packages/scipy/stats/__pycache__/ |
Upload File : |
�
d�c8 � � � d Z ddlmZ ddlZddlZddlmZ ddlm Z m
Z
ddgZ d d!d�Z ddddd�d"d�Z
d� Zd � Zd � Zd � Zd � ZdS )#z5
Created on Fri Apr 2 09:06:05 2021
@author: matth
� )� annotationsN)�special)�Optional�Union�entropy�differential_entropy�pk�np.typing.ArrayLike�qk� Optional[np.typing.ArrayLike]�base�Optional[float]�axis�int�return� Union[np.number, np.ndarray]c �� � |�|dk rt d� � �t j | � � } d| z t j | |d�� � z } |�t j | � � }n^t j |� � }t j | |� � \ } }d|z t j ||d�� � z }t j | |� � }t j ||�� � }|�|t j |� � z }|S )a�
Calculate the Shannon entropy/relative entropy of given distribution(s).
If only probabilities `pk` are given, the Shannon entropy is calculated as
``H = -sum(pk * log(pk))``.
If `qk` is not None, then compute the relative entropy
``D = sum(pk * log(pk / qk))``. This quantity is also known
as the Kullback-Leibler divergence.
This routine will normalize `pk` and `qk` if they don't sum to 1.
Parameters
----------
pk : array_like
Defines the (discrete) distribution. Along each axis-slice of ``pk``,
element ``i`` is the (possibly unnormalized) probability of event
``i``.
qk : array_like, optional
Sequence against which the relative entropy is computed. Should be in
the same format as `pk`.
base : float, optional
The logarithmic base to use, defaults to ``e`` (natural logarithm).
axis : int, optional
The axis along which the entropy is calculated. Default is 0.
Returns
-------
S : {float, array_like}
The calculated entropy.
Notes
-----
Informally, the Shannon entropy quantifies the expected uncertainty
inherent in the possible outcomes of a discrete random variable.
For example,
if messages consisting of sequences of symbols from a set are to be
encoded and transmitted over a noiseless channel, then the Shannon entropy
``H(pk)`` gives a tight lower bound for the average number of units of
information needed per symbol if the symbols occur with frequencies
governed by the discrete distribution `pk` [1]_. The choice of base
determines the choice of units; e.g., ``e`` for nats, ``2`` for bits, etc.
The relative entropy, ``D(pk|qk)``, quantifies the increase in the average
number of units of information needed per symbol if the encoding is
optimized for the probability distribution `qk` instead of the true
distribution `pk`. Informally, the relative entropy quantifies the expected
excess in surprise experienced if one believes the true distribution is
`qk` when it is actually `pk`.
A related quantity, the cross entropy ``CE(pk, qk)``, satisfies the
equation ``CE(pk, qk) = H(pk) + D(pk|qk)`` and can also be calculated with
the formula ``CE = -sum(pk * log(qk))``. It gives the average
number of units of information needed per symbol if an encoding is
optimized for the probability distribution `qk` when the true distribution
is `pk`. It is not computed directly by `entropy`, but it can be computed
using two calls to the function (see Examples).
See [2]_ for more information.
References
----------
.. [1] Shannon, C.E. (1948), A Mathematical Theory of Communication.
Bell System Technical Journal, 27: 379-423.
https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
.. [2] Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information
Theory (Wiley Series in Telecommunications and Signal Processing).
Wiley-Interscience, USA.
Examples
--------
The outcome of a fair coin is the most uncertain:
>>> import numpy as np
>>> from scipy.stats import entropy
>>> base = 2 # work in units of bits
>>> pk = np.array([1/2, 1/2]) # fair coin
>>> H = entropy(pk, base=base)
>>> H
1.0
>>> H == -np.sum(pk * np.log(pk)) / np.log(base)
True
The outcome of a biased coin is less uncertain:
>>> qk = np.array([9/10, 1/10]) # biased coin
>>> entropy(qk, base=base)
0.46899559358928117
The relative entropy between the fair coin and biased coin is calculated
as:
>>> D = entropy(pk, qk, base=base)
>>> D
0.7369655941662062
>>> D == np.sum(pk * np.log(pk/qk)) / np.log(base)
True
The cross entropy can be calculated as the sum of the entropy and
relative entropy`:
>>> CE = entropy(pk, base=base) + entropy(pk, qk, base=base)
>>> CE
1.736965594166206
>>> CE == -np.sum(pk * np.log(qk)) / np.log(base)
True
Nr �+`base` must be a positive number or `None`.g �?T�r �keepdims�r ) �
ValueError�np�asarray�sumr �entr�broadcast_arrays�rel_entr�log)r r r
r �vec�Ss �6/usr/lib/python3/dist-packages/scipy/stats/_entropy.pyr r s� � �d ��D�A�I�I��F�G�G�G� ��B� � �B� �R��"�&��$��6�6�6� 6�B� �z��l�2� � ���
�Z��^�^���$�R��,�,���B�
��V�b�f�R�d�T�:�:�:�
:��� �r�2�&�&��
��s�� � � �A� �� �R�V�D�\�\��� �H� �auto)�
window_lengthr
r �method�valuesr% �
Optional[int]r& �strc � � t j | � � } t j | |d� � } | j d }|�)t j t j |� � dz � � }dd|z cxk r|k sn t d|� d|� d�� � �|�|dk rt d � � �t j | d�
� � }t t t t t d �}|�
� � }||vr!d t |� � � �}t |� � �|d
k r|dk rd}n |dk rd}nd} || ||� � } |�| t j |� � z } | S )aV Given a sample of a distribution, estimate the differential entropy.
Several estimation methods are available using the `method` parameter. By
default, a method is selected based the size of the sample.
Parameters
----------
values : sequence
Sample from a continuous distribution.
window_length : int, optional
Window length for computing Vasicek estimate. Must be an integer
between 1 and half of the sample size. If ``None`` (the default), it
uses the heuristic value
.. math::
\left \lfloor \sqrt{n} + 0.5 \right \rfloor
where :math:`n` is the sample size. This heuristic was originally
proposed in [2]_ and has become common in the literature.
base : float, optional
The logarithmic base to use, defaults to ``e`` (natural logarithm).
axis : int, optional
The axis along which the differential entropy is calculated.
Default is 0.
method : {'vasicek', 'van es', 'ebrahimi', 'correa', 'auto'}, optional
The method used to estimate the differential entropy from the sample.
Default is ``'auto'``. See Notes for more information.
Returns
-------
entropy : float
The calculated differential entropy.
Notes
-----
This function will converge to the true differential entropy in the limit
.. math::
n \to \infty, \quad m \to \infty, \quad \frac{m}{n} \to 0
The optimal choice of ``window_length`` for a given sample size depends on
the (unknown) distribution. Typically, the smoother the density of the
distribution, the larger the optimal value of ``window_length`` [1]_.
The following options are available for the `method` parameter.
* ``'vasicek'`` uses the estimator presented in [1]_. This is
one of the first and most influential estimators of differential entropy.
* ``'van es'`` uses the bias-corrected estimator presented in [3]_, which
is not only consistent but, under some conditions, asymptotically normal.
* ``'ebrahimi'`` uses an estimator presented in [4]_, which was shown
in simulation to have smaller bias and mean squared error than
the Vasicek estimator.
* ``'correa'`` uses the estimator presented in [5]_ based on local linear
regression. In a simulation study, it had consistently smaller mean
square error than the Vasiceck estimator, but it is more expensive to
compute.
* ``'auto'`` selects the method automatically (default). Currently,
this selects ``'van es'`` for very small samples (<10), ``'ebrahimi'``
for moderate sample sizes (11-1000), and ``'vasicek'`` for larger
samples, but this behavior is subject to change in future versions.
All estimators are implemented as described in [6]_.
References
----------
.. [1] Vasicek, O. (1976). A test for normality based on sample entropy.
Journal of the Royal Statistical Society:
Series B (Methodological), 38(1), 54-59.
.. [2] Crzcgorzewski, P., & Wirczorkowski, R. (1999). Entropy-based
goodness-of-fit test for exponentiality. Communications in
Statistics-Theory and Methods, 28(5), 1183-1202.
.. [3] Van Es, B. (1992). Estimating functionals related to a density by a
class of statistics based on spacings. Scandinavian Journal of
Statistics, 61-72.
.. [4] Ebrahimi, N., Pflughoeft, K., & Soofi, E. S. (1994). Two measures
of sample entropy. Statistics & Probability Letters, 20(3), 225-234.
.. [5] Correa, J. C. (1995). A new estimator of entropy. Communications
in Statistics-Theory and Methods, 24(10), 2439-2449.
.. [6] Noughabi, H. A. (2015). Entropy Estimation Using Numerical Methods.
Annals of Data Science, 2(2), 231-241.
https://link.springer.com/article/10.1007/s40745-015-0045-9
Examples
--------
>>> import numpy as np
>>> from scipy.stats import differential_entropy, norm
Entropy of a standard normal distribution:
>>> rng = np.random.default_rng()
>>> values = rng.standard_normal(100)
>>> differential_entropy(values)
1.3407817436640392
Compare with the true entropy:
>>> float(norm.entropy())
1.4189385332046727
For several sample sizes between 5 and 1000, compare the accuracy of
the ``'vasicek'``, ``'van es'``, and ``'ebrahimi'`` methods. Specifically,
compare the root mean squared error (over 1000 trials) between the estimate
and the true differential entropy of the distribution.
>>> from scipy import stats
>>> import matplotlib.pyplot as plt
>>>
>>>
>>> def rmse(res, expected):
... '''Root mean squared error'''
... return np.sqrt(np.mean((res - expected)**2))
>>>
>>>
>>> a, b = np.log10(5), np.log10(1000)
>>> ns = np.round(np.logspace(a, b, 10)).astype(int)
>>> reps = 1000 # number of repetitions for each sample size
>>> expected = stats.expon.entropy()
>>>
>>> method_errors = {'vasicek': [], 'van es': [], 'ebrahimi': []}
>>> for method in method_errors:
... for n in ns:
... rvs = stats.expon.rvs(size=(reps, n), random_state=rng)
... res = stats.differential_entropy(rvs, method=method, axis=-1)
... error = rmse(res, expected)
... method_errors[method].append(error)
>>>
>>> for method, errors in method_errors.items():
... plt.loglog(ns, errors, label=method)
>>>
>>> plt.legend()
>>> plt.xlabel('sample size')
>>> plt.ylabel('RMSE (1000 trials)')
>>> plt.title('Entropy Estimator Error (Exponential Distribution)')
���Ng �?� zWindow length (z7) must be positive and less than half the sample size (z).r r r )�vasicek�van es�correa�ebrahimir$ z`method` must be one of r$ �
r. i� r0 r- )r r �moveaxis�shape�math�floor�sqrtr �sort�_vasicek_entropy�_van_es_entropy�_correa_entropy�_ebrahimi_entropy�lower�setr )
r' r% r
r r& �n� sorted_data�methods�message�ress
r" r r � s� � �` �Z��
�
�F�
�[���r�
*�
*�F�� �R��A�� � �
�4�9�Q�<�<�#�#5�6�6�
� ��M�!� %� %� %� %�A� %� %� %� %��
0�m�
0�
0�*+�
0�
0�
0�
�
�
�
��D�A�I�I��F�G�G�G��'�&�r�*�*�*�K�*�(�(�,�'� )� )�G�
�\�\�^�^�F�
�W� � �;�S��\�\�;�;����!�!�!�
���� ��7�7� �F�F�
�$�Y�Y� �F�F� �F�
�'�&�/�+�}�
5�
5�C� �� �r�v�d�|�|��� �Jr# c �� � t j | j � � }||d< t j | ddgf |� � }t j | ddgf |� � }t j || |fd�� � S )z9Pad the data for computing the rolling window difference.r+ .r r )r �arrayr3 � broadcast_to� concatenate)�X�mr3 �Xl�Xrs r" �_pad_along_last_axisrK Q so � �
�H�Q�W� � �E��E�"�I� � ��3���8��e� ,� ,�B� � ��3���9� �u� -� -�B�
�>�2�q�"�+�B� /� /� /�/r# c �� � | j d }t | |� � } | dd|z d�f | ddd|z �f z
}t j |d|z z |z � � }t j |d�� � S )z:Compute the Vasicek estimator as described in [6] Eq. 1.3.r+ .r, N���r )r3 rK r r �mean)rG rH r> � differences�logss r" r8 r8 [ sx � � ��� �A� �Q��"�"�A��C��Q����K�.�1�S�)�B��F�)�^�#4�4�K�
�6�!�Q�q�S�'�K�'� (� (�D�
�7�4�b� !� !� !�!r# c � � | j d }| d|d�f | dd| �f z
}d||z
z t j t j |dz |z |z � � d�� � z }t j ||dz � � }|t j d|z � � z t j |� � z t j |dz � � z
S )z1Compute the van Es estimator as described in [6].r+ .N� r )r3 r r r �arange)rG rH r> �
difference�term1�ks r" r9 r9 d s� � �
��� �A��3����7��a��S�q�b�S��k�)�J�
�q��s�G�b�f�R�V�Q�q�S�!�G�j�$8� 9� 9��C�C�C� C�E�
� �!�Q�q�S���A� �2�6�!�A�#�;�;� ������ *�R�V�A�a�C�[�[� 8�8r# c � � | j d }t | |� � } | dd|z d�f | ddd|z �f z
}t j d|dz � � � t
� � }t j |� � dz }d|||k dz
|z z |||k < d|||||z
dz k z
|z z ||||z
dz k < t j ||z ||z z � � }t j |d�� � S )z3Compute the Ebrahimi estimator as described in [6].r+ .r, NrM rR r ) r3 rK r rS �astype�float� ones_liker rN )rG rH r> rO �i�cirP s r" r; r; o s � �
��� �A� �Q��"�"�A��C��Q����K�.�1�S�)�B��F�)�^�#4�4�K�
� �!�Q�q�S��� � ��'�'�A� � �a� � �� �B��a��Q��i�!�m�Q�&�&�B�q�A�v�J��a�!�A��1��Q��J�-� /�� 2�2�B�q�A��E�A�I�~��
�6�!�k�/�R�!�V�,� -� -�D�
�7�4�b� !� !� !�!r# c �� � | j d }t | |� � } t j d|dz � � }t j | |dz � � dd�df }||z }||z dz
}t j | d|f dd�� � }| d|f |z
}t j ||z d�� � } |t j |d z d�� � z }
t j t j | |
z � � d�� � S )
z1Compute the Correa estimator as described in [6].r+ rR N.rM Tr r r, )r3 rK r rS rN r r ) rG rH r> r[ �dj�j�j0�XibarrT �num�dens r" r: r: � s� � �
��� �A� �Q��"�"�A�
� �!�Q�q�S���A� ��A�2�q��s� � �A�A�A�t�G� $�B� �B��A�
�Q����B� �G�A�c�2�g�J�R�$� 7� 7� 7�E��3��7��e�#�J�
�&��B� �R�
(�
(�
(�C�
�B�F�:�q�=�r� *� *� *�
*�C� �G�B�F�3�s�7�O�O�"� -� -� -� -�-r# )NNr )
r r
r r r
r r r r r ) r' r
r% r( r
r r r r& r) r r )�__doc__�
__future__r r4 �numpyr �scipyr �typingr r �__all__r r rK r8 r9 r; r: � r# r" �<module>rk s ��� � #� "� "� "� "� "� � � � � � � � � � � � � � � "� "� "� "� "� "� "� "� �,�
-�� 15�$(��A
� A
� A
� A
� A
�N $(� ���
y� y� y� y� y� y�x0� 0� 0�"� "� "�9� 9� 9�"� "� "�".� .� .� .� .r#