Package weka.experiment

# Class PairedTTester

java.lang.Object
weka.experiment.PairedTTester

All Implemented Interfaces:
  `Serializable`, `OptionHandler`, `RevisionHandler`, `Tester`

Direct Known Subclasses:
  `PairedCorrectedTTester`

public class PairedTTester extends Object implements OptionHandler, Tester, RevisionHandler
Calculates T-Test statistics on data stored in a set of instances.

Valid options are:

     -D <index,index2-index4,...>
      Specify list of columns that specify a unique
      dataset.
      First and last are valid indexes. (default none)

     -R <index>
      Set the index of the column containing the run number

     -F <index>
      Set the index of the column containing the fold number

     -G <index1,index2-index4,...>
      Specify list of columns that specify a unique
      'result generator' (eg: classifier name and options).
      First and last are valid indexes. (default none)

     -S <significance level>
      Set the significance level for comparisons (default 0.05)

     -V
      Show standard deviations

     -L
      Produce table comparisons in Latex table format

     -csv
      Produce table comparisons in CSV table format

     -html
      Produce table comparisons in HTML table format

     -significance
      Produce table comparisons with only the significance values

     -gnuplot
      Produce table comparisons output suitable for GNUPlot

Version:
  $Revision: 11542 $
Author:
  Len Trigg (trigg@cs.waikato.ac.nz)
See Also:
  - Serialized Form

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `PairedTTester()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `assign(Tester tester)`
  retrieves all the settings from the given Tester
  `PairedStats`
  `calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn)`
  Computes a paired t-test comparison for a specified dataset between two resultsets.
  `boolean`
  `displayResultset(int index)`
  Checks whether the resultset with the given index shall be displayed.
  `Range`
  `getDatasetKeyColumns()`
  Get the value of DatasetKeyColumns.
  `int[]`
  `getDisplayedResultsets()`
  Gets the indices of the the datasets that are displayed (if `null` then all are displayed).
  `String`
  `getDisplayName()`
  returns the name of the tester
  `int`
  `getFoldColumn()`
  Get the value of FoldColumn.
  `Instances`
  `getInstances()`
  Get the value of Instances.
  `int`
  `getNumDatasets()`
  Gets the number of datasets in the resultsets
  `int`
  `getNumResultsets()`
  Gets the number of resultsets in the data.
  `String[]`
  `getOptions()`
  Gets current settings of the PairedTTester.
  `ResultMatrix`
  `getResultMatrix()`
  Gets the instance that produces the output.
  `Range`
  `getResultsetKeyColumns()`
  Get the value of ResultsetKeyColumns.
  `String`
  `getResultsetName(int index)`
  Gets a string descriptive of the specified resultset.
  `String`
  `getRevision()`
  Returns the revision string.
  `int`
  `getRunColumn()`
  Get the value of RunColumn.
  `boolean`
  `getShowStdDevs()`
  Returns true if standard deviations have been requested.
  `double`
  `getSignificanceLevel()`
  Get the value of SignificanceLevel.
  `int`
  `getSortColumn()`
  Returns the column to sort on, -1 means the default sorting.
  `String`
  `getSortColumnName()`
  Returns the name of the column to sort on.
  `String`
  `getToolTipText()`
  returns a string that is displayed as tooltip on the "perform test" button in the experimenter
  `String`
  `header(int comparisonColumn)`
  Creates a "header" string describing the current resultsets.
  `Enumeration<Option>`
  `listOptions()`
  Lists options understood by this object.
  `static void`
  `main(String[] args)`
  Test the class from the command line.
  `String`
  `multiResultsetFull(int baseResultset, int comparisonColumn)`
  Creates a comparison table where a base resultset is compared to the other resultsets.
  `String`
  `multiResultsetRanking(int comparisonColumn)`
  returns a ranking of the resultsets
  `String`
  `multiResultsetSummary(int comparisonColumn)`
  Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
  `int[][]`
  `multiResultsetWins(int comparisonColumn, int[][] nonSigWin)`
  Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
  `String`
  `resultsetKey()`
  Creates a key that maps resultset numbers to their descriptions.
  `void`
  `setDatasetKeyColumns(Range newDatasetKeyColumns)`
  Set the value of DatasetKeyColumns.
  `void`
  `setDisplayedResultsets(int[] cols)`
  Sets the indicies of the datasets to display (`null` means all).
  `void`
  `setFoldColumn(int newFoldColumn)`
  Set the value of FoldColumn.
  `void`
  `setInstances(Instances newInstances)`
  Set the value of Instances.
  `void`
  `setOptions(String[] options)`
  Parses a given list of options.
  `void`
  `setResultMatrix(ResultMatrix matrix)`
  Sets the matrix to use to produce the output.
  `void`
  `setResultsetKeyColumns(Range newResultsetKeyColumns)`
  Set the value of ResultsetKeyColumns.
  `void`
  `setRunColumn(int newRunColumn)`
  Set the value of RunColumn.
  `void`
  `setShowStdDevs(boolean s)`
  Set whether standard deviations are displayed or not.
  `void`
  `setSignificanceLevel(double newSignificanceLevel)`
  Set the value of SignificanceLevel.
  `void`
  `setSortColumn(int newSortColumn)`
  Set the column to sort on, -1 means the default sorting.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### PairedTTester

    public PairedTTester()

- ## Method Details

  - ### setResultMatrix

    public void setResultMatrix(ResultMatrix matrix)
    Sets the matrix to use to produce the output.
    Specified by:
      `setResultMatrix` in interface `Tester`
    Parameters:
      `matrix` - the instance to use to produce the output
    See Also:
      - `ResultMatrix`

  - ### getResultMatrix

    public ResultMatrix getResultMatrix()
    Gets the instance that produces the output.
    Specified by:
      `getResultMatrix` in interface `Tester`
    Returns:
      the instance to produce the output

  - ### setShowStdDevs

    public void setShowStdDevs(boolean s)
    Set whether standard deviations are displayed or not.
    Specified by:
      `setShowStdDevs` in interface `Tester`
    Parameters:
      `s` - true if standard deviations are to be displayed

  - ### getShowStdDevs

    public boolean getShowStdDevs()
    Returns true if standard deviations have been requested.
    Specified by:
      `getShowStdDevs` in interface `Tester`
    Returns:
      true if standard deviations are to be displayed.

  - ### getNumDatasets

    public int getNumDatasets()
    Gets the number of datasets in the resultsets
    Specified by:
      `getNumDatasets` in interface `Tester`
    Returns:
      the number of datasets in the resultsets

  - ### getNumResultsets

    public int getNumResultsets()
    Gets the number of resultsets in the data.
    Specified by:
      `getNumResultsets` in interface `Tester`
    Returns:
      the number of resultsets in the data

  - ### getResultsetName

    public String getResultsetName(int index)
    Gets a string descriptive of the specified resultset.
    Specified by:
      `getResultsetName` in interface `Tester`
    Parameters:
      `index` - the index of the resultset
    Returns:
      a descriptive string for the resultset

  - ### displayResultset

    public boolean displayResultset(int index)
    Checks whether the resultset with the given index shall be displayed.
    Specified by:
      `displayResultset` in interface `Tester`
    Parameters:
      `index` - the index of the resultset to check whether it shall be displayed
    Returns:
      whether the specified resultset is displayed

  - ### calculateStatistics

    public PairedStats calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn) throws Exception
    Computes a paired t-test comparison for a specified dataset between two resultsets.
    Specified by:
      `calculateStatistics` in interface `Tester`
    Parameters:
      `datasetSpecifier` - the dataset specifier
      `resultset1Index` - the index of the first resultset
      `resultset2Index` - the index of the second resultset
      `comparisonColumn` - the column containing values to compare
    Returns:
      the results of the paired comparison
    Throws:
      `Exception` - if an error occurs

  - ### resultsetKey

    public String resultsetKey()
    Creates a key that maps resultset numbers to their descriptions.
    Specified by:
      `resultsetKey` in interface `Tester`
    Returns:
      a value of type 'String'

  - ### header

    public String header(int comparisonColumn)
    Creates a "header" string describing the current resultsets.
    Specified by:
      `header` in interface `Tester`
    Parameters:
      `comparisonColumn` - a value of type 'int'
    Returns:
      a value of type 'String'

  - ### multiResultsetWins

    public int[][] multiResultsetWins(int comparisonColumn, int[][] nonSigWin) throws Exception
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
    Specified by:
      `multiResultsetWins` in interface `Tester`
    Parameters:
      `comparisonColumn` - the index of the comparison column
      `nonSigWin` - for storing the non-significant wins
    Returns:
      a 2d array where element [i][j] is the number of times resultset j performed significantly better than resultset i.
    Throws:
      `Exception` - if an error occurs

  - ### multiResultsetSummary

    public String multiResultsetSummary(int comparisonColumn) throws Exception
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other. The results are summarized in a table.
    Specified by:
      `multiResultsetSummary` in interface `Tester`
    Parameters:
      `comparisonColumn` - the index of the comparison column
    Returns:
      the results in a string
    Throws:
      `Exception` - if an error occurs

  - ### multiResultsetRanking

    public String multiResultsetRanking(int comparisonColumn) throws Exception
    returns a ranking of the resultsets
    Specified by:
      `multiResultsetRanking` in interface `Tester`
    Parameters:
      `comparisonColumn` - the column to compare with
    Returns:
      the ranking
    Throws:
      `Exception` - if something goes wrong

  - ### multiResultsetFull

    public String multiResultsetFull(int baseResultset, int comparisonColumn) throws Exception
    Creates a comparison table where a base resultset is compared to the other resultsets. Results are presented for every dataset.
    Specified by:
      `multiResultsetFull` in interface `Tester`
    Parameters:
      `baseResultset` - the index of the base resultset
      `comparisonColumn` - the index of the column to compare over
    Returns:
      the comparison table string
    Throws:
      `Exception` - if an error occurs

  - ### listOptions

    public Enumeration<Option> listOptions()
    Lists options understood by this object.
    Specified by:
      `listOptions` in interface `OptionHandler`
    Returns:
      an enumeration of Options.

  - ### setOptions

    public void setOptions(String[] options) throws Exception
    Parses a given list of options.

    Valid options are:

         -D <index,index2-index4,...>
          Specify list of columns that specify a unique
          dataset.
          First and last are valid indexes. (default none)

         -R <index>
          Set the index of the column containing the run number

         -F <index>
          Set the index of the column containing the fold number

         -G <index1,index2-index4,...>
          Specify list of columns that specify a unique
          'result generator' (eg: classifier name and options).
          First and last are valid indexes. (default none)

         -S <significance level>
          Set the significance level for comparisons (default 0.05)

         -V
          Show standard deviations

         -L
          Produce table comparisons in Latex table format

         -csv
          Produce table comparisons in CSV table format

         -html
          Produce table comparisons in HTML table format

         -significance
          Produce table comparisons with only the significance values

         -gnuplot
          Produce table comparisons output suitable for GNUPlot

    Specified by:
      `setOptions` in interface `OptionHandler`
    Parameters:
      `options` - an array containing options to set.
    Throws:
      `Exception` - if invalid options are given

  - ### getOptions

    public String[] getOptions()
    Gets current settings of the PairedTTester.
    Specified by:
      `getOptions` in interface `OptionHandler`
    Returns:
      an array of strings containing current options.

  - ### getResultsetKeyColumns

    public Range getResultsetKeyColumns()
    Get the value of ResultsetKeyColumns.
    Specified by:
      `getResultsetKeyColumns` in interface `Tester`
    Returns:
      Value of ResultsetKeyColumns.

  - ### setResultsetKeyColumns

    public void setResultsetKeyColumns(Range newResultsetKeyColumns)
    Set the value of ResultsetKeyColumns.
    Specified by:
      `setResultsetKeyColumns` in interface `Tester`
    Parameters:
      `newResultsetKeyColumns` - Value to assign to ResultsetKeyColumns.

  - ### getDisplayedResultsets

    public int[] getDisplayedResultsets()
    Gets the indices of the the datasets that are displayed (if `null` then all are displayed). The base is always displayed.
    Specified by:
      `getDisplayedResultsets` in interface `Tester`
    Returns:
      the indices of the datasets to display

  - ### setDisplayedResultsets

    public void setDisplayedResultsets(int[] cols)
    Sets the indicies of the datasets to display (`null` means all). The base is always displayed.
    Specified by:
      `setDisplayedResultsets` in interface `Tester`
    Parameters:
      `cols` - the indices of the datasets to display

  - ### getSignificanceLevel

    public double getSignificanceLevel()
    Get the value of SignificanceLevel.
    Specified by:
      `getSignificanceLevel` in interface `Tester`
    Returns:
      Value of SignificanceLevel.

  - ### setSignificanceLevel

    public void setSignificanceLevel(double newSignificanceLevel)
    Set the value of SignificanceLevel.
    Specified by:
      `setSignificanceLevel` in interface `Tester`
    Parameters:
      `newSignificanceLevel` - Value to assign to SignificanceLevel.

  - ### getDatasetKeyColumns

    public Range getDatasetKeyColumns()
    Get the value of DatasetKeyColumns.
    Specified by:
      `getDatasetKeyColumns` in interface `Tester`
    Returns:
      Value of DatasetKeyColumns.

  - ### setDatasetKeyColumns

    public void setDatasetKeyColumns(Range newDatasetKeyColumns)
    Set the value of DatasetKeyColumns.
    Specified by:
      `setDatasetKeyColumns` in interface `Tester`
    Parameters:
      `newDatasetKeyColumns` - Value to assign to DatasetKeyColumns.

  - ### getRunColumn

    public int getRunColumn()
    Get the value of RunColumn.
    Specified by:
      `getRunColumn` in interface `Tester`
    Returns:
      Value of RunColumn.

  - ### setRunColumn

    public void setRunColumn(int newRunColumn)
    Set the value of RunColumn.
    Specified by:
      `setRunColumn` in interface `Tester`
    Parameters:
      `newRunColumn` - Value to assign to RunColumn.

  - ### getFoldColumn

    public int getFoldColumn()
    Get the value of FoldColumn.
    Specified by:
      `getFoldColumn` in interface `Tester`
    Returns:
      Value of FoldColumn.

  - ### setFoldColumn

    public void setFoldColumn(int newFoldColumn)
    Set the value of FoldColumn.
    Specified by:
      `setFoldColumn` in interface `Tester`
    Parameters:
      `newFoldColumn` - Value to assign to FoldColumn.

  - ### getSortColumnName

    public String getSortColumnName()
    Returns the name of the column to sort on.
    Specified by:
      `getSortColumnName` in interface `Tester`
    Returns:
      the name of the column to sort on.

  - ### getSortColumn

    public int getSortColumn()
    Returns the column to sort on, -1 means the default sorting.
    Specified by:
      `getSortColumn` in interface `Tester`
    Returns:
      the column to sort on.

  - ### setSortColumn

    public void setSortColumn(int newSortColumn)
    Set the column to sort on, -1 means the default sorting.
    Specified by:
      `setSortColumn` in interface `Tester`
    Parameters:
      `newSortColumn` - the new sort column.

  - ### getInstances

    public Instances getInstances()
    Get the value of Instances.
    Specified by:
      `getInstances` in interface `Tester`
    Returns:
      Value of Instances.

  - ### setInstances

    public void setInstances(Instances newInstances)
    Set the value of Instances.
    Specified by:
      `setInstances` in interface `Tester`
    Parameters:
      `newInstances` - Value to assign to Instances.

  - ### assign

    public void assign(Tester tester)
    retrieves all the settings from the given Tester
    Specified by:
      `assign` in interface `Tester`
    Parameters:
      `tester` - the Tester to get the settings from

  - ### getToolTipText

    public String getToolTipText()
    returns a string that is displayed as tooltip on the "perform test" button in the experimenter
    Specified by:
      `getToolTipText` in interface `Tester`
    Returns:
      the tool tip

  - ### getDisplayName

    public String getDisplayName()
    returns the name of the tester
    Specified by:
      `getDisplayName` in interface `Tester`
    Returns:
      the display name

  - ### getRevision

    public String getRevision()
    Returns the revision string.
    Specified by:
      `getRevision` in interface `RevisionHandler`
    Returns:
      the revision

  - ### main

    public static void main(String[] args)
    Test the class from the command line.
    Parameters:
      `args` - contains options for the instance ttests