How to Handle Noisy Data in preprocessing of data?

-


Binning method:(one of the method)
  • first sort data and partition into (equi-depth) bins
  • then one can smooth by bin means,  smooth by bin median, smooth by bin boundaries, etc.
Equal-width (distance) partitioning:
  • It divides the range into N intervals of equal size: uniform grid
  • if A and B are the lowest and highest values of the attribute, the width of intervals will be: W = (B-A)/N.
  • The most straightforward
  • But outliers may dominate the presentation
  • Skewed data is not handled well.
Equal-depth (frequency) partitioning:
  • It divides the range into N intervals, each containing approximately same number of samples
  • Good data scaling
  • Managing categorical attributes can be tricky.
Code for binning (if needed we can edit for user input instead of random) : Feel free to comment about mistakes and doubts.

Comments

Post a Comment

Popular posts from this blog

Splitting criteria

-
Image
This package helps to find splitting criteria in C4.5 Information Gain ID3 uses information gain as its attribute selection measure. This measure is based on pioneering work by Claude Shannon on information theory, which studied the value or “information content” of messages. Let node N represent or hold the tuples of partition D. The attribute with the highest information gain is chosen as the splitting attribute for node N. This attribute minimizes the information needed to classify the tuples in the resulting partitions and reflects the least randomness or “impurity” in these partitions. Such an approach minimizes the expected number of tests needed to classify a given tuple and guarantees that a simple (but not necessarily the simplest) tree is found. The expected information needed to classify a tuple in D is given by: where pi is the probability that an arbitrary tuple inDbelongs to classCi and is estimated by |Ci,D|/|D|. A log function to the base 2 is used becaus...
Read more

Embedding the GitHub Gists codes into blogs

-
Image
Create the gist of your codes: Then get the embedding link. Click the HTML link and place the embedded link at the desired position. Code your code snippet appears as magic with git link in the below. Above process works for dynamic themes like Contempo in bloggers, but it fails for other dynamic themes so go for step 5. https://stackoverflow.com/questions/18788724/issues-adding-github-gist-to-my-blogusing-google-blogger/21355714#21355714  is the link where you can find the information to display the codes of Github with Gists in dynamic blogs.
Read more