Programming Collective Intelligence is easy to read, small but concise, and its only major flaw is the title; and that is because it is misleading. The book touches quite heavily on using collective information and social site APIs, but what it is really about is data mining. It may not be a flaw with the majority of readers, but personally I wouldn't care about the collective, the Facebook API or anything like that, but I was really interested in the different ways to analyse data. In that sense, this book can be taken as a reference guide on data mining.
Each algorithm and idea is accompanied by Python sources. I personally dislike Python as a language, but the author afirms he chose it intentionally because the algorithms look clear and the source is small, with its purpose unhindred by many language artefacts. The book was so interesting, though, that I plan (if I ever find the time :( ) to take all the examples and do them in C#, then place them on Github.
The book covers classification and feature extraction, supervised and unsupervised algorithms, filtering and discovery and it also has exercises at the end of each chapter. Here is a short list:
Making Recommendations - about the way one can use data from user preferences in order to create recommendations. Distance metrics and finding similar items to the ones we like or people with similar tastes.
Discovering Groups - about classifying data into different groups. Supervised and unsupervised methods are described, hierarchical clustering, dendograms, column clustering, K-Means clustering and diferent methods of visualisation.
Searching and Ranking - it basically explains step by step how to make a search engine. Word frequency, word distance, location of a document, counting methods, artifical neural networks, the Google PageRank algorithm, extraction of information from link text, and learning from user clicks can be found in this chapter.
Optimization - simulated annealing, hill climbing, genetic algorithms are described and exampled here. The chapter talks about optimizing problems like travel schedules and the example uses data from Kayak.
Document Filtering - a chapter about filtering documents based on preferences or getting rid of spam. You can find here Bayesian filtering and the Fisher method.
Decision Trees - a very interesting method of splitting information items into groups that have a hierarchical connection between them. The examples use the Zillow API
Advanced Classification - Kernel Methods and Support Vector Machines. This is a great chapter and it show some pretty cool uses of data mining using the Facebook API
Finding Independent Features - reviews Bayesian classification and clustering, then proposes Non-Negative Matrix Factorisation, a method invented circa the late 90s, a powerful algorithm which uses matrix algebra to find features in a data set
Evolving Intelligence - bingo! Genetic Programming made easy. Really cool.
Algorithm Summary, Third Party Libraries and Mathematical Formulas - if you had any doubts you can use this book as a data mining reference book, the last three chapters eliminate them. An even more concise summary of the methods explained in the book, listing every math formula and obscure library used in the book
Conclusion: I really loved the book and I can hardly wait to take it apart with a computer in hand.
The scenario is as follows: you want to attach a collection to an element and fill it via XAML. The objects that you add to the collection should inherit the element DataContext.
One issue arising is that if you use an attached property for a collection you cannot use a default value in the metadata (since it would be used for all the instances of the property) and that if you don't instantiate it, any attempt to add stuff to it via the markup will result in a collection null exception. If this were a mere dependency property from inside the control, the solution would have been to create the collection in the element constructor. Since it is an attached one, we don't have this mechanism available.
Another solution would be to instantiate it in the property getter, like a normal property with a backing field, but in the case of XAML, the dependency and attached properties are accessed directly via the DependencyObject.GetValue method, bypassing the parent class property getter entirely. The only solution to force going through the getter is to register the attached property with a name string that is different from the convention.
Above we are registering an attached property of type ObservableCollection<MyItem> ingeniously named MyAttachedCollection from a class named ParentClass. The convention says we whould register it using a name that is the DependencyProperty name without the ending "Property". I, however, have added an "Internal" string, which forces the property system to go through the getter! Here is the getter:
Voila! This allows you to instantiate in the getter the troublesome collection.
Now, the other request in the scenario is to inherit the DataContext from the element the collection is attached to. To do that we at least need that the collection and the items in it to be DependencyObjects in order to have a DataContext, but we are going even further: we need to define them as Freezable! The Freezable abstract class requires you to implement a CreateInstanceCore method. Just make the MyItem class inherit Freezable and throw a NotImplementedException in the CreateInstanceCore method. For the collection itself, we need to inherit from another useful class: FreezableCollection<T>. After replacing ObservableCollection with FreezableCollection, the collection and the items have the same DataContext property as the element the property is attached to. An interesting fact here is that Freezable does NOT inherit from FrameworkElement, but it is a DependencyObject and the DataContext property does propagate down through the element tree.
One last interesting thing: the FreezableCollection class implements INotifyCollectionChanged, but explicitly! In order to use the CollectionChanged event you need to cast it in code to INotifyCollectionChanged.
It's been a while since I've last posted some music. This is not the kind of music I would listen to, mostly, but it is all Japanese music, 5 seconds of each band, a lot of bands. I thought it was a nice overview of a type of music I know almost nothing about. Enjoy!
What it basically says is that the Internet is open only as the huge private companies that control it are willing to allow this openness. Governments and companies alike can pressure key points in order to control the spread of information. The laws (which set of laws, btw?) are vague, allowing a limbo in which only the powerful have the upper hand. Two services we take for granted, like DNS and the newly found fab cloud computing are easily attacked or pressured into blocking access or revealing information.
But what I found even more troubling is the way this challenge of Wikileaks (because what else can you call wearing the underpants of the biggest bully as your flag) has been answered so mindlessly by the US. The government that is trying to get its hands and make an example out of Gary McKinnon had his most secret documents openly exposed, making it look vulnerable, weak. Its response is nothing less than angry mindless rage: denial of service attacks on the Wikileaks DNS, harrassment of anybody supporting financially or technically the Wikileaks organization, very convenient rape charges against Julian Assange and so on. This is the behemoth that, behind nice faces like Obama's, does stuff like Guantanamo and has that huge inertia that would almost push humanity to extinction during the Cold War: "you mess with me, I mess with you".
However, this is a battle that any government has already lost. Short of a global appocalypse, the rabbit is out of the hat and the Wikileaks model will live on, regardless of who runs it and what structure it has. People have been shown to actually make a difference. All that media and movie onslaught of images of the evil government that can kill anybody at will and make everything dissappear has been proven a myth. They are not invulnerable. Even worse, they can't handle the stress, they are sore losers. They lost information, but also face and honor. And the funny thing is, they did it to themselves.
I didn't know about this until today, but the technique itself is so simple that it must have worked on every browser from the first introduction of the :visited CSS selector. Think about it: in your site you have a css style that colors the visited links with a specific color. Then you place in your page a list of links to other sites (maybe in a hidden container). Using Javascript, you inquire the computed style for each of the links. Taadaa! You now know the links in your list that the visitor has recently visited.
You can download a PHP demo of the technique from here. Ain't this neat?
Update: I since found another interesting article, showing how a site could (on certain browsers that are not Internet Explorer >:) )see if you are logged on to another. Here is the link.
Ok, so I had to say something about Julian Assange and Wikileaks. I will not speculate on the probably bogus rape arrest warrant for Assange (oh, it seems I did :) ), but instead focus on one of his quotes: "If governments would prefer to not have such information surface they have two choices: don't engage in wars that even their own military employees find reprehensible, and don't rely on secrecy as a method of governance.". Sounds like the old "Nothing to hide, nothing to fear" thing, used by so many people with power to justify their actions. Well, payback's a bitch, isn't it?
Before you go on, let me summarize this long post for you: while it is possible to clone a control template in order to change just some things in it, it is a difficult and error prone process. The code at the end of the post is a proof of concept thing, which works for simple scenarios, but needs additional work for complicated controls.
I was exploring the option of not overwriting the ControlTemplate of a WPF Control when I try adding stuff to it. Instead, I tried to get the ControlTemplate and manipulate it before putting it back. It is not as easy as it seems. Even more, people stack over each other to advise everybody not to do it. I am not saying it is an easy option, so my advice is to try other alternatives, if you have them, but the idea is: it can be done!
Let's take it step by step. In order to get the control template we should get it as soon as it is available, but perhaps before applying it. One could override OnApplyTemplate and do it there or, as it is my case (trying to do it via attached properties and lacking an ApplyingTemplate event), do it once when the control is initializing. The control template is easily obtained via the Template property. If you try to change anything in it, though, you will get an exception, because the template is sealed. So the only option is to clone it, change stuff in it, then set the control template to that clone.
The template is of type ControlTemplate, but it doesn't seem to contain much. It has Resources and Triggers properties, also a VisualTree property and a LoadContent method. There is also a Template property in the ControlTemplate class... try to set it and a null exception will be thrown, so forget it. The first two are easy to use, just iterate through the collections. VisualTree is of the weird and undocumented type FrameworkElementFactory, while LoadContent is a method that returns a DependencyObject.
Well, the idea is that LoadContent will return the content of the template which you should use to set the VisualTree property, but the process of getting a DependencyObject and getting a FrameworkElementFactory tree is not simple.
First things first: get a new ControlTemplate. Its contructor gets a Type parameter which we take from the TargetType property of the original template. We then add any resources from the original template to the resources of the new one. Next step is to take the content, using LoadContent, which will get us the first child of the element tree. In order to traverse it we will use the VisualTreeHelper static class which exposes the GetChildrenCount and GetChild methods.
The next step is to create a FrameworkElementFactory. It has a constructor which receives a Type and another which gets a Type and a name string. We will use the first, since the Name can be set afterwards. The type we get from the type of the DependencyObject returned by LoadContent. The VisualTree of the new control template will have to be this new factory object, but it also needs all the properties of the original object as well as all its children.
In order to get the dependency and attached properties of each element we will use the MarkupWriter.GetMarkupObjectFor method, which returns a MarkupObject. Each of its Properties will have a DependencyProperty property which will give us the properties. However, the value of the property is not so easy to get. If we use GetValue, any binding or markup extensions will be evaluated and probably give wrong results (since the control has not been initialized yet). Using ReadLocalValue brings us pretty close, only that for certain objects like Binding we don't get a BindingBase object, but a BindingExpressionBase. We need to cast the value we get to BindingExpressionBase and TemplateBindingExpression and get to the underlying binding object.
Now that we've got the properties and the correct values, we use the factory SetValue method to set it. A special case is Name which must be set directly to the Name property. We use AppendChild to add a factory to a parent factory.
The last step is to get the Triggers from the original template and copy it in the new one. Now Seal it and you have yourself a clone. Not sure how one would manipulate the template to get a usable and maintainable template manipulation, but this is how you start.
I know you are suckers for code, so here it is:
Update:Actually the collapsed code below doesn't work except for the simplest of templates. There are several reasons for it and I will explore them below.
The first problem I found was dependency properties registered as read only that could only be set from XAML, like VisualStateManager.VisualStateGroups. When trying to use the FrameworkElementFactory SetValue method it would throw an error. Funny enough, the only reason that happened is because said method is checking if the property is read only and throws an exception. I had to use reflection to circumvent this, and it worked, albeit really ugly.
The second problem was more basic. Not every property is a dependency property. Such a simple property is Grid.ColumnDefinitions! Not only it is not a dependency property, but it is also read only. So I had to find another mechanism to fix this. At this point you probably realise this method is not a good one to employ, but if you are really desperate (or stubborn, like me) there is a way. The solution I found is to save all the properties that I need to set into a list and then set them in a RoutedEventHandler invoked from the Loaded event!
And if this is not enough, simply setting the value from the template in the control doesn't always work. In the generated template control the ColumnDefinition objects are already in the ColumnDefinitionCollection of the control. Adding them to a control that the factory generates results in an error. What I did here is a simple value=XamlReader.Parse(XamlWriter.Save(value)).
In other cases, like the Border.Child property, it must be completely ignored! So a list of properties to be ignored is needed.
Conclusion: Some improvements have been done in the code, but it's a little larger than before. The complicated way in which this works makes it cumbersome to be used, and I would not recommend it, but it works and it has extension points where errors with properties can be handled. Here is the new code:
#region Using directives
using System.Collections.Generic; using System.ComponentModel; using System.Windows; using System.Windows.Controls; using System.Windows.Markup;
#endregion
namespace BestPractices { /// <summary> /// Base class for ControlTemplate transforming classes /// </summary> publicabstractclass BaseTemplateTransformer : BaseControlTransformer { #region Public Methods
/// <summary> /// Clones the ControlTemplate of a control and allows for its manipulation /// </summary> /// <param name="control"></param> publicoverridevoid Transform(Control control) { ControlTemplate template = control.Template; if (template == null) { return; } // create new template ControlTemplate newTemplate = new ControlTemplate(template.TargetType); // copy the resources foreach (object key in template.Resources.Keys) { newTemplate.Resources.Add(key, template.Resources[key]); } //get the VisualTree factory from the original template content DependencyObject content = template.LoadContent(); newTemplate.VisualTree = OnGetElementFactory(content); // copy the triggers foreach (TriggerBase trigger in template.Triggers) { newTemplate.Triggers.Add(trigger); } // allow for template manipulation OnBeforeSeal(newTemplate); // seal the template and set it back newTemplate.Seal(); control.Template = newTemplate; }
/// <summary> /// Creates a custom ControlTransformFactory for the content object. /// Override in order to replace elements in the initial template. /// </summary> /// <param name="content"></param> /// <returns></returns> publicvirtual ControlTransformFactory OnGetElementFactory(DependencyObject content) { if (content == null) { returnnull; } // use the object type ControlTransformFactory factory = new ControlTransformFactory(content, this); return factory; }
/// <summary> /// Returns a safe value for setting on the control /// </summary> /// <param name="item">The value from the template</param> /// <returns></returns> publicvirtualobject GetSafeValue(object item) { return getSafeValue((dynamic) item); }
/// <summary> /// Returns true if a property needs to be saved and set when the control loads. /// Defaults to false, except for ColumnDefinitions and RowDefinitions /// </summary> /// <param name="propertyDescriptor"></param> /// <returns></returns> publicvirtualbool MustSetProperty(PropertyDescriptor propertyDescriptor) { return sMustSetProperties.Contains(propertyDescriptor.Name); }
#endregion
#region Protected Methods
/// <summary> /// Allows for the manipulation of a control template /// </summary> /// <param name="newTemplate"></param> protectedvirtualvoid OnBeforeSeal(ControlTemplate newTemplate) { }
#endregion
#region Statics
privatestaticreadonly List<string> sMustSetProperties = new List<string> { "ColumnDefinitions", "RowDefinitions" };
/// <summary> /// Transforms a DependencyObject tree into a FrameworkElementFactory tree /// </summary> /// <param name="content"></param> /// <returns></returns> publicstatic ControlTransformFactory CreateElementFactory(DependencyObject content) { if (content == null) { returnnull; } // use the object type ControlTransformFactory factory = new ControlTransformFactory(content); return factory; }
/// <summary> /// default return the same value /// </summary> /// <param name="value"></param> /// <returns></returns> privatestaticobject getSafeValue(objectvalue) { returnvalue; }
using System; using System.Collections; using System.Collections.Generic; using System.ComponentModel; using System.Reflection; using System.Windows; using System.Windows.Markup.Primitives; using System.Windows.Media;
#endregion
namespace BestPractices { /// <summary> /// FrameworkElementFactory used in control template transformers /// </summary> publicclass ControlTransformFactory : FrameworkElementFactory { #region Nested
/// <summary> /// List of non dependency properties that will be set when the control loads /// </summary> protected List<MarkupProperty> SimpleProperties { get { if (mSimpleProperties == null) { mSimpleProperties = new List<MarkupProperty>(); } return mSimpleProperties; } }
#endregion
#region Constructors
public ControlTransformFactory(DependencyObject content, BaseTemplateTransformer templateTransformer = null) : base(content.GetType()) { mTemplateTransformer = templateTransformer ?? new NoTemplateTransformer(); // set its name string name = content.GetValue(FrameworkElement.NameProperty) asstring; if (!string.IsNullOrWhiteSpace(name)) { Name = name; } // copy the properties foreach (MarkupProperty propertyItem in getProperties(content)) { SetProperty(propertyItem); } // do it recursively int count = VisualTreeHelper.GetChildrenCount(content); for (int i = 0; i < count; i++) { DependencyObject child = VisualTreeHelper.GetChild(content, i); AppendChild(mTemplateTransformer.OnGetElementFactory(child)); } }
#endregion
#region Public Methods
/// <summary> /// SetValue that uses reflection to force DependencyProperties that were registered as read only /// </summary> /// <param name="dependencyProperty"></param> /// <param name="value"></param> /// <param name="forceSetReadOnly"></param> publicvoid SetValue(DependencyProperty dependencyProperty, objectvalue, bool forceSetReadOnly = false) { if (!forceSetReadOnly) { base.SetValue(dependencyProperty, value); } else { forceSetValue(dependencyProperty, value); } }
/// <summary> /// Sets a value from a MarkupProperty object. /// Dependency properties will be set via SetValue and the others via a Loaded handler on the object /// </summary> /// <param name="propertyItem"></param> publicvoid SetProperty(MarkupProperty propertyItem) { DependencyProperty property = propertyItem.DependencyProperty; if (property == null) { setSimpleProperty(propertyItem); } else { objectvalue = propertyItem.Value; if (value == DependencyProperty.UnsetValue) { return; } SetValue(property, value, property.ReadOnly); } }
#endregion
#region Private Methods
/// <summary> /// Force set value in the factory, even if the property is ReadOnly /// </summary> /// <param name="dp"></param> /// <param name="value"></param> privatevoid forceSetValue(DependencyProperty dp, objectvalue) { object resourceKey = getResourceKey(value); if (resourceKey == null) { updatePropertyValueList(dp, valueis TemplateBindingExtension ? "TemplateBinding" : "Set", value); } else { updatePropertyValueList(dp, "Resource", value); } }
/// <summary> /// Set a property to have its value set at load time /// </summary> /// <param name="markupProperty"></param> privatevoid setSimpleProperty(MarkupProperty markupProperty) { if (markupProperty.PropertyDescriptor == null) { return; } if (!mTemplateTransformer.MustSetProperty(markupProperty.PropertyDescriptor)) { return; } SimpleProperties.Add(markupProperty); if (mLoadHandler == null) { mLoadHandler = new RoutedEventHandler(simplePropertyHandler); AddHandler(FrameworkElement.LoadedEvent, mLoadHandler); } }
/// <summary> /// Handler on the Loaded event of the control /// </summary> /// <param name="sender"></param> /// <param name="args"></param> privatevoid simplePropertyHandler(object sender, RoutedEventArgs args) { foreach (MarkupProperty propertyItem in SimpleProperties) { PropertyDescriptor propertyDescriptor = propertyItem.PropertyDescriptor; if (propertyDescriptor == null || propertyItem.Value == null) { continue; } if (propertyDescriptor.IsReadOnly) { IList list = propertyItem.Value as IList; if (list != null) { IList destinationList = propertyDescriptor.GetValue(sender) as IList; if (destinationList != null) { foreach (object item in list) { destinationList.Add(mTemplateTransformer.GetSafeValue(item)); } } else { //shouldn't happend } } else { // what now? } } else { propertyDescriptor.SetValue(sender, mTemplateTransformer.GetSafeValue(propertyItem.Value)); } } FrameworkElement element = (FrameworkElement) sender; element.RemoveHandler(FrameworkElement.LoadedEvent, mLoadHandler); }
#endregion
#region Statics
static ControlTransformFactory() { // Get the types and methods that will be used in Reflection scenarios sUpdatePropertyValueListProperty = typeof (FrameworkElementFactory).GetMethod("UpdatePropertyValueList", BindingFlags.NonPublic | BindingFlags.Instance); sPropertyValueTypeType = typeof (FrameworkElementFactory).Assembly.GetType("System.Windows.PropertyValueType"); }
privatestaticreadonly MethodInfo sUpdatePropertyValueListProperty; privatestaticreadonly Type sPropertyValueTypeType;
/// <summary> /// get the properties set on a DependencyObject /// </summary> /// <param name="content"></param> /// <returns></returns> privatestatic IEnumerable<MarkupProperty> getProperties(DependencyObject content) { MarkupObject markupObject = MarkupWriter.GetMarkupObjectFor(content); return markupObject.Properties; }
/// <summary> /// Get the ResourceKey property from an object, if it exists /// </summary> /// <param name="target"></param> /// <returns></returns> privatestaticobject getResourceKey(object target) { if (target == null) { returnnull; } Type type = target.GetType(); PropertyInfo property = type.GetProperty("ResourceKey"); if (property == null) { returnnull; } return property.GetValue(target, newobject[] {}); }
I am a .NET programmer living and working in Bucharest, Romania.
Posts are divided into programming and misc.
Check out the icons above on how to contact or chat with me.
Unless otherwise specified, all code or any type of work you find on this blog is under MIT license. While I welcome attribution, I don't require it. Just use anything in any way you see fit. Consider it completely and utterly shared for the lulz.
Programming Collective Intelligence is easy to read, small but concise, and its only major flaw is the title; and that is because it is misleading. The book touches quite heavily on using collective information and social site APIs, but what it is really about is data mining. It may not be a flaw with the majority of readers, but personally I wouldn't care about the collective, the Facebook API or anything like that, but I was really interested in the different ways to analyse data. In that sense, this book can be taken as a reference guide on data mining.