Thursday, 13 April 2017

Lambdas, LInQ, Javascript and so on

As a .NET developer I am very familiar with LInQ, or Language Integrated Queries, which is a collection of fluent interface methods that deal with querying data from collections. However, so many people outside the .NET ecosystem are not familiar with the concept or they use it as disparate functions in their language of choice. What makes it even more confusing is that the same concept is implemented in other languages with different names. Let me give you an example:
var arr=[1,2,3,4,5,6];
var result=arr
.Where(v=>v%2==0) //get only even values
.Select(v=>v*10) //return their values multiplied with 10
.Aggregate(15,(s,v)=>v+s); //aggregate their value into a sum that starts with a seed of 15
// result should be 15+2*10+4*10+6*10=135

We see here the use of three of these methods:
  • Where - filters the values on a condition
  • Select - changes the values it returns
  • Aggregate - creates an aggregate value using an operation on all the values in the collection

Let me write you the same C# code without using these methods:
var arr=[1,2,3,4,5,6];
var result=15;
foreach(var v in arr) {
if (v%2==0) {
result+=v*10;
}
}

In this case, some people might prefer the second version, but it is only an example. LInQ is not a silver bullet that replaces all loops, only a tool amongst many in a large toolset. Some advantages of using such a method are concise code, better readability, a common API for iterating, filtering and querying collections, etc. For example in the largely used Entity Framework or its previous incarnations such as Linq over SQL, the queries would look the same, but they would be translated into SQL and sent to the database and executed just once. So it would not get a list of thousands of records to filter it in memory, instead it would translate the expression of the function sent to the query into SQL and execute it there. The same sort of operations can be used on streams of data, rather than fixed collections, like in the case of Reactive Extensions.

Some other methods in this set include:
  • First/Last - getting the first or last element in an enumerable that satisfies a boolean condition
  • Skip - ignoring a number of values in a collection
  • Take - returning a number of values in a collection
  • Any/All - returning true if at least one or all of the items satisfy a boolean condition
  • Average/Sum/Min/Max - specific aggregating methods for the elements in the collection
  • OrderBy/OrderByDescending - sorting
  • Count - counting

There are many others, you can look them up here.

Does this system of querying data seem familiar to you? To SQL developers it will feel second nature. In SQL the same result from above would be achieved by using something like:
SELECT 15+SUM(SELECT v*10 FROM table WHERE v%2=0)

Note that other than putting the source of the data in front, LInQ syntax is almost identical.

However, in other languages this sort of data query is called map/reduce and in fact there is a very used programming model called MapReduce that applies in big data processing. In Java, the function that filters data is called filter, the one that alters the values is called map and the one that aggregates data is called reduce. Similar in Javascript. Here is the same code in Javascript:
var arr=[1,2,3,4,5,6];
var result=arr
.filter(v=>v%2==0) //get only even values
.map(v=>v*10) //return their values multiplied with 10
.reduce((s,v)=>v+s,15); //aggregate their value into a sum that starts with a seed of 15
// result should be 15+2*10+4*10+6*10=135
Note that the lambda syntax of writing functions used here is new in ECMA Script version 6. Before you would have to use the function(x) { return [something with x]; } syntax.

In Haxe, the concept is achieved by using the Lambda library and the functions are again named differently: filter for filtering, map for altering and fold for aggregating.

There is another sort of people that would instantly recognize this model of data querying: functional programming people. Indeed, SQL is a functional programming language at its core and the same standard for data querying is used very efficiently in functional programming languages, since they know whether a function is pure or not (has side effects). When only dealing with pure functions, some optimizations can be made on the query by the compiler before anything is even executed. Haskell has the same naming as Haxe (filter, map, fold) for example.

So whenever I get to review other people's code, especially people that have little experience with either SQL or C#, I cringe to see stuff like this:
var max=-1;
for (var i=0; i<arr.length; i++) {
if (max<arr[i]) max=arr[i];
}
In my head this should be simply arr.max(); And considering how easy it is to implement something like this in Javascript, for example, it's a crime for not using it:
Array.prototype.max=function() { return Math.max.apply(null,this); }

Yet there is more to this than my personal preference for reading code. Composition, for example. Because this works like a fluent API or a builder pattern, one can keep adding conditions to a query. Imagine you have to filter a list of strings based on a Google like query string. At the very minimum you would need to split the query into strings and filter repeatedly on each one. Something like this:
var arr=['this is my special query string','this is a string','my query string is this awesome','no query strings here, move along','these are not the strings you are looking for'];
var query="this is a query string";
var splits=query.split(/\s+/g);
var result=arr;
splits.forEach(s=>result=result.filter(a=>a.includes(s)));
console.log(result);

There is a lot of stuff I could be saying about this subject, but I need to summarize it. It's all about inverting loops. Instead of having to go through a collection, a stream or some other data source, then executing some code for each element, this method allows you to encapsulate the operations you want to execute on those elements, pass them around, compose them, translate them, then use them on any data source in the same way. A common API means reusability, better readability of code, less written code and a simpler declaration of intent. Because we get out of the loop system, we can expand the use for other paradigms, such as infinite data streams or event buses.

Fear and Hope

It occurred to me recently that the opposite of fear is hope. Well, of course, you will say, didn't you know that? I did, but I also didn't fully grasp the concept. It doesn't help that fear is considered an emotion, yet hope a more complicated idea.

I was thinking about the things that go wrong in my country and some of it, a large part, comes from bad laws. And I was trying to understand what a "bad law" is. I tried some examples, like the dog leash one - I know, I have a special personal hate for that one in particular - but I noticed a pattern. It's not about the content of the law as it is about its trigger. You see, lawmen don't propose and pass laws because they like work, but because there was an event that triggered the need for that law. Law is always reactive, not proactive. It could be proactive, but there is a lot more effort involved, like convincing people that there is an actual problem that needs addressing. It's much easier to wait for the problem to manifest and then try (or pretend) to fix it.

Anyway, the pattern that I noticed was related to the trigger for individual laws. The bad laws were the ones that came out of fear. One kid got killed by stray dogs, kill them all and institute mandatory leashes on pets. The good laws, on the other hand, come from hope. Lower taxes so people are more inclined to work and thus produce more and so get more tax in. Hopefully people will not be lazy.

And it's not only related to laws, but to personal decisions as well. Will I try a new thing, hoping that it will make me better, teach me something, be fun, or will I not try it because it is dangerous, somebody might get hurt, I may lose precious time, etc? When it is so abstract it's almost a given that you will take the first choice, yet when it is more personal fear tends to paralyze.

Fear is also contagious. The people who want us to be afraid are afraid themselves. Control freaks, power hungry people, they don't want to take us to a better place because they are afraid to lose that control, because they are afraid of what might happen. And their toolkit is based on fear, too. Something exploded and killed people, some asshole drove a car into people: we must ban explosives, cars and - just to be safe - people. Don't go to space because people might die, although they die every second and most of the time you don't care about it. Let's hoard money and things because we might not get another chance to have them, because we might lose them, because we are so afraid. The fear people don't know any other language but fear and they will use it against you. Much easier to instill fear than to give hope, so hope is not that contagious. It is fragile and it is precious.

I submit that while fear might keep us safe it will never make us happy. The very expression "to keep safe" implies stagnation, keeping, holding, controlling, restricting freedom.

So here is my solution. As Saint-Exupery said, perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. Let's strictly define our safe zone, or the area we need to be safe in order to not be afraid. Personally, as a group, as a country, as a planet, let's set the minimum requirements to being safe, a place or situation we can always retreat to and not be afraid. Whether it is a place that is your own, or a lack of debt, or a job or business that will give you just enough money to survive and not spiral out of control, a relationship or some other safety net, everyone needs it. But beyond it, let's abandon fear and instead use hope. Hope that you can do more, you can be better, you can live more or have fun, that other people will act good rather than badly, that strangers will help rather than harm you, that the unknown will reveal beauty rather than terror.

I will choose to define good decisions as coming from hope. Will that hope be proven to be unfounded? Maybe. But a decision based on fear will never ever be good enough. And if all else fails, I have my safe zone to get back to. And I know, I very much know that having a place to get back to from failure is a luxury, that not many people have it as good as I do, but to have it and still live in fear, that's just stupid.

Saturday, 8 April 2017

Song of Kali, by Dan Simmons

book cover A friend of mine recommended this as one of his favorite books, so of course I went into it with very high expectations and of course I was disappointed. That doesn't mean it's a bad book, just that I expected more than I've got.

In Song of Kali, Dan Simmons describes Calcutta as a place of evil, in a culture of filth and senseless violence and death. He goes there with his Indian wife and their infant child when he is called to retrieve a new manuscript from a supposedly dead Indian poet. A lot of culture shock, a lot of weird mystical events and some weird and horrible people that do horrible things is what the book is about.

In 1985 this was perhaps a fantastic story, I don't know, but now it feels a little bit cliché: American man goes somewhere he sees as completely alien and where he feels out of place, usually going there with the family, so that the empathy and horror can be heightened, and where abnormal things he has no control over happen. It also part of a category of stories that I personally dislike: the "something that can't be explained or controlled" category, which implies absolutely no character growth other than realizing there are situations like that in which one can find themselves. And indeed the book is all like that: stories that make little sense, but somehow are linked to the perceptions and experiences of the protagonist, mysterious characters that do things that mean little unless the story takes them exactly to a certain point, at which you are left wondering how did they know to do that thing, and a lot of extraneous details that are there only to reinforce the feeling of disgust and dread that the character feels, but do little to further the story.

In the end, it is just some weird ass plot that makes no sense, a bunch of characters that you can't empathize with (some of them you can't even understand) and a big fat "It is so because I feel it is so", which is so American and has little to do with me. Others agree that the book is most effective when describing the humid fetid heat of the city and the inhumanity of its inhabitants and less with the so called "horror" in the text or the connection the reader feels with the characters. It brings to mind Lovecraft and his strong feelings about things that now are banal and CGI in every movie. Some are even more vehement in their dislike of the book. Here is another review in the same vein.

So how come so many people speak highly of the novel? Well, my guess is that it affects the reader if they are in the right frame of mind. My friend told me about the part that he liked in the book and, frankly, that part is NOT in the book, so whatever literary hallucination he had when reading the book I had none of it. My rating of it cannot be but average, even considering it's a debut novel that won the 1986 World Fantasy Award.

Saturday, 1 April 2017

Breaking Bad, the Movie!

So I heard that there is this fan made cut of the series, two hours long, that encompasses the entire story of Breaking Bad. I got a hold of it and watched it. Pretty good. Just some lazy editing in some places, but overall good quality. Therefore, if you want to see what happened in the series overall, without bingeing on 62 hours of TV show, you might want to check it out.

My problem with the film is that it validated my decision to stop watching the series. It focused primarily on Walter's decision points, which were mostly related to his problems with his family (mostly his bitch of a wife that I believe is one of the most irritating characters of all time), friends and coworkers. The only part that I really enjoyed about the series was the first season, where there was actual chemistry involved. Just like other shows that start off with a brilliant specialist that is rather annoying otherwise but gets away with it because he is a flawless craftsman, it begins great then devolves in stories about his personal life. Why would anyone want to for years follow the personal issues of someone who they only became interested in because of their work story is beyond me, but this is what happens. Dr. House, Numb3rs, Elementary, Weeds, even lawyer, doctor and cop shows slowly force their heroes to stop doing their work and instead deal with all kinds of problems in their off duty life; they all lose me at season two, usually. Because of this focus on personal life, the chemistry part got removed from the movie, which makes is all the more boring.

Anyway, my duty is complete on informing the Internet on this film. It is amazing how people spend their time doing something like this for nothing as much as recognition (because then lawyers would bust their chops about using copyrighted content), but do such a lovely job. I would love to have this sort of edits for every show on the planet. Then I would be able to keep up with all of them! :D Also interesting is that there is an IMDb page for the movie found by Google, but then when you navigate to it you get a big 404 page, meaning someone probably created it and then it promptly got deleted. Even if illegal, it is still a movie, assholes! Here is the Google cached version, for how long it will work.

And BTW, if you want to still write a review on the movie, as deleted as it is, you can do so by following this link. Maybe that will force the guys from IMDb to undelete the page.

Tuesday, 28 March 2017

Fevre Dream, by George R. R. Martin

book cover In Fevre Dream, George R. R. Martin writes about a fat bearded guy with a large appetite and a passion for food that loves to be a boat captain. Write what you know, they say. Anyways, this book about vampires in the bayou feels really dated. It has been described as "Bram Stoker meets Mark Twain", so you can imagine how much; written in 1982, it feels like written by a Lovecraft contemporary.

I love Lovecraft, but it gets worse. None of the characters in the book except maybe the main protagonist are likable. They come off as either high and mighty or ridiculously servile. And I understand that in a story where vampires have a master that can be all controlling this is to be expected, but at the same time the hero of the story, without being "compelled", still acts like a servant, enthralled (pardon my pun) by the aristocratic majesty of his vampire friend. One has to get through pages of tedious description of architecture and food and home improvement to get to the succulent part (OK, couldn't help that one) but which then feels cloyed and unsatisfactory. So many interesting characters get just a few scenes, while most of the book is how much the captain loves his food and his ship. And while it discusses some social issues, like slavery and how easily people died or disappeared at the time, it also promotes this idea of personal nobility that justifies other people getting used. This focus on aristocracy is something one sees in A Song of Fire and Ice as well, but less pronounced.

I could have given it an average to good rating if not for the abysmal ending. While at the beginning I had applauded the way the author was building tension and apparently providing a solution only to snatch it away at the last moment, the ending destroys all of it by pretty much invalidating much of the foil of the characters and a major part of the story. The time displacement also accentuates this feeling, as I thought "waited so much for this?!", and by that I mean both me as a reader and the main character in the book.

Bottom line: uninteresting vampires in a slow paced story that probably appeals to Martin fans only. It manages to insert the reader in the eighteen hundreds and the river boat mentality, but there is nothing much else to learn or enjoy in the book beyond that.

Sunday, 19 March 2017

Learning ASP.Net MVC - Part 6 - Dependency Injection and Services

Learning ASP.Net MVC series:
  1. Setup
  2. MVC Concepts
  3. Authentication
  4. Entity Framework Fundamentals
  5. Upgrading project to .NET Core 1.1
  6. Dependency Injection and Services

Previously on Learning ASP.Net MVC...


Started with the idea of a project that would use user configurable queries to do Google searches, store the information in the results and then perform various data analysis on them and display them based on what the user wants. However, I first implemented a Google authentication and then went to write some theoretical posts. Lastly, I've upgraded the project from .NET Core 1.0 to version 1.1.

Well, it took me a while to get here because I was at a crossroads. I like the idea of dependency injectable services to do data access. At the same time there is the entire Entity Framework tutorial path that kind of wants to strongly integrate EF with my projects. I mean, if I have a service that gives me the list of all items in the database and then I want to get only a few items, it would be bad design to filter the entire list. As such, I would have to write a different method that allows me to get the items based on some kind of filters. On the other hand, Entity Framework code looks just like that "give me all you have, filtered by this" which is then translated into an efficient query to the database. One possibility would be to have my service return IQueryable <T>, so I could also use the system to generate the database code on the fly.

The Design


I've decided on the service architecture, against an EF type IQueryable way, because I want to be able to replace that service with anything, including something that doesn't work with a database or something that doesn't know how to dynamically create queries. Also, the idea that the service methods will describe exactly what I want appeals to me more than avoiding a bit of duplicated code.

Another thing to define now is the method through which I will implement the dependency injection. Being the control freak that I am, I would go with installing my own library, something like SimpleInjector, and configure it myself and use it explicitly. However, ASP.Net Core has dependency injection included out of the box, so I will use that.

As defined, the project needs queries to pass on to Google and a storage service for the results. It needs data services to manage these entities, as well as a service to abstract Google itself. The data gathering operation itself cannot be a simple REST call, since it might take a while, it must be a background task. The data analysis as well. So we need a sort of job manager.

As per a good structured design, the data objects will be stored in a separate project, as well as the interfaces for the services we will be using.

Some code, please!


Well, start with the code of the project so far: GitHub and let's get coding.

Before finding a solution to actually run the background code in the context of ASP.Net, let's write it inside a class. I am going to add a folder called Jobs and add a class in it called QueryProcessor with a method ProcessQueries. The code will be self explanatory, I hope.
public void ProcessQueries()
{
var now = _timeService.Now;
var queries = _queryDataService.GetUnprocessed(now);
var contentItems = queries.AsParallel().WithDegreeOfParallelism(3)
.SelectMany(q => _contentService.Query(q.Text));
_contentDataService.Update(contentItems);
}

So we get the time - from a service, of course - and request the unprocessed queries for that time, then we extract the content items for each query, which then are updated in the database. The idea here is that, for the first time a query is defined or when the interval from the last time the query was processed, the query will be sent to the content service from which content items will be received. These items will be stored in the database.

Now, I've kept the code as concise as possible: there is no indication yet of any implementation detail and I've written as little code as I need to express my intention. Yet, what are all these services? What is a time service? what is a content service? Where are they defined? In order to enable dependency injection, we will populate all of these fields from the constructor of the query processor. Here is how the class would look in its entirety:
using ContentAggregator.Interfaces;
using System.Linq;

namespace ContentAggregator.Jobs
{
public class QueryProcessor
{
private readonly IContentDataService _contentDataService;
private readonly IContentService _contentService;
private readonly IQueryDataService _queryDataService;
private readonly ITimeService _timeService;

public QueryProcessor(ITimeService timeService, IQueryDataService queryDataService, IContentDataService contentDataService, IContentService contentService)
{
_timeService = timeService;
_queryDataService = queryDataService;
_contentDataService = contentDataService;
_contentService = contentService;
}

public void ProcessQueries()
{
var now = _timeService.Now;
var queries = _queryDataService.GetUnprocessed(now);
var contentItems = queries.AsParallel().WithDegreeOfParallelism(3)
.SelectMany(q => _contentService.Query(q.Text));
_contentDataService.Update(contentItems);
}
}
}

Note that the services are only defined as interfaces which we declare in a separate project called ContentAggregator.Interfaces, referred above in the usings block.

Let's ignore the job processor mechanism for a moment and just run ProcessQueries in a test method in the main controller. For this I will have to make dependency injection work and implement the interfaces. For brevity I will do so in the main project, although it would probably be a good idea to do it in a separate ContentAggregator.Implementations project. But let's not get ahead of ourselves. First make the code work, then arrange it all nice, in the refactoring phase.

Implementing the services


I will create mock services first, in order to test the code as it is, so the following implementations just do as little as possible while still following the interface signature.
public class ContentDataService : IContentDataService
{
private readonly static StringBuilder _sb;

static ContentDataService()
{
_sb = new StringBuilder();
}

public void Update(IEnumerable<ContentItem> contentItems)
{
foreach (var contentItem in contentItems)
{
_sb.AppendLine($"{contentItem.FinalUrl}:{contentItem.Title}");
}
}

public static string Output
{
get { return _sb.ToString(); }
}
}

public class ContentService : IContentService
{
private readonly ITimeService _timeService;

public ContentService(ITimeService timeService)
{
_timeService = timeService;
}

public IEnumerable<ContentItem> Query(string text)
{
yield return
new ContentItem
{
OriginalUrl = "http://original.url",
FinalUrl = "https://final.url",
Title = "Mock Title",
Description = "Mock Description",
CreationTime = _timeService.Now,
Time = new DateTime(2017, 03, 26),
ContentType = "text/html",
Error = null,
Content = "Mock Content"
};
}
}

public class QueryDataService : IQueryDataService
{
public IEnumerable<Query> GetUnprocessed(DateTime now)
{
yield return new Query
{
Text="Some query"
};
}
}

public class TimeService : ITimeService
{
public DateTime Now
{
get
{
return DateTime.UtcNow;
}
}
}

Now all I have to do is declare the binding between interface and implementation. The magic happens in ConfigureServices, in Startup.cs:
services.AddTransient<ITimeService, TimeService>();
services.AddTransient<IContentDataService, ContentDataService>();
services.AddTransient<IContentService, ContentService>();
services.AddTransient<IQueryDataService, QueryDataService>();

They are all transient, meaning that for each request of an implementation the system will just create a new instance. Another popular method is AddSingleton.

Using dependency injection


So, now I have to instantiate my query processor and run ProcessQueries.

One way is to set QueryProcessor as a service. I extract an interface, I add a new binding and then I give an interface as a parameter of my controller constructor:
[Authorize]
public class HomeController : Controller
{
private readonly IQueryProcessor _queryProcessor;

public HomeController(IQueryProcessor queryProcessor)
{
_queryProcessor = queryProcessor;
}

public IActionResult Index()
{
return View();
}

[HttpGet("/test")]
public string Test()
{
_queryProcessor.ProcessQueries();
return ContentDataService.Output;
}
}
In fact, I don't even have to declare an interface. I can just use services.AddTransient<QueryProcessor>(); in ConfigureServices and it works as a parameter to the controller.

But what if I want to use it directly, resolve it manually, without injecting it in the controller? One can use the injection of a IServiceProvider instead. Here is an example:
[Authorize]
public class HomeController : Controller
{
private readonly IServiceProvider _serviceProvider;

public HomeController(IServiceProvider serviceProvider)
{
_serviceProvider = serviceProvider;
}

public IActionResult Index()
{
return View();
}

[HttpGet("/test")]
public string Test()
{
var queryProcessor = _serviceProvider.GetService<QueryProcessor>();
queryProcessor.ProcessQueries();
return ContentDataService.Output;
}
}
Yet you still need to use services.Add... in ConfigureServices and inject the service provider in the constructor of the controller.

There is a way of doing it completely separately like this:
var serviceProvider = new ServiceCollection()
.AddTransient<ITimeService, TimeService>()
.AddTransient<IContentDataService, ContentDataService>()
.AddTransient<IContentService, ContentService>()
.AddTransient<IQueryDataService, QueryDataService>()
.AddTransient<QueryProcessor>()
.BuildServiceProvider();
var queryProcessor = serviceProvider.GetService<QueryProcessor>();

This would be the way to encapsulate the ASP.Net Dependency Injection in another object, maybe in a console application, but clearly it would be pointless in our application.

The complete source code after these modifications can be found here. Test the functionality by going to /test on your local server after you start the app.

Tuesday, 14 March 2017

The Call (the Gray Land #1), by Peadar Ó Guilín

book cover I've seen several very positive reviews of The Call, by Peadar Ó Guilín so I started reading it. A few hours later I had finished it. It was good: well written, with compelling characters, a fresh idea and a combination of young adult and body horror mixed with Irish mythology that hooked me immediately. I was sorry it had ended and simultaneously hoped for and cursed the idea of "trilogizing" it.

So the book follows this girl who can't use her legs because of polio. She is a happy child until her parents explain to her the realities: Ireland is separated from the world by an impassible barrier and the Aes Sidhe, the Irish fairies, are kidnapping each adolescent kid once, hunt them and hurt them in horrific ways, as revenge for the Irish banishing them to a hellish world. When "the call" comes, the child disappears, leaving back anything that is not part of their bodies and returns in 184 seconds. However, they experience an entire day in the colorless, ugly and cruel world of the Sidhe where they have to fight for their lives. In response, the Irish nation organizes in order to survive, with mandatory child births and training centers where teens are being prepared for the call in hope they will survive.

One might think this is something akin to young adult novels like The Maze, but this is much better. The main character has to overcome her disability as well as the condescending pity or disgust of others. She must manage her crush on a boy in school as well as the rules, both societal and self imposed, about expressing emotion in a world where any friend you have may just disappear in front of you and returned a monster or dead. Her friends are equally well defined, without the book being overly descriptive. The fairies have the ability to change the human body with a mere touch, so even the few kids who survive returned mentally and bodily deformed. The gray world itself is filled with horrors, with an ecosystem of carnivorous plants and animals that are actually made of altered humans, from hunting dogs and mounts to worms and spiders which somehow still maintain some sort of sentience so they can feel pain. I found the Aes Sidhe incredibly compelling: they are incredibly beautiful people and are full of joy and merriment, even as they maim and torture and kill and even when they are themselves in pain or dying, a race of psychotic vengeful people that know nothing but hate.

So I really liked the book and recommend it highly.