Saturday 25 May 2013

Adblock EasyList filter and action files for Privoxy

On   In , , ,    → 23 comments  edit
Update 12 Feb 2016: As an alternative, if you have access to your hosts file, you can use a generated list of domains to immediately block any access to those IPs, courtesy of Steven Black: StevenBlack/hosts, and you don't have to install anything and works on most operating systems.

Update 19 Feb 2016: Here are the latest files to directly download and use. Do read the rest of the article, as these files might not be what you are looking for:
user.action
user.filter

Now for the rest of the article:

I discovered today a new tool called Privoxy. It is a proxy software that has extra features, like ad blocking and extra privacy. What that means is that you can install the proxy, point your browser to that proxy and have an almost ad free untracked by marketing firms or FaceBook experience. The only problem, though, is that the default filters are not so comprehensive. Would it be great if one could take the most used list for ad blocking (Adblock Plus' Easylist) and convert it to Privoxy? Well it would, only that no one seems to want to do it for Windows. You get a few folks that have created Linux scripts to do the conversion, but they won't work for Windows. Nor do they deem it necessary to make an online tool or a web service or at least publish the resulting files so that we, Windows people, can also use the list!

Well, I intend to do a small script that will allow for this, preferably embedded in this blog post, but until then, I had no script, no files, only Privoxy installed. Luckily, I also have Cygwin installed, which allows me to run a ridiculous flavour of Linux inside Windows. I had to hack the script in order for it to work on Cygwin, but in the end, at last, I managed to make it work. For now, I will publish the resulting files here and give you instructions to install them. When they become obsolete, send me a comment and I will refresh them.

So, the instructions:
  • Install Privoxy
  • Go to the installation folder of Privoxy and look for files named 'user.action' and 'user.filter'
  • Download the user.action file from here and replace the default one.
  • Download the user.filter file from here and replace the default one.
  • Restart Privoxy
  • Of course, then you have to go to your browser settings and set the proxy to Privoxy (usually localhost, port 8118)

Warning! The filter file is quite big and it will cause some performance issues. You might want to use only the action file with the filter actions removed.

Update: If you can't download the files, let me know. I am using Github pages and it seems sometimes it doesn't work as I expected.

Also, I have lost faith that AdBlockPlus rules can be completely and correctly translated to Privoxy and I lack the time, so I am publishing my crappy program as it is and if you want to use it or fix it, be my guest. You can find the program here: AdblockPlus to Privoxy. I would ask that you share with me those modifications, though, so that everybody can benefit from them.

Update October 2014: Other people have contributed by making their own translation software. Here are the links for the binary and source code of adblock2privoxy made by Zubr, in Haskell mind you, which is pretty cool:

Binary: https://www.dropbox.com/s/69u1iqbubzft1yl/adblock2privoxy-1.2.4.rar?dl=0
Source code: https://projects.zubr.me/wiki/adblock2privoxy

23 comments:

  1. Roquene26 June 2013 at 08:58

    Thanks!

    Exactly what i'm looking for :)
    Found the Linux Script today but don't have the Know-How to get it running under Windows. Many thanks!

    ReplyDelete
  2. JAY TRIPATHI11 October 2013 at 01:26

    thank you very much... now adblock wont eat up my resources.... this is the first time i have commented online....

    so i guess we will have to update easylist manually for this... any instructions on how to do so??

    also do you know of any way i can extract the userscript from any extension and use it with tampermonkey...

    ReplyDelete
  3. Siderite11 October 2013 at 02:27

    Well, I did it via Cygwin and the online script. I thought of making a C# program for it, but was too lazy.

    ReplyDelete
  4. ChriZN18 October 2013 at 13:31

    Hell yeah,
    you made my day. Also don't have a Linux around here, and only like to update with the easylists.

    I hope you will walk your plan with an Windows-App for update the lists.

    Thank you for what you have done, and maybe you'll have a surprise for the privoxy-community.

    Regards, ChriZN

    ReplyDelete
  5. Anonymous6 March 2014 at 08:13

    Time to refresh .... ;)

    ReplyDelete
  6. Siderite10 March 2014 at 04:57

    Thank you for reminding me of this. I will create a Windows app to do this for you. Give me some time, though.

    ReplyDelete
  7. Siderite18 March 2014 at 08:30

    I am still working on the application, but now you have some beta files you can download. If you experience performance issues, don't use the user.filter file.

    ReplyDelete
  8. Anonymous4 June 2014 at 08:04

    very interested in this, any update?

    ReplyDelete
  9. Siderite4 June 2014 at 09:18

    Unfortunately the conversion is not so straightforward. The two systems are very different. In order to implement filters in Privoxy, for example, I have to find elements in the HTML using regular expressions. It's slow and ugly. The application I built is not pretty either :)

    Since I see there is interest, I will try to do something about it. I wanted to put it on CodePlex, but it's such a makeshift code that I was embarrassed, honestly.

    ReplyDelete
  10. Anonymous4 June 2014 at 22:13

    There seems to be scripts that do this but available only for nix and osx. :( Ive moved away from mac so your help will be greatly appreciated!!

    ReplyDelete
  11. Siderite5 June 2014 at 03:50

    But you could download the files in the post, right? Or are you interested in the application as well, as a developer?

    ReplyDelete
  12. Anonymous10 June 2014 at 04:54

    heheh i am not a developer, looking for an app that can update rules.

    I appreciate that you coverted them for us to download but I oftentimes hope to have the latest/updated rules.

    hence i look for an app that can do this.

    thanks!

    ReplyDelete
  13. Skylar Loeper6 July 2014 at 06:58

    Hey, I was wondering if you have had any progress on those scripts? As of now, when I use the user.filter provided on March 18, it throws a fatal error in my face ( Fatal error: can't load re_filterfile '/etc/privoxy/user.filter': Invalid argument ). I am currently using DD-WRT (Brainslayer's build) build 24461 on a Netgear WNDR4300, with Privoxy 3.0.21 built-in if that helps any.

    Other than that, kudos to you for building such a complex script to achieve this. This is definitely one of the reasons I respect coders and programmers so much.

    ReplyDelete
  14. Siderite6 July 2014 at 08:18

    Published the latest files, which seem to work on my computer with minor errors in some filter rules. Also published the project, as it is.

    Feel free to download, use, change, etc the program.

    ReplyDelete
  15. Skylar Loeper6 July 2014 at 09:42

    Thanks a lot man, it truly does mean a lot for what you have done. Who knows, someone might see the code and take a crack at it.

    If I ever manage to start learning how to program without my brain going into meltdown, I'll look at it as well.

    Once again, thanks.

    ReplyDelete
  16. Anonymous26 August 2014 at 17:25

    hi there, i ve used the files in freetz for fritz.box but it don´t seems to right i get a lot of errors, a very slow internet connection and high load between 6-10

    here are some log lines:

    2014-08-27 02:12:10.295 2aab0000 Info: Privoxy version 3.0.21
    2014-08-27 02:12:10.295 2aab0000 Info: Program name: privoxy
    2014-08-27 02:12:10.296 2aab0000 Info: Loading filter file: /mod/etc/privoxy/default.filter
    2014-08-27 02:12:10.322 2aab0000 Info: Loading filter file: /tmp/flash/privoxy/user.filter
    2014-08-27 02:12:10.339 2aab0000 Info: Loading actions file: /mod/etc/privoxy/match-all.action
    2014-08-27 02:12:10.340 2aab0000 Info: Loading actions file: /mod/etc/privoxy/default.action
    2014-08-27 02:12:10.375 2aab0000 Info: Loading actions file: /tmp/flash/privoxy/user.action
    2014-08-27 02:12:10.377 2aab0000 Info: Listening on port 8118 on IP address 0.0.0.0
    2014-08-27 02:13:05.596 2ac87520 Error: 192.168.2.105's request: 'GET /subscribe?host_int=676407892&ns_map=271243323_2723280508987&user_id=169690204&nid=285&ts=1409098385 HTTP/1.1' is invalid. Privoxy isn't configured to accept intercepted requests.

    2014-08-27 02:10:54.141 2ad07520 Info: Compressed content detected, content filtering disabled. Consider recompiling Privoxy with zlib support or enable the prevent-compression action.

    ReplyDelete
  17. Siderite26 August 2014 at 23:20

    As I understand it, Privoxy can be configured to proxy intercepted requests, so that it behaves as a transparent proxy. The accept-intercepted-requests 1 configuration option enables it.

    Also, as I said in the post, decompressing all incoming calls and doing a lot of regular expression changes on the content is very expensive. That is why I recommend NOT using the user.filter file except for a few specific lines that take care of the things that annoy you the most.

    The way it works is this:
    1. can the call (to a page or a resource like an image, etc) be made? (user.action comes into play here). If not, block the call.
    From here user.filter is the one that does the work
    2. If yes, is it an html page?
    3. If yes, is it compressed? Decompress it if so.
    4. Does the entire content (hundreds of kilobytes of text) match any of the regular like expressions in the user.filter?
    5. If yes, then use regular expression replacing (which is efficient for text, but in this case are very complex in order to match HTML, which is not efficient)
    6. Serve the content

    So you see, the difference in processing between user.action and user.filter is the difference between processing a URL string and an entire page content, which you previously decompressed.

    ReplyDelete
  18. Siderite26 October 2014 at 23:22

    Great news! Thanks for leaving the comment. Now people have another way to do this.

    ReplyDelete
  19. Anonymous29 October 2014 at 06:09

    Hi Siderite
    Privoxy is a supper awesome software, it can do almost everything, the fastest Proxy software because it use just plain RegEx engine and there is no built in engine like AdGuard, Proxomitron so it is the fastest ad filtering software so I really want to make it up to a higher level by supporting it.

    Here is my thread about how to make Privoxy filter HTTPS, I would like to share the thread to you and other people here because HTTPS filtering a feature that Privoxy's user really want to have for a long time.

    https://sourceforge.net/p/ijbswa/support-requests/1667/

    Have fun!
    Cattleya

    ReplyDelete
  20. Cattleya12 January 2015 at 13:41

    Hi Siderite,

    I finally finish my AdBlock2Privoxy tool, I named it "convert2privoxy"

    Here you can download: https://projects.zubr.me/ticket/16

    And just copy paste AdBlock rule into the tool GUI and click AdBlock2Privoxy.

    The tool cannot and will ignore $ type filter, I must ignore these filter because partial implement it will cause even worse false positive.

    Good bye,

    ReplyDelete
  21. Unknown23 March 2016 at 23:32

    Download user.action and user.filter from link below but not working
    --------------------------------------
    Update 19 Feb 2016: Here are the latest files to directly download and use. Do read the rest of the article, as these files might not be what you are looking for:
    user.action
    user.filter
    -------------------------l-------------

    However download user.action and user.filter from link below working OK
    ---------------------------------------
    So, the instructions:
    Install Privoxy
    Go to the installation folder of Privoxy and look for files named 'user.action' and 'user.filter'
    Download the user.action file from here and replace the default one.
    Download the user.filter file from here and replace the default one.
    ---------------------------------------

    It seems those user.action / user.filter are different.
    Which one is updated one? Why the first one not working?

    ReplyDelete
  22. Anonymous25 March 2016 at 21:05

    Thanks for doing this. Your work is appreciated..

    ReplyDelete
  23. Siderite10 April 2016 at 02:00

    I don't know why the links won't work for some people, they work perfectly for me. I've changed the URL format, try again.

    ReplyDelete