Netflix - Application based routing over another VPN

By Freeaqingme on Tuesday 17 September 2013 03:28 - Comments (4)
Categories: Linux, Networking, Views: 9.221

In my last blogpost I already showed how you could use a VPN connection for one application, while using your normal WAN for all other applications. Although I'm using Netflix as an example, you could use the same mechanism for virtually any other IP-based application.
The idea
For many people the suggested solution will suffice. However, as more and more people do these days, I proxy all my traffic through my (EU-based) VPS. This is perfectly doable with the setup I described in my last blogpost. But it becomes tedious to set up if you have multiple devices. Therefore, I want to do all my routing on my VPS, as outlined in this diagram:

                                               |----------|       |----------|    
                                               |          |       |          |    
                                        /------|   VPN    |------>| Netflix  | 
|----------|       |----------|        /       |          |       |          |
|          |       |          |-------/        |----------|       |----------|
| Desktop  |-----> |   VPS    | 
|          |       |          |-------\          /------^-----\
|----------|       |----------|        \        /              \
                                        \-------|   Internet   |
                                                 \             /
                                                  \-----------/
 


The connection between my Desktop and VPS is an OpenVPN tunnel. The connection between the VPS and VPN is also an OpenVPN tunnel provided by Private Internet Access (PIA). We cannot simply duplicate the exact same logic from my last blogpost in this situation, because it's based off the UID of the Netflix application, and in the current setup the routing is done on a different machine than where the Netflix application runs. In this particular example the advantage of routing to the VPN service from PIA are limited, but once you get multiple devices (laptop, mobile phone...) on the left side of the VPS, and start using multiple VPN services or locations, it starts to pay off.
IPIP
We need to find a solution to tell the VPS what gateway to take. It would of course be possible to set up another VPN connection between my Desktop and VPS but that would be tedious, especially once the number of clients and gateways increases. In my current VPN setup OpenVPN has been configured in routed mode (using TUN interfaces rather than TAP) and that means that we cannot use any layer-2 specific features (like abusing the QoS bit to indicate the desired gateway). In my case I decided to with an IPIP (Ip over Ip) tunnel. When an IP Packet is sent through an IPIP-tunnel, the only thing that happens is that an extra Source and Destination header is added to the IP Packet, as is illustrated in the Wireshark capture below (click to enlarge):
IPIP wireshark example
Setting it up
In the steps outlined below I'll assume you've already set up a VPN connection between the Desktop and the VPS. If you have not, you could take a look at this blogpost. On both the client and server the interfaces are called tun0. Furthermore, I'll assume you've already configured a VPN on the VPS towards the US, called tunUS, as described in my last post.
Creating the IPIP-tunnel
Because the IPIP-tunnel can only be initialized once the OpenVPN connection between the Desktop (Client) and VPS (Server) has been established, I put the logic of creating this IPIP-tunnel in a script that would be called by OpenVPN once the connection is initialized (using the OpenVPN 'up' directive).

On the client side:

code:
1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

# make sure all routes have been initialized
sleep 5

ip tunnel add tunUS-in-tun0 mode ipip remote 172.31.254.1 local 172.31.254.10
ip address add 172.31.252.2/30 dev tunUS-in-tun0
ip link set tunUS-in-tun0 up

ip route add default via 172.31.252.1 src 172.31.252.2 table 2
ip route add 172.31.252.0/30 dev tunUS  proto kernel  scope link  src 172.31.252.2 table 2



The 172.31.254.1 and .10 addresses are the OpenVPN gateway and client ip. I'd have preferred to use the variables that OpenVPN passes on to this script, but for some reason the gateway ip that's supplied is wrong (or technically right, but cannot be used for this purpose). On the server side we have a script that is almost the same:

code:
1
2
3
4
5
6
7
8
9
10
#!/bin/bash

# make sure all routes have been initialized
sleep 5

ip tunnel add tunUS-in-tun0 mode ipip remote 172.31.254.10 local 172.31.254.1
ip address add 172.31.252.1/30 dev tunUS-in-tun0
ip link set tunUS-in-tun0 up

ip route add 172.31.252.0/30 dev tunUS-in-tun0  proto kernel  scope link  src 172.31.252.1 table 2

Directing traffic on the client
Next up, we need to do direct the Netflix traffic from the Netflix application into the IPIP-tunnel. You could very well add this to the same script as the IPIP-tunnel on the client:

code:
1
2
3
iptables -t mangle -A OUTPUT -m owner --gid-owner netflix  -j MARK --set-mark 2
iptables -t nat -A POSTROUTING -o tunUS-in-tun0 -j SNAT --to 172.31.252.2
ip rule add fwmark 2 table 2



To verify that everything so far works, you can open a ping as user netflix on the client to a random (public) ip, and verify using tcpdump on the server that the traffic is coming in from the right ip on the tunUS-in-tun0 interface.
Forwarding traffic from Client to VPN on VPS
To get the traffic from the VPS to Netflix via the VPN service we still need to route all traffic from the tunUS-in-tun0 to the tunUS interface on the VPS:

code:
1
2
3
4
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 172.31.252.0/24 -o tunUS -j SNAT --to-source 10.198.1.6
ip rule add from 172.31.252.0/24 lookup 2
ip route add default via 10.198.1.5 dev tunUS  src 10.198.1.6



In this case, the 10.198.1.5 and .6 are my gateway and client ip from my VPN Service, I retrieved those using:

code:
1
2
3
4
# ip a s tunUS
17: tunUS: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.198.1.6 peer 10.198.1.5/32 scope global tunUS



I'm thinking you could perfectly well retrieve those from OpenVPN in its export variables when invoking it using the up-script configuration directive. Beware, in the case of PIA these addresses are changed once a day or so, so hardcoding them may not be the way to go.
RP_Filter
Linux has a feature called reverse-path filtering. This feature drops traffic from any ip it has no route for on that interface. Because it does not keep into account the use of multiple routing tables, we'll need to disable it: on both the client and the server.


code:
1
2
3
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
    echo 0 > $f
    done

Troubleshooting the setup
With all debugging, you begin initiating traffic on the client, as the Netflix application does:

code:
1
client# sudo -u netflix ping 8.8.8.8



Then, you should see the traffic going out on the right interface:

code:
1
2
3
4
5
6
client# sudo tcpdump -i tunUS-in-tun0 -ns0 icmp
[sudo] password for dolf: 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunUS, link-type RAW (Raw IP), capture size 65535 bytes
03:13:47.599418 IP 172.31.252.2 > 8.8.8.8: ICMP echo request, id 25294, seq 115, length 64
03:13:47.835465 IP 8.8.8.8 > 172.31.252.2: ICMP echo reply, id 25294, seq 115, length 64



Please take note of the IP addresses. If these don't match, things are bound to go wrong.

So once you've confirmed the traffic leaves your desktop from the right interface using the right ip addresses, it's time to hop on to the VPS, see if the traffic comes in there:

code:
1
2
3
4
5
6
7
server# sudo tcpdump -i tunUS-in-tun0 -ns0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunUS-in-tun0, link-type RAW (Raw IP), capture size 65535 bytes
03:12:38.890815 IP 172.31.252.2 > 8.8.8.8: ICMP echo request, id 25294, seq 64, length 64
03:12:39.104200 IP 8.8.8.8 > 172.31.252.2: ICMP echo reply, id 25294, seq 64, length 64
03:12:39.894865 IP 172.31.252.2 > 8.8.8.8: ICMP echo request, id 25294, seq 65, length 64
03:12:40.108261 IP 8.8.8.8 > 172.31.252.2: ICMP echo reply, id 25294, seq 65, length 64



The traffic you see here should match 1:1 with the output of the previous command. If it does not (or you have no traffic at all on this interface), there's something wrong with your IPIP tunnel.

The last step to see if all goes as it should, is to see if the traffic towards the VPN service leaves your VPS correctly:

code:
1
2
3
4
5
6
7
server# sudo tcpdump -i tunUS -ns0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunUS, link-type RAW (Raw IP), capture size 65535 bytes
03:17:38.141751 IP 10.198.1.6 > 8.8.8.8: ICMP echo request, id 25294, seq 363, length 64
03:17:38.355633 IP 8.8.8.8 > 10.198.1.6: ICMP echo reply, id 25294, seq 363, length 64
03:17:39.136641 IP 10.198.1.6 > 8.8.8.8: ICMP echo request, id 25294, seq 364, length 64
03:17:39.350106 IP 8.8.8.8 > 10.198.1.6: ICMP echo reply, id 25294, seq 364, length 64



The outgoing IP here should match that of the tunUS address. If you don't see any traffic at all, it's likely that one of your iptables, rule or routes was dropped. This can happen if your tunUS interface is removed (this happens if you e.g. restart OpenVPN - or PIA does).
RP_Filter
If you do see traffic coming in at one place, but it doesn't reach the application or next hop, it may not be related to any of your firewall rules, ip rules, or routes. It will probably mean the return path filtering is kicking in. You can debug this using:

code:
1
echo 1 >/proc/sys/net/ipv4/conf/<interfacename>/log_martians




Edit September 24, 2013: I did some more digging. It turns out the rp_filtering will by defintion have to be disabled. The post has been modified to reflect this.

Netflix - Using a VPN for just one application

By Freeaqingme on Wednesday 11 September 2013 23:46 - Comments (17)
Categories: Linux, Networking, Views: 40.213

(note: if you're on Windows, you could take a look at this blogpost, or install Ubuntu first)

Some companies have not really yet grasped the concept of a global Internet, and like to restrict things to the public of just one country. In this case, the movie industry comes to mind, which allows Netflix to only air certain movies and tv shows in certain countries. Although I am located in The Netherlands, and Netflix recently opened their doors there as well, I'm still affected by geo-restrictions. In The Netherlands I'm served Dutch subs, and some tv shows are not available. Others, are available on DVD only in the US, whereas I can simply stream them in The Netherlands.

There's a simple solution of course, a VPN service. With most VPN services, you simply download a small application, and click & run. The VPN application will direct all your traffic to a host in a location of your choosing, and for the outside world it will look as if you were located at that location.

However, even though that works, I don't want to route all of my traffic through a US host (for you know, there's the NSA, MPAA, they got it all). So I needed a way to route all of the Netflix traffic through the VPN, while leaving the rest of my traffic unaffected.

My first attempt did not succeed. Most routing is based on destination IP, but Netflix uses Amazon and other generic CDNs for content distribution. This brings along two problems; The ip-adresses used are constantly changed, so it's near-impossible to track specific IP-addresses. Especially not, because Amazon hosts more than one website, which I'd like to access via my regular IP. The first attempt therefore used a hosts file that mapped the Amazon hostnames to alternative IP's, and based on those IP's I then did some destination based routing. The problem is that netflix uses many hostnames, therewith not really maintainable (I stopped collecting them after two-dozen). I didn't try configuring an HTTP proxy in Firefox, because I figured it wouldn't work in Silverlight. My first attempt can be found here.
Setting it up
So given that using destination IP's wasn't really an option to make any distinction on, I opted to do it based on UID. IPtables has the ability to have a firewall rule match on the user that initiated the traffic, so running netflix as its own user enables one to use a whole set of firewall rules dedicated to just Netflix.

We begin by installing netflix-desktop and setting up the Netflix-user:

code:
1
2
3
4
apt-add-repository ppa:ehoover/compholio
apt-get update
apt-get install netflix-desktop
useradd -m netflix

Allow a sudo'ed application to use X
Netflix will not start though. It requires access to X, and we haven't done anything with the network yet either. To allow Netflix to use X, it needs access to your 'xauthority' file. You can find the location of this file by running the following command in Bash.


code:
1
2
export | grep xauthority -i
declare -x XAUTHORITY="/run/gdm/auth-for-dolf-7osFEz/database"



Then when you have this file, you can grant the Netflix user access:

code:
1
2
chmod 640 /run/gdm/auth-for-dolf-7osFEz/database
chown dolf:netflix /run/gdm/auth-for-dolf-7osFEz/database



Now you should be able to run Netflix with the following command. That is, you get a nice window, telling you that you do not have access from your country/ip.

code:
1
DISPLAY=":0" XAUTHORITY="/run/gdm/auth-for-dolf-7osFEz/database" sudo -u netflix -H netflix-desktop

Networking
Next up is the networking. First you'll need a VPN service. I went with Private Internet Access (PIA) because it works with Netflix, is cheap, and offers an OpenVPN service (some are stuck in the PPTP era). To ensure not all your traffic is automatically redirected to the VPN, and to not have to type your password everytime, you do have to modify the default config files PIA provides. E.g.:

code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
client
dev tunUS   # renamed the interface name for brevity.
proto udp
remote us-california.privateinternetaccess.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
tls-client
remote-cert-tls server
auth-user-pass
comp-lzo
verb 1
reneg-sec 0

ca /etc/openvpn/privateinternetaccess/ca.crt

# Add these lines:
auth-user-pass /etc/openvpn/privateinternetaccess/passwd
route-nopull
keepalive 1 30



Linux supports multiple routing tables. When you show the contents of the routing table ('ip route show'), it shows the contents of the default routing table. Just compare the output of the following two commands:

code:
1
2
3
4
> ip route show
default via 192.168.1.1 dev eth0  proto static 
> ip route show table 254
default via 192.168.1.1 dev eth0  proto static



So after you've set up the VPN service, and you know there is more than one routing table you can use, all you need to do is configure a new routing table that sends all traffic through the VPN, and tell Linux to use that routing table for all traffic originating from the Netflix user:

code:
1
2
3
ip route add default dev tunUS table 2
iptables -t mangle -A OUTPUT -m owner --gid-owner netflix  -j MARK --set-mark 2
ip rule add fwmark 2 table 2

RP_Filter
Linux has a feature called reverse-path filtering. This feature drops traffic from any ip it has no route for on that interface. Because it does not keep into account the use of multiple routing tables, we'll need to disable it:


code:
1
2
3
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
    echo 0 > $f
    done

Audio through PulseAudio
If you start Netflix as netflix user, you should get a nice GUI, and no more errors about being in the wrong country. However, once you start to play a movie or an episode of a tv serie, you'll have no sound yet. Assuming a default Ubuntu installation, Pulseaudio is used to manage all audio, and by default it doesn't support multiple users. Now you can try to get that right the right way, or you can go the quick-and-dirty way (my favorite). The only disadvantage is that if Netflix is playing, no other applications can use the sound system.

code:
1
2
3
4
sudo usermod  -a -G audio netflix
sudo ln -s /home/dolf/.pulse-cookie /home/netflix/.pulse-cookie
sudo chown dolf:audio .pulse-cookie 
sudo chmod 640 .pulse-cookie



Now you should be good to go, and all should be running.

I myself proxy all my traffic via my own VPS (located in the EU as well), and from there I route my traffic via the appropriate VPN service. But because the Netflix application runs locally, I cannot make a distinction based on GID on my VPS. In a next blog I'll show how you can still get that done. EDIT: New blogpost: Netflix - Application based routing over another VPN

Edit September 24, 2013: I did some more digging. It turns out the rp_filtering will by defintion have to be disabled. Therefore, the section 'RP_Filter' has been added.
Edit September 26, 2013: Added the keep-alive statement to the OpenVPN config so there are less timeouts and your routes etc should be more stable due to less changing IP's.