HOWTO: Netscreen and Windows XP native VPN client

  • Hello,

    as the subject of this topic states I want to create a dialup VPN to a Netscreen Firewall. To make it more complex I will choose the onboard Windows VPN client. At the moment only a few documentations exist and most of them contain only pieces of the whole setup process. Different threads in discussion groups out there state that the whole XP-VPN/Netscreen thing will not be possible but they lack the proof that this cannot be done. In the next weeks I will take some time to investigate all the traps that may occur in this context and will continue this thread whenever I have reached a new “milestone”. Please be patient with some modes of expression but I am not an firewall expert at all. I know some basic concepts but from time to time I may choose the wrong explanation. Maybe I will fail, maybe I will need some magic, but at least a can say afterwards that I have tried it. So let the headache begin …

    STEP 1: Checking prerequisites

    At the moment there are only two software requirements that have to be met. You will need a Netscreen with ScreenOS 5.1 and higher. This is the first firmware that will support a VPN client behind a router that has no official internet IP address (NAT-T). On the client side you will need Windows XP with at least SP2. It contains the so called L2TP/IPSec-NAT-T-Update (MS KB818043) that will enable the client to handle this situation correctly.

    From the design aspect we will have to use a certificate based IPsec authentication. Preshared keys will not work for Windows XP roadwarriors as the builtin client only sends its IP or its FQDN as Phase 1 ids. This does not matter as certificates should always be the choice to go.

    STEP 2: Creating a root certificate.

    When working with certificates we will have to obtain a root certificate (or certificate authority = CA) that can be used to check the trustworthiness of any user or firewall certificate that we need in the later setup process. As we want to take the cheap way, we will generate the root certificate ourselves with openssl on a linux machine. Nothing easier than that. First we will create a private key for the root certificate.

    linux > openssl genrsa -des3 -out ca.key 4096
    Generating RSA private key, 4096 bit long modulus
    e is 65537 (0x10001)
    Enter pass phrase for ca.key: <password>
    Verifying - Enter pass phrase for ca.key: <password>
    linux > ls -l
    total 4
    -rw-r--r-- 1 root root 3311 Jan 13 19:40 ca.key</password></password>

    Quite fine. Now we have the root key and we will use it to create our root certificate. A lifetime of 20 years (or 7300 days) will keep maintenance low. Let’s assume we are working for ACME Inc. and so we will insert some nice data into this certificate.

    linux > openssl req -new -x509 -days 7300 -key ca.key -out ca.crt
    Enter pass phrase for ca.key: <password>
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:Florida
    Locality Name (eg, city) []:Tampa
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:ACME Inc.
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:ACME Inc.
    Email Address []:
    collnx05:~/x # ls -l
    total 8
    -rw-r--r-- 1 root root 2114 Jan 13 19:45 ca.crt
    -rw-r--r-- 1 root root 3311 Jan 13 19:40 ca.key</password>

    And voila ca.crt is our root certificate. This could be a good basis for printing money if we convince someone to sign his certificates by us. But the only thing we want to do at the moment is to load this certificate into the Netscreen. Simply logon to your firewall an go to Objects > Certificates. Search for your ca.crt File and click on load. When you change the view to Show CA afterwards you should see your just uploaded certificate.

    Enough for today. Stay tuned for updates …

  • Is oakley.dll part of Windows 7 as well.  I do not see it when searching in Win 7?  Is it possible to use windows 7 client to make an L2TP IPSec vpn connection to a Juniper SSG-140 without the use of certificates?  If not, how can I adapt this “How To” article to a windows 7 client if I do not have “oakley.dll” in win 7?  Any help would be greatly appreciated .  Thanks



  • Yes, it does work in 6.2 and 6.3 either.

    To get the complete log go to the Console and enter the follwing commands:

    1. clear db
    2. debug ike all

    try to connect via VPN

    4. undebug all
    5. get db st

    than you see the complete log.

    I also found an article from the Microsoft-VPN running with NAT-T on Win XP:

    and the same for Win Vista (works for me on Win7 as well) and Server 2008:

    This makes the DLL-Changes unneccessary.

  • Hello,
    Does this work on Win 7? I tried following the steps but it stops with this log and an 789 error on the client:

    2012-07-17 21:01:03 info IKE [client IP] Phase 2 msg ID 00000001: Responded to the peer’s first message.
    2012-07-17 21:01:02 info IKE [client IP] Phase 1: Completed Main mode negotiations with a 28800-second lifetime.
    2012-07-17 21:01:02 info IKE [client IP] Phase 1: Completed for user username.
    2012-07-17 21:01:02 notif PKI: No revocation check, per config, for cert with subject name,CN=username,.
    2012-07-17 21:01:02 info IKE [client IP] phase 1:The symmetric crypto key has been generated successfully.
    2012-07-17 21:01:02 info IKE [client IP] Phase 1: Responder starts MAIN mode negotiations.

    Is there any way to get more detailed logs? Or, can anyone say something about these log entries?


  • Thanks for the article, but does it work on ScreenOS 6.2 firmware? I’m getting error: “General error, the operation failed” when i try to upload a cert described in step 4.

  • Hello,

    just to sweep away any concerns: Editing oakley.dll in the wrong way will not break your Windows and prevent it from booting but only the IPSEC implementation. Give it a try.

    Best regards.

  • Hi,
    ok, I’ve done. I have used Notepad++ with the hex editor plugin.
    The version of my source file is 5.1.2600.5512 in italian language, it’s a problem?
    Is there a way to post my patched file in this forum?

    Best regards.

  • I’m sorry to tell you that I will not spread a modified oakley.dll as I do not want to redistribute a software that is not mine with patches that I have applied. The fix is easy and well documentend. If you have come all the way till there it should be no problem to come over this. All you need is a HEX editor and search/replace the mentioned bytes.

    Best regards.

  • Hi,
    I’m not able to modify oakley.dll, may I have a patched file version, please?
    So I can test your solution.

    Best regards.

  • Funny, after moths of silence the thread has found its way back to the first page of the forum.

    As I stated earlier in my article you have to set the AutoKEY IKE definition to transport to enable L2TP. Additionally you must set the NAT-T flag in the gateway definition. Who ever has written Juniper KB6439 may be right but reality proves that truth must be somewhere beyond this article. For about one year me and some coworkers use L2TP over IPSEC from behind routers without any problems. Components are XP SP3 native client and SSG140 Firmware 6.1.0r4.

      Prerequisites for L2TP over IPSEC:
        1. Certificates
        2. a patched Windows oakley.dll

    Best regards.

  • This is to bring to your notice… that L2tp over IPSEC is not supported if the client is behind the nat device.

    Refer to “” which clearly explains the reason why nat-t is not supported.

    To be brief, l2tp over uses transport mode unlike normal ipsec which uses tunnel mode.

    The firmware on the firewall has no role to play in this case as it is a client which used transport mode.

    The issue was tested in the lab and the same results were observed.


    Prerequisites of l2tp over ipsec:

    1. certificates
                2. routable public ip address (with no nat device between the client and the firewall)

  • I think I get the answer

    “Enable NAT-Traversal”,  it is for vpn client without public ip if your pc behind firewall, you must enable and use NS/XP ways

    2.if your pc dial up with public ip,you can use juniper kb10939

    But I don’t like juniper, it make VPN configure difficult than netscreen.
    and I need to modify the dll file. very unfriendly.

    My home firewall’s VPN, I just check VPN button, then config user
    then it’s OK.

    WAWAWA, this take me two days for try errors.

    last, 3q NS/XP again.
    I hate myself, why not trust him at begining. why am I so stupid to
    trust juniper kb. and other old version netscreen vpn paper.
    O mygod

  • Hi,

    why do I check “Enable NAT-Traversal”, there is error

    “Rejected an IKE packet on ethernet0/0
    from to with cookies
    811a892d285a5d6f and b7dcc6d168a5194f
    because The peer did not send a proxy

    if I don’t enable , it can pass phrase 2, but connect fail win error 678.
    IKE Phase 2 msg ID
    c3ca8c69: Completed negotiations with
    SPI ec6af547, tunnel ID 32772, and
    lifetime 3600 seconds/250000 KB.

    How can I do?

  • Hi.

    Changing the DLL should be straight forward. you should only need an hex editor and open the file. Afterwards search for the hexadecimal string and you should find one. if that is not the case maybe you have another oakley.dll than me (mine has version 5.1.2600.2180).

    If you have no luck you will need tools like IDA Pro to disassemble the code. This will reveal the assembler code in which you have to find the use “GetComputerFQDN failed %d” message. At this location you will see the required bytes you have to change to modify the jump destination.

    Best regards.

  • I will include the neccessary conversion you have to apply to the oakley.dll. But remember: YOU WILL DO THIS ON YOUR OWN RISK:

    system32\oakley.dll (Windows XP 32 Bit): replace 3BC7741C5068 by 3BC7751C5068
    system32\oakley.dll (Windows XP 64 Bit): replace 85C0743389 by 85C0753389

    Pay attention that the byte sequences only occur once in the oakley.dll. If you want to patch the file you will have to take care of the safety copies Windows stores in system32\dllcache and/or ServicePackFiles. Just search all occurrences of the library and fix them in the right order. Reboot your machine and check that the changes in system32\oakley.dll have been made permanent. Finally you have a fully working VPN client for your Netscreen.

    Would you be able to go into a little more detail on how to do this, I don’t regularly edit dll’s :). Even after disassembling the dll I cant seem to find 3BC7741C5068.


  • I realy miss this configuration for smart card using windows xp client without ns-remote and I followed this article and it is working but I dunno how to switch this for using a smart card!
    my boss wants me to have a solution for clients smart card using native windows xp vpn and if I can’t do it I will loose my job please inform me if you have any comment on this feel free to contact me at

  • ISSUE 2: NAT-T

    Starting this topic I had never thought that it would be so challenging. Now I know why it has been covered so rarely in the past. The less work you want to have on the client side, the harder you have to fight on all the other fronts. But why should I use a memory wasting VPN client bloated with features I do not need. Of course all programs have their existence authorization but the longer I work in the IT the more I’m attracted to clean and easy tools on my workstation. Different strokes for different folks.

    After all I managed to fix the problem that kept me busy the last days. While writing these lines I feel a great satisfaction. Man has won over machine. But what it is all about you may ask. So we will go into detail for the last time.

    In phase 2 of the IKE handshake the client and the server of a VPN connection have to decide how they will identify themselfes. Possible values are email address, IP address or FQDN, … The Netscreen accepts an IP address and this is no problem if you use the Netscreen client. The Windows native client provides an IP address as long as it has an official IP address and NAT-T is not required. If the client works behind a router this behaviour changes dramatically. All of a sudden it will simply send an FQDN and the Netscreen will block any further packet. I asked why so often during my analysis but it led me to no conclusion. All the normal tricks faded away as soon as I tried them out. If you look at the interesting part in the IKE debug trace it reads like this:

    12:31:22 : IKE <w.x.y.z >Recv*: [HASH] [SA] [NONCE] [ID] [ID] [NAT_OA] 
    12:31:22 : IKE<w.x.y.z >   extract payload (352): 
    12:31:22 : IKE<w.x.y.z >   QM in state OAK_QM_SA_ACCEPT.
    12:31:22 : IKE <w.x.y.z >ERROR: Cannot handle this id type, 2!</w.x.y.z ></w.x.y.z ></w.x.y.z ></w.x.y.z >

    So what to do if everything fails? We have to fix the client! Call it what you want for me it is nothing more than a program enhancement to fit my needs. To show you what I mean I will give you an extract of the oakley.dll:

    push    eax
    mov     [ebp+var_10C], 100h
    call    _GetComputerFQDN
    add     esp, 0Ch
    cmp     eax, edi
    jz      short loc_756CD5F7
    push    eax
    push    offset aGetcomputerfqd ; "GetComputerFQDN failed %d"
    push    edi
    push    edi
    push    edi
    push    edi
    push    80000000h
    call    _IKEDbgMsg
    add     esp, 1Ch
    jmp     loc_756CD55B

    If you are a little bit into programming you can see what happens here. The library tries to determine the FQDN of the client. If this succeeds (return code in register EAX=0) it will continue in the default way otherwise it will write a message to the log. The normal behaviour will be that the GetComputerFQDN finishes without errors and the client can use the FQDN in IKE phase 2. And exactly at this point we will place a small modification (nothing more than a single bit). We will convert the conditional JZ jump into a JNZ. This means the client thinks it cannot determine the FQDN. Luckily the routine will not abort but will send its IP address to the firewall instead. I cried out loud when I saw the first NAT-T connect to my rebellious Netscreen. To make the solution easy for the non assembler oriented reader I will include the neccessary conversion you have to apply to the oakley.dll. But remember: YOU WILL DO THIS ON YOUR OWN RISK:

    system32\oakley.dll (Windows XP 32 Bit): replace 3BC7741C5068 by 3BC7751C5068
    system32\oakley.dll (Windows XP 64 Bit): replace 85C0743389 by 85C0753389

    Pay attention that the byte sequences only occur once in the oakley.dll. If you want to patch the file you will have to take care of the safety copies Windows stores in system32\dllcache and/or ServicePackFiles. Just search all occurrences of the library and fix them in the right order. Reboot your machine and check that the changes in system32\oakley.dll have been made permanent. Finally you have a fully working VPN client for your Netscreen.

    My short trip into the deeps of VPN client connectivity have been very instructive. I learned quite a lot about my firewall and its operating principle but this peek should be enough for the next couple of months. Nevertheless I know now that a Netscreen & a Windows XP native VPN client will be the team that supports me best. Hopefully you enjoyed reading this article as much as it was fun for me writing it. A very special thanks goes to Mike and Henning for providing me the test environment.

    But now its time to close this article with best regards.


  • ISSUE 1: Split Tunneling

    Already mentioned above all client traffic will be routed by default into the VPN connection. This is a security aspect that one cannot neglect. When you are connected to the network of ACME Inc. the infrastructure of the company should provide you all the neccesary services you need for working. So if you want to surf the internet when being connected by VPN the requests should go the way laptop > VPN > company proxy > internet. Additionaly one can argue that a normal PC in the ACME building underlies the same restrictions. It is not allowed to participate in the intranet on the one hand and have an second network card to be connected to the an insecure network of the neighbour company. So by default the configuration will be absolutely correct for the person with security in mind.

    Let’s assume you want to allow the roadwarriors to acces your intranet and use their current network connections at the same time. This could be helpful if the companies internet connection has only a slow uplink and a lot of teleworkers do not only work on the inside resources but also surf the internet through the VPN. In this case the uplink bandwidth represents a bottleneck that can cause severe slowdown.

    The magic word here is split tunneling. It describes the ability of the client to route all traffic to the company network via VPN and all other traffic to the normal network connection. Basically it is nothing more than enhancing the default routing table with a new entry that routes all traffic to the ACME intranet ( through the VPN. This can be done manually by the user every time he connects - a solution no one really wants - or automatically. Sadly the automatism is not implemented in the windows client by default. Therefore we need CMAK.

    CMAK or known as Connection Management Administration Kit is a sofware component on the Windows 2003 Server CD. It provides a wizard to create a self installing binary that an user can execute without any further input on his laptop. It will create the VPN connection and all its settings so that the user may directly launch the VPN afterwards. No other efforts required. A detailed description of this tool will go far beyond the scope of this article. Therefore I just reference to the best documentation I found throughout the net in my opinion. If you are interested take some time and read more at

    The essential point of the configuration process is the step Routing Table Update. There you can provide a file with a new routing table. Everytime a client will connect to the CMAK created VPN connection it will automatically update its routing table corresponding to the contents of this file. The exact syntax of this file is more or less an abbreviation of the windows route command. In our case the file wll contain only one line:

    ADD MASK default METRIC default IF default

    It tells the client that only the traffic to the ACME intranet will go through the VPN. I guess you cannot make it easier than this. Nevertheless it took me some time to find this secret solution any normal VPN client is capable of by default. That only leaves a single issue unsolved …

  • STEP 13: Create the VPN connection

    Now that the laptop is prepared we can setup the new VPN connection. Open the network configuration wizard and choose Connect to the network at my workplace as the network type. On the next page we will select VPN connection (what else) and continue to name the connection ACME VPN (or whatever you like). As we will manage the standard internet connection (dialup or via router) ourselves we do not want to dial an initial connection automatically. In the next input field we will enter the address of the Netscreen outside interface. In this case it is On the last page we say that we don’t need a smardcard and finish the wizard. As the connection is configured we have to adapt the VPN type. Right click on ACME VPN, choose Properties and switch to the Network tab. It is strange that the default Automatic VPN type cannot build a L2TP over IPsec connection. Maybe Microsoft knows… So we select L2TP over IPsec manually and save the changes.

    STEP 14: Connect ot the VPN

    The time has come to build the initial connection to the ACME VPN network. If you are directly connected to the internet and the laptop has an official IP address this will be possible by now. Otherwise you have to wait for my upcoming posts that will cover two unsolved issues. Open the newly created connection and provide the username and the password. Both of them must match the user settings in the Netscreen. Hit Connect and the connection should be established.

    At this point we have come exactly as far as the few documentations I have found in the internet. Nevertheless I hope I managed to give you a far more complete view of this topic. Hopefully we will go beyond that the next days…

  • STEP 12: Import certificates to user laptop

    A user that wants to logon to the ACME intranet needs two certificates. His user certificate (stored in john.doe.p12) and the root certificate root.crt. It is not possible to hand out the user certificate alone because we need a so called valid certificate chain. This means a program that wants to check the validity of a certificate will go back the signing chain until it finds a trusted root certificate. In our case we have only one dependency and therefore we need two certificates. Start the Microsoft management console (mmc.exe) on your windows client and open the dialogue Add/Remove Snap-in. Choose Add and select Certificates in the following window to insert the certificate snap-in to the management console.

    You now have the choice which certificates you want to manage. Here you must select the computer account. If your company has high security standards this should concern you. The Windows VPN client will always search for the certificates in the user independend computer account. This means that anyone who has access to the machine can at least finish the IKE handshake (phase 1 and 2) if he knows the IP or DNS name of the Netscreen outside interface. To build a connection via a L2TP tunnel to your network he needs a username and a password of course. This should always be remembered when implementing a VPN with the standard Windows client. If you want to proceed click on Next. Mark the radiobutton This Snap-in manages the local computer and return to the previous dialogue by selecting Finish. You don’t need any other snap-ins so click on Close and shut the last open window with Ok.

    You should now see the certificate tree in the console. Pay attention that the root node in the left pane reads Certificates (Local Computer). Everything else will be the wrong place for your certificates. To import the root certifcate root.crt go down the tree Trusted Root Certification Authorities > Certificates. There you will see a lot of root certificates your machine already trusts. Right click on the leaf Certificates in the left pane and select All Tasks > Import … A wizard will open and asks for the location of the certificate you want to import. Provide the full path to the file root.crt. Click on Next and select the radio button Import certificates into the following store. Don’t change anything else. Just select Next and afterwards Finish. The root certificate of ACME Inc. is now in place and should show up in the right pane.

    We have to repeat the same steps for the user certificate. Therefore right click on the leaf Personal > Certificates and select All tasks > Import … again. Do everything what you made before for the root certificate but this time select the file john.doe.p12. As the PKCS12 certificate is password protected the wizard will ask for that. Simply enter the password you chose during the creation process. After finishing the wizard your user certificate will be on the laptop too.

    This guide is slowly coming to its end and I must admit that I’m still working hard on the NAT-T miracle. I don’t like the idea to be defeated by this lousy oakley.dll doing its FQDN handshake all the time.