HOWTO: Netscreen and Windows XP native VPN client

  • Hello,

    as the subject of this topic states I want to create a dialup VPN to a Netscreen Firewall. To make it more complex I will choose the onboard Windows VPN client. At the moment only a few documentations exist and most of them contain only pieces of the whole setup process. Different threads in discussion groups out there state that the whole XP-VPN/Netscreen thing will not be possible but they lack the proof that this cannot be done. In the next weeks I will take some time to investigate all the traps that may occur in this context and will continue this thread whenever I have reached a new “milestone”. Please be patient with some modes of expression but I am not an firewall expert at all. I know some basic concepts but from time to time I may choose the wrong explanation. Maybe I will fail, maybe I will need some magic, but at least a can say afterwards that I have tried it. So let the headache begin …

    STEP 1: Checking prerequisites

    At the moment there are only two software requirements that have to be met. You will need a Netscreen with ScreenOS 5.1 and higher. This is the first firmware that will support a VPN client behind a router that has no official internet IP address (NAT-T). On the client side you will need Windows XP with at least SP2. It contains the so called L2TP/IPSec-NAT-T-Update (MS KB818043) that will enable the client to handle this situation correctly.

    From the design aspect we will have to use a certificate based IPsec authentication. Preshared keys will not work for Windows XP roadwarriors as the builtin client only sends its IP or its FQDN as Phase 1 ids. This does not matter as certificates should always be the choice to go.

    STEP 2: Creating a root certificate.

    When working with certificates we will have to obtain a root certificate (or certificate authority = CA) that can be used to check the trustworthiness of any user or firewall certificate that we need in the later setup process. As we want to take the cheap way, we will generate the root certificate ourselves with openssl on a linux machine. Nothing easier than that. First we will create a private key for the root certificate.

    linux > openssl genrsa -des3 -out ca.key 4096
    Generating RSA private key, 4096 bit long modulus
    e is 65537 (0x10001)
    Enter pass phrase for ca.key: <password>
    Verifying - Enter pass phrase for ca.key: <password>
    linux > ls -l
    total 4
    -rw-r--r-- 1 root root 3311 Jan 13 19:40 ca.key</password></password>

    Quite fine. Now we have the root key and we will use it to create our root certificate. A lifetime of 20 years (or 7300 days) will keep maintenance low. Let’s assume we are working for ACME Inc. and so we will insert some nice data into this certificate.

    linux > openssl req -new -x509 -days 7300 -key ca.key -out ca.crt
    Enter pass phrase for ca.key: <password>
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:Florida
    Locality Name (eg, city) []:Tampa
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:ACME Inc.
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:ACME Inc.
    Email Address []:
    collnx05:~/x # ls -l
    total 8
    -rw-r--r-- 1 root root 2114 Jan 13 19:45 ca.crt
    -rw-r--r-- 1 root root 3311 Jan 13 19:40 ca.key</password>

    And voila ca.crt is our root certificate. This could be a good basis for printing money if we convince someone to sign his certificates by us. But the only thing we want to do at the moment is to load this certificate into the Netscreen. Simply logon to your firewall an go to Objects > Certificates. Search for your ca.crt File and click on load. When you change the view to Show CA afterwards you should see your just uploaded certificate.

    Enough for today. Stay tuned for updates …

  • This is already covered in Juniper KB8536 ( In the L2TP over IPSec section, there is a technical note which describes how to do this from Windows and from ScreenOS sides. The technical note mentions Win2K and ScreenOS 4.0, but the details are still accurate.

    The caveats are, as mentioned, you must either use peer ID type as IP address or ASN-DN (certificates). Since most VPN clients are usually from a dynamic IP, then that means pretty much you have to use PKI certs. As mentioned, this is Windows client limitation (no provision to specify FQDN or U-FQDN). The second caveat is there is a chance that you could experience issues if behind a NAT device that has a problem translating ESP (IP proto 50) traffic. This is because with L2TP over IPSec, you must use transport mode on the NetScreen which doesn’t support Nat-traversal.

    Hope this helps.

  • Thanks for your remarks MaxPipeline. My documentation will partly depend on this source. Sadly these types of documentation always lack the description for getting around the caveats. What I want to build here is some kind of memo for myself that I can read again in a few months and do not have to search the whole web for additional tips and tricks. But now back to the setup process.

    STEP 3: Certificate Revocation List

    When you give away certificates that where signed with a root certificate it may be required to mark the certificate invalid although it’s desired lifetime is not over yet. This ist done by a CRL (certificate revocation list). A CRL of a root certificate is nothing more than all the signed certificates that should no longer be trusted. A Netscreen will handle CRLs but do we need them? In my simple setup this will never be the case for the following reason. In IKE phase 1 a client will send its certificate that the netscreen has to match in its user database. This is done by comparing the contents of the certificate (Email, CN, Username, …) with the information that is stored in each dialup user. The handshake will fail if no corresponding user data is found. In the reversal conclusion we can say a certificate will be invalid as soon as the designated user is deleted from the Netscreen.

    To disable the CRL for a root certificate we have to logon to the device by telnet or ssh and have to determine the internal certificate id. It si only because I did not find a feature in the web interface to accomplish the task.

    netscreen-> get pki x509 list ca-cert
    Getting CA CERT ...
    IDX  ID num     X509 Certificate Subject Distinguish Name
    0003 89129012   CA CERT friendly name <52>
                    CN=ACME Inc.,O=ACME Inc.,L=Tampa,ST=Florida,C=US,
                    Expire on 01- 8-2028 18:45, Issued By:
                    CN=ACME Inc.,O=ACME Inc.,L=Tampa,ST=Florida,C=US,

    Our root certificate is stored with the id 89129012. With this we can disable the revocation check in the following way.

    netscreen-> set pki authority 89129012 cert-status revocation-check none

    To save the changes we just have to logoff from the device

    netscreen-> exit
    Configuration modified, save? [y]/n y

    Now our root certificate is in place.

  • STEP 4: Firewall certificate

    Creating an L2TP over IPsec Dialup VPN with certificates means that both sides (client and firewall) need their own certificate. Luckily the netscreen supports the generation of a private key and a signing request. So we will jump into the web interface to objects > certificates > new and create a new one. Here you normally can enter any text you like. Nevertheless I want to implement the FQDN feature that signal15 described in his thread (MAC OS VPN client & Netscreen). What is the idea behind that? Let’s assume our firewalls outside interface can be reached via DNS name If the client tries to do the IPSec handshake it will receive the certificate of the firewall. Anywhere in this certificate the DNS name should occur. Only if this condition is met the client can trust its partner because the certificate was created exclusively for this domain. So we will insert the following data into the certificate subject information:

    Name : ACME Firewall
    Organization : ACME Inc.
    FQDN :

    Don’t forget to set the FQDN to what the client will point to later. Set the key to RSA 2048 bits and let the netscreen generate a private key and a certificate signing request. We will copy the request out of the following screen and store it side by side to our root certificate as file fw.csr. Tries over tries led me to the conclusion that an offical signing request will place the FQDN automatically into the additional certificate field called alternative name (subjectAltName). Unfortunately this does not apply if one is using self signed certificates. So we will have to add the field manually. For this reason we have to create the file subj.txt with the following content.


    With this file we finally can sign our firewall certificate

    linux > openssl x509 -req -days 7300 -in fw.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out fw.crt -extfile subj.txt
    Signature ok
    subject=/O=ACME Inc./CN=0096052005000522/CN=rsa-key/ Firewall
    Getting CA Private Key
    Enter pass phrase for ca.key: <password></password>

    The above step creates the new certificate file fw.crt with a lifetime of 20 years that is “officially” signed by our own root certificate. Pay attention that the serial you provide in the command line is not used by another certificate that is already loaded into your netscreen. Before transferring the file to the firewall we can take the last chance to check if the subjectAltName has found its way into the certificate.

    linux > openssl x509 -in fw.crt -text -noout 
            X509v3 extensions:
                X509v3 Subject Alternative Name:

    This worked well and we can log on to the netscreen web interface again and go to objects > certificates. Provide the path to the file fw.crt and upload it to the firewall. I think this was not to difficult and saved quite a lot of money. Btw. if anyone experiences problems when generating and signing certificates with 64bit Linux machines should try the same on a 32bit device.

  • STEP 5: Creating a user group

    Before we can setup the IKE firewall configuration we need a user group. For this we simply jump to objects > users > local groups in the web interface of the netscreen. There we create a new group ACME Teleworkers. Remote users will be assigned to the group later.

    STEP 6: IKE gateway

    Now it is time for the configuration of IKE phase 1. The webinterface provides the required dialog at location VPNs > Autokey Advanced > Gateway. As one would expect we create a new entry and call it GW-ACME-Teleworkers. The dialup users will connect to our firewall from changing IP addresses and so the only solution for setting a remote gateway type will be Dialup User Group. Choose the new item ACME Teleworkers from the group drop down list. All other fields on this page are not from interest. Preshared keys will not be used and our local id is stored in the firewall certificate.

    From here we directly go to the advanced configuration page. For simplicity I set the security level to custom and provide only rsa-g2-3des-sha as phase 1 proposal. The tree other proposal fields can be set to none. The Windows VPN client only supports main mode negotation (a handshake with more phases) so we have to enable that radio button. Activate the Enable NAT-Traversal checkbox and set keepalive frequencies or heartbeat settings to whatever fits your needs best. Our szenario does not need any XAuth settings so we can set this parameter to none.

    Finally we must choose the preferred certificates for this gateway. The local certificate will be the firewall certificate (CN=ACME Firewall) and the peer CA must be the corresponding root certificate (CN=ACME Inc). Don’t forget to set the certificate type to X509-SIG. Selecting the Return button we come back to the basic settings where we choose Ok to save the new gateway.

  • STEP 7: IKE phase 2

    Now it is time to define the phase 2 parameters for the new gateway (phase 1). This can be done in VPNs > AutoKey IKE. For readability we call this VPN IKE-ACME and assign the predefined gateway GW-ACME-Teleworkers to it. In the advanced options there is again some work to do. We set the security level to custom with the proposals nopfs-esp-3des-sha, nopfs-esp-3des-md5, nopfs-esp-des-sha and nopfs-esp-des-md5. I think we won’t need them all, but this time a had not leisure to test it out. The flag Transport Mode has to be checked because the encryption of packet headers in tunnel mode would break the communication in a NAT environment.

    Binding has to be set to none and not only proxy ids but also the local and remote ips have to be left blank. The reason should be clear. We are building a policy based routing and the client will connect with a random IP as proxy id. Leave the other fields as they are and click on Return. Back in the normal Autkey IKE data we choose Save to make the changes permanent.

    STEP 8: Defining an IP pool

    When providing an dialup network the outside teleworker will receive an IP address on its virtual adapter VPN that is different from the inside network. This is due to the fact that otherwise an inside machine would not answer incoming TCP/IP packets by sending them back to the gateway (the Netscreen) that will forward them via the VPN tunnel. Let’s assume ACME Inc. has an inside network with IP adresses in the range The teleworkers should be assigned the address range from In Objects > IP Pools one can define the required address intervall. We choose New and name the IP pool IPPool-Teleworkers. Insert as the desired interval and save the settings.

    Make sure that there are no overlapping networks when creating an IP pool. If you have multiple gateways also remember that the default gateway in your inside topology for the VPN network is pointing to the internal Netscreen interface.

  • Very thorough, at least so far as you’ve written to date. Thanks.

    I’ve been butting heads with this exact issue, and was happy to see someone else addressing it at the top of the forum when I checked.

    Max, the KnowledgeBase article you reference seems to not work right now. I’d love to read it, but when I click on it takes me to the blank page instead. Searching the knowedge base for article 8356 doesn’t seem to work, either.

  • Thank you for your acknowledgement. Just to inform everyone what is up behind the scenes I’m glad to say that I’m still confident to satisfy everyones expectations of this article. But there are two showstoppers I’m working on currently. First of all Windows routes all traffic into the VPN tunnel after I connect to my firewall. This may be desired but I do not like it and I guess I have found an automatic workaround by now. The second caveat will be much harder. Everything is working fine when the client has an official IP address. When being behind a NAT router the client sends an FQDN as phase 2 id instead of an IP address and the firewall stops the handshake. This is covered in Juniper KB3294. I have a strong belief that a miracle will happen to make it work. It should not concern you at the moment but praying might help. And now back to something complete different …

    STEP 8: L2TP Tunnel

    After defining the IPsec parameters for our connection it is time to manage the L2TP tunnel parameters. Enter VPNs > L2TP > Tunnel and create a new one called Tunnel-ACME-Teleworkers. We don’t want to bother with L2TP default settings of the Netscreen so we choose custom settings this time. The users will be maintained in the firewall so the authentication server must be set to local and we will allow access for the dialup user group ACME Teleworkers. Everything that deals with interface, peer IP, host and secret is not in our focus. Just leave the fields as they are. For the IP pool name we take the newly created IPPool-Teleworkers. The ACME company maintains a DNS/WINS server which can be reached by IP and that is exactly what we will provide in the corresponding fields of the web dialogue. Choose save and again a step is finished.

    STEP 9: Policy

    To allow traffic from the VPN notebooks to our internal network we still need a policy. This will be the simplest part of the whole configuration. Open the Policies page, choose the direction “from untrust to trust” and create the object Policy-ACME-Teleworkers. The source address is our address book entry Dial-Up VPN. It is an predefined entry in your Netscreen and it will fit to any address that matches not otherwise. Your local LAN should be the destination address and the allowed services can be set to ANY - supposed you want to give full access to the dialup clients. Because we have not defined a tunnel interface yet we will choose the action Tunnel. For this tunnel the VPN has to be set to IKE-ACME and L2TP needs be Tunnel-ACME-Teleworkers. Activate logging if you like and choose Ok to save your changes.

    With this the firewall internal ruleset is built and we will care for the client configuration in the next posts.

  • Regarding the issue that everything goes over the VPN Tunnel: That’s default behavior for Windows VPN. Full Tunnel is viewed as more secure. To enable split tunnelling, go into the TCP/IP properties on the VPN connection, hit the advanced button, and uncheck “Use default gateway on remote network”.

    For a more full explanation, see here:

  • Apparently the above knowledge base article link is failing because the article number is typoed. It should be:
    …or better yet, the tech note linked from there: 4.0.0/VPN-400-003.htm

    Found with the help of a Juniper support agent.

  • I agree with you that unchecking the default gateway in the TCP/IP options of the VPN connection allows me to prevent the client from routing all its traffic into the tunnel. The negative side effect of this is that the routing table on the client side is not updated at all and no traffic will go into the VPN. But we will cover this later on.

    STEP 10: Creating dialup users

    Now that everything is in place on the firewall we can create the users that will be allowed to connect to the ACME company network. For each user we go to Objects > Users > Local and hit the New button. The first person we want to grant access is our employee of the month John Doe. So we name the user John.Doe and enable him. This will be the user id the client has to provide in the username field of the dialup dialogue. As this entry should be taken into account for IKE handshakes we check the flag IKE User. Mark the Use Distinguished Name for ID radio button and enter John.Doe.01 for the CN and as the email. These texts will have to match the contents of the user certificate we will hand out to the client. You may ask why I chose to enter the number 01 into these text fields but the reason behind that is simple. If John Does laptop will be stolen one day and someone else has access to his certificate I simply change the 01 to a 02 and create a new user certificate. This is nothing more than the above mentioned “manual management” of certificate revocation lists. If the firewall is hit during IKE phase 1 with the leaked certificate entries CN=John.Doe.01 & it will not find a matching user id and aborts the IKE phase. Check L2TP User so that the account will be allowed to open an L2TP tunnel after its IPsec negotiation is over. Finally we provide the user password that has to be entered on the client afterwards and save the the account. Jump to Objects > Users > Local Groups and add John Doe to the group ACME Teleworkers.

    STEP 11: Client certificate

    A user who wants to identify himself to the firewall needs a certificate that matches an entry in the firewall. For this the firewall will compare any subject field of the certificate (CN,email,OU,…) with the user data in its database and will grant access if it finds an entry that has the same field contents. So we have to create a certificate that fits to our new user John Doe. Again I use openssl to generate a private key file john.doe.key for the user.

    linux > openssl genrsa -des3 -out john.doe.key 4096
    Generating RSA private key, 4096 bit long modulus
    e is 65537 (0x10001)
    Enter pass phrase for john.doe.key: <password>
    Verifying - Enter pass phrase for john.doe.key: <password></password></password>

    Insert any password you like. It is only needed during the creation process of the certificate and needs not to be hand out to the user. The next step is clear. To obtain a certificate from a private key we need a signing request john.doe.csr. Nothing easier than this.

    linux > openssl req -new -key john.doe.key -out john.doe.csr -subj "/CN=John.Doe.01/"
    Enter pass phrase for john.doe.key:<password></password>

    Hopefully you did not throw away your root certificate. We need it at this point to sign the user certificate. In contrast to the root certificate I only create a user certitificate with a validity of one year. This will help to keep the maintenance process easy if users leave ACME Inc. and the firewall administrator is not informed of this.

    linux > openssl x509 -req -days 365 -in john.doe.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out john.doe.crt
    Signature ok
    Getting CA Private Key
    Enter pass phrase for ca.key: <password></password>

    Now we have a client certificate john.doe.crt but sadly the Windows client cannot handle this type of file correctly. What we need is a PKCS12 certificate file. This file is a password encrypted bundle of an unencrypted private key and the matching certificate. Or to say it the other way round we take the john.doe.key file and merge it with the john.doe.crt file into a new john.doe.p12 file that is secured by a new password.

    linux > openssl pkcs12 -inkey john.doe.key -in john.doe.crt -export -out john.doe.p12
    Enter pass phrase for john.doe.key: <password>
    Enter Export Password: <password handed="" out="" to="" the="" user="">
    Verifying - Enter Export Password: <password handed="" out="" to="" the="" user=""></password></password></password>

    I guess you can see clearly that we have to hand out the PKCS12 password to the user afterwards and not the key password. This is because the key will be decrypted before storing it in the PKCS12 file.

    Congratulations. If you could follow this topic until here you managed the hardest part. The setup process on the client side will be much easier but nevertheless I will continue this topic with all required detailed descriptions.

  • STEP 12: Import certificates to user laptop

    A user that wants to logon to the ACME intranet needs two certificates. His user certificate (stored in john.doe.p12) and the root certificate root.crt. It is not possible to hand out the user certificate alone because we need a so called valid certificate chain. This means a program that wants to check the validity of a certificate will go back the signing chain until it finds a trusted root certificate. In our case we have only one dependency and therefore we need two certificates. Start the Microsoft management console (mmc.exe) on your windows client and open the dialogue Add/Remove Snap-in. Choose Add and select Certificates in the following window to insert the certificate snap-in to the management console.

    You now have the choice which certificates you want to manage. Here you must select the computer account. If your company has high security standards this should concern you. The Windows VPN client will always search for the certificates in the user independend computer account. This means that anyone who has access to the machine can at least finish the IKE handshake (phase 1 and 2) if he knows the IP or DNS name of the Netscreen outside interface. To build a connection via a L2TP tunnel to your network he needs a username and a password of course. This should always be remembered when implementing a VPN with the standard Windows client. If you want to proceed click on Next. Mark the radiobutton This Snap-in manages the local computer and return to the previous dialogue by selecting Finish. You don’t need any other snap-ins so click on Close and shut the last open window with Ok.

    You should now see the certificate tree in the console. Pay attention that the root node in the left pane reads Certificates (Local Computer). Everything else will be the wrong place for your certificates. To import the root certifcate root.crt go down the tree Trusted Root Certification Authorities > Certificates. There you will see a lot of root certificates your machine already trusts. Right click on the leaf Certificates in the left pane and select All Tasks > Import … A wizard will open and asks for the location of the certificate you want to import. Provide the full path to the file root.crt. Click on Next and select the radio button Import certificates into the following store. Don’t change anything else. Just select Next and afterwards Finish. The root certificate of ACME Inc. is now in place and should show up in the right pane.

    We have to repeat the same steps for the user certificate. Therefore right click on the leaf Personal > Certificates and select All tasks > Import … again. Do everything what you made before for the root certificate but this time select the file john.doe.p12. As the PKCS12 certificate is password protected the wizard will ask for that. Simply enter the password you chose during the creation process. After finishing the wizard your user certificate will be on the laptop too.

    This guide is slowly coming to its end and I must admit that I’m still working hard on the NAT-T miracle. I don’t like the idea to be defeated by this lousy oakley.dll doing its FQDN handshake all the time.

  • STEP 13: Create the VPN connection

    Now that the laptop is prepared we can setup the new VPN connection. Open the network configuration wizard and choose Connect to the network at my workplace as the network type. On the next page we will select VPN connection (what else) and continue to name the connection ACME VPN (or whatever you like). As we will manage the standard internet connection (dialup or via router) ourselves we do not want to dial an initial connection automatically. In the next input field we will enter the address of the Netscreen outside interface. In this case it is On the last page we say that we don’t need a smardcard and finish the wizard. As the connection is configured we have to adapt the VPN type. Right click on ACME VPN, choose Properties and switch to the Network tab. It is strange that the default Automatic VPN type cannot build a L2TP over IPsec connection. Maybe Microsoft knows… So we select L2TP over IPsec manually and save the changes.

    STEP 14: Connect ot the VPN

    The time has come to build the initial connection to the ACME VPN network. If you are directly connected to the internet and the laptop has an official IP address this will be possible by now. Otherwise you have to wait for my upcoming posts that will cover two unsolved issues. Open the newly created connection and provide the username and the password. Both of them must match the user settings in the Netscreen. Hit Connect and the connection should be established.

    At this point we have come exactly as far as the few documentations I have found in the internet. Nevertheless I hope I managed to give you a far more complete view of this topic. Hopefully we will go beyond that the next days…

  • ISSUE 1: Split Tunneling

    Already mentioned above all client traffic will be routed by default into the VPN connection. This is a security aspect that one cannot neglect. When you are connected to the network of ACME Inc. the infrastructure of the company should provide you all the neccesary services you need for working. So if you want to surf the internet when being connected by VPN the requests should go the way laptop > VPN > company proxy > internet. Additionaly one can argue that a normal PC in the ACME building underlies the same restrictions. It is not allowed to participate in the intranet on the one hand and have an second network card to be connected to the an insecure network of the neighbour company. So by default the configuration will be absolutely correct for the person with security in mind.

    Let’s assume you want to allow the roadwarriors to acces your intranet and use their current network connections at the same time. This could be helpful if the companies internet connection has only a slow uplink and a lot of teleworkers do not only work on the inside resources but also surf the internet through the VPN. In this case the uplink bandwidth represents a bottleneck that can cause severe slowdown.

    The magic word here is split tunneling. It describes the ability of the client to route all traffic to the company network via VPN and all other traffic to the normal network connection. Basically it is nothing more than enhancing the default routing table with a new entry that routes all traffic to the ACME intranet ( through the VPN. This can be done manually by the user every time he connects - a solution no one really wants - or automatically. Sadly the automatism is not implemented in the windows client by default. Therefore we need CMAK.

    CMAK or known as Connection Management Administration Kit is a sofware component on the Windows 2003 Server CD. It provides a wizard to create a self installing binary that an user can execute without any further input on his laptop. It will create the VPN connection and all its settings so that the user may directly launch the VPN afterwards. No other efforts required. A detailed description of this tool will go far beyond the scope of this article. Therefore I just reference to the best documentation I found throughout the net in my opinion. If you are interested take some time and read more at

    The essential point of the configuration process is the step Routing Table Update. There you can provide a file with a new routing table. Everytime a client will connect to the CMAK created VPN connection it will automatically update its routing table corresponding to the contents of this file. The exact syntax of this file is more or less an abbreviation of the windows route command. In our case the file wll contain only one line:

    ADD MASK default METRIC default IF default

    It tells the client that only the traffic to the ACME intranet will go through the VPN. I guess you cannot make it easier than this. Nevertheless it took me some time to find this secret solution any normal VPN client is capable of by default. That only leaves a single issue unsolved …

  • ISSUE 2: NAT-T

    Starting this topic I had never thought that it would be so challenging. Now I know why it has been covered so rarely in the past. The less work you want to have on the client side, the harder you have to fight on all the other fronts. But why should I use a memory wasting VPN client bloated with features I do not need. Of course all programs have their existence authorization but the longer I work in the IT the more I’m attracted to clean and easy tools on my workstation. Different strokes for different folks.

    After all I managed to fix the problem that kept me busy the last days. While writing these lines I feel a great satisfaction. Man has won over machine. But what it is all about you may ask. So we will go into detail for the last time.

    In phase 2 of the IKE handshake the client and the server of a VPN connection have to decide how they will identify themselfes. Possible values are email address, IP address or FQDN, … The Netscreen accepts an IP address and this is no problem if you use the Netscreen client. The Windows native client provides an IP address as long as it has an official IP address and NAT-T is not required. If the client works behind a router this behaviour changes dramatically. All of a sudden it will simply send an FQDN and the Netscreen will block any further packet. I asked why so often during my analysis but it led me to no conclusion. All the normal tricks faded away as soon as I tried them out. If you look at the interesting part in the IKE debug trace it reads like this:

    12:31:22 : IKE <w.x.y.z >Recv*: [HASH] [SA] [NONCE] [ID] [ID] [NAT_OA] 
    12:31:22 : IKE<w.x.y.z >   extract payload (352): 
    12:31:22 : IKE<w.x.y.z >   QM in state OAK_QM_SA_ACCEPT.
    12:31:22 : IKE <w.x.y.z >ERROR: Cannot handle this id type, 2!</w.x.y.z ></w.x.y.z ></w.x.y.z ></w.x.y.z >

    So what to do if everything fails? We have to fix the client! Call it what you want for me it is nothing more than a program enhancement to fit my needs. To show you what I mean I will give you an extract of the oakley.dll:

    push    eax
    mov     [ebp+var_10C], 100h
    call    _GetComputerFQDN
    add     esp, 0Ch
    cmp     eax, edi
    jz      short loc_756CD5F7
    push    eax
    push    offset aGetcomputerfqd ; "GetComputerFQDN failed %d"
    push    edi
    push    edi
    push    edi
    push    edi
    push    80000000h
    call    _IKEDbgMsg
    add     esp, 1Ch
    jmp     loc_756CD55B

    If you are a little bit into programming you can see what happens here. The library tries to determine the FQDN of the client. If this succeeds (return code in register EAX=0) it will continue in the default way otherwise it will write a message to the log. The normal behaviour will be that the GetComputerFQDN finishes without errors and the client can use the FQDN in IKE phase 2. And exactly at this point we will place a small modification (nothing more than a single bit). We will convert the conditional JZ jump into a JNZ. This means the client thinks it cannot determine the FQDN. Luckily the routine will not abort but will send its IP address to the firewall instead. I cried out loud when I saw the first NAT-T connect to my rebellious Netscreen. To make the solution easy for the non assembler oriented reader I will include the neccessary conversion you have to apply to the oakley.dll. But remember: YOU WILL DO THIS ON YOUR OWN RISK:

    system32\oakley.dll (Windows XP 32 Bit): replace 3BC7741C5068 by 3BC7751C5068
    system32\oakley.dll (Windows XP 64 Bit): replace 85C0743389 by 85C0753389

    Pay attention that the byte sequences only occur once in the oakley.dll. If you want to patch the file you will have to take care of the safety copies Windows stores in system32\dllcache and/or ServicePackFiles. Just search all occurrences of the library and fix them in the right order. Reboot your machine and check that the changes in system32\oakley.dll have been made permanent. Finally you have a fully working VPN client for your Netscreen.

    My short trip into the deeps of VPN client connectivity have been very instructive. I learned quite a lot about my firewall and its operating principle but this peek should be enough for the next couple of months. Nevertheless I know now that a Netscreen & a Windows XP native VPN client will be the team that supports me best. Hopefully you enjoyed reading this article as much as it was fun for me writing it. A very special thanks goes to Mike and Henning for providing me the test environment.

    But now its time to close this article with best regards.


  • I realy miss this configuration for smart card using windows xp client without ns-remote and I followed this article and it is working but I dunno how to switch this for using a smart card!
    my boss wants me to have a solution for clients smart card using native windows xp vpn and if I can’t do it I will loose my job please inform me if you have any comment on this feel free to contact me at

  • I will include the neccessary conversion you have to apply to the oakley.dll. But remember: YOU WILL DO THIS ON YOUR OWN RISK:

    system32\oakley.dll (Windows XP 32 Bit): replace 3BC7741C5068 by 3BC7751C5068
    system32\oakley.dll (Windows XP 64 Bit): replace 85C0743389 by 85C0753389

    Pay attention that the byte sequences only occur once in the oakley.dll. If you want to patch the file you will have to take care of the safety copies Windows stores in system32\dllcache and/or ServicePackFiles. Just search all occurrences of the library and fix them in the right order. Reboot your machine and check that the changes in system32\oakley.dll have been made permanent. Finally you have a fully working VPN client for your Netscreen.

    Would you be able to go into a little more detail on how to do this, I don’t regularly edit dll’s :). Even after disassembling the dll I cant seem to find 3BC7741C5068.


  • Hi.

    Changing the DLL should be straight forward. you should only need an hex editor and open the file. Afterwards search for the hexadecimal string and you should find one. if that is not the case maybe you have another oakley.dll than me (mine has version 5.1.2600.2180).

    If you have no luck you will need tools like IDA Pro to disassemble the code. This will reveal the assembler code in which you have to find the use “GetComputerFQDN failed %d” message. At this location you will see the required bytes you have to change to modify the jump destination.

    Best regards.

  • Hi,

    why do I check “Enable NAT-Traversal”, there is error

    “Rejected an IKE packet on ethernet0/0
    from to with cookies
    811a892d285a5d6f and b7dcc6d168a5194f
    because The peer did not send a proxy

    if I don’t enable , it can pass phrase 2, but connect fail win error 678.
    IKE Phase 2 msg ID
    c3ca8c69: Completed negotiations with
    SPI ec6af547, tunnel ID 32772, and
    lifetime 3600 seconds/250000 KB.

    How can I do?

  • I think I get the answer

    “Enable NAT-Traversal”,  it is for vpn client without public ip if your pc behind firewall, you must enable and use NS/XP ways

    2.if your pc dial up with public ip,you can use juniper kb10939

    But I don’t like juniper, it make VPN configure difficult than netscreen.
    and I need to modify the dll file. very unfriendly.

    My home firewall’s VPN, I just check VPN button, then config user
    then it’s OK.

    WAWAWA, this take me two days for try errors.

    last, 3q NS/XP again.
    I hate myself, why not trust him at begining. why am I so stupid to
    trust juniper kb. and other old version netscreen vpn paper.
    O mygod