Building A CCIE Security VRACK

Introduction

 

Since my recent twitter update showing off my new CCIE security vrack script, I have been getting a flutter of messages from readers and subscribers asking about how I put everything together.  I hesitated to write this post, because frankly there is so much involved and it took so much time, I knew the blog post would be a monster to write.  Also, I’m just afraid I’m going to forget a bunch of stuff I figured out “on the fly” along the way. However,  I’ve decided to write up my playbook on how to put together a CCIE security v3.0 study lab!

 

Requirements & Resources

 

  • Quad Core CPU (I use an intel core2 quad q6600 CPU)
  • A ton of RAM (I run 8GB DDR2 RAM in my machine)
  • Basic knowledge of linux operating system
  • Basic knowledge of dynamips/dynagen
  • Basic knowledge of shell scripting
  • Basic knowledge of VMware and VMware networking
  • My shell scripts and net files (Download HERE)
  • QEMU 0.11 source code
  • QEMU 0.11 patch code
  • ASA / IPS files (kernel, initrd, disk images, etc)
  • 3725 IOS for the routers (12.4.15T10 Advanced Enterprise is what I run)
  • Windows XP Professional
  • Windows Server 2003 Enterprise
  • Cisco ACS 4.2
  • Patience
  • Time
  • Acceptance of the fact that this is only a simulation and that everything doesn’t work the first time around and that some things namely in QEMU ASA / IPS emulation just plain won’t work at all yet.  I don’t recommend you forget about real equipment all together.
  • This is a very personal solution custom tailored for me on my system.  Your results may vary
  • I could not have done this without the help of the following people / resources
    • Antonio Soares – Antonio is sort of THE guy in the community that sat down and figured a lot of this stuff out.  You will find the information on his web site and various forums invaluable in your journey.  He already has downloadable dynamips .net files for all the CCIE security topologies I talked about.  My own net files are basically copied from his examples and slightly modified for my environment.
    • inetpro.org tutorial – A very good tutorial about this stuff with a lot of great information.  His overall topology is quite different than mine, but you can learn a lot here

 

VRACK Summary

 

I am going to break down each piece of the lab in separate sections, but lets have a look at the final product before we jump so that we know what we are trying to build.  Here is a logical diagram of what things will ultimately look like

The easiest way for me to explain this is from the outside in.  Everything within the dotted rectangle is running virtually inside the linux host machine.  You can see on the far left the eth0 interface of the linux machine is plugged into my home LAN.  You might also notice that eth0 has basically become a virtual bridge through VMware.  We will get to that, but basically this allows us to have the host linux machine and our guest virtual machines all on the home LAN.  Then we have our WinXP and Win2k3 VM’s.  The Win2k3 VM runs Cisco ACS 4.2 and will act as the AAA/CA server in our lab environment.  The XP workstation runs XP pro and is used as our lab test PC.  Notice they both have 2 NICs — One is bridged to our home LAN (so we can access these VMs remotely, and have internet access on them, etc) and the other is connected to a separate virtual switch that ultimately connects them into a dynamips virtual switch.  The “Dynamips” cloud there can represent a few different things.  I have net files setup for INE, IPX and Yusuf CCIE security topologies.  QEMU is the software that runs the ASA and IPS emulations.  It hooks into dynamips essentially by using UDP connectors.  At a high level, this is what we are dealing with.

Once everything is working, we can ultimately launch an IPX, INE or Yusuf CCIE security VRACK from a shell script.  You will see how this shell script utilizes all the tools necessary to completely start up your vrack.  It can even start and stop vmware virtual machines built and run using the FREE vmware player completely in the background with the right hacks in place. Ultimately, we run a shell script and things look like this:

********************************************************************
*                                                                  *
* ######  ####  ######   ######   #######                          *
* ##    ##  ##  ##    ## ##    ## ##     ##                        *
* ##        ##  ##       ##       ##     ##         |       |      *
* ##        ##   ######  ##       ##     ##        |||     |||     *
* ##        ##        ## ##       ##     ##        |||     |||     *
* ##    ##  ##  ##    ## ##    ## ##     ##       |||||   |||||    *
*  ######  ####  ######   ######   #######     ||||||||||||||||||| *
*                                                                  *
*   ASTORINO NETWORKS CISCO VRACKS                                 *
*   http://www.astorinonetworks.com                                *
********************************************************************

1. Load INE SECURITY VRACK
2. Load IPX SECURITY VRACK
3. Load Yusuf SECURITY VRACK
4. Load INE RS VRACK
5. Load IPX RS VRACK
6. Exit

Make A Selection:

If you are still interested, put the coffee on and keep reading. Time to get started!

 

Getting Started With Dynamips

 

Fortunately, UBUNTU Linux has an awesome package management tool called apt-get that makes it very easy to download and install packages. It even automatically goes out and pulls down all the necessary dependencies. The first thing you want to do is open a shell and go download and install dynamips and dynagen.  To do this execute the following commands:

sudo apt-get install dynamips
sudo apt-get install dynagen

To get started running a lab, fire up dynamips using the command dynamips -H 7200 then start up dynagen with your net file of choice using dynagen whatever.net.  If you don’t have any errors in your net file dynagen should come up and you can start playing with your routers. All my net files use 4 dynamips processes so when I fire things up the basic commands look like this. Notice the & character. This tells your shell to start the process in the background.

/usr/bin/dynamips -H 7200 &
/usr/bin/dynamips -H 7201 &
/usr/bin/dynamips -H 7202 &
/usr/bin/dynamips -H 7203 &
/usr/bin/dynagen /home/joe/cisco-labs/security/ine_sec.net

Dynagen NET File Global Settings

All my NET files are in the tar file linked at the beginning of this post in the resources section.  I am not going to cover every line here but, I will hit the important pieces.  First, we will look at the global values under each dynamips process [localhost:720x].  Again, I would like to say that these files are basically copies of Antonio’s wonderful net files.  Most of this work is HIS WORK not mine.

 

GhostIOS, Sparsemem, Workingdir

 

You will notice that I have commented out the lines enabling ghostios and sparsemem.  Why?  We don’t need them, and it is a DRAMATIC performance increase for your virtual routers when you have the balls in your system to do this.  Why?

By default, dynamips emulates the virtual routers’ RAM in a file created on the hard disk.  Guess what goes into RAM? Eventually, the IOS gets loaded from flash on a real router into RAM. Your running-configuration runs out of RAM too.  That means by default every router in your topology has to load the IOS and configuration into “RAM” which is really just a file on your hard drive.  When every router has to do that, and they are all utilizing the same IOS image that is a waste of resources.  GhostIOS is an awesome tool to make that more efficient.  What it does is allow all the virtual routers running the same image to all utlize the same single IOS file instead of everybody needing their own.  Sounds good, so why is it disabled?  This fits in with the mmap = false option under the platform options in the net file.

By default dynamips does memory mapping with the default setting mmap = true.  What this means again is that by default the router RAM is emulated in a file on your hard disk.  Why?  Because most people don’t have 8GB of RAM in their PC and if you dedicated say 192MB of real RAM per router you would run out of system memory.  The downside is, running virtual RAM off a hard disk file is WAY slower than real RAM. That is just the nature of computer hardware.  Fortunately for us, we DO have 8GB of RAM so I can pawn off 192MB of REAL RAM per router no problem.  This means if we disable mmap by setting mmap = false our IOS and everything stored in the virtual router RAM is no longer running off a hard drive file…it is running out of real super fast system memory!  With that in mind, we also have no need to run ghostios because every router has it’s own dedicated real RAM where the IOS will be loaded.  That brings us to sparsemem.  This is an option used to optimize systems that again don’t have a ton of memory.  It lets the virtual router sort of dynamically allocate and unallocate memory.  So it is a grow as you go situation.  You might allocate 192MB RAM for your router, but if it is only using 32MB at the time, only 32MB is needed at that instant instead of the whole 192MB.  Again, we have PLENTY of RAM so we don’t need to deal with that

You will notice I hard set my workingdir as well.  This will make dynamips save all the temporary files and router emulation files to that specific directory.  This includes the router ROM files.  When you do a write memory to save your configs, the config gets written to that ROM file.  By doing things this way I can write memory and write erase things as I please.  When I come back and load that net file, everything is always just as I left it, just like real hardware.

 

Dynagen NET File Platform Settings

 

I use all 3725 routers.  At the beginning of each dynamips process you will find a section that sets values related to all the 3725 routers in that section.

 

Image and RAM

 

This is self explanatory, but I wanted to add one tip.  For the 3700 platform you do NOT need to uncompress your IOS image like you do on the 2600 platforms.  I personally do no uncompress mine. Although according to CCO 12.4.15T10 Advanced Enterprise requires 256MB RAM, I have had great luck running with 192MB.

 

IdlePC Values

 

When you are emulating routers in dynamips, by default the software doesn’t really know when the virtual router CPU is idle.  The long story short is that by default your virtual router is going to spike your CPU to 100% and it is going to suck.  To fix this, we use dynagen to find a value called “idlepc”.  We put this into our .net files.  There are plenty of good resources on how to do this, but I thought I would share my “secrets”.  I can run an entire CCIE rack of routers at about 10% CPU using this method.  Boot up a net file containing only a single 3725 router. Make sure it’s configuration is BLANK so it prompts you to enter initial setup mode.  Answer no, and press enter until you get to the router> prompt.  Let ALL the initial startup log messages scroll by.  Make sure you are sitting on a router with a blank config doing NOTHING at the router> prompt.  At this point, find your idlepc value by running idlepc get <router> from dynagen.  Apply this to your net file.  This was only half the battle.  Even with a great idlepc, I needed one more thing.  That magical thing is the idlemax = 100 command you see in the .net files.  I’m not entirely sure what this does, but for me it works like magic in combination with the idlepc to give me an awesome setup.  Note that my net files run the same exact platform and IOS for every router, which makes things easy, but idlepc values are platform and IOS specific. Also note, do NOT USE MY IDLEPC VALUES!  That will not help you, as they are specific to YOUR machine and YOUR hardware.  If you just copy my values, your lab is going to run like crap. When you are done if it takes more than a few seconds to do a show running-config something is not right.  You should be able to run a full CCIE lab in dynamips without breaking a sweat on a system with the hardware and settings I have specified.

 

Dynagen NET File NIO Connectors

 

NIO connectors are how we link virtual machines, virtual IPS interfaces, and virtual firewall interfaces to our dynamips routers.  If you look under the configuration for the switches (which are really just 3725 routers with 16 port switch modules) you will see these.  There are two basic flavors I use, NIO_linux and NIO_udp.

 

NIO_Linux

 

NIO linux is specifically how we are going to link the LAB virtual NIC of our virtual machines into a dynamips switch interface.  Here is an example line from the ine_sec.net file from SW1

# VMNet2, Cisco ACS
  f1/11 = NIO_linux_eth:vmnet2

What does this mean? It says port f1/11 on this “switch” is essentially bridged to the vmnet2 adapter on our linux host.  If that is a mindjob, go look at the diagram and picture it like this:  “Whatever traffic leaves SW1 F1/11 in dynamips comes out the VMNET2 adapter on the host machine”.  The VMNET2 adapter is hooked into the VMNET2 vswitch, and hooked into that vswitch is the virtual machine itself.

 

NIO_udp

 

NIO_udp is how we connect the ASA and IPS interfaces to our dynamips.  Here is another example from the ine_sec.net file under SW1:

# qemu-vlan2, ASA1 e0/1
  f1/13 = NIO_udp:20002:127.0.0.1:30002

This says “F1/13 on the switch talks to UDP port 30002 and sourced things from UDP 20002.” Wow, OK what does that mean? Think of it as a connector. If an ASA virtual interface is plugged in here and this switch interface wants to talk to it the switch talks to UDP port 30002 and sources those packets from UDP 20002. The virtual ASA interface is “listening” on port 30002. When the ASA talks to the switch the reverse happens. You will see that come together later in the QEMU section

That about does it for the dynamips/dynagen section.

 

VMware Architecture

 

Let’s talk about the VMware side of things now.  First, we’ll talk about the architecture, then about the specific implementation and how we can “hack” vmware player to start/stop virtual machines from the CLI and how to get it to allocate to us the proper virtual vmnet adapters.

Let’s start from the beginning on the far left of the diagram.  First you must understand vmnet0 or vmware virtual bridging.  vmnet0 bridging allows your host machine and your guest VMs to all be on the same broadcast domain and ultimately the same IP subnet by using bridging.  What it does is essentially make your physical host NIC into a virtual bridge.  That virtual bridge is hooked into a virtual switch we call vmnet0.  Hooked into that vmnet0 switch are your host and guest NICs.  In my case, when somebody on the home network ARPs for 10.1.10.11 for example, the broadcast hits my host ubuntu machine physical NIC.  Since that NIC is now really a virtual bridge, the bridge forwards the broadcast to the vmnet0 vswitch and that vswitch forwards it to the actual host.  This means that from an upstream physical switch perspective, my Ubuntu host and both guest VMs live off the same switch port.  It’s like my ubuntu host is a downstream switch. The vmnet0 bridging piece gives our guest VMs and host machine access to the network and ultimately the internet.  This means we can now SSH or RDP into these machines : )

Now, the second piece is a little bit more “hairy”.  Each guest VM has it’s own dedicated vswitch for their LAB nics.  Why?  If we hooked both VMs into the same vswitch they could talk to each other directly, without passing through the dynamips lab switches and we don’t want that.  If the XP workstation wants to talk to the ACS machine we want to ensure the traffic goes from the XP workstation to the dynamips virtual switch environment and finally to the ACS machine.  The LAB NIC in both machines should be setup in vmware as a host-only NIC.  What this means is illustrated in the diagram.  It means your host machine has a virtual network adapter called either vmnet1 or vmnet2 in this case that plugs into a private vswitch with the guest VM.  Everything is isolated inside the machine.  Usually this is done when you do NOT want your VM to have access to anything outside the vm host machine.  Except…ultimately we want the guest VMs to “plug into” a virtual dynamips switch , so what gives?  That is the magic of the NIO_linux connector in the net file.  Let’s look at that example again:

# VMNet2, Cisco ACS
  f1/11 = NIO_linux_eth:vmnet2

OK.  What this does is yet MORE bridging.  It says “whatever comes out of f1/11 on our dynamips switch gets bridged to the vmnet2 host adapter”  So even though we use host-only networking for these NICs, that host only NIC is bridged to dynamips in software.  Is your head spinning yet? Now, technically when you allocate the vmnet1 and vmnet2 adapter in vmware you HAVE TO give it an IP address and mask.  That means technically, even though that vmnet adapter is being bridged to dynamips, it still has an IP and that means if that IP happens to be on the same logical subnet as your VM guest machines, the VM guest machines could in theory talk to the host machine through the LAB interface.  Minor setback.

Now, here comes the real fun part.  If you have VMware workstation you are probably all set, but I am cheap.  VMware workstation cost money.  VMware player is free on linux and windows.  However, it is very limited.  When you create a virtual machine, and allocate the NICs it only lets you choose bridged (vmnet0), NAT (another thing all together we don’t care about) or host-only VMNET1.  That’s right you can only use VMNET1 out of the box.  There is no option to add another host-only vmnet adapter like VMNET2.  In our case we HAVE TO HAVE separate independent host only vmnet adapters to make this work right.   So, what to do?  We hack it : )

 

Installing VMware Player

 

You can go download vmware player for linux for free at vmware.com.  Unfortunately, it is a bit irritating because you have to actually register to get the download.  Oh well…You will end up with a .package file you need to install.  To do this run this command:

sudo gksudo sh /home/joe/Downloads/VMware-Player-4.0.1-528992.x86_64.bundle

 

Adding VMNET2 Adapter In VMware Player

There are basically two things you need to do.  First, allocate your two NICs to each VM in vmware player.  For your first VM you will use the built in vmnet1 host-only for your first NIC (LAB NIC).  The second NIC will be bridged to vmnet0 as we talked about. You are good to go on the first machine.

When you create the second VM, you won’t have an option to bind the first NIC to another vmnet adapter, so we need to create our own.  Initially set the NICs of the second VM exactly like you did the first one.  This means that as of now both LAB NICs are hooked into vmnet1.  We need to hack that.  Open up the .vmx file for the 2nd VM and look for the lines for ethernet0.  Make the following changes:

ethernet0.connectionType = "custom"
ethernet0.vnet = "vmnet2"

We have told the LAB NIC it is using a “custom” setup and that it is bound to vmnet2. Now we need to actually setup the vmnet2 virtual adapter. To do this edit /etc/vmware/networking.  Here is mine.  By default you will only see parameters for VMNET1 and VMNET8 since that is what vmplayer sort of locks you into.  We simply add parameters for another adapter called VMNET2

VERSION=1,0
answer VNET_1_DHCP no
answer VNET_1_HOSTONLY_NETMASK 255.255.255.0
answer VNET_1_HOSTONLY_SUBNET 172.16.183.0
answer VNET_1_VIRTUAL_ADAPTER yes
answer VNET_1_VIRTUAL_ADAPTER_ADDR 172.16.183.1
answer VNET_2_DHCP no
answer VNET_2_HOSTONLY_NETMASK 255.255.255.0
answer VNET_2_HOSTONLY_SUBNET 172.16.184.0
answer VNET_2_VIRTUAL_ADAPTER yes
answer VNET_2_VIRTUAL_ADAPTER_ADDR 172.16.184.1
answer VNET_8_DHCP yes
answer VNET_8_DHCP_CFG_HASH 4FF8EEADB654C70A9452D1DE4E445C2B5FBA286B
answer VNET_8_HOSTONLY_NETMASK 255.255.255.0
answer VNET_8_HOSTONLY_SUBNET 192.168.170.0
answer VNET_8_NAT yes
answer VNET_8_VIRTUAL_ADAPTER yes

By default, host-only networking also provides DHCP services, which you probably don’t want so you will notice we disabled that. This is also where you MUST define IP address settings for the virtual adapters.  If you don’t, things won’t work.  Finally, you need to restart the vmware services to get it to read the new information by running

sudo /etc/init.d/vmware restart

 

Starting & Stopping VMs from CLI

 

At this point, you should be able to get your virtual machines up and running, but you have to actually open a separate instance of the VMware player GUI for each VM.  It’s mildly irritating.  By default, vmware player offers nothing to make this less painful.  We’ll just have to hack it again : )  Enter the VMware VIX API.  From what I understand the VIX API is basically an API that allows the control of vmware VMs from the CLI and from scripts…exactly what we want.  I think it comes standard with the VMware stuff you actually pay for, but what they don’t really tell you is that this API can be used for VMs running with vmware player as well : )  All we need to do is install the API and then learn a few commands.

First, install the VIX API.  You can download it from vmware.com for free as well.

sudo gksudo sh /home/joe/Downloads/VMware-VIX-1.11.0-471780.x86_64.bundle

That is basically it!  You now have access to a VERY powerful set of tools invoked by a program called vmrun.  Unfortunately, it appears the vmrun utility is not yet updated to run with VMware player 4.0.1 so we need to hack that too.  Fortunately for us, that is easy.  Go edit /usr/lib/vmware/vixwrapper-product-config.txt and you should find this section at the bottom:

# Workstation 8.0.0 and Player 4.0.0
ws        14  vmdb  8.0.0 Workstation-8.0.0-and-vSphere-5.0.0
player    14  vmdb  4.0.0 Workstation-8.0.0-and-vSphere-5.0.0
ws-shared 14  none  8.0.0 Workstation-8.0.0-and-vSphere-5.0.0
ws        14  vmdb  e.x.p Workstation-8.0.0-and-vSphere-5.0.0
player    14  vmdb  e.x.p Workstation-8.0.0-and-vSphere-5.0.0
ws-shared 14  none  e.x.p Workstation-8.0.0-and-vSphere-5.0.0

All you need to do is edit the player section at the top to “4.0.1” or whatever version you have and vmrun will work flawlessly. If you miss this step, vmrun will just fail to run with vmware player 4.0.1.

See the start_acs and start_xp shell scripts in the tar file I have at the beginning of this post.  Here is what you need to know

!START A VM FROM THE COMMAND LINE WITH NO GUI POPUP
vmrun -T player start "/home/joe/vmware/Win2K3 Enterprise/Win2K3 Enterprise.vmx" nogui

!STOP A VM FROM THE COMMAND LINE
vmrun -T player stop "/home/joe/vmware/Win2K3 Enterprise/Win2K3 Enterprise.vmx" soft

!LIST ANY RUNNING VMs
vmrun -T player list

You now have the ability to start and stop virtual machines strictly from the command line without ANYTHING ever even popping up…pretty much like some of the more “enterprise” vmware tools. This also means we can script the start and stop of virtual machines. Awesome…just awesome.

 

QEMU / ASA & IPS Emulation

 

This is probably the part that gave me the most headache.  Again, I had a lot of help from great friends and others that have gone before me.  I will cover the basics of what you need to know here.

 

Patching QEMU v0.11

 

This is VERY important.  Basically, the QEMU program source code needs to be patched to properly run the ASA and the IPS.  For the ASA you might find some source code patches for versions that are not v0.11 and they may work, but for me personally I found that I HAD to use v0.11 to run the IPS properly anyways so that is what I would recommend.  Do NOT install QEMU from apt-get like a normal package!  Why?  Because you need to actually patch the source code before the package gets built or nothing will work.  Specifically, I think it is the NIO UDP stuff that needs baked into the code. So , what do you do?

First, you go find the source code for QEMU v0.11.  That should not be hard to do.  I would post it here, but it is over 3.5MB and the limit on files on this blog is 2MB.  Anyways, go find that and untar it somewhere like your home directory. You should have a directory called qemu-0.11.0 and inside that directory you will find all the C source files.  Now, you need the patch file.  The patch file is called qemu-0.11.0-olive.patch and it is in my tar file.  copy this file into the qemu-0.11.0 directory and run it by using the patch command like so:

sudo patch -p1 -i qemu-0.11.0-olive.patch

If everything goes OK it should patch your source code without any issues.  Now you need to actually compile the source code and install the program.  Before you do this, you may need to install dependencies:

sudo apt-get install build-essential libncurses5-dev libsdl1.2-dev libpcap-dev zlib1g-dev

Now configure for installation, compile your code, and install the program. Note, when you run ./configure you can set the target-list parameter which tells the compiler what platform you are compiling for.  I think the default is i386-softmmu which is basically 32-bit CPU.  In my case I used the 64-bit version which is x86_64-softmmu

./configure
make
make install

If everything compiles OK you should now have your QEMU patched binaries in /usr/local/bin unless you specified another directory during the ./configure process.

ASA Emulation

 

QEMU is sort of like VMware in that it runs virtual machines.  Since the ASA is really nothing more than a linux box with special hardware and software, with enough hacking we can emulate one with QEMU.  I am no expert, but to emulate the ASA you essentially need three files to start with that are called during the QEMU initialization.  These files are:

  • The IOS kernel
  • The IOS initrd
  • The ASA hard disk image

Now, many people much smarter then me have figured out how to build these files.  Essentially, they are taking an IOS bin image and manually extracting the linux kernel and initrd files, then modifying them.  That means you need to either know how to extract the kernel and initrd and also how to build the ASA hard disk image, or you have to know where to get them.  Since you don’t exactly download this sort of thing from CCO, and because I really don’t feel like getting sued by Cisco, I will NOT provide those for you.  Trust me — If you are smart and you are patient, and you look in the right places you will find what you need.  That is only half the battle though. Once you have the files, you need to know how to use them.  You will find “start_asa1.sh” and “start_asa2.sh” in the tar file included here.  These are the scripts I use to start the ASA.  Let’s look at one.

#!/bin/sh
/usr/local/bin/qemu-system-x86_64 \
        -hda /home/joe/cisco-labs/asa/new-asa1.disk \
        -kernel /home/joe/cisco-labs/asa/new-asa1.kernel \
        -initrd /home/joe/cisco-labs/asa/new-asa1.initrd.gz \
        -nographic \
        -append "auto nousb ide1=noprobe bigphysarea=16384 console=ttyS0,9600n8 hda=980,16,32" \
        -m 256 -cpu coreduo -icount auto \
        -net nic,vlan=1,model=pcnet,macaddr=00:aa:00:00:01:01 -net udp,vlan=1,sport=30001,dport=20001,daddr=127.0.0.1 \
        -net nic,vlan=2,model=pcnet,macaddr=00:aa:00:00:01:02 -net udp,vlan=2,sport=30002,dport=20002,daddr=127.0.0.1 \
        -net nic,vlan=3,model=pcnet,macaddr=00:aa:00:00:01:03 -net udp,vlan=3,sport=30003,dport=20003,daddr=127.0.0.1 \
        -net nic,vlan=4,model=pcnet,macaddr=00:aa:00:00:01:04 -net udp,vlan=4,sport=30004,dport=20004,daddr=127.0.0.1 \
        -serial telnet:10.1.10.11:2009,server &

Let’s break that down a bit here. the -hda parameter points to the hard disk image, simple enough.  The kernel and initrd parameters are also self explanatory.  -nographic tells qemu to suppress the graphical interface that pops up by default.  The -append line is absolutely necessary and passes proper values to the kernel that I don’t fully understand. -m 256 gives the ASA virtual machine 256MB of RAM. -cpu coreduo sets the CPU the virtual machine runs.  In this case we are telling it to emulate an intel coreduo CPU.  Why? There are many options you can try here, but I found coreduo to work the best for me as far as CPU utilization.

Now, the -icount auto command is HUGE.  This parameter does something with the virtual CPU clock cycles again that I don’t fully understand.  What I can tell you is that by adding this option I cut the CPU utilization that the ASA uses nearly in half.  The next four lines setup our ASA interfaces.  Basically you are setting up four ASA interfaces and tying them into the NIO UDP stuff we talked about earlier.  Also notice we hard code MAC addresses for the interfaces.  This will prove to be limiting on the ASA when you look at topics like active/standby failover because the emulation has a hard time with dynamically changing MAC addresses.  Look at the first line:

-net nic,vlan=1,model=pcnet,macaddr=00:aa:00:00:01:01 -net udp,vlan=1,sport=30001,dport=20001,daddr=127.0.0.1 \

This sets up the ASA e0/0 interface.  It gives it a MAC address and basically says “When I want to communicate out e0/0 send data to UDP port 20001 and source it from UDP 30001”.  Note this is the opposite of what we had in the dynamips net file.  This is how the ASA virtual interface talks to the switch.  Finally you have -serial telnet:10.1.10.11:2009, server.  This tells QEMU that the ASA console will be available by telnetting to 10.1.10.11 port 2009. In my case 10.1.10.11 is my linux machine that runs all this.

Once you actually execute the script, you still need to actually telnet to port 2009 in this case to bootstrap the ASA.  As soon as you telnet in, you will see the ASA start to boot up.  IF YOU HAVE THE CORRECT image, you should see a nice little menu like this before it boots:  This specific image seems to work great for dynamic routing protocols, and even multiple context mode. It is ASA 8.0(2).

=====================[ Rebuild on 23 Dec 2010, by N!NiX ]===================
= Cisco ASA5520 - ASDM without Proxy - Multiple Security Contexts - WebVPN =
=     ( Powered by Jeremy Grossmann, Pedro Flor, nekkro-kvlt, NiNiX )      =
=     ( Forums: 7200emu.hacki.at, www.gns3.net/phpBB, blog.gns3.net )      =
============================================================================

[Menu] - ASA last/default boot: (MULTIPLE MODE, NETWORK PROMISC)
1) - Enter Linux Shell
2) - Boot in ciscoasa(config)# mode single   - Only Once
3) - Boot in ciscoasa(config)# mode single   - Always
4) - Boot in ciscoasa(config)# mode multiple - Only Once
5) - Boot in ciscoasa(config)# mode multiple - Always
6) - Toggle eth0-eth5 promiscuous mode
7) - Format flash disk0:
8) - Info message
9) - Shutdown/Halt

Enter a option:

Make sure to toggle promiscuous mode ON the first time you boot, or multicast type things will totally not work (EIGRP, OSPF, RIP and other stuff). Once the ASA boots up, run the following commands:

copy running-config disk0:/.private/startup-config
boot config disk0:/.private/startup-config

This will allow you to do “write memory” and save your configurations.

 

 

IPS Emulation

 

The IPS emulation is alot like the ASA emulation in terms of how we run it, so I won’t repeat myself.  You need the patched QEMU v0.11 for this to work.  Here is my IPS startup script which you can also find in the tar file.

#!/bin/sh
/usr/local/bin/qemu-system-x86_64 \
        -name IPS4235 -hda /home/joe/cisco-labs/ips/ips-disk1.img \
        -hdb /home/joe/cisco-labs/ips/ips-disk2.img -m 1024 -cpu coreduo -icount auto -nographic \
        -smbios type=0,vendor="Phoenix Technologies Ltd.",version="1.10",date="09/30/2002",release="A04" \
        -smbios type=1,product="IDS-4235",manufacturer="Cisco Systems",version="1.0",serial="12345789012",uuid="E0A32395-8DFE-D511-8C31-001FC641BA6B",sku="011",family="IDS-4235/4250" \
        -net nic,vlan=9,macaddr=00:aa:00:00:01:09,model=e1000 -net udp,vlan=9,sport=30009,dport=20009,daddr=127.0.0.1 \
        -net nic,vlan=10,macaddr=00:aa:00:00:01:10,model=e1000 -net udp,vlan=10,sport=30010,dport=20010,daddr=127.0.0.1 \
        -net nic,vlan=11,macaddr=00:aa:00:00:01:11,model=e1000 -net udp,vlan=11,sport=30011,dport=20011,daddr=127.0.0.1 \
        -net nic,vlan=12,macaddr=00:aa:00:00:01:12,model=e1000 -net udp,vlan=12,sport=30012,dport=20012,daddr=127.0.0.1 \
        -net nic,vlan=13,macaddr=00:aa:00:00:01:13,model=e1000 -net udp,vlan=13,sport=30013,dport=20013,daddr=127.0.0.1 \
        -serial telnet:10.1.10.11:2011,server &

Note that the SMBIOS command parameters and quotes have to be EXACTLY correct or you will get an error when booting the IPS.

 

VRACK Scripting

 

After I actually got all this working, I wanted something that would automate all the tedious command line stuff I had to do to start and stop everything.  I essentially wanted a quick script that would allow me to launch either an IPX, INE or Yusuf CCIE security rack with minimal effort.  After I got THAT working, I modified it to include INE and IPX R&S topologies as well.  You will find the script “start_vrack.sh” in the tar file.  Here it is. Now, before you all email me and tell me how crappy my programming is and how I shouldn’t be using “kill -9” to terminate processes just let me say this — I am not a programmer by trade. I have a degree in computer science, so I can get by. Half of what I do in here is probably either wrong or inefficient and I’m sure somebody out there will probably now write a buffer overflow for it : ) I do enough to get by, and it works great for what I need. I am not going to go into the detailed explanation of this script. Take it, use it, modify it, play with it and figure it out.

One note — Make sure to run the script using sudo (running the script as root)

#!/bin/bash
function killdyn() {
# This function kills all existing dynamips and dynagen processes
 echo "Killing any existing Dynamips processes ..."
 pkill -9 dynamips
 pkill -9 dynagen
}
#
function killqemu() {
# This function kills all existing QEMU processes and shuts down the virtual machines
 echo "Killing any existing QEMU processes ..."
 pkill -9 qemu
}
function killvm() {
# This function gracefully shuts down the ACS and XP VMs
 echo "Gracefully shutting down Virtual Machines ..."
 vmrun -T player stop "/home/joe/vmware/Win2K3 Enterprise/Win2K3 Enterprise.vmx" > /dev/null 2>&1
 vmrun -T player stop "/home/joe/vmware/WinXP Pro/WinXP Pro.vmx" > /dev/null 2>&1
}
#
function startdyn() {
# This function starts the dynamips hypervisors
echo "Initializing Dynamips Hypervisors ..."
/usr/bin/dynamips -H 7200 > /dev/null 2>&1 &
/usr/bin/dynamips -H 7201 > /dev/null 2 /dev/null 2 /dev/null 2 /dev/null 2>&1
 vmrun -T player start "/home/joe/vmware/WinXP Pro/WinXP Pro.vmx" nogui > /dev/null 2>&1
}
function startasaips() {
# This function starts the ASA/IPS QEMU Emulations
 echo "Initializing ASA1, ASA2 and IPS ..."
 /home/joe/cisco-labs/scripts/start_asa1.sh > /dev/null 2>&1
 /home/joe/cisco-labs/scripts/start_asa2.sh > /dev/null 2>&1
 /home/joe/cisco-labs/scripts/start_ips.sh > /dev/null 2>&1
}
#
clear
while :
do
 clear
 echo "********************************************************************"
 echo "*                                                                  *"
 echo "* ######  ####  ######   ######   #######                          *"
 echo "* ##    ##  ##  ##    ## ##    ## ##     ##                        *"
 echo "* ##        ##  ##       ##       ##     ##         |       |      *"
 echo "* ##        ##   ######  ##       ##     ##        |||     |||     *"
 echo "* ##        ##        ## ##       ##     ##        |||     |||     *"
 echo "* ##    ##  ##  ##    ## ##    ## ##     ##       |||||   |||||    *"
 echo "*  ######  ####  ######   ######   #######     ||||||||||||||||||| *"
 echo "*                                                                  *"
 echo "*   ASTORINO NETWORKS CISCO VRACKS                                 *"
 echo "*   http://www.astorinonetworks.com                                *"
 echo "********************************************************************"
 echo
 echo " 1. Load INE SECURITY VRACK"
 echo " 2. Load IPX SECURITY VRACK"
 echo " 3. Load Yusuf SECURITY VRACK"
 echo " 4. Load INE RS VRACK"
 echo " 5. Load IPX RS VRACK"
 echo " 6. Exit"
 echo
 echo -n "Make A Selection:  "
#
 read opt
 case $opt in
   1)killdyn;
     killqemu;
     killvm;
     cd /home/joe/cisco-labs/security/working/ine;
     startdyn;
     startasaips;
     startvm;
     /usr/bin/dynagen /home/joe/cisco-labs/security/ine_sec.net;;
   2)killdyn;
     killqemu;
     killvm;
     cd /home/joe/cisco-labs/security/working/ipx;
     startdyn;
     startasaips;
     startvm;
     /usr/bin/dynagen /home/joe/cisco-labs/security/ipx_sec.net;;
   3)killdyn;
     killqemu;
     killvm;
     cd /home/joe/cisco-labs/security/working/yusuf;
     startdyn;
     startasaips;
     startvm;
     /usr/bin/dynagen /home/joe/cisco-labs/security/yusuf.net;;
   4)killdyn;
     killqemu;
     killvm;
     cd /home/joe/cisco-labs/rs/working/ine;
     startdyn;
     /usr/bin/dynagen /home/joe/cisco-labs/rs/ine_rs.net;;
   5)killdyn;
     killqemu;
     killvm;
     cd /home/joe/cisco-labs/rs/working/ine;
     startdyn;
     /usr/bin/dynagen /home/joe/cisco-labs/rs/ipx_rs.net;;
   6)killdyn;
     killqemu;
     killvm;
     exit 1;;
   *)echo "$opt is not a valid option";
     echo "Press [enter] key to continue...";
     read enterKey;;
 esac
done

29 Comments

  • Mike says:

    Joe,

    Thanks for the incredible post, you did a good job on this one as usual, your website is simply amazing.
    I just got a question for you, I got stuck on the ASA Qemu part. I did manage to get the initrd and kernel files on the net, but I couldn’t find the ASA hard disk. I tried to create that using the following:

    qemu-img create new-asa1.disk 1000M

    But when I run the script start_asa1.sh, qemu complains that it cannot open the hda file.
    Should I create this file or download it?

    Thank you very much
    Cheers!
    Keep up the good work

  • […] boys, time to pop the cherry on that fancy new security vrack : )  I tried to come up with a simple topology that would allow me to test all the NAT features I […]

  • Mike says:

    Hi Joe. It is me again :) Sorry for asking so many questions, but maybe you can help me out with this one. I have made some progress since last time we talked. I am trying to run your vrack script now, but it is giving me the following error:

    root@cisco:/home/cisco/cisco-labs/scripts# sudo sh start_vrack.sh
    start_vrack.sh: 2: Syntax error: “(” unexpected

    I am using Ubuntu 10.10 64bits. Am I doing it wrong?

    Thanks!
    Mike

    • Joe Astorino says:

      Yes, you have a basic problem there. Do it with out the “sh” at the beginning. Just go to your scripts directory and run “sudo ./start_vrack.sh” you don’t need to call the sh shell in the middle of it. I tried it your way and got the same error. My way it runs great.

  • Mike says:

    Hi Joe!
    I am still having two problems and if you could shed some light. The first one is related to ASA, the first_start menu does not appear, so everytime the start_asa.sh script runs it loads ASA on single context mode. Is there any file I need to delete after labbing, so the first_start meny appears and thus give me the ability to select the mode I wanna run.
    Second issue is with IPS. It works if I use just:

    qemu -hda ips-disk1.img -hdb ips-disk2.img -m 1024

    I am able to login to the IPS and config it. However, when I use your script, which binds the interfaces to the other network devices, the script runs, I am able to telnet on the port that the script is listening on, but when I put the username and password, I get the following error message:

    Warning: MainApp has not started, please try again later.
    Would you like to run cidDump?[no]:

    And this brings me back to the login prompt. I wonder if there is any other thing I need to tweak.

    Well, these are the only things that are not working for me, I am actively searching the internet to get those sorted out, but couldn’t find anything as of yet. Maybe you have some advice on this as well.

    Thanks a lot in advance.
    Mike

    • Joe Astorino says:

      My guess is you don’t have the same ASA or IPS images I do but I can’t be sure. I don’t have those problems. Every time I boot my ASA I get the following boot menu which makes it really easy and great. I know there are other images out there that do NOT include this:

      =====================[ Rebuild on 23 Dec 2010, by N!NiX ]===================
      = Cisco ASA5520 – ASDM without Proxy – Multiple Security Contexts – WebVPN =
      = ( Powered by Jeremy Grossmann, Pedro Flor, nekkro-kvlt, NiNiX ) =
      = ( Forums: 7200emu.hacki.at, http://www.gns3.net/phpBB, blog.gns3.net ) =
      ============================================================================

      [Menu] – ASA last/default boot: (SINGLE MODE, NETWORK PROMISC)
      1) – Enter Linux Shell
      2) – Boot in ciscoasa(config)# mode single – Only Once
      3) – Boot in ciscoasa(config)# mode single – Always
      4) – Boot in ciscoasa(config)# mode multiple – Only Once
      5) – Boot in ciscoasa(config)# mode multiple – Always
      6) – Toggle eth0-eth5 promiscuous mode
      7) – Format flash disk0:
      8) – Info message
      9) – Shutdown/Halt

      Enter a option:

      For the IPS, I simply do not have that issue. It boots up and I login just fine with no issues. Obviously, we would be using the same scripts so the only thing I can think of that would be different is your image. Good luck.

  • Mike says:

    Thanks Joe! I have searched everywhere and was not able to find them, I believe the issues I am seeing are with the images I am using.
    I guess I am reverting back to Windows this time, until I am able to make this work.
    Thanks very much for your help!
    Keep up the good work

  • charlie says:

    Fantastic write-up, thank you Joe!

    Took me about 3 hours from start to finish and things look like they’re working perfectly.

    Two issues I ran into that may help others:

    I had to use dos2unix on your asa/ips shell scripts before they would run.

    Also, setting up the IPS required some additional work. For anyone stuck, google “emulate cisco ips” and go to the first link. You’ll have to edit some network config files. Tip: Boot up the IPS without the “nographic” option to get into the boot menu.

    Again, this is a wonderful write-up and I’m amazed I have a virtual security lab running.

    • Joe Astorino says:

      rock n’ roll Charlie : ) Obviously, I wrote the scripts in linux, but at some stage I had them and I think I tarred them using 7zip in windows so the ctrl-lf probably got screwed up. Lab on.

  • Naeima says:

    Hi ,
    thank you for your complete instruction . is it possible for you do same for windows user ? . i will be thankful .

  • Alex says:

    “NOT USE MY IDLEPC VALUES!”

    as you too said they are specific to the IOS and platform, I would say just to IOS because the idle PC just identifies a point in the code where the software does not do anything. So theoretically, if you had found the point then that idle PC could be used whenever that specific image is loaded in dynamips… the problem is that when you run the script to find the point where the IOS should be idle, that point in the code is not always accurate.
    So IMO the uncertainty comes from how I determine the idle PC not from the fact that it varies with the HW or software that hosts dynamips… and if there were somebody with a machine just dedicated to such task (testing IOS images searching for idle PCs) and could share the results, well that would be great :)

    my 2 cents

  • Nesta says:

    Awesome post!!! I can only imagine how much time you needed to compile this. I’m planning to bring up a full CCIE Sec lab but with IOU instead Dynamips because of the full L2 features and much better performance of IOU. Whether it’s possible I’m not sure at this moment but your guide will save me quite some time in the process. Have you ever thought of trying this?

  • games says:

    First of all I would like to say superb blog!

    I had a quick question which I’d like to ask if you do not mind. I was interested to know how you center yourself and clear your thoughts prior to writing. I have had difficulty clearing my thoughts in getting my thoughts out there. I truly do enjoy writing however it just seems like the first 10 to 15 minutes are wasted just trying to figure out how to begin. Any recommendations or tips? Cheers!

    • Joe Astorino says:

      For me it usually starts with a technology I am fascinated by and working with at the time. Then I try to think of a real world example ahead of time, and draw that out and think about how it works. Then I will spend time going through it and making sure I fully understand. Then it is just a matter of putting down on paper what the process was and trying to explain it in an easy to follow and fun to read way. Also, I proof read all my stuff quite a bit usually before posting it and usually end up with some edits to make it look cleaner or whatever. Hope that helps.

  • Michael Mbuthia says:

    Awesome post Joe!!, giving myself two weeks to replicate your VRACK design.. Wish me luck!!

  • […] that I have borrowed heavily from Joe Astorino over here – No need to reinvent the wheel when it […]

  • JeremyRom says:

    Oh! its really nice. Thank you for sharing with us.
    CCIE Rack Rental

  • tony says:

    can this be made possible for us that run windows. I really love your work.

  • Ayedh says:

    A wonderful tutorial for building CCIE Security v3, but it is outdated. Any planing to update it for CCIE Security v4 VRack based on INE rack. No one tackle it yet, I am not sure where I should start?

  • Good article! We will be linking to this great post on
    our website. Keep up the great writing.

  • SAMI says:

    Hi very good work you have i just try to build a ccie topology but i`m having some issues can you please help me here is the output
    ============
    SW2#
    cisco Systems, Inc.
    170 West Tasman Drive
    San Jose, California 95134-1706

    Cisco IOS Software, 3700 Software (C3725-ADVENTERPRISEK9-M), Version 12.4(15)T5, RELEASE SOFTWARE (fc4)
    Technical Support: http://www.cisco.com/techsupport
    Copyright (c) 1986-2008 by Cisco Systems, Inc.
    Compiled Wed 30-Apr-08 18:27 by prod_rel_team
    Image text-base: 0x60008930, data-base: 0x6363A000

    BIST FAILED…
    Unknown file system detected.
    Use format command to format the card as DOS File System.
    Or use erase command to format the card as Low End File System.

    SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!

    %Software-forced reload
    ===================
    I`m using windows 8.1 ram 8gb processor i7 ,gns3 ver 0.8.7
    devises are = 6routers,6switches,4pix,

  • […] boys, time to pop the cherry on that fancy new security vrack : )  I tried to come up with a simple topology that would allow me to test all the NAT features I […]

  • Cerys says:

    Thanks for finally talking about >Building A CCIE
    Security VRACK – Astorino Networks <Liked it!

Leave a Reply