Mixing Assembly language with Visual Basic

Jun 02, 2020 by Nitin Yadav

This tutorial will help explain the basics & also the details needed if you want to mix assembly code with Visual Basic. This tutorial will not explain how to use assembly language. If you need to learn that, then you can find many tutorials and help in other assembly related sites. The code used here is done with NASM 0.97, which is freely available on the internet (but help for it is not so common).

Be warned however: Mixing assembly code with your Visual Basic programs can seriously make it much harder to debug and use. A mistake can easily cause the entire Visual Basic environment to shut-down, freeze the computer, make GPF’s, and even restart the computer (I have found all these out the hard way!).

Not to mention that there aren’t any debugging tools that you can use to fix the problems. In fact, if your assembly code doesn’t restore everything after it is done, things like using the Debug window in Visual Basic will interfere, and cause the VB environment to shut down.

Assembly language can be very difficult to learn and program. However, it can also be easier in some cases, because you get a much better understanding of what you are doing.

What you need

Visual Basic (I have only tried with VB5, but VB4 or above should work). If you dont have Visual Basic, then you can get the “Express Edition” (was called the “Custom Control Edition”) for free at the Microsoft site!
NASM (I have only tried with v0.97). If you dont have NASM, then you can get it for free at the NASM site (http://www.nasm.us/).
A simple text editor. You can use Notepad, but I recommend something better like PFE, Scite or Notepad++ (http://notepad-plus.sourceforge.net/uk/download.php).

Useful Links

Get NASM (free)
Get Visual Basic Express Edition (free)
Patrice Scribe’s VBA51 site. This site shows how to use DirectX with VB, using Patrice Scribe’s Type Libraries. It also shows how to use assembly language DLL’s in VB, mixed with DirectX, to get excellent, fast 3D graphics.
Mixing VB and C++ (NEW). Article on writing C++ DLL’s for VB. Explains how to pass VB arrays & Strings.
Unlimited Realities. A site with some great tutorials on graphics & game related programs, including tutorials on using a method of accessing a pictures pixels as if it was a normal array. Mixing this method and assembly DLL’s can show great outcomes.
VB Explorer. A good page for learning VB
VB Helper. Another good VB page with many tips.
Hugo Elias’ DOS ASM page. Some graphics/game related pages.
NASM help file (“Nasmdoc.hlp”)
Newsgroups for ASM programming such as:
– comp.lang.asm.x86
– alt.lang.asm
– comp.os.ms-windows.programmer.win32
Newsgroups for VB programming such as:
– microsoft.public.vb.winapi
– microsoft.public.vb.winapi.graphics
– microsoft.public.win32.programmer.gdi
Search for ‘assembly programming’ from any search engine (such as Lycos or Yahoo)

Using NASM with VB

To use NASM with Visual Basic, you have to make a DLL with NASM, and then you can use that DLL with Visual Basic (VB3 doesn’t let you use dll’s). You can use the DLL’s you made, the same way as if you where using a Windows system DLL, such as ‘user32.dll’ or ‘gdi32.dll’.

To make a DLL with NASM, there are three steps to it:

You first need to write your NASM code
Then you must compile your NASM code and link it with Visual Basic’s linker.
Then you can use it with VB.

For example, here we will make a DLL to add two Long integers (DWords): Save the following code in a text editor as myDLL.asm into a new directory, such as C:\VB5\samples\ASM : (if you are going to copy and paste this code from here, select it from the left edge of this window, otherwise the carriage returns will be stuffed up.)

	SEGMENT code USE32
	GLOBAL _DllMain		;Just a small routine that gets called.
_DllMain:	mov	eax, 1  ;Dont worry about this.
		retn	12	;


;Sub addLongs (ByRef number1 As Long, ByVal number2 As Long)
	GLOBAL addLongs
addLongs:
	enter	0, 0
	mov	eax, [ebp+8]	;pointer to number1
	mov	ecx, [eax]	;ecx = number1
	add	ecx, [ebp+12]	;ecx = number1 + number2
	mov	[eax], ecx	;number1 = number1 + number2
	leave
	retn 8			;return, with 8 bytes of arguments (2 DWords)


	ENDS

The first line means tells NASM that it is a 32 bit Windows program. This MUST be in all of your DLL’s, before any code.
The second line tells the linker that _DLLMain will be a global name. The linker will allow that name to be called by Visual Basic. You must declare all your procedures as GLOBAL, otherwise they wont be seen by Visual Basic.
The third line has the name (label) called _DllMain, so that the linker knows where _DllMain is.
The two lines of code that _DllMain does is simply making eax equal to 1, removing 12 bytes for its arguments and then returning to whoever called the routine. This is a special routine that you dont have to worry about.
The first line of the 2nd routine starts with a ; (semicolon) to show that the rest of the line is a remark statement, just like a ‘ (apostraphe) does in Visual Basic. The line is only there to show what it would be called as in Visual Basic. Notice that the first argument is passed by reference, while the other argument is passed by value. This means that the actual memory location of the first argument is sent, and the actual value of the second argument is sent. The first variable is sent by reference, so that it can be changed. The second variable cant be changed by this routine, because we only have its value, but we have the actual location of the first variable, because it is passed by reference.
The second line (of the 2nd routine) shows that addLongs is also going to be a label that Visual Basic will be able to see.
The third line shows where addLongs is.
The fourth line will save the ebp register, and set it to point to the start of the call stack. (This is only necessary if you are going to use arguments). EBP will now point to things like the caller’s memory address, and also the arguments that were sent. EBP+8 will point to the first argument. Since Windows 9x is 32 bit, each argument must be 32 bits, which is 4 bytes. Since the arguments are one after the other, then the next argument will be at EBP+12, and the third argument will be at EBP+16, then at EBP+20, etc…
The fifth line will set the EAX register to equal the first argument. Since the first argument was passed by reference, EAX will be equal to the location in memory where the first variable is stored.
The sixth line will set the ECX register to equal the value of the first argument.
The seventh line will add the value of the second argument to ECX, and so ECX will then hold the value of the first argument plus the second argument.
The eigth line will set the first variable to equal ECX, which was the two numbers added together. Therefore ‘number1’, which is used by the program in Visual Basic, will now have ‘number2’ added to it.
The second last line of the routine will undo what enter did. It must be used in the routine if enter was used.
The last line of the 2nd routine marks the end of the routine, and it will return back to the caller (Visual Basic), and it must also show the amount of bytes of argument that were sent to it.
The very last line (ENDS) means that it is the end of all your code for the file.

Once this is all typed (names ARE case-sensitive in NASM!) and saved, it is ready to be compiled.

To compile it, I made a batch file in the same directory, called “MakeDLL.bat” to automatically type in the arguments for compiling the DLL’s, and I put the following into it:

C:\Nasm\NasmW.exe -f coff myDLL.asm
C:\VB5\Link.exe /dll /export:addLongs /entry:DllMain myDLL.o
del myDLL.exp
del myDLL.lib
del myDLL.o

The first two lines are the important ones, but the last three lines will remove wasted files if you dont use them.
The first line compiles your assembly code. The arguments to it say that it is to be compiled to the COFF format, which is the format that the Visual Basic linker uses. Obviously, you should replace the directory of NASM (in brown) with the directory where it is on your computer. If you wanted to see the machine code listing of your DLL, then you can also add ‘ -l myDLL.lst’ onto the end of the first line.
The second line will link your COFF format ‘.o’ file into a ‘.dll’ file that can be used with Visual Basic. The first argument ‘/dll’ tells it to make it a DLL file. Without it, it will make a EXE file. After that, the ‘/export:addLongs’ says that the name ‘addLongs’ will be visible to Visual Basic. You must do this for every one of your routines in the DLL, otherwise Visual Basic wont be able to find them. The next argument ‘/entry:DllMain’ says where that there is a ‘DLLMain’ routine. This should always be used. The last argument ‘myDLL.o’ must always be there, to show what file to link. Obviously, you should replace the directory given here (in brown) with the directory of Visual Basic on your computer, which should have a file called ‘link.exe’.
This should compile and link your ASM file into a DLL file. Using a batch file like this can be very handy, because you will have to run it each time you make any modification to your asm code. Obviously, to use compile a different NASM file, you would have to change wherever it says ‘myDLL’ to whatever the new file is called, and to change the ‘/export’ routines.
Once you have save this batch file, you can run it. A lot of messages will come up onto the screen while it is compiling & linking. You should get familiar with what it says when everything works, so you can quickly tell when something has gone wrong. (Dont worry if the linker says that ‘_DllMain’ is not ‘__stdcall’ with 12 bytes of arguments. I can only get rid of it by making an EXE file instead of a DLL file).

You should now have a file called “myDLL.dll”

You can now use the DLL in Visual Basic, just like any other DLL. Make a new EXE project in Visual Basic. Save it into the same directory as the DLL file you made. Type in the following:

Option Explicit

Private Declare Sub addLongs Lib "samples\ASM\myDLL" (ByRef number1 As Long , ByVal number2 As Long)

Private Sub Form_Click()
    Dim x As Long, y As Long
    x = 200
    y = 5
    Print "x = "; x
    Print "y = "; y
    addLongs x, y
    'If it reached this line, then its probably perfect.
    Print "Added y to x, so now x = "; x
    'The answer better be 205, otherwise you stuffed something up!
End Sub

The second line will declare the routine so that you can use it in your VB program. You need to do one of these declarations for each of the routines you use, even if they are all in the same DLL. (Here it is declared as a ‘Private’ Declaration, meaning that only this VB form will be able to access it. You could put it into a seperate Module (‘.bas’), so that all the forms in your VB project can access it. If you do this, then remove the word ‘Private’). Obviously you should change the directory (relative to the VB directory) that it says the DLL is in, if it is somewhere different on your computer.

This should all run perfectly, and print 205 on the screen. If it didnt work perfectly, then try out the troubleshooting section below.

Doing more than a simple addition in your DLL

(It is expected that by this point, you have got correctly addLongs to work perfectly. If it doesnt, then try out the troubleshooting section below.)

There is obviously a lot more that you can do in your DLL than this simple addition. You can do almost anything that you could do in DOS, except that you cant use interrupts, and also bear in mind that you are programming in protected mode. If you havent ever programmed in protected mode asm, then dont worry. All you have to consider is that the segments arent 64k, they are enormous, and so you only ever need to use the one segment, that will give you access to megabytes of memory! (I didn’t even realise that I was programming in protected mode, until I learned more about protected mode asm in DOS, that I realised that Windows was protected mode!).

This means that you can do most of the things you might have done in DOS, as well as a few more things.

There are many very useful things to do in an assembly DLL, where you can do things that VB wont let you do directly, such as:

Make your own modified BitBlt or StretchBlt routines
Write your own system DLL routines
Use only the lower byte of an Integer
Get the location in memory of a variable
Print/Use the binary data of text or floating-point numbers instead of their ASCII values
Perform shifts instead of multiplies & divides
Optimise your code for a particular type of computer
Optimise your inner loops
much, much more …

To be able to do these, you may need some more information, so here it is.

Making a Function instead of a Sub

If you have read about mixing assembly code with other programs such as C/C++ or Pascal, then you may already know that function values are retuned in registers. For example, if a function returns a Long integer, then it would be returned in the register EAX. This means that if you change the declaration of any Sub-routine in your DLL into a Function that returns a Long integer, then you will see whatever the value was in EAX when the routine finished. The NASM documentation has more information about this. Mixing VB & NASM is almost the exact same as mixing VC & NASM, except that with VB, you dont need to start your NASM routines with an underscore as the first letter. Therefore:

A Byte function is sent back through AL
A Integer function is sent back through AX
A Long function is sent back through EAX
A Single function is sent back through ST0
A Double function is sent back through ST0
A UDT function is sent back through EDX:EAX

If you want to send back more than one variable, you will have to send back a UDT (User-Defined-Type), which is sent ByVal through EDX:EAX.

For example:

Private Type buffer
	num1 As Long
	num2 As Long
End Type

Private Function loadBuffer Lib "myDLL" () As

Using this example, the value of buffer.num1 will be EAX, and the value of buffer.num2 will be EDX. However, if you try using a different combination of variables, things start getting confusing. The important thing to remember is that the entire User-Defined-Type comes out of EDX:EAX. Therefore, you cant send back more than 8 bytes worth of data. If you tried using three Long integers, then you will get a VB error saying ‘bad DLL calling convention’, because all the data can only come from EDX:EAX, and so the third Long integer cant be passed. Be aware that if you put a Single or a Double variable into your UDT, then you wont get the number from ST0, but instead, you will still get the value from EDX:EAX. You can do things like make a UDT of 2 Integers, and 4 Bytes (or even an array of 4 Bytes), because that will all fit into the 8 bytes of EDX:EAX. However, you cant make a number from both EDX and EAX. In other words, a UDT of an Integer, a Long integer and another Integer, will add up to 8 bytes, but it wont work, because the Long integer will have to access from both EDX and EAX. Because of this, if you try something like a UDT of an Integer and then a Long integer, VB will align it so that the Integer will access AX, and the Long integer will access EDX.

Therefore, if you want to fill a UDT with values in the DLL, then you must first create a variable of a UDT, then pass that variable (by reference) to the DLL, so the DLL can fill its values up with its information.

For example:

Private Type buffer
	num1 As Double
	num2 As Byte
	num3(1 to 4) As Single
	num4 As Single
End Type
Private Declare loadBuffer Lib "myDLL" (ByRef myBuffer As Double)

Private Sub Form_Click ()
	Dim myBuffer As buffer
	Print myBuffer.num4		' <-- Is empty
	loadBuffer myBuffer.num1
	Print myBuffer.num4		' <-- Will now have a value
End Sub

Notice that to pass the UDT as an argument, you pass it a reference to the first variable in the UDT, because that will be where the UDT starts, and then the next variables will be right after that.

For example, the following could go into your NASM DLL:

;Sub loadBuffer (ByRef myBuffer As Double)
	GLOBAL loadBuffer
loadBuffer:
	enter	0, 0
	PUSH	EBX
	mov	ebx, [ebp+8]	;EBP+8 points to myBuffer
	mov	[ebx], eax	;EBX points to myBuffer.num1
	mov	[ebx+8], bl	;EBX+8 points to myBuffer.num2
	mov	[ebx+9], ebx	;EBX+9 points to myBuffer.num3(1)
	mov	[ebx+13], ecx	;EBX+13 points to myBuffer.num3(2)
	mov	[ebx+17], edx	;EBX+17 points to myBuffer.num3(3)
	mov	[ebx+21], esi	;EBX+21 points to myBuffer.num3(4)
	mov	[ebx+25], edi	;EBX+25 points to myBuffer.num4
	leave
	retn	4		;4 bytes of arguments (1 DW)

Note: The exact same applies for passing arrays. You pass it the first item in the array, by reference. For example:

Private Sub passArray Lib "myDLL" (ByRef anArray As Byte)

Private Sub Form_Click ()
	Dim myArray(1 to 5) As Byte
	passArray myArray(1)		' <-- This will send a reference to myArray
	Print myArray(4)
End Sub

Using internal variables in your DLL

You can have your own set of variables inside your DLL routines. For example, you can have variables that are accessible by all of your routines, and they will stay the same value between calls (like ‘Global’ & ‘Static’ variables of VB).

You can also have local variables, that are only accessible within the one routine, and are destroyed after each call. To use local variables, you can either ‘push’ and ‘pop’ them, or you can set space in the stack directly, by changing the ‘enter 0, 0’ command. You can read the help that comes with NASM, on doing this.

If you want to use the ‘global’ internal variables, then this is what you do: (It is the same as for DOS)

	[SECTION .data]
myByteData:		db 12, 34, 'a', 0xF3, 'Numbers and Strings in one variable!'
myWordData:		dw 12, 1234, 0xAB12
myDWordData:		dd 12, 0x12345678, 12345678

	[SECTION .bss]
myByteVariable:		resb 1
myWordVariable:		resw 1
myDWordVariable:	resd 1
myDWordArray:		resd 5

	SEGMENT code USE32

.... all your routines ....

;Function getLowerByte (ByVal number1 As Long) As Long
	GLOBAL getLowerByte
getLowerByte:
	enter	0, 0
	mov	eax, dword 0	;clear eax
	mov	al, [ebp+8]	;ebp+8 points to number1
				;eax = lower Byte of number1
	leave
	retn	4		;4 bytes of arguments (1 DW)

;Function getmyWordData () As Long
	GLOBAL getmyWordData
getmyWordData:
	enter	0, 0
	mov	eax, dword 0	;clear eax
	mov	ax, [myWordData] ;eax = lower word of myWordData (= 12)				
	leave
	retn	4		;4 bytes of arguments (1 DW)

	ENDS

Using interrupts

You can’t use interrupts in Windows, no matter what language you use.

If you wanted to use DOS / BIOS / Interrupts, then you will have to find out how to do what you want in Visual Basic first, and then when that is working, you could write a DLL function that quickly prepares the data for VB to use.

For example, if you want to write some data to a file, then instead of using DOS interrupts, you could write an Assembly function to store all of the data you want to save into an array, then save that array in Visual Basic.
I don’t know much about other hardware related operations such as directly accessing I/O ports or the Printer port, etc. I think these still work the same in Windows, but I’m not sure.

If you know something is possible to be done in Windows in another language (such as VB, VC++, etc), then it should be possible with assembly language, but handy things like interrupts aren’t allowed in the Windows operating system.

Troubleshooting

There can be many different problems that happen when you are using your own assembly DLL’s. However, luckily a lot of the problems can cause warning messages, that can help you find out what went wrong. The following are some guidelines, about what various error messages mean (in order of what you should check):

NASM.exe won’t compile your assembly code

This means that you misspelt some of your NASM code, or more likely: you used a command wrong. For help with these problems, you can find help from other sources, because it will be the same problem if you were writing a DOS program. For example, you might have forgotten to specify if something was a Byte or a Word or DWord.

LINK.exe won’t link your assembly file

This is unlikely to happen often. If it wont link correctly, first check to make sure that NASM did compile it. If NASM wont compile it, then LINK wont be able to link it either, and so the ‘.dll’ file will also be deleted, and VB wont run your program.

The only other likely problem with LINK not linking is if you have done something such as not declared a label as ‘GLOBAL’ in your assembly code. If LINK says that it cant find a specific label to export (such as un-resolved externals), then first check your assembly code to make sure that there is a statement saying ‘GLOBAL ‘ and the label name, and also check the batch file you are using, to make sure that it has an argument that says ‘/export:’ and the label name (without any spaces in between).

If LINK cant find your file, then dont forget to check if NASM compiled it correctly first, then check the batch program to make sure the directories are correct.

VB says it can’t find your DLL

This is annoying, because VB takes the current directory as the directory that VB is in, and so if the DLL file is in another directory, it usually cant find it. This doesn’t happen if you make an EXE file from your Visual Basic program, and the DLL is in the same directory. It only happens in the Visual Basic environment. There are three ways to fix this:

If you are only going to use the DLL on your hard-drive, you will obviously know the full path to the DLL. You can type the entire path name for all of your VB declarations. This is handy while you are working on it, and then you can change it later.
You can always keep the DLL in the Windows system directory. When you declare a DLL in VB, it first looks in the current directory (the VB directory), then the Windows directory, then the Windows System directory. Therefore, you dont even have to specify the entire path, because it will search the system directory automatically. Then when you distribute the program, you can always move the DLL into the Windows System directory.
My preferred choice is to leave the DLL in the same directory as its VB program, and use the first option, of naming the full path directory whenever you declare it in VB, and before you distribute it, simply delete the full path, leaving just its name. This way you can have the DLL simply in the same directory as the program, and if you want other programs to use the same DLL, then you can move it into the Windows System directory.

VB says it ‘Can’t find DLL entry point’ in your DLL

This means that you have declared a routine from that DLL that it cant see. There are two possible reasons for this:

The routine does not exist.
You haven’t added the routine to the exports in the batch file for the linker. You should check the batch file (or the commands you are using) to make sure that the linker has the argument ‘/export:’ and the name of the routine (without a space between the two).

VB says that the routine has a ‘bad DLL calling convention’

This gave me a lot of trouble at first, until I started realising what it meant. It means that your assembly routine is doing something wrong, such as not returning using the right amount of bytes for arguments. The problem is that there are so many possible causes to give this error message. Here are the possible causes that I know about:

You have retn with the wrong number of arguments

For example, if you have a routine that takes 8 bytes of arguments, and at the bottom of the routine, you say ‘retn’ or ‘retn 4’ or ‘retn 9’ or anything other than 8, then you will get this message.

You have modified an important register, without restoring it

If you are lucky, you can also get this message if you modify an important register such as EBP, ESP, EDI, ESI, EDX, or even EBX, without restoring them. (If you are unlucky, then say VB will be replaced by GPF’s 🙂
Dont get too worried. You can modify these registers, so long as you restore them to their original state. The good thing is that you may not have to save them all. Some times, you dont have to save most of them, but it all depends on what VB decides to use. Here are some things that I found:

EBP MUST ALWAYS BE RESTORED!
I havent played around with ESP (I’m not that crazy)
EDI, ESI, EDX & EBX behave differently if you are using it in the VB environment, or if it is in a compiled EXE. I have forgot which ones need restoring when, but to be on the safe side, you should always restore all these registers, and when you are finished with the project, you can start deleting each restoration, one by one, checking to see if it still works. (You may have to test the routines several times to makes sure that it works perfectly).

You have the wrong number of arguments in your VB declaration

You can get this if you use the wrong number of arguments in the VB declaration. Check with your asm code to make sure that you haven’t forgotten something.

You have passed an argument that isn’t 32-bits

Since VB 4+ are 32bit, just like Windows 9x, you have to pass EVERYTHING using 32-bits. This means you cant have an Integer argument, because that is only 16-bit, you cant have a Double argument, because that is 64-bit, and you cant pass a Byte argument, because that is only 8-bit. You must pass arguments either as a Long integer, or pass it by reference (so that it still uses a Long integer to represent it’s location).
For example, the following applies to arguments:

(ByVal X as Byte) must be converted to:		(ByRef X As Byte)
(ByVal X as Integer) must be converted to:	(ByRef X As Integer)
(ByVal X as Long) will work
(ByVal X as Single) will work
(ByVal X as Double) must be converted to:	(ByRef X As Double)
(ByVal X() as Long) must be converted to:	(ByRef X As Long)
(ByVal X(5) as Long) must be converted to:	(ByRef X As Long)

Tagged with: visual basic