Don't be worried.
If you right click the SQL instance (from sql management studio), you can set the maximum ram used by SQL.
SQL works in this way: when it starts, it uses some RAM (let's say 1-2 gb), everytime a query is executed, SQL tries to "load" the database in ram. This process is called buffering (it's like a video streaming, more or less). Once the query is executed, SQL doesn't free the used ram, it can overwrite it.
When you restart your server, the RAM is cleaned, but you should be able to do the same thing by restarting the SQL service.
Everything depends on how much your query are long and complicated.
In other words, you can move the SQL on another VM, but this scenario will be the same.
You should assure to your OS at least 4 GB, if you have 24 GB, limit SQL to use 19 GB (SQL is not really precise about memory limit :P).
There are people who load the entire DB on RAM, this is possible by adding RAM. If your DB is 30 GB, you can give SQL 35 GB of ram and SQL will load the entire DB in RAM.
Hope this help.