C
cbelwal
How to build a Copilot for Security API Plugin – Part II
Copilot for Security (Copilot) is a large language model (LLM) based Generative Artificial Intelligence (GAI) system for cybersecurity, compliance, identity and management use cases. Copilot is not a monolithic system but is an ecosystem running on a platform that allows data requests from multiple sources using a unique plugin mechanism. Plugins allow Copilot to not only reason on data from Microsoft products but also from third-parties.
In part-1 of this series, we discussed building an API plugin using a single GET call. In this article, we expand on Part-I and look at building API plugins that make more advanced GET calls using parameters. If you have not read part-I, we encourage you to do so first, as several parts in this article assumes familiarity with the code and other details that were mentioned in part-I. In this blog, we will only discuss API plugins and more information on the other types of Copilot plugins can be found here.
GET calls with Query Parameters
Let us add another function to the Flask website we had first created in part-I. While we can use any standard application that exposes a REST API, it is easier and clearer from the server-side if we have full control of the REST service. The new function will be used for a GET call that will take in three parameters, two of them in the query and one in the path. A new Class is required to handle this additional data, code for which is given below:
ReflectorJson assigns the passed values to internal properties and returns a dictionary of properties in the ReflectorJson.getDict() function. The dictionary is converted to a JSON by the jsonify() function and returned as a HTTP response. Hence the get_params_data() function returns the JSON serialization of the ReflectorJson object with the serialization including the values passed to it. To better understand the output, let us run this site manually in a local machine. We have bind the webservice to all network interfaces and will run it on port 5000. When we see the following log output in the Python console the webserver is up and ready to service requests.
Since we are passing in multiple parameters it will be easier if we use a REST client like Boomerang or Postman. Since Boomerang has an easy-to-use interface available as a plugin for Microsoft Edge we will use that.
In Boomerang, we add the two values (they should be named ‘value1’ and ‘value2’ as the Flask app extracts their value based on these names), and call the path http://127.0.0.1:5000/params/testData, where ‘testData’ is the value that will be assigned to the ‘data’ variable inside the ‘get_params_data’ function.
When we send this request, the Flask website returns the response in a JSON which is shown in Boomerang’s Response tab:
The variables “userdata”, “value1” and “value2” are the ones explicitly passed by our GET call and reflected back by the Flask webservice. With our REST endpoint now working, we are ready to make a plugin that will make a GET call and pass in the 3 variables. Note that as in part-I, if you intend on using the Flask webservice to test this plugin we must host the Flask website where it’s accessible from the Internet allowing Copilot can communicate with it. This can be done either by hosting the Flask webservice as an Azure App Service, Azure VM or some other means. We can also use other webservices that service GET calls, but make sure to change the plugin YAML files given in next section accordingly.
Plugin YAML files
The main plugin file in YAML format is given below:
We give the plugin a unique name starting with ‘Elman’ in honor of Jeff Elman who designed the first Recurrent Neural Network. The above description will use the OpenAPI specification defined in the file ‘API_Plugin_Reflection_OAI_GET_Params.yaml’.
To generate the OpenAPI specification, we will use the same approach as we did in part-I, which is to use Bing Copilot. However, this time we will give a more detailed prompt which contains the output JSON so Bing Copilot has all the nuanced details to generate the file. The prompt to give Bing Copilot to generate the OpenAPI specification is given below:
Write an OpenAPI spec document that takes a GET call to http://127.0.0.1:5000/params/{data} where {data} is a variable, along with two query parameters value1 and value2, and returns the following JSON output. The JSON schema should be defined in a separate schema section in path /components/schemas/ReflectionDataParamsPluginResponse that is referenced with $ref. JSON schema should only contains the type and description properties for each value:
{
"object": "Reflector Json",
"sourceip": "127.0.0.1",
"useragent": "",
"userdata": "testData",
"value1": "This is Value 1",
"value2": "This is Value 2"
}
Partial output for the above prompt in Bing Copilot is shown below:
After copying the above OpenAPI generated file and making slight modifications (mainly in title and description fields) the final OpenAPI specification is given below. We upload this OpenAPI specification in the location specified in the ‘OpenAPISpecURL’ field of the main YAML document. Remember that this location should be publicly accessible over the Internet.
The OpenAPI spec file is given below:
With the OpenAPI specification file ready let us now upload the plugin. Click on the sources icon as highlighted in red circle below:
In Custom section, select ‘Upload Plugin’:
Select the ’Copilot for Security Plugin':
After selecting the main YAML file for the plugin, press the ‘Add’ button to complete the upload:
Note: If you would like your API custom plugin to be used by others within your tenant, please change the “Who can use this plugin?” from ‘Just Me’ to ‘Everyone’. For more information, see Copilot for Security Authentication.
If the plugin upload is successful, a Plugin added confirmation will be shown. If the plugin fails to upload an error message that may be accompanied by a code will be displayed. Incorrectly formatted YAML files are one of the common causes of error, and if you have an error code more information is available here:
Since we have used the Flask API to also serve the OpenAPI specification file we can see the call made by Copilot to download it from the URL given for ‘OpenAPISpecURL’ field, in the server logs.
With the plugin uploaded, now it’s time to validate it. When a new plugin is added it is more efficient to invoke its skill directly and manually pass the parameters, rather than giving a prompt and have Copilot parse the prompt (prompt engineering comes into play to make sure correct parameters are extracted from your prompt!).
To invoke a skill directly click on the ‘Prompts’ icon as shown below:
A popup comes up showing all Promptbooks and Skills available, select ‘See all system capabilities’ to view all the skills:
For API plugins the values specified in the ‘operationId’ and the ‘summary’ or ’description’ fields assigned to each skill (each skill corresponds to a unique REST API endpoint) are displayed in system capabilities. We can search by the 'operationId', which in our case is ‘ReflectionDataGETParams’ as seen in the OpenAPI specification. Searching by the first few keywords brings it up, we then can click on the name.
This brings a new window where you can directly enter the values of the parameters that we want to pass to the skill (these values will then be passed to the REST API):
After entering the values for the parameters, click the ‘Submit’ button:
Copilot will invoke the skill directly and make a REST call with the parameters to our server, which we can verify on the server logs:
The REST call will return a JSON similar to the one we get when making the call directly from Boomerang. Copilot formats the JSON in a nicely formed paragraph:
Now we invoke the skill via a prompt that contains all the fields required by the API call. The prompt is:
Get reflection data for newTestInput, newParamValue1 and newParamValue2
Copilot passes the correct parameter values to the API which we verify in the server:
The output JSON is also nicely formatted in bulleted form.
One observation from the previous prompt is that Copilot assigns the parameters values in a sequential order. In the prompt we can also specify the input field each value corresponds with, which leads to a better prompt by removing ambiguity on value assignment (hint: this is prompt engineering!).
In the following prompt, we reverse the order of passing values but have the prompt explicitly specify the value corresponding to each input parameter.
Get reflection data where value2 is newParamValue2, value1 is newParamValue1 and input is TestInput
From the prompt output we can see even though TestInput was passed last, it was correctly assigned to the ‘user data’ output variable. We can also verify the order of parameters by looking at the GET call in the server:
So far, we have been passing all the required inputs in our prompts. What happens if our prompt does not include all the parameters? Let us run the following prompt in a new session and find out:
Get reflection data for TestInput
The above prompt is missing the values for ‘value1’ and ‘value2’, Copilot correctly passes the TestInput value but the values for ‘Value 1’ and ‘Value 2’ are random and obviously not correct. Server log shows the raw GET call.
Since we did not specify the 3 parameters that are required by the ‘ReflectionDataGETParams’ skills, Copilot uses other values from the prompt or from the current session to fill those values. In certain cases, it is possible that the skill is not even selected since the number of required inputs are missing.
Note that in this case we ran the prompt in a new session. If we run it in an existing session some of the previous outputs can be inserted in for value1 and value2. This may lead either to a correct or a completely incorrect result depending on what previous values were picked up. This is why prompt engineering is important, as it requires framing the prompt correctly so that required inputs for a skill is present in the prompt or the session.
One way to mitigate arbitrary values to be passed for missing values, is to assign default values for each input.
Using default values for parameters
Copilot for Security allows assignment of a default value and one of the ways to do that is specifying the default value in natural language for the ‘description’ field of the input. To set default values for ‘value1’ we change the description field to ‘Value Parameter 1, default to "Dummy Value 1"’ (original description was ‘Value Parameter 1’). This sets the string “Dummy value 1” as default for ‘value1’, similarly the ‘description’ field for ‘value2’ is ‘Value Parameter 1, default to "Dummy Value 1"’. These are the only changes required and the updated OpenAPI specification file is given below:
Delete the current plugin and reimport it, so the new OpenAPI specification document is used.
In a new session, let us give the same prompt as last time, where only one of the three required inputs are specified:
Get reflection data for TestInput
The only input present in the prompt is assigned to ‘User Data’ while Value 1 and Value 2 are assigned their respective default values. Server log shows the REST call made with the default values.
In this article, we showed how to make GET calls with parameters. So far, we have not discussed making REST API calls with authentication or API and that will be the topic of discussion in part-III, stay tuned.
Continue reading...
Copilot for Security (Copilot) is a large language model (LLM) based Generative Artificial Intelligence (GAI) system for cybersecurity, compliance, identity and management use cases. Copilot is not a monolithic system but is an ecosystem running on a platform that allows data requests from multiple sources using a unique plugin mechanism. Plugins allow Copilot to not only reason on data from Microsoft products but also from third-parties.
In part-1 of this series, we discussed building an API plugin using a single GET call. In this article, we expand on Part-I and look at building API plugins that make more advanced GET calls using parameters. If you have not read part-I, we encourage you to do so first, as several parts in this article assumes familiarity with the code and other details that were mentioned in part-I. In this blog, we will only discuss API plugins and more information on the other types of Copilot plugins can be found here.
GET calls with Query Parameters
Let us add another function to the Flask website we had first created in part-I. While we can use any standard application that exposes a REST API, it is easier and clearer from the server-side if we have full control of the REST service. The new function will be used for a GET call that will take in three parameters, two of them in the query and one in the path. A new Class is required to handle this additional data, code for which is given below:
Code:
# Use this class to reflect back the parameters passed via GET query
class ReflectorJson:
def __init__(self,data,json,ip,useragent
self.object = "Reflector Json"
self.userdata = data
self.value1 = json["value1"]
self.value2 = json["value2"]
self.sourceip = ip
self.useragent=useragent
def getDict(self
return self.__dict__
# This method accepts query parameters, passes them to create a ReflectorJson JSON object
@app.route('/params/<data>', methods=['GET'])
def get_params_data(data
args = request.args
jsonData = args
obj = ReflectorJson(data,jsonData,request.remote_addr,request.user_agent.string)
response = jsonify(obj.getDict())
return responseReflectorJson assigns the passed values to internal properties and returns a dictionary of properties in the ReflectorJson.getDict() function. The dictionary is converted to a JSON by the jsonify() function and returned as a HTTP response. Hence the get_params_data() function returns the JSON serialization of the ReflectorJson object with the serialization including the values passed to it. To better understand the output, let us run this site manually in a local machine. We have bind the webservice to all network interfaces and will run it on port 5000. When we see the following log output in the Python console the webserver is up and ready to service requests.
Since we are passing in multiple parameters it will be easier if we use a REST client like Boomerang or Postman. Since Boomerang has an easy-to-use interface available as a plugin for Microsoft Edge we will use that.
In Boomerang, we add the two values (they should be named ‘value1’ and ‘value2’ as the Flask app extracts their value based on these names), and call the path http://127.0.0.1:5000/params/testData, where ‘testData’ is the value that will be assigned to the ‘data’ variable inside the ‘get_params_data’ function.
When we send this request, the Flask website returns the response in a JSON which is shown in Boomerang’s Response tab:
The variables “userdata”, “value1” and “value2” are the ones explicitly passed by our GET call and reflected back by the Flask webservice. With our REST endpoint now working, we are ready to make a plugin that will make a GET call and pass in the 3 variables. Note that as in part-I, if you intend on using the Flask webservice to test this plugin we must host the Flask website where it’s accessible from the Internet allowing Copilot can communicate with it. This can be done either by hosting the Flask webservice as an Azure App Service, Azure VM or some other means. We can also use other webservices that service GET calls, but make sure to change the plugin YAML files given in next section accordingly.
Plugin YAML files
The main plugin file in YAML format is given below:
Code:
#Filename: API_Plugin_Reflection_GET_Params.yaml
Descriptor:
Name: Elman's Reflection API plug-in using GET params v1
DisplayName: Elman's Reflection API plug-in using GET params v1
Description: Skills for getting a GET REST API call reflection based on parameters that are passed v1
DescriptionForModel: Skills for getting a GET REST API call reflection based on parameters that are passed. This can be called with a prompt like "Get Elman's Reflection Data for data1 with value1 and value2
SkillGroups:
- Format: API
Settings:
# Replace this with your own URL where the OpenAPI spec file is located.
OpenApiSpecUrl: http://<URL>/file/API_Plugin_Reflection_OAI_GET_Params.yamlWe give the plugin a unique name starting with ‘Elman’ in honor of Jeff Elman who designed the first Recurrent Neural Network. The above description will use the OpenAPI specification defined in the file ‘API_Plugin_Reflection_OAI_GET_Params.yaml’.
To generate the OpenAPI specification, we will use the same approach as we did in part-I, which is to use Bing Copilot. However, this time we will give a more detailed prompt which contains the output JSON so Bing Copilot has all the nuanced details to generate the file. The prompt to give Bing Copilot to generate the OpenAPI specification is given below:
Write an OpenAPI spec document that takes a GET call to http://127.0.0.1:5000/params/{data} where {data} is a variable, along with two query parameters value1 and value2, and returns the following JSON output. The JSON schema should be defined in a separate schema section in path /components/schemas/ReflectionDataParamsPluginResponse that is referenced with $ref. JSON schema should only contains the type and description properties for each value:
{
"object": "Reflector Json",
"sourceip": "127.0.0.1",
"useragent": "",
"userdata": "testData",
"value1": "This is Value 1",
"value2": "This is Value 2"
}
Partial output for the above prompt in Bing Copilot is shown below:
After copying the above OpenAPI generated file and making slight modifications (mainly in title and description fields) the final OpenAPI specification is given below. We upload this OpenAPI specification in the location specified in the ‘OpenAPISpecURL’ field of the main YAML document. Remember that this location should be publicly accessible over the Internet.
The OpenAPI spec file is given below:
Code:
openapi: 3.0.0
info:
title: REST API Reflection using GET params
description: Skills for getting reflection input for a GET REST API call using Params
version: "v1"
servers:
# Replace this with your own URL where the OpenAPI spec file is located.
- url: http://172.13.112.25:5000
paths:
/params/{input}:
get:
operationId: ReflectionDataGETParams
summary: A Reflection Data Plugin that reads values from URL Params and returns them
parameters:
- in: path
name: input
schema:
type: string
required: true
description: Parameter Input
- in: query
name: value1
schema:
type: string
required: true
description: Value Parameter 1
- in: query
name: value2
schema:
type: string
required: true
description: Value Parameter 2
responses:
"200":
description: OK
content:
application/json:
schema:
$ref: "#/components/schemas/ReflectionDataParamsPluginResponse"
# This is referred to by $ref
components:
schemas:
ReflectionDataParamsPluginResponse:
type: object
properties:
objecttype:
type: string
description: Object type
userdata:
type: string
description: Userdata
value1:
type: string
description: Reflected Parameter 1
value2:
type: string
description: Reflected Parameter 2
sourceip:
type: string
description: The Source IP
useragent:
type: string
description: The User AgentWith the OpenAPI specification file ready let us now upload the plugin. Click on the sources icon as highlighted in red circle below:
In Custom section, select ‘Upload Plugin’:
Select the ’Copilot for Security Plugin':
After selecting the main YAML file for the plugin, press the ‘Add’ button to complete the upload:
Note: If you would like your API custom plugin to be used by others within your tenant, please change the “Who can use this plugin?” from ‘Just Me’ to ‘Everyone’. For more information, see Copilot for Security Authentication.
If the plugin upload is successful, a Plugin added confirmation will be shown. If the plugin fails to upload an error message that may be accompanied by a code will be displayed. Incorrectly formatted YAML files are one of the common causes of error, and if you have an error code more information is available here:
Since we have used the Flask API to also serve the OpenAPI specification file we can see the call made by Copilot to download it from the URL given for ‘OpenAPISpecURL’ field, in the server logs.
With the plugin uploaded, now it’s time to validate it. When a new plugin is added it is more efficient to invoke its skill directly and manually pass the parameters, rather than giving a prompt and have Copilot parse the prompt (prompt engineering comes into play to make sure correct parameters are extracted from your prompt!).
To invoke a skill directly click on the ‘Prompts’ icon as shown below:
A popup comes up showing all Promptbooks and Skills available, select ‘See all system capabilities’ to view all the skills:
For API plugins the values specified in the ‘operationId’ and the ‘summary’ or ’description’ fields assigned to each skill (each skill corresponds to a unique REST API endpoint) are displayed in system capabilities. We can search by the 'operationId', which in our case is ‘ReflectionDataGETParams’ as seen in the OpenAPI specification. Searching by the first few keywords brings it up, we then can click on the name.
This brings a new window where you can directly enter the values of the parameters that we want to pass to the skill (these values will then be passed to the REST API):
After entering the values for the parameters, click the ‘Submit’ button:
Copilot will invoke the skill directly and make a REST call with the parameters to our server, which we can verify on the server logs:
The REST call will return a JSON similar to the one we get when making the call directly from Boomerang. Copilot formats the JSON in a nicely formed paragraph:
Now we invoke the skill via a prompt that contains all the fields required by the API call. The prompt is:
Get reflection data for newTestInput, newParamValue1 and newParamValue2
Copilot passes the correct parameter values to the API which we verify in the server:
The output JSON is also nicely formatted in bulleted form.
One observation from the previous prompt is that Copilot assigns the parameters values in a sequential order. In the prompt we can also specify the input field each value corresponds with, which leads to a better prompt by removing ambiguity on value assignment (hint: this is prompt engineering!).
In the following prompt, we reverse the order of passing values but have the prompt explicitly specify the value corresponding to each input parameter.
Get reflection data where value2 is newParamValue2, value1 is newParamValue1 and input is TestInput
From the prompt output we can see even though TestInput was passed last, it was correctly assigned to the ‘user data’ output variable. We can also verify the order of parameters by looking at the GET call in the server:
So far, we have been passing all the required inputs in our prompts. What happens if our prompt does not include all the parameters? Let us run the following prompt in a new session and find out:
Get reflection data for TestInput
The above prompt is missing the values for ‘value1’ and ‘value2’, Copilot correctly passes the TestInput value but the values for ‘Value 1’ and ‘Value 2’ are random and obviously not correct. Server log shows the raw GET call.
Since we did not specify the 3 parameters that are required by the ‘ReflectionDataGETParams’ skills, Copilot uses other values from the prompt or from the current session to fill those values. In certain cases, it is possible that the skill is not even selected since the number of required inputs are missing.
Note that in this case we ran the prompt in a new session. If we run it in an existing session some of the previous outputs can be inserted in for value1 and value2. This may lead either to a correct or a completely incorrect result depending on what previous values were picked up. This is why prompt engineering is important, as it requires framing the prompt correctly so that required inputs for a skill is present in the prompt or the session.
One way to mitigate arbitrary values to be passed for missing values, is to assign default values for each input.
Using default values for parameters
Copilot for Security allows assignment of a default value and one of the ways to do that is specifying the default value in natural language for the ‘description’ field of the input. To set default values for ‘value1’ we change the description field to ‘Value Parameter 1, default to "Dummy Value 1"’ (original description was ‘Value Parameter 1’). This sets the string “Dummy value 1” as default for ‘value1’, similarly the ‘description’ field for ‘value2’ is ‘Value Parameter 1, default to "Dummy Value 1"’. These are the only changes required and the updated OpenAPI specification file is given below:
Code:
openapi: 3.0.0
info:
title: REST API Reflection using GET params
description: Skills for getting reflection input for a GET REST API call using Params
version: "v1"
servers:
# Replace this with your own URL where the OpenAPI spec file is located.
- url: http://172.13.112.25:5000
paths:
/params/{input}:
get:
operationId: ReflectionDataGETParams
summary: A Reflection Data Plugin that reads values from URL Params and returns them
parameters:
- in: path
name: input
schema:
type: string
required: true
description: Parameter Input
- in: query
name: value1
schema:
type: string
required: true
description: Value Parameter 1, default is "Dummy Value 1"
- in: query
name: value2
schema:
type: string
required: true
description: Value Parameter 2,default is "Dummy Value 2"
responses:
"200":
description: OK
content:
application/json:
schema:
$ref: "#/components/schemas/ReflectionDataParamsPluginResponse"
# This is referred to by $ref
components:
schemas:
ReflectionDataParamsPluginResponse:
type: object
properties:
objecttype:
type: string
description: Object type
userdata:
type: string
description: Userdata
value1:
type: string
description: Reflected Parameter 1
value2:
type: string
description: Reflected Parameter 2
sourceip:
type: string
description: The Source IP
useragent:
type: string
description: The User AgentDelete the current plugin and reimport it, so the new OpenAPI specification document is used.
In a new session, let us give the same prompt as last time, where only one of the three required inputs are specified:
Get reflection data for TestInput
The only input present in the prompt is assigned to ‘User Data’ while Value 1 and Value 2 are assigned their respective default values. Server log shows the REST call made with the default values.
In this article, we showed how to make GET calls with parameters. So far, we have not discussed making REST API calls with authentication or API and that will be the topic of discussion in part-III, stay tuned.
Continue reading...