ArtificialDeveloper 1.1.0

Welcome to the ArtificialDeveloper app! - created and maintained by Sascha Wildgrube.

Overview

The ArtificialDeveloper performs "Test Driven Code Generation" (also referred to as "Test-Driven Generation" or "TDG") - a human developer writes the test and GenAI writes the code to fit the test.

The ArtificialDeveloper app is the world's first enterprise production ready implementation of the Test-Driven Code Generation pattern - on the ServiceNow platform!

Test-Driven Code Generation

LLMs have made considerable progress since ChatGPT was made available to the public for the first time. And the quality of generated code improved greatly. But yet, the created code by far does not yet meet production standards and is not fit-for-purpose in many cases.

The core challenge is that code - in order to function properly - must be an expression of an algorithmic idea - and not the random summary of something that someone (or something) read before. The latter may be the starting point, but it is not sufficient for a final version.

Whether tests are written as code or executed manually by a developer, in most cases a (human) developer runs in cycles of building a hypothesis, then writing code, testing the resulting program and analyzing the result of the test - and then starts over.

Test-Driven Development is the practice of automating parts of that cycle by writing test code early in the process and shorten the feedback loop between the hypothesis and the analysis, allowing for greater productivity and more robust results.

When the quality of generated code is assessed, we compare apples and oranges. Generated code usually results from a single prompt, containing a description of the requirement, eventually the current version of the code. Do we really expect AI to deliver a result that is comparable to what an experienced human developer would provide after running through the cycle several times? Is that fair?

The more relevant questions are: are we asking AI the right way? Do we make the best use of the technology?

Test-Driven Code Generation (also referred to as "Test-Driven Generation" or "TDG") is to integrate the generative power of AI in the same procedure that the human developer would go through. Where Test-Driven Development automates the testing part in the cycle, Test-Driven Code Generation aims at automating the part where code is actually written.

  1. Human writes boilerplate code and describes the outcome.
  2. Human writes a test.
  3. The test is run.
  4. If the test passes, end here or return to step 2 to add more test cases.
  5. If the test fails, ask AI to implement the code based on the description, the interface, the current version of the code, the test code and the latest test results.
  6. Return to step 3.

As of now, the Artificial Developer app supports Test-Driven Code generation for Script Includes only. As there are no built-in references between Script Includes and ATF Tests, the app assumes a naming convention that requires an ATF test to be called after the app the Script Include is contained in, then a dash, then the name of the Script Include. E.g.: "Artificial Developer - TestDrivenCodeGeneration".

For disambiguation: The term "Test-Driven Generation" has sometimes been used to describe the process of creating test cases (not the code to fit these tests) using AI. However, more recent publications use the term consistently for for code (not test) generation.

Developer Efficiency Gains

The big question is: what to expect from AI augmented coding? As always, the answer is: it depends.

It depends on the LLM, the specific prompt and the difficulty of the problem to be solved. To make reasonable statements about solution quality, the following problem taxonomy is proposed:

Class 1 Problems

These are low complexity non-domain specific problems, where many fit-for-purpose solutions are available on public sources. Solutions do not depend on additional data sources or the system environment. Resulting code delivers fully deterministic results.

Examples are simple math problems (linear algebra, geometry, Newton physics) like calculating the volume of bodies or gravitational vectors, or well-known algorithms like sorting or searching.

Class 2 Problems

These are high complexity non-domain specific problems or low complexity domain specific problems. Solutions are available on public sources, but not all of them are fit-for-purpose.

Examples are complex math problems with partially non-deterministic results, optimization problems, data validation that require implicit knowledge (e.g. emails, urls, ISINs, IBANs), problems that require data retrieval via database interfaces or REST APIs, low complexity domain-specific problems that involve ServiceNow specific APIs like retrieving records based on filter criteria.

Class 3 Problems

These are high complexity domain specific problems. Fit-for-purpose solutions are not available on public sources or are difficult to find due to varying terminology or context.

An example is to retrieve an API token from a ServiceNow connection alias and then to use it with a REST API that requires multiple requests to perform a task.

As of now - at the time of the release of this application version - Test-Driven Code Generation using GTP4 works very well for class 1 problems, works most of the time for class 2 problems but fails on class 3 problems - although provides code that can serve as a basis for further manual work.

As the Test-Driven Code generation approach requires developers to split capabilities into functions and produce automated tests, the efficiency gain for class 1 problems may be neglectable but for class 2 problems it can be significant. One could argue though that enforcing the refactoring of code into Script Include functions and creation of automated tests is per se a design pattern that saves costs on the long run. As a result, the adoption of TDG and hence indirectly adopting TDD already represents an efficiency gain - even if a particular problem cannot be fully solved through TDG.

Disclaimer

ArtificialDeveloper is NOT an officially supported ServiceNow product.

ArtificialDeveloper does NOT come with any kind of warranty. If you use it, you own it!

System Requirements

Installation

  1. Navigate to Plugins
  2. Click button "Sync with Store"
  3. Search for Now Assist for Creator
  4. Install "Now Assist for Creator"
  5. Navigate to the Generic Prompt Capability
  6. Select the related list "OneExtend Definition Configs".
  7. Set the preferred LLM as the default.
  8. Open the OneExtend Capability Definition.
  9. Open the Connection and Credential Alias.
  10. Select "Create Connection and Credential" and add authentication credentials if not done already.
  11. Create an account on GitHub - if not done already.
  12. Create a personal access token for your GitHub account.
  13. Add credentials to access GitHub - use "Basic Auth".
  14. Fork the repository https://github.com/saschawildgrube/servicenow-devtools.
  15. Go to Studio and import the DevTools application from source control.
  16. Perform all installation steps for the DevTools application documented here: https://www.wildgrube.com/servicenow-devtools/ 
  17. Fork the repository https://github.com/saschawildgrube/servicenow-ad.
  18. Go to Studio and import the ArtificialDeveloper application from source control.
  19. Execute installation scripts.
    Run the following script as a background script in scope x_snc_ad:
  20. x_snc_devtools.InstallApp("x_snc_ad");
  21. Assign the "x_snc_ad.developer" role to users who should have access to the "Test-Driven Code Generation" capability.

Reference

Configuration Options

Roles

UI Actions

Script Includes

  • AppGetDependencies

    Retrieves information about the app's dependencies to other apps and required versions.

  • AppGetProperty

    Gets a system property of this application.

  • AppInstall

    This script installs ArtificialDeveloper.

  • AppSetDefaults

    Sets all system properties to default values.

  • AppSetProperty

    Sets a system property of this application.

  • ArtificialDeveloperClientAPI

    Exposes Ajax end-points to clients scripts.

  • ArtificialDeveloperWorkerAPI

    The ArtificialDeveloperWorkerAPI class contains proxy functions to functions that do lengthy operations and hence may be executed through a process worker.

  • Debug

    Produces a log output in the application log using the DevTools Debug function.

  • DevTools

    Implements the extension point for DevTools.

  • DevToolsGetApplicationStatus

    The function DevToolsGetApplicationStatus is called by the DevTools extension point instance class and returns the application status object with additional status information to be displayed on the "DevTools Application Overview" page.

  • DevToolsGetFormatConfig

    The DevToolsGetFormatConfig function is called by the DevTools extension point instance class and returns an object with field formatting information.

  • DevToolsGetInstancePipelineName

    The function DevToolsGetInstancePipelineName is called by the DevTools extension point instance class and returns the pipeline name of an instance based on the given actual name.

  • DevToolsGetLinkDirectory

    The DevToolsGetLinkDirectory function is called by the DevTools extension point instance class and returns an extended (or modified) link directory object that serves as the basis for the DevTools Link page.

  • DevToolsGetParentRecord

    The DevToolsGetParentRecord function is called by the DevTools extension point instance class and returns a parent record for the given record if it can be determined.

  • DevToolsRenderDebugDump

    The function DevToolsRenderDebugDump is called by the DevTools extension point instance class and returns additional debug information provided by the application.

  • IsActive

    Returns true if the application is active, otherwise false.

  • IsTestDrivenCodeGenerationAvailable

    Returns true if the ArtificalDeveloper app is active and Test-Driven Code generation is supported for the given record, otherwise false.

  • Log

    Produces a log output in the application log.

  • LogError

    Produces an error log output in the application log.

  • LogWarning

    Produces a warning log output in the application log.

  • TestDrivenCodeGeneration

    Initiates the test-driven code generation loop based on the given record and performs the loop as often as it takes up to the maximum amount of iterations.

Dependencies

Technical Debts

No technical debt has been documented for this application.

License

Copyright 2024 by Sascha Wildgrube

Licensed under the Apache License, Version 2.0 (the "License")

You may not use ArtificialDeveloper except in compliance with the License.

You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Release Notes

1.1.0 - 2024-11-10

  1. Now also supports responses provided by NowLLM Mixtral.
  2. Minor prompt improvements.
  3. Added chapter on Developer Efficiency Gains and problem classes to the manual.

1.0.1 - 2024-11-03

  1. Updated wording in manual and app description.

1.0.0 - 2024-11-02

First baselined version