Welcome to the ArtificialDeveloper app! - created and maintained by Sascha Wildgrube.
The ArtificialDeveloper performs "Test Driven Code Generation" (also referred to as "Test-Driven Generation" or "TDG") - a human developer writes the test and GenAI writes the code to fit the test.
The ArtificialDeveloper app is the world's first enterprise production ready implementation of the Test-Driven Code Generation pattern - on the ServiceNow platform!
LLMs have made considerable progress since ChatGPT was made available to the public for the first time. And the quality of generated code improved greatly. But yet, the created code by far does not yet meet production standards and is not fit-for-purpose in many cases.
The core challenge is that code - in order to function properly - must be an expression of an algorithmic idea - and not the random summary of something that someone (or something) read before. The latter may be the starting point, but it is not sufficient for a final version.
Whether tests are written as code or executed manually by a developer, in most cases a (human) developer runs in cycles of building a hypothesis, then writing code, testing the resulting program and analyzing the result of the test - and then starts over.
Test-Driven Development is the practice of automating parts of that cycle by writing test code early in the process and shorten the feedback loop between the hypothesis and the analysis, allowing for greater productivity and more robust results.
When the quality of generated code is assessed, we compare apples and oranges. Generated code usually results from a single prompt, containing a description of the requirement, eventually the current version of the code. Do we really expect AI to deliver a result that is comparable to what an experienced human developer would provide after running through the cycle several times? Is that fair?
The more relevant questions are: are we asking AI the right way? Do we make the best use of the technology?
Test-Driven Code Generation (also referred to as "Test-Driven Generation" or "TDG") is to integrate the generative power of AI in the same procedure that the human developer would go through. Where Test-Driven Development automates the testing part in the cycle, Test-Driven Code Generation aims at automating the part where code is actually written.
As of now, the Artificial Developer app supports Test-Driven Code generation for Script Includes only. As there are no built-in references between Script Includes and ATF Tests, the app assumes a naming convention that requires an ATF test to be called after the app the Script Include is contained in, then a dash, then the name of the Script Include. E.g.: "Artificial Developer - TestDrivenCodeGeneration".
For disambiguation: The term "Test-Driven Generation" has sometimes been used to describe the process of creating test cases (not the code to fit these tests) using AI. However, more recent publications use the term consistently for code (not test) generation.
The big question is: what to expect from AI augmented coding? As always, the answer is: it depends.
It depends on the LLM, the specific prompt and the difficulty of the problem to be solved. To make reasonable statements about solution quality, the following problem taxonomy is proposed:
Class 1 Problems
These are low complexity non-domain specific problems, where many fit-for-purpose solutions are available on public sources. Solutions do not depend on additional data sources or the system environment. Resulting code delivers fully deterministic results.
Examples are simple math problems (linear algebra, geometry, Newton physics) like calculating the volume of solid bodies or gravitational vectors, or well-known algorithms like sorting or searching.
Class 2 Problems
These are high complexity non-domain specific problems or low complexity domain specific problems. Solutions are available on public sources, but not all of them are fit-for-purpose.
Examples are complex math problems with partially non-deterministic results, optimization problems, data validation that require implicit knowledge (e.g., emails, urls, ISINs, IBANs), problems that require data retrieval via database interfaces or REST APIs, low complexity domain-specific problems that involve ServiceNow specific APIs like retrieving records based on filter criteria.
Class 3 Problems
These are high complexity domain specific problems. Fit-for-purpose solutions are not available on public sources or are difficult to find due to varying terminology or context.
An example is to retrieve an API token from a ServiceNow connection alias and then to use it with a REST API that requires multiple requests to perform a task.
As of now - at the time of the release of this application version - Test-Driven Code Generation using GTP4 works very well for class 1 problems, works most of the time for class 2 problems but fails on class 3 problems - although provides code that can serve as a basis for further manual work.
As the Test-Driven Code generation approach requires developers to split capabilities into functions and produce automated tests, the efficiency gain for class 1 problems may be neglectable but for class 2 problems it can be significant. One could argue though that enforcing the refactoring of code into Script Include functions and creation of automated tests is per se a design pattern that saves costs on the long run. As a result, the adoption of TDG and hence indirectly adopting TDD already represents an efficiency gain - even if a particular problem cannot be fully solved through TDG.
ArtificialDeveloper is NOT an officially supported ServiceNow product.
ArtificialDeveloper does NOT come with any kind of warranty. If you use it, you own it!
x_snc_devtools.InstallApp("x_snc_ad");Controls if the ArtificialDeveloper application is active.
The OOTB SysID of the "Script" variable in the ATF Test Step "Run Server Side Script".
Controls the verbosity of log output produced by the ArtificialDeveloper application.
Defines how many iterations should be performed during Test-Driven Code Generation by default.
A user that can get support from the artifical developer.
Initiates the test-driven code generation loop for the current record.
Retrieves information about the app's dependencies to other apps and required versions.
Gets a system property of this application.
This script installs ArtificialDeveloper.
Sets all system properties to default values.
Sets a system property of this application.
Exposes Ajax end-points to clients scripts.
|
Methods:
* start()
The ArtificialDeveloperWorkerAPI class contains proxy functions to functions that do lengthy operations and hence may be executed through a process worker.
Produces a log output in the application log using the DevTools Debug function.
Implements the extension point for DevTools.
|
Methods:
* initialize()
* GetParentRecord()
* GetLinkDirectory()
* GetFormatConfig()
* GetApplicationStatus()
* RenderDebugDump()
* GetInstancePipelineName()
* GetEncodedQueryToPreventUselessChecks()
* AppVersionBaselineOnFinish()
* Prompt()
The function DevToolsGetApplicationStatus is called by the DevTools extension point instance class and returns the application status object with additional status information to be displayed on the "DevTools Application Overview" page.
The DevToolsGetLinkDirectory function is called by the DevTools extension point instance class and returns an extended (or modified) link directory object that serves as the basis for the DevTools Link page.
The DevToolsGetParentRecord function is called by the DevTools extension point instance class and returns a parent record for the given record if it can be determined.
The function DevToolsRenderDebugDump is called by the DevTools extension point instance class and returns additional debug information provided by the application.
Returns true if the application is active, otherwise false.
Returns true if the ArtificalDeveloper app is active and Test-Driven Code generation is supported for the given record, otherwise false.
Produces a log output in the application log.
Produces an error log output in the application log.
Produces a warning log output in the application log.
Initiates the test-driven code generation loop based on the given record and performs the loop as often as it takes up to the maximum amount of iterations.
Compatible Versions: 1.108.0
Repository url: https://github.com/saschawildgrube/servicenow-devtools.git
No technical debt has been documented for this application.
Copyright 2024 by Sascha Wildgrube
Licensed under the Apache License, Version 2.0 (the "License")
You may not use ArtificialDeveloper except in compliance with the License.
You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
First baselined version