Back to previous page

Grey Box Testing With A Janio QA Engineer

Prasan Shetty is a Quality Assurance (QA) Engineer at Janio. He’s had the extraordinary experience of starting his job right in the middle of a pandemic – from home. Let’s take a quick look at his journey and how he uses one of the many testing methods in his arsenal to ensure that our products are in working order.

An Eventful Journey at Janio

I knew that joining the company from home during a pandemic period was always going to be challenging, but a good team and supportive colleagues has made my journey much easier. They’ve been supportive in helping me transition smoothly into my QA role by explaining what our B2B (Business-to-Business) product is and its processes in detail.

Furthermore, they provided good documentation, coupled with daily standups, sprint planning meetings, and biweekly catch-up calls with my manager to ensure that my tasks are clear and in sync with the team. They also provided me with the necessary help when I ran into roadblocks with my tasks. I'm also happy to have had the opportunity to participate in the expansion of my team as an interviewer, and to explore the technologies used in product testing at Janio.

Reviewing some JIRA tickets

My role as a QA Engineer in Janio

My journey began with Janio’s B2B product. I had to write and maintain test cases using Test Rail, regression and retesting of features. There was also the aspect of defect management using JIRA to ensure that quality features are being pushed during weekly releases. Currently, I’m working on the Shopify App Integration (which is now live!) with the B2C (Business-to-Consumer) and B2B products.

I’m now up to speed with the automation framework used at Janio through writing and maintaining test scripts in Robot Framework for the B2B product, as well as troubleshooting any issues in test scripts for the B2C product. It has been exciting so far and I hope to continue to contribute and learn more in Janio.

To bring in an example of the testing method used in my day-to-day workflow, I’ll explain what Grey Box testing is, and why we use it.

Note: There are many stylistic representations of ‘Grey Box’, but in this article, we’ll use ‘Grey-box’.

Grey-box Testing

Grey-box testing is a software testing method used to confirm that it meets the requirements defined, by the business, for the software. It is called grey-box because it's a combination of both white-box testing and black-box testing.

Let’s first pause here to explore what white-box and black-box testing actually is.

White-box & Black-box Testing

In white-box testing, the structure, design, and coding of the software are tested to verify the flow of input–output (missing word?). It also targets room for improvement for the software's design, usability, and security. This is done with full visibility of the code, with the test cases being designed around it.

On the other hand, black-box testing focuses on the input–output of the software, and is based around the software requirements and specifications without the knowledge of the internal structure and code.

A summary of black-box vs white-box testing

At Janio, most of our products are web-based to ensure accessibility from any device, and to ensure quick deployment of updates across all our partners, clients, and customers. Since grey-box testing is carried out with limited or partial knowledge of the internal workings of the software application, it’s an ideal fit for web-based applications, and is arguably one of the best techniques for functional testing.

Testing Strategy

Grey-box testing, as described above, does not necessarily require the tester to obtain the source code in order to design test cases. To carry out this testing process, test cases can be designed based on the algorithm, knowledge of architectures, internal states, or other advanced descriptions of the program behaviour.

Additionally, it utilises all the clear-cut techniques of black-box testing for function testing. Generating a test case is based on requirements and pre-setting all the conditions through the assertion method*.

* The assertion method is a method used to identify the set of inputs for which assertion will fail, and enables testers to guard against possible errors in function.

Grey-box Testing Techniques

I will now elaborate on some grey-box testing techniques in this section, using data tables where available.

Matrix Testing

Matrix testing is a technique within Grey-box testing that defines all the used variables of a particular program. In any program, variables are the essential elements through which values move through the program.

The variables should be on par with the requirement, without which, the readability of the program and speed of the software will be reduced.

A few reasons why matrix testing is used:

  • To eliminate uninitialised and unused variables by identifying used variables from the program.
  • To examine inherent risks (i.e. technical risks and business risks) that are associated with the variables of different frequencies as labeled by the software developer.

The design of test cases become smoother and easier when all of this information is summarised in two types of tables shown below.

[Tables can be scrolled from left to right.]



From the information in the two tables above, the testing analyst can immediately identify the technical and business aspect of the code (namely, saving and deleting records) that requires testing. This is because there is substantial risk involved.

The intention behind this testing is to locate defective logic in the system by providing maximum coverage with code and User Interface (UI) functions, while minimising the number of test cases in a statistical and organised way of software testing.

Complex applications and supply chain/eCommerce products, like B2B applications, can be tested with this technique. To demonstrate, we’ll use an Orthogonal Array Testing example. It is composed of an array of values where a variable is represented in each column and a test case is represented in each row.

A simple example is as follows.

To test all possible combinations exhaustively, the tester would have to create a total of 27 test combinations (3³) as described above.

However, by assigning values for each factor and then extrapolating for combined pairing, we can select a subset of the total, bringing down the number of test cases from 27 to 9. This effective technique helps in maximising the required testing coverage, with few test cases.

You can read more about Orthogonal Array Testing here.

Pattern Testing

This testing is carried out by using the record of analysis on the historical data of the previous system defects. These analyses may contain specific reasons for the defect or bug with information on the problem that has been addressed, (an) applicable situation(s), or generic test cases, and so on.

Unlike black-box testing, grey-box testing uses historical data and analyses to discover, within the code, reasons for the failure so that they can be fixed in the next iteration of the software. Notably, pattern testing is only applicable to such a type of software that was developed following the same pattern of the previous one, as the possibility of similar defects would only occur in the current software version.

Regression Testing

Regression testing is carried out after executing a functional development or repair to the program. This is done to ensure that previously developed-and-tested software is still performant after a change, and did not regress.

So, to verify whether the modification in any of the previous versions of the software has regressed or caused unintended, adverse side effect(s) in other aspects of the program’s new version, these test strategies can be executed (in no specific order):

  • Retesting within a firewall where dependencies are analysed for choosing baseline tests
  • Retesting risky use cases where the risk factor is considered
  • Retesting all existing test cases
  • Retesting by profile where time is allocated in proportion to the operational profile
  • Retesting changed segment where code changes are compared for choosing baseline tests

There are instances where, at some stage in confirmation testing (retesting of a bug that is fixed by a developer), a defect appears to be rectified and begins functioning as intended, when in fact, the rectified defect might have initiated a different defect somewhere else in the software.

Regression testing helps take care of these types of defects by utilising the above-mentioned testing strategies.

Other Applicable Scenarios

Grey-box Testing is also said to be performed when the internal codes for two modules or units are studied for designing test cases (White-box Testing method) and then actual tests are conducted using the exposed interfaces (Black-box Testing method).

Another example would be the testing of a B2B web application containing links. If an error crops up while clicking the links, changes can be made in the HTML code for further testing and checking. Here, the user is carrying out white-box testing by altering the code and using black-box testing by observing the output on the front end.

Conclusion

Grey-box testing is a staple in our workflow, since it is a hybrid of black-box and white-box testing. Because it's a combination of two common testing methods, its effectiveness will inevitably be double than that of any normal testing methods.

Some of its other advantages include:

  1. Better user validation
  2. Excellent test scenarios that can be designed
  3. Testing is performed with high level database diagram and data flow management

It offers the advantage of understanding the front-end, while also shedding light on some of the internal workings (i.e. code, structure, architecture).

All in all, Grey-box testing provides an excellent way to conduct functional testing at a high level, but it can and should be combined with more extensive, deeper, white-box testing to ensure quality and test completion.

The Road Ahead

In recent times, automation and testing have always been mentioned in the same breath. For Janio’s QA team, we are working towards having more test cases automated with Robot Framework. Additionally, to also build up our automation pipeline with all major test cases integrated into CI/CD (Continuous Integration/Continuous Deployment) pipeline, and to ensure quality features gets pushed into production without manual intervention.

-----

Interested in a QA role in Janio? We’re hiring!

Back to previous page