XUtils

Schemathesis

A tool for automatic property-based testing of web applications built with Open API / Swagger specifications.

Python
https://github.com/kiwicom/schemathesis

Quick Demo

Schemathesis Demo

With a summary right in your PRs:

image

Getting Started

Choose from multiple ways to start testing your API with Schemathesis.

💡 Your API schema can be either a URL or a local path to a JSON/YAML file.

💻 Command-Line Interface

Quick and easy for those who prefer the command line.

Python

  1. Install via pip: python -m pip install schemathesis
  2. Run tests
st run --checks all https://example.schemathesis.io/openapi.json

Docker

  1. Pull Docker image: docker pull schemathesis/schemathesis:stable
  2. Run tests
docker run schemathesis/schemathesis:stable
   run --checks all https://example.schemathesis.io/openapi.json

🐍 Python Library

For more control and customization, integrate Schemathesis into your Python codebase.

  1. Install via pip: python -m pip install schemathesis
  2. Add to your tests:
import schemathesis

schema = schemathesis.from_uri("https://example.schemathesis.io/openapi.json")


@schema.parametrize()
def test_api(case):
    case.call_and_validate()

💡 See a complete working example project in the /example directory.

How it works

Here’s a simplified overview of how Schemathesis operates:

  1. Test Generation: Using the API schema to create a test generator that you can fine-tune to your testing requirements.
  2. Execution and Adaptation: Sending tests to the API and adapting through statistical models and heuristics to optimize subsequent cases based on responses.
  3. Analysis and Minimization: Checking responses to identify issues. Minimizing means simplifying failing test cases for easier debugging.
  4. Stateful Testing: Running multistep tests to assess API operations in both isolated and integrated scenarios.
  5. Reporting: Generating detailed reports with insights and cURL commands for easy issue reproduction.

Research Findings on Open-Source API Testing Tools

Our study, presented at the 44th International Conference on Software Engineering, highlighted Schemathesis’s performance:

  • Defect Detection: identified a total of 755 bugs in 16 services, finding between 1.4× to 4.5× more defects than the second-best tool in each case.

  • High Reliability: consistently operates seamlessly on any project, ensuring unwavering stability and reliability.

Explore the full paper at https://ieeexplore.ieee.org/document/9793781 or pre-print at https://arxiv.org/abs/2112.10328

Testimonials

“The world needs modern, spec-based API tests, so we can deliver APIs as-designed. Schemathesis is the right tool for that job.”

Emmanuel Paraskakis - Level 250

“Schemathesis is the only sane way to thoroughly test an API.”

Zdenek Nemec - superface.ai

“The tool is absolutely amazing as it can do the negative scenario testing instead of me and much faster! Before I was doing the same tests in Postman client. But it’s much slower and brings maintenance burden.”

Luděk Nový - JetBrains

“Schemathesis is the best tool for fuzz testing of REST API on the market. We are at Red Hat use it for examining our applications in functional and integrations testing levels.”

Dmitry Misharov - RedHat

“There are different levels of usability and documentation quality among these tools which have been reported, where Schemathesis clearly stands out among the most user-friendly and industry-strength tools.”

Testing RESTful APIs: A Survey - a research paper by Golmohammadi, at al

Acknowledgements

Schemathesis is built on top of Hypothesis, a powerful property-based testing library for Python.

Additional content

Papers

Videos

Articles

  • coming soon...