Mimic

An API Compatible Mock Service For OpenStack

Slides: https://glyph.im/mimic-paris

Lekha

software developer in test

(at) Rackspace

github:	lekhajee
freenode irc:	lekha
twitter:	@lekha_j1

Glyph

software developer

(at) Rackspace

github:	glyph
freenode irc:	glyph
twitter:	@glyph

What?

Who?

Sneak Peek

Why?

Rackspace Auto Scale

An open source project

Source: https://github.com/rackerlabs/otter

Dependencies

(of Auto Scale)

Rackspace
Identity

Rackspace
Cloud Servers

Rackspace
Cloud Load Balancers

Rackspace Identity

is API compatible with

Openstack Identity v2

Rackspace Cloud Servers

is powered by

Openstack Compute

Rackspace Cloud Load Balancers

is a

Custom API

Testing

(for Auto Scale)

Functional

API contracts

System Integration

	`↱`	Identity
Auto Scale	`→`	Compute
	`↳`	Load Balancers

System Integration

Success

Failure

Testing Problems

Test run time ≥ server build time

BUILD → ACTIVE ERROR ACTIVE

unknown errors

However...

Improving test coverage

Tests → gate

And...

Slow, flaky tests

Unhappy peers

We've Had Enough!

(on Auto Scale)

There And Back Again

Specific	→	General
Auto Scale		Mimic
General	→	Specific
Mocking Failure		Mimic Mimicking OpenStack

General →

Negative Path Testing

Real Services

FAIL

But Not When You

WANT THEM TO

Succeeding

At Success

Means Failing

At Failure

Mimic Succeeds

At Failure!

😈 ☁

(Production)

😇 ☁

(Staging)

import unittest

`test_stuff ... [OK]`

try:
    result = service_request()
except:
    return error
else:
    return ok(result)

try:
    result = service_request()
except:
    return error
else:
    return ok(result)

if not os.chdir(ca_folder(project_id)):
    raise exception.ProjectNotFound(
        project_id=project_id)

if not os.chdir(ca_folder(project_id)):
    raise exception.ProjectNotFound(
        project_id=project_id)

Glyph: The bug here is that chdir does not actually return a value.

@mock.patch.object(os, 'chdir', return_value=True)
def test_revoke_cert_process_execution_error(self):
    "..."

@mock.patch.object(os, 'chdir', return_value=False)
def test_revoke_cert_project_not_found_chdir_fails(self):
    "..."

Glyph: Because the unit tests introduced with that change construct their own mocks for chdir, Nova's unit tests properly cover all the code, but the code is not integrated with a system that is verified in any way against what the real system (in this case, Python's chdir) does.

Glyph: However, standing up an OpenStack cloud is significantly more work than running mkdir. If you are developing an application against OpenStack, deploying a real cloud to test against can be expensive, error-prone, and slow, as Auto Scale's experience shows.

The Best Of

Both Worlds?

→Specific

Mimic

Version 0.0

Pretending

...

Pretending

to authenticate

Pretending

to Boot Servers

Pretending

is faster

in-memory

minimal
dependencies

(almost entirely pure Python)

Service Dependencies

Configuration

self-contained

Demo!

Nova command-line client

Lekha: Lets see how we can run the python nova command-line client against Mimic

config.sh

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=11111
export OS_AUTH_URL=http://localhost:8900/identity/v2.0/tokens

config.sh

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=11111
export OS_AUTH_URL=http://localhost:8900/identity/v2.0/tokens

config.sh

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=11111
export OS_AUTH_URL=http://localhost:8900/identity/v2.0/tokens

Using Mimic

(on Auto Scale)

The Results!

(Functional tests using Mimic)

Functional Tests:

15 minutes

against a real system

vs.

30 seconds

against Mimic

The Results!

(Integration tests using Mimic)

Integration Tests:

3 hours or more

against a real system

vs.

3 minutes

against Mimic

✈

What about
negative paths?

Mimic does
simulate errors

Error injection using `metadata`

Retry On Errors

Lekha: For the purposes of Auto Scale it was important that we have right number of servers on a scaling group, even if a number of attempts to create one failed. We chose to use metadata for error injection so that requests with injected errors could also be run against real services. For Auto Scale, the expected end result is the same number of servers created, irrespective of the number of failures. But this behavior may also be useful to many other applications because retrying is a common pattern for handling errors.

Mimic 0.0 was...

Too Limited

Mimic 0.0 was...

Single Region

Beyond Auto Scale:

Refactoring Mimic

YAGNI

E(ITO)YAGNI

Plugins!

Identity

Is the Entry Point

(Not A Plugin)


            http://localhost:8900/mimicking/ NovaApi-78bc54/ORD/ v2/tenant_id_f15c1028/servers

Plugin Interface:
“API Mock”

class YourAPIMock():

  def catalog_entries(...)

  def resource_for_region(...)

(that's it!)

Glyph: (click) catalog_entries (click) and resource_for_region (click) That's it!.

def catalog_entries(self,
                    tenant_id):

return [
    Entry(
        tenant_id, "compute", "cloudServersOpenStack",
        [
            Endpoint(tenant_id, region="ORD",
                     endpoint_id=text_type(uuid4()),
                     prefix="v2"),
            Endpoint(tenant_id, region="DFW",
                     endpoint_id=text_type(uuid4()),
                     prefix="v2")
        ]
    )
]

def resource_for_region(
    self, region, uri_prefix,
    session_store
):
    return (YourRegion(...)
            .app.resource())

Glyph: resource_for_region takes (CLICK) the name of a region, (CLICK) a URI prefix - produced by Mimic core to make URI for each service unique, so you can generate URLs to your services in any responses which need them - (CLICK) and a session store where the API mock may look up state of the resources it pretended to provision for the respective tenants. (CLICK) resource_for_region returns an HTTP resource associated with the top level of the given region. This resource then routes this request to any tenant- specific resources associated with the full URL path.

class YourRegion():

    app = MimicApp()

    @app.route('/v2/<string:tenant_id>/servers',
               methods=['GET'])

    def list_servers(self, request, tenant_id):
        return json.dumps({"servers": []})

Tell Mimic

To Load It

# mimic/plugins/your_plugin.py

from your_api import YourAPIMock
the_mock_plugin = YourAPIMock()

Glyph: To register your plugin with Mimic, you just need to drop an instance of it into any module of the mimic.plugins package.

Mimic Remembers

(until you restart it)

session = session_store.session_for_tenant_id(tenant_id)

class YourMockData():
    "..."

your_data = session.data_for_api(your_api_mock,
                                 YourMockData)

session = session_store.session_for_tenant_id(tenant_id)

from mimic.plugins.other_mock import (other_api_mock,
                                      OtherMockData)

other_data = session.data_for_api(other_api_mock,
                                  OtherMockData)

Errors As A Service

Error Conditions Repository

Discovering Errors

Against Real Services

Record Those Errors

Within Mimic

Discover More Errors

Against Real Services

Record Those Errors

For The Next Project

Share A Repository

For Future Projects

Mimic Is A Repository

Mimic Endpoint

/mimic/v1.0/presets

Control

Now & Later

Now

`now()`

`/mimic/v1.1/tick`

{
    "amount": 1.0
}

{
    "advanced": 1.0,
    "now": "1970-01-01T00:00:01.000000Z"
}

{
  "server": {
    "status": "BUILD",
    "updated": "1970-01-01T00:00:00.000000Z",
    "OS-EXT-STS:task_state": null,
    "user_id": "170454",
    "addresses": {},
    "...": "..."
  }
}

{
  "server": {
    "status": "ACTIVE",
    "updated": "1970-01-01T00:00:01.000000Z",
    "OS-EXT-STS:task_state": null,
    "user_id": "170454",
    "addresses": {},
    "...": "..."
  }
}

`--realtime`

Glyph: You can ask Mimic to actually pay attention to the real clock with the --realtime command-line option; that disables this time-advancing endpoint, but it will allow any test suites that rely on real time passing to keep running.

Later

Error Injection

Currently: Metadata-Based

Error Injection

Currently: In-Band

Error Injection

Future: Separate Catalog Entry

Error Injection

Future: Out-Of-Band

Error Injection

Future: With Your Help

Even Later...

Future Possibilities,
Crazy Features!

Even Later...

Real SSH Server

For Fake Servers

Even Later...

Real DNS Server

For Fake Zones

Mimic for OpenStack

It's Easy!

We need your help!

Source:	https://github.com/rackerlabs/mimic
Issues:	https://github.com/rackerlabs/mimic/issues
Chat:	##mimic on Freenode

Mimic

An API Compatible Mock Service For OpenStack

Lekha

software developer in test

(at) Rackspace

Glyph

software developer

(at) Rackspace

What?

Who?

Sneak Peek

Why?

Rackspace Auto Scale

Dependencies

(of Auto Scale)

RackspaceIdentity

RackspaceCloud Servers

RackspaceCloud Load Balancers

Rackspace Identity

is API compatible with

Openstack Identity v2

Rackspace Cloud Servers

is powered by

Openstack Compute

Rackspace Cloud Load Balancers

is a

Custom API

Testing

(for Auto Scale)

Functional

API contracts

System Integration

↱

Identity

Auto Scale

→

Compute

↳

Load Balancers

System Integration

Success

Failure

Testing Problems

Test run time ≥ server build time

BUILD → ACTIVE ERROR ACTIVE

unknown errors

However...

Improving test coverage

Tests → gate

And...

Slow, flaky tests

Unhappy peers

We've Had Enough!

(on Auto Scale)

There And Back Again

Specific

→

General

General

→

Specific

General →

Negative Path Testing

Real Services

FAIL

But Not When You

WANT THEM TO

Succeeding

At Success

Means Failing

At Failure

Mimic Succeeds

At Failure!

😈 ☁

(Production)

😇 ☁

(Staging)

test_stuff ... [OK]

The Best Of

Both Worlds?

Rackspace
Identity

Rackspace
Cloud Servers

Rackspace
Cloud Load Balancers

`↱`

`→`

`↳`

`test_stuff ... [OK]`

minimal
dependencies

What about
negative paths?

Mimic does
simulate errors

Error injection using `metadata`

Plugin Interface:
“API Mock”