Paging Morpheus API in Python
Using Morpheus' API is something that comes up very often, and there have been a few bad habits that I've noticed. I myself have been guilty of using large maximum values when hitting endpoints. eg. <api>/networks?max=10000
I came up with the following code. It's a function than concatenates the results into one variable and returns it. I may write this into the pymorpheus module with a proper cursor one day.
from pymorpheus import MorpheusClient
# Set morpheusUrl and morpheusToken before this
mclient = MorpheusClient(morpheusUrl, token=morpheusToken)
def getAllResults(mclient, apipath):
# Get first page of results sorting by ID
result = mclient.call("get", apipath, options=[("sort", "id")])
# If the total is within the max results, no need to paginate
if result['meta']['total'] <= result['meta']['max']:
results = result
else:
# API calls in Morpheus return a key with a list, this grabs that first key
mainkey = list(result.keys())[0]
# Set the default offset
offset = result['meta']['max']
moreResults = True
while moreResults is True:
# Get next result set, adding the offset
nextResult = mclient.call("get", apipath, options=[("sort", "id"), ("offset", str(offset))])
# Add the nextResult list to the original result
result[mainkey] = result[mainkey] + nextResult[mainkey]
# If the offset plus the latest result size equals the total returned items, then exit
if nextResult['meta']['offset'] + nextResult['meta']['size'] >= result['meta']['total']:
moreResults = False
# Else increment the offset and run again
else:
offset = offset + nextResult['meta']['max']
return(result)
allNetworks = getAllResults(mclient, "/networks")