README
¶
KServe V2 REST Proxy
This REST Proxy leverages gRPC-Gateway to create a reverse-proxy server which translates a RESTful HTTP API into gRPC. This allows sending inference requests using the KServe V2 REST Predict Protocol to platforms that expect the gRPC V2 Predict Protocol.
Note: This is currently a work in progress, and is subject to performance and usability issues.
Install the ProtoBuf compiler
The protocol buffer compiler, protoc is required to compile the .proto files.
To install it, follow the instructions here.
Generate the gRPC gateway stubs
After changing the grpc_predict_v2.proto file, run the protoc compiler to regenerate
the gRPC gateway code stubs. It's recommended to use the developer image which has
all the required libraries pre-installed.
make run generate
Build the Docker image
After regenerating the gRPC gateway stubs, rebuild the rest-proxy Docker image.
make build
Push the Docker image
Before pushing the new rest-proxy image to your container registry, re-tag the
image created by make build in the step above.
DOCKER_USER="kserve"
DOCKER_TAG="dev"
docker tag kserve/rest-proxy:latest ${DOCKER_USER}/rest-proxy:${DOCKER_TAG}
docker push ${DOCKER_USER}/rest-proxy:${DOCKER_TAG}
Update your ModelMesh deployment
In order to use the newly built rest-proxy image in a ModelMesh Serving deployment update the restProxy.image in config/default/config-defaults.yaml and (re)deploy the ModelMesh Serving.
To update a running deployment of ModelMesh serving, add or update the restProxy.image section in the model-serving-config ConfigMap as described the ModelMesh Serving configuration instructions.
Directories