// SPDX-FileCopyrightText: (C) 2004-2019 Retroshare Team // SPDX-License-Identifier: CC0-1.0 RetroShare JSON API =================== :Cxx: C++ == How to use RetroShare JSON API Look for methods marked with +@jsonapi+ doxygen custom command into +libretroshare/src/retroshare+. The method path is composed by service instance pointer name like +rsGxsChannels+ for +RsGxsChannels+, and the method name like +createGroup+ and pass the input paramethers as a JSON object. .Service instance pointer in rsgxschannels.h [source,cpp] -------------------------------------------------------------------------------- /** * Pointer to global instance of RsGxsChannels service implementation * @jsonapi{development} */ extern RsGxsChannels* rsGxsChannels; -------------------------------------------------------------------------------- .Method declaration in rsgxschannels.h [source,cpp] -------------------------------------------------------------------------------- /** * @brief Request channel creation. * The action is performed asyncronously, so it could fail in a subsequent * phase even after returning true. * @jsonapi{development} * @param[out] token Storage for RsTokenService token to track request * status. * @param[in] group Channel data (name, description...) * @return false on error, true otherwise */ virtual bool createGroup(uint32_t& token, RsGxsChannelGroup& group) = 0; -------------------------------------------------------------------------------- .paramethers.json [source,json] -------------------------------------------------------------------------------- { "group":{ "mMeta":{ "mGroupName":"JSON test group", "mGroupFlags":4, "mSignFlags":520 }, "mDescription":"JSON test group description" }, "caller_data":"Here can go any kind of JSON data (even objects) that the caller want to get back together with the response" } -------------------------------------------------------------------------------- .Calling the JSON API with curl on the terminal [source,bash] -------------------------------------------------------------------------------- curl -u $API_USER --data @paramethers.json http://127.0.0.1:9092/rsGxsChannels/createGroup -------------------------------------------------------------------------------- .JSON API call result [source,json] -------------------------------------------------------------------------------- { "caller_data": "Here can go any kind of JSON data (even objects) that the caller want to get back together with the response", "retval": true, "token": 3 } -------------------------------------------------------------------------------- Even if it is less efficient because of URL encoding HTTP +GET+ method is supported too, so in cases where the client cannot use +POST+ she can still use +GET+ taking care of encoding the JSON data. With +curl+ this can be done at least in two different ways. .Calling the JSON API with GET method with curl on the terminal [source,bash] -------------------------------------------------------------------------------- curl -u $API_USER --get --data-urlencode jsonData@paramethers.json \ http://127.0.0.1:9092/rsGxsChannels/createGroup -------------------------------------------------------------------------------- Letting +curl+ do the encoding is much more elegant but it is semantically equivalent to the following. .Calling the JSON API with GET method and pre-encoded data with curl on the terminal -------------------------------------------------------------------------------- curl -u $API_USER http://127.0.0.1:9092/rsGxsChannels/createGroup?jsonData=%7B%0A%20%20%20%20%22group%22%3A%7B%0A%20%20%20%20%20%20%20%20%22mMeta%22%3A%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupName%22%3A%22JSON%20test%20group%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupFlags%22%3A4%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mSignFlags%22%3A520%0A%20%20%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%20%20%22mDescription%22%3A%22JSON%20test%20group%20description%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%22caller_data%22%3A%22Here%20can%20go%20any%20kind%20of%20JSON%20data%20%28even%20objects%29%20that%20the%20caller%20want%20to%20get%20back%20together%20with%20the%20response%22%0A%7D -------------------------------------------------------------------------------- Note that using +GET+ method +?jsonData=+ and then the JSON data URL encoded are added after the path in the HTTP request. == JSON API authentication Most of JSON API methods require authentication as they give access to RetroShare user data, and we don't want any application running on the system eventually by other users be able to access private data indiscriminately. JSON API support HTTP Basic as authentication scheme, this is enough as JSON API server is intented for usage on the same system (127.0.0.1) not over an untrusted network. If you need to use JSON API over an untrusted network consider using a reverse proxy with HTTPS such as NGINX in front of JSON API server. If RetroShare login has been effectuated through the JSON API you can use your location SSLID as username and your PGP password as credential for the JSON API, but we suggests you use specific meaningful and human readable credentials for each JSON API client so the human user can have better control over which client can access the JSON API. .NewToken.json [source,json] -------------------------------------------------------------------------------- { "token": "myNewUser:myNewPassword" } -------------------------------------------------------------------------------- .An authenticated client can authorize new tokens like this -------------------------------------------------------------------------------- curl -u $API_USER --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/authorizeToken -------------------------------------------------------------------------------- .An unauthenticated JSON API client can request access with -------------------------------------------------------------------------------- curl --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/requestNewTokenAutorization -------------------------------------------------------------------------------- When an unauthenticated client request his token to be authorized, JSON API server will try to ask confirmation to the human user if possible through +mNewAccessRequestCallback+, if it is not possible or the user didn't authorized the token +false+ is returned. == Offer new RetroShare services through JSON API To offer a retroshare service through the JSON API, first of all one need find the global pointer to the service instance and document it in doxygen syntax, plus marking with the custom doxygen command +@jsonapi{RS_VERSION}+ where +RS_VERSION+ is the retroshare version in which this service became available with the current semantic (major changes to the service semantic, changes the meaning of the service itself, so the version should be updated in the documentation in that case). .Service instance pointer in rsgxschannels.h [source,cpp] -------------------------------------------------------------------------------- /** * Pointer to global instance of RsGxsChannels service implementation * @jsonapi{development} */ extern RsGxsChannels* rsGxsChannels; -------------------------------------------------------------------------------- Once the service instance itself is known to the JSON API you need to document in doxygen syntax and mark with the custom doxygen command +@jsonapi{RS_VERSION}+ the methods of the service that you want to make available through JSON API. .Offering RsGxsChannels::getChannelDownloadDirectory in rsgxschannels.h [source,cpp] -------------------------------------------------------------------------------- /** * Get download directory for the given channel * @jsonapi{development} * @param[in] channelId id of the channel * @param[out] directory reference to string where to store the path * @return false on error, true otherwise */ virtual bool getChannelDownloadDirectory( const RsGxsGroupId& channelId, std::string& directory ) = 0; -------------------------------------------------------------------------------- For each paramether you must specify if it is used as input +@param[in]+ as output +@param[out]+ or both +@param[inout]+. Paramethers and return value types must be of a type supported by +RsTypeSerializer+ which already support most basic types (+bool+, +std::string+...), +RsSerializable+ and containers of them like +std::vector+. Paramethers passed by value and by reference of those types are both supported, while passing by pointer is not supported. If your paramether or return +class+/+struct+ type is not supported yet by +RsTypeSerializer+ most convenient approach is to make it derive from +RsSerializable+ and implement +serial_process+ method like I did with +RsGxsChannelGroup+. .Deriving RsGxsChannelGroup from RsSerializable in rsgxschannels.h [source,cpp] -------------------------------------------------------------------------------- struct RsGxsChannelGroup : RsSerializable { RsGroupMetaData mMeta; std::string mDescription; RsGxsImage mImage; bool mAutoDownload; /// @see RsSerializable virtual void serial_process( RsGenericSerializer::SerializeJob j, RsGenericSerializer::SerializeContext& ctx ) { RS_SERIAL_PROCESS(mMeta); RS_SERIAL_PROCESS(mDescription); RS_SERIAL_PROCESS(mImage); RS_SERIAL_PROCESS(mAutoDownload); } }; -------------------------------------------------------------------------------- You can do the same recursively for any member of your +struct+ that is not yet supported by +RsTypeSerializer+. Some Retroshare {Cxx} API functions are asyncronous, historically RetroShare didn't follow a policy on how to expose asyncronous API so differents services and some times even differents method of the same service follow differents asyncronous patterns, thus making automatic generation of JSON API wrappers for those methods impractical. Instead of dealing with all those differents patterns I have chosed to support only one new pattern taking advantage of modern {Cxx}11 and restbed features. On the {Cxx}11 side lambdas and +std::function+s are used, on the restbed side Server Side Events are used to send asyncronous results. Lets see an example so it will be much esier to understand. .RsGxsChannels::turtleSearchRequest asyncronous API [source,cpp] -------------------------------------------------------------------------------- /** * @brief Request remote channels search * @jsonapi{development} * @param[in] matchString string to look for in the search * @param multiCallback function that will be called each time a search * result is received * @param[in] maxWait maximum wait time in seconds for search results * @return false on error, true otherwise */ virtual bool turtleSearchRequest( const std::string& matchString, const std::function& multiCallback, std::time_t maxWait = 300 ) = 0; -------------------------------------------------------------------------------- +RsGxsChannels::turtleSearchRequest(...)+ is an asyncronous method because it send a channel search request on turtle network and then everytime a result is received from the network +multiCallback+ is called and the result is passed as parameter. To be supported by the automatic JSON API wrappers generator an asyncronous method need a parameter of type +std::function+ called +callback+ if the callback will be called only once or +multiCallback+ if the callback is expected to be called more then once like in this case. A second mandatory parameter is +maxWait+ of type +std::time_t+ it indicates the maximum amount of time in seconds for which the caller is willing to wait for results, in case the timeout is reached the callback will not be called anymore. [IMPORTANT] ================================================================================ +callback+ and +multiCallback+ parameters documentation must *not* specify +[in]+, +[out]+, +[inout]+, in Doxygen documentation as this would fool the automatic wrapper generator, and ultimately break the compilation. ================================================================================ .RsFiles::turtleSearchRequest asyncronous JSON API usage example [source,bash] -------------------------------------------------------------------------------- $ cat turtle_search.json { "matchString":"linux" } $ curl --data @turtle_search.json http://127.0.0.1:9092/rsFiles/turtleSearchRequest data: {"retval":true} data: {"results":[{"size":157631,"hash":"69709b4d01025584a8def5cd78ebbd1a3cf3fd05","name":"kill_bill_linux_1024x768.jpg"},{"size":192560,"hash":"000000000000000000009a93e5be8486c496f46c","name":"coffee_box_linux2.jpg"},{"size":455087,"hash":"9a93e5be8486c496f46c00000000000000000000","name":"Linux.png"},{"size":182004,"hash":"e8845280912ebf3779e400000000000000000000","name":"Linux_2_6.png"}]} data: {"results":[{"size":668,"hash":"e8845280912ebf3779e400000000000000000000","name":"linux.png"},{"size":70,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.txt.sha1sum"},{"size":3076767744,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.iso"},{"size":2780872,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.bin"},{"size":917504,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.lzma"},{"size":2278404096,"hash":"e8845280912ebf3779e400000000000000000000","name":"gentoo-linux-livedvd-amd64-multilib-20160704.iso"},{"size":151770333,"hash":"e8845280912ebf3779e400000000000000000000","name":"flashtool-0.9.23.0-linux.tar.7z"},{"size":2847372,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.elf"},{"size":1310720,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.gz"},{"size":987809,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux-lzma.elf"}]} -------------------------------------------------------------------------------- By default JSON API methods requires client authentication and their wrappers are automatically generated by +json-api-generator+. In some cases methods need do be accessible without authentication such as +rsLoginHelper/getLocations+ so in the doxygen documentaion they have the custom command +@jsonapi{RS_VERSION,unauthenticated}+. Other methods such as +/rsControl/rsGlobalShutDown+ need special care so they are marked with the custom doxygen command +@jsonapi{RS_VERSION,manualwrapper}+ and their wrappers are not automatically generated but written manually into +JsonApiServer::JsonApiServer(...)+. == Quirks === 64 bits integers handling While JSON doesn't have problems representing 64 bits integers JavaScript, Dart and other languages are not capable to handle those numbers natively. To overcome this limitation JSON API output 64 bit integers as an object with two keys, one as proper integer and one as string representation. .JSON API 64 bit integer output example [source,json] -------------------------------------------------------------------------------- "lobby_id": { "xint64": 6215642878098695544, "xstr64": "6215642878098695544" } -------------------------------------------------------------------------------- So from languages that have proper 64bit integers support like Python or C++ one better read from `xint64` which is represented as a JSON integer, from languages where there is no proper 64bit integers support like JavaScript one can read from `xstr64` which is represented as JSON string (note that the first is not wrapped in "" while the latter is). When one input a 64bit integer into the JSON API it first try to parse it as if it was sent the old way for retrocompatibility. .JSON API 64 bit integer deprecated format input example [source,json] -------------------------------------------------------------------------------- "lobby_id":6215642878098695544 -------------------------------------------------------------------------------- This way is *DEPRECATED* and may disappear in the future, it is TEMPORALLY kept only for retrocompatibiliy with old clients. If retrocompatible parsing attempt fail then it try to parse with the new way with proper JSON integer format. .JSON API 64 bit integer new proper integer format input example [source,json] -------------------------------------------------------------------------------- lobby_id": { "xint64": 6215642878098695544 } -------------------------------------------------------------------------------- If this fails then it try to parse with the new way with JSON string format. .JSON API 64 bit integer new string format input example [source,json] -------------------------------------------------------------------------------- "lobby_id": { "xstr64": "6215642878098695544" } -------------------------------------------------------------------------------- [WARNING] ================================================================================ Clients written in languages without proper 64bit integers support must use *ONLY* the string format otherwise they will send approximated values and get unexpected results from the JSON API, because parsing will success but the value will not be exactly the one you believe you sent. ================================================================================ == A bit of history === First writings about this The previous attempt of exposing a RetroShare JSON API is called +libresapi+ and unfortunatley it requires a bunch of boilerplate code when we want to expose something present in the {Cxx} API in the JSON API. As an example here you can see the libresapi that exposes part of the retroshare chat {Cxx} API and lot of boilerplate code just to convert {Cxx} objects to JSON https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L44 To avoid the {Cxx} to JSON and back conversion boilerplate code I have worked out an extension to our {Cxx} serialization code so it is capable to serialize and deserialize to JSON you can see it in this pull request https://github.com/RetroShare/RetroShare/pull/1155 So first step toward having a good API is to take advantage of the fact that RS is now capable of converting C++ objects from and to JSON. The current API is accessible via HTTP and unix socket, there is no authentication in both of them, so anyone having access to the HTTP server or to the unix socket can access the API without extra restrictions. Expecially for the HTTP API this is a big risk because also if the http server listen on 127.0.0.1 every application on the machine (even rogue javascript running on your web browser) can access that and for example on android it is not safe at all (because of that I implemented the unix socket access so at least in android API was reasonably safe) because of this. A second step to improve the API would be to implement some kind of API authentication mechanism (it would be nice that the mechanism is handled at API level and not at transport level so we can use it for any API trasport not just HTTP for example) The HTTP server used by libresapi is libmicrohttpd server that is very minimal, it doesn't provide HTTPS nor modern HTTP goodies, like server notifications, websockets etc. because the lack of support we have a token polling mechanism in libresapi to avoid polling for every thing but it is still ugly, so if we can completely get rid of polling in the API that would be really nice. I have done a crawl to look for a replacement and briefly looked at - https://www.gnu.org/software/libmicrohttpd/ - http://wolkykim.github.io/libasyncd/ - https://github.com/corvusoft/restbed - https://code.facebook.com/posts/1503205539947302/introducing-proxygen-facebook-s-c-http-framework/ - https://github.com/cmouse/yahttp taking in account a few metrics like modern HTTP goodies support, license, platform support, external dependencies and documentation it seemed to me that restbed is the more appropriate. Another source of boilerplate code into libresapi is the mapping between JSON API requests and C++ API methods as an example you can look at this https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L158 and this https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ApiServer.cpp#L253 The abstract logic of this thing is, when libreasapi get a request like +/chat/initiate_distant_chat+ then call +ChatHandler::handleInitiateDistantChatConnexion+ which in turn is just a wrapper of +RsMsgs::initiateDistantChatConnexion+ all this process is basically implemented as boilerplate code and would be unnecessary in a smarter design of the API because almost all the information needed is already present in the C++ API +libretroshare/src/retroshare+. So a third step to improve the JSON API would be to remove this source of boilerplate code by automatizing the mapping between C++ and JSON API call. This may result a little tricky as language parsing or other adevanced things may be required. Hope this dive is useful for you + Cheers + G10h4ck === Second writings about this I have been investigating a bit more about: [verse, G10h4ck] ________________________________________________________________________________ So a third step to improve the JSON API would be to remove this source of boilerplate code by automatizing the mapping between C++ and JSON API call ________________________________________________________________________________ After spending some hours investigating this topic the most reasonable approach seems to: 1. Properly document headers in +libretroshare/src/retroshare/+ in doxygen syntax specifying wihich params are input and/or output (doxygen sysntax for this is +@param[in/out/inout]+) this will be the API documentation too. 2. At compile time use doxygen to generate XML description of the headers and use the XML to generate the JSON api server stub. http://www.stack.nl/~dimitri/doxygen/manual/customize.html#xmlgenerator 3. Enjoy