To homepage To homepage
Solved

Connecting to Authenticated REST Endpoint

E
Rank

I have been trying for quite a while to connect to an REST endpoint requiring authentication. Have gone through the the cdata documentation and various ressources on the support site.

I have working setups both in postman and SSIS Script Task, but is very keen on migrating to a more native TX setup (maintainability), will be migratning several more in the time to come.

The process is as follows:

  1. Get Authentication Token (HTTP POST)
    1. The POSTed information is send as ‘x-www-form-urlencoded’ in the body part (Postman)
    2. The response is a json object, containing the token in a attribute
  2. Get Actual Data using the Authentication Token (HTTP GET with token as a url parameter)

The source in this case i ArcGIS, and the documentation does not mention anything about a standardised authentication scheme.

Is there a way around implementing this in any of the standardised authentication schemes, or do I need to go in the direction of RSD files?
Please advice
Thanks in advance.

icon

Best answer by Thomas Lind 21 February 2023, 16:02

View original

10 replies

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi

When it is not a standard method, which seems to be more common with how many issues I have been getting in regarding this, there is usually only one way.

That is to create a RSD file that does the post call.

Usually you will have some settings that needs to be sent to a specific URL that then returns a token, which will be set as a header for the call for the real data.

Sometimes you can add these settings as options in the OAuth Params field. So:

connection_id=<id code>,connection_secret=<secret code>,extra_value=<value>

This will make it so you don’t need to create a RSD file.

If it is coming through as a JSON statement. Like this:

{"AcceptToken": "<token id>","Username": "<user>","Password": "<password>"}

Then you will have to use a post command in a RSD file to get the data.

Here is how it is done in the file.

<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
	<api:info title="Countries" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
	    <attr name="Id"   xs:type="integer" readonly="false" other:xPath="/json/Id"   />
	    <attr name="Name" xs:type="string"  readonly="false" other:xPath="/json/Name" />
    	<attr name="Area" xs:type="string"  readonly="false" other:xPath="/json/Area" />
	</api:info>

	<api:set attr="DataModel" value="DOCUMENT" />
	<api:set attr="URI" value="https://api.com/Countries" />
	<api:set attr="elementmappath#" value="json/Token" />
	<api:set attr="elementmapname#" value="Token" />

	<api:script method="GET">
		<api:set attr="TMP.token" value="" />
		<api:set attr="method" value="POST"/>
		<api:set attr="URI" value="https://api.com/Authorize" />
		<api:set attr="contenttype" value="application/json" />
		<api:set attr="data">
			{"AccessToken": "12345678987654321","Username": "John","Password": "John123456"}
		</api:set>
		<api:call op="jsonproviderGet" out="login">
			<api:set attr="TMP.token" value="[login.token]" />
		</api:call>
		<api:set attr="method" value="GET"/>	
		<api:set attr="Header:Name#" value="Token" />
		<api:set attr="Header:Value#" value="[TMP.token]" />
		<api:call op="jsonproviderGet">
			<api:push/>
		</api:call>
	</api:script>
</api:script>

Regarding what you mentioned with HTTPPost, you can do that instead of adding a jsonproviderGet call and instead of a data attribute you can do a postdata one.

If you get stuck with the file, send it to me and I will look at it. If you feel like it contains private info send it to me as a private message.

E
Rank

Hi Thomas,

Thank you for your reply, it was also my belief, that RSD files was the path to go down.

I have tried simply getting a token (with 1 hour TTL) through Postman and getting this call to work without the authentication handled by the cdata data source.

I have tried with the following URL:

https://agis.evida.dk/server/rest/services/DGD/DGD_I_X/MapServer/2/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&distance=&units=esriSRUnit_Meter&relationParam=&outFields=OBJECTID,SHAPE,ASSETGROUP,ASSETTYPE%2Cassetid%2Cname%2Cownedby%2Cmaintby%2Clocationdescription%2Clifecyclestatus%2CASSETUNITCODE%2CSystemSubnetworkName%2Cregulatorytype%2Cinstallationdate%2Cinservicedate%2CCREATIONDATE%2CCREATOR%2CLASTUPDATE%2CUPDATEDBY%2CDGD_NET_ID%2CDGD_STATION_ID%2CDGD_STATION_NM%2CDGD_F_MR_NAVN%2CDGD_F_MR_NR&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=4326&havingClause=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&historicMoment=&returnDistinctValues=false&resultOffset=&resultRecordCount=&returnExtentOnly=false&datumTransformation=¶meterValues=&rangeValues=&quantizationParameters=&featureEncoding=esriDefault&f=pjson&token=<token received from Auth Call>

In the CDATA REST 2022 provider, I have tried configuring the “URI” and “Custom URL Params” options only, the copy connection string gives the following (Token replaced):

Custom Url Params="where=1%3D1;text=;objectIds=;time=;geometry=;geometryType=esriGeometryEnvelope;inSR=;spatialRel=esriSpatialRelIntersects;distance=;units=esriSRUnit_Meter;relationParam=;outFields=OBJECTID,SHAPE,ASSETGROUP,ASSETTYPE%2Cassetid%2Cname%2Cownedby%2Cmaintby%2Clocationdescription%2Clifecyclestatus%2CASSETUNITCODE%2CSystemSubnetworkName%2Cregulatorytype%2Cinstallationdate%2Cinservicedate%2CCREATIONDATE%2CCREATOR%2CLASTUPDATE%2CUPDATEDBY%2CDGD_NET_ID%2CDGD_STATION_ID%2CDGD_STATION_NM%2CDGD_F_MR_NAVN%2CDGD_F_MR_NR;returnGeometry=true;returnTrueCurves=false;maxAllowableOffset=;geometryPrecision=;outSR=4326;havingClause=;returnIdsOnly=false;returnCountOnly=false;orderByFields=;groupByFieldsForStatistics=;outStatistics=;returnZ=false;returnM=false;gdbVersion=;historicMoment=;returnDistinctValues=false;resultOffset=;resultRecordCount=;returnExtentOnly=false;datumTransformation=;parameterValues=;rangeValues=;quantizationParameters=;featureEncoding=esriDefault;f=pjson;token=<Token received from AUTH Call>";URI=https://agis.evida.dk/server/rest/services/DGD/DGD_I_X/MapServer/2/query

When testing connection, I get the following error:
“Destinationen for en aktivering udløste en undtagelse.
Invalid JSON markup. Expected json, but instead found [text/html].”

For the authentication part, I can easily provide the HTTP calls involved via email. I believe you have access to my user profile.

Thanks in advance.

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi René

The error means that the response it gets from using the URI is coming back with HTML encoding and not JSON. I can see you use a lot of these. If you set it in the custom URL params field it should be able to convert it automatically when it is used. Have you tried to use the real characters instead of the HTML encoded values?

Also please create a log with verbosity level 3 and recreate the error it will be far more descriptive regarding what the error is.

Regards
Thomas Lind

E
Rank

Hi Thomas,

I have replaced the HTML URL Encoded Characters, which gives me the following connection string:

Custom Url Params="where=1=1;text=;objectIds=;time=;geometry=;geometryType=esriGeometryEnvelope;inSR=;spatialRel=esriSpatialRelIntersects;distance=;units=esriSRUnit_Meter;relationParam=;outFields=OBJECTID,SHAPE,ASSETGROUP,ASSETTYPE,assetid,name,ownedby,maintby,locationdescription,lifecyclestatus,ASSETUNITCODE,SystemSubnetworkName,regulatorytype,installationdate,inservicedate,CREATIONDATE,CREATOR,LASTUPDATE,UPDATEDBY,DGD_NET_ID,DGD_STATION_ID,DGD_STATION_NM,DGD_F_MR_NAVN,DGD_F_MR_NR;returnGeometry=true;returnTrueCurves=false;maxAllowableOffset=;geometryPrecision=;outSR=4326;havingClause=;returnIdsOnly=false;returnCountOnly=false;orderByFields=;groupByFieldsForStatistics=;outStatistics=;returnZ=false;returnM=false;gdbVersion=;historicMoment=;returnDistinctValues=false;resultOffset=;resultRecordCount=;returnExtentOnly=false;datumTransformation=;parameterValues=;rangeValues=;quantizationParameters=;featureEncoding=esriDefault;f=pjson;token=<removed>";URI=https://agis.evida.dk/server/rest/services/DGD/DGD_I_X/MapServer/2/query

Still same error….

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi René

You mentioned I could try this? Did you send something me with a private message?

 

E
Rank

How do I PM you?

Thomas Lind
Rank
Userlevel 6
Badge +5

Click on my user icon and click send message.

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi René

To close this. Here is the final solution to how you do a post call with a x-www-form-urlencoded part.

<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
	<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
	<api:info title="Post_x-www-form-urlencoded" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
		<!-- You can modify the name, type, and column size here. -->
		<attr name="id"       xs:type="integer"  readonly="false" other:xPath="/json/data/id"           />
		<attr name="extra_id" xs:type="integer"  readonly="false" other:xPath="/json/data/extra_id"              />
		<attr name="name"     xs:type="string"   readonly="false" other:xPath="/json/data/name"            />
		<attr name="date"     xs:type="date"     readonly="false" other:xPath="/json/data/date"        />
		<attr name="datetime" xs:type="datetime" readonly="false" other:xPath="/json/data/datetime"         />
		<attr name="value"    xs:type="long"     readonly="false" other:xPath="/json/data/value"              />
		<attr name="bool"     xs:type="boolean"  readonly="false" other:xPath="/json/data/bool"        />
		<attr name="token"    xs:type="string"   other:ispseudocolumn="true"   />
	</api:info>

	<api:set attr="EnablePaging" value="true" />
    <api:set attr="DataModel" value="RELATIONAL" />
	<api:set attr="DefaultURI" value="https://api.rest.com/v2/data/query" />
	<api:set attr="_connection._AddToURI" value="f=json&option=1option&transformation=1transformation&token={token}" />
	<api:set attr="BASEURI" value="[DefaultURI]?[_connection._AddToURI]" />
	<api:set attr="BASEJSONPath" value="$.data" />
	<api:set attr="elementmappath#" value="/json/token" />
	<api:set attr="elementmapname#" value="token" />

  <!-- The GET method corresponds to SELECT. Here you can override the default processing of the SELECT statement. The results of processing are pushed to the schema's output. See SELECT Execution for more information. -->
	<api:script method="GET">
		<api:set attr="tmp.token" value="" />
		<api:set attr="method" value="POST"/>
		<api:set attr="URI" value="https://api.rest.com/v2/data/generateToken" />
		<api:set attr="JSONPath" value="$" />
		<api:set attr="contenttype" value="application/x-www-form-urlencoded" />
		<api:set attr="EncodePostData" value="false" />
		<api:set attr="data">username=user&password=123456789&api_key=key12345678987654321&grant_type=password</api:set>
		<api:call op="jsonproviderGet" out="login">
			<api:set attr="tmp.token" value="[login.token]" />
		</api:call>

        <api:set attr="method" value="GET"/>
		<api:set attr="URI" value="[BASEURI | replace('{token}', '[tmp.token | urlencode()]')]"/>
		<api:set attr="JSONPath" value="[BASEJSONPath]"/>
		<api:call op="jsonproviderGet">
			<api:push/>
		</api:call>
	</api:script>
</api:script>

The important parts are these three rows.

<api:set attr="contenttype" value="application/x-www-form-urlencoded" /> <api:set attr="EncodePostData" value="false" />
<api:set attr="data">username=user&password=123456789&api_key=key12345678987654321&grant_type=password</api:set>

They specify that the call will be done with the array of options.

 

E
Rank

I can confirm that this does in fact work with my endpoint.

Thanks for your patience and help, Thomas

Regards, René

Christian Hauggaard
Rank
Userlevel 6
Badge +5

Hi @EvidaXrem can you please help us select a best answer by clicking the “best answer” button on one of the answers? Thanks!

 

Reply


Terms and ConditionsCookie settings