We are pulling records from salesforce and at the max it could be 1 million. We are creating flatfile from the payload and pushing to the VM queue. If the record size is small like 5K then pushing to VM queue takes just few seconds. But, if the record size exceeds like 15K publishing the message to vm queue is taking unexpected long time it's not getting completed.
I commented out the VM to check whether DW is taking more time but it's just take seconds to process the records.
I am pasting the config and code below. Why there is a bottle neck while pushing it to VM? How I can resolve it?
VM config:
<vm:config name="VM_Config" doc:name="VM Config" doc:id="43bc7e5d-f387-4ad5-a1a8-bde183165f2c" >
<vm:queues >
<vm:queue queueName="lms" queueType="PERSISTENT"/>
</vm:queues>
</vm:config>
<flow name="create_flatFile" doc:id="63912901-4623-4b21-b659-d686b94e5fc6" >
<set-variable value="#[0]" doc:name="Counter" doc:id="a0765cec-ad3b-4262-9681-f7c31a2391d7" variableName="batchCounter" />
<ee:transform doc:name="Json to VPS File" doc:id="2e091d10-c526-43f0-af55-e42f10be9316">
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
import * from dw::core::Strings
output application/flatfile schemaPath = "schemas/VPS.ffd", structureIdent="MultiSegment", missingValues="spaces", deferred=true, bufferSize=16384
fun getFieldData(item) = {
"Statement_Message_ID__c": "014205",
"Option_Set__c": "019303",
("ASC__c": "011206") if(!isBlank(item.ASC__c)),
("CDF23__c": "008902") if(!isBlank(item.CDF23__c ))
}
---
{
vps: {
vps: [
{
":USRDSB:-SOURCE": {
":USRDSB:-SOURCE": "SOURCE",
":USRDSB:-SOURCE-ID ": "MULESOFT"
},
":USRDSB:-HEADER": {
":USRDSB:-HEADER": "HEADER",
":USRDSB:-TRANSMISSION-ID": "SPLNSCRB"
},
":USRDSB:-DETAIL-RECORD": vars.eligibleRecords flatMap (item)->
entriesOf(getFieldData(item)) map {
":USRDSB:-FIELD-IND": $.value,
":USRDSB:-CARD-NBR": rightPad(leftPad(item.TSYS_ID__c as Number ,11,0),19,' '),
":USRDSB:-FIELD-DATA": if($.value ~="014205") item.Statement_Message_ID__c
else if($.value ~="019303") item.Option_Set__c
else if($.value ~="008902")leftPad(item.CDF23__c,2,0)
else item.ASC__c
}
},
]
}
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<vm:publish doc:name="Publish" doc:id="cf145508-87c4-4b43-a786-7d153a426659" queueName="lms" timeout="30" config-ref="VM_Config" transactionalAction="NOT_SUPPORTED">
</vm:publish>
<logger level="INFO" message="Aggregation complete" />
</flow>